08-28-2008
I told you before you should check for DOS carriage returns. Those are the reason the earlier script didn't work correctly.
Shamrock's code was correct for the k[$1] case, you added too many fields there. You should only be adding $2 and $3 from the duplicate lines.
Last edited by era; 08-28-2008 at 06:15 PM..
Reason: Actually it was only on the previous screen five hours ago
10 More Discussions You Might Find Interesting
1. Shell Programming and Scripting
i would like to scan file in for duplicate lines, and print the duplicates to another file,
oh and it has to be case insensitive.
example
line1
line2
line2
line3
line4
line4
outputfile:
line2
line4
any ideas (5 Replies)
Discussion started by: nixguy
5 Replies
2. Shell Programming and Scripting
Hi, I have a noob question . Can someone help me how to concatenate line by line using this variables?
var1:
Apple|
Banana|
var2:
Red
Yellow
then how can I concatenate both line by line? in which the result would be:
Apple|Red
Banana|Yellow
just to generate a row result i was... (6 Replies)
Discussion started by: hagdanan
6 Replies
3. Shell Programming and Scripting
Hi guys, I'm really happy to find this forum
I have a log file, and I have to find all lines that have "error" word, and then save this output in file, the output file has to have just only one line to any Duplicated lines and counter that show how many time this lines duplicated?
I already... (2 Replies)
Discussion started by: wax_light
2 Replies
4. Shell Programming and Scripting
I have a file a.txt having content like
deepak
ram
sham
deepram
sita
kumar
I Want to delete the first line containing "deep" ...
I tried using...
grep -i 'deep' a.txt
It gives me 2 rows...I want to delete the first one..
+ need to know the command to delete the line from... (5 Replies)
Discussion started by: saluja.deepak
5 Replies
5. Shell Programming and Scripting
I have several hundreds of tiny files which need to be concatenated into one single line and all those in a single file. Some files have several blank lines. Tried to use this script but failed on it.
awk 'END { print r } r && !/^/ { print FILENAME, r; r = "" }{ r = r ? r $0 : $0 }' *.txt... (8 Replies)
Discussion started by: sdf
8 Replies
6. Shell Programming and Scripting
I need help in figuring out hoe to delete lines in a data file. The data file is huge. I am currently using "vi" to search and delete the lines - which is cumbersome since it takes lots of time to save that file (due to its huge size).
Here is the issue. I have a data file with the following... (4 Replies)
Discussion started by: GosarJunk
4 Replies
7. Shell Programming and Scripting
Hello to all,
I'm new to perl, I have input file that contains the string below:
315350535ff450000014534130101ff4500ff45453779ff450ff45545f01ff45ff453245341ff4500000545000This string has as line separator "ff45". So, I want to print each line but the code below is not working.
perl -pe '... (2 Replies)
Discussion started by: Ophiuchus
2 Replies
8. Shell Programming and Scripting
I have a script that builds a database ~30 million lines, ~3.7 GB .cvs file. After multiple optimzations It takes about 62 min to bring in and parse all the files and used to take 10 min to remove duplicates until I was requested to add another column. I am using the highly optimized awk code:
awk... (34 Replies)
Discussion started by: Michael Stora
34 Replies
9. Shell Programming and Scripting
Hi,
Do anybody experience how to concatenate multiple line into one line by using awk or perl command?
Input file:
>set1
QAWEQRQ@EWQEASED
ASDAEQW
QAWEQRQTQ
ASRFQWRGWQ
From the above Input file, it got 5 lines
Desired output file:
>set1... (6 Replies)
Discussion started by: perl_beginner
6 Replies
10. UNIX for Dummies Questions & Answers
Hello All !
I need your help on this case,
I have a csv file with this:
ITEM105;ARI FSR;2016-02-01 08:02;243
ITEM101;ARI FSR;2016-02-01 06:02;240
ITEM032;RNO TLE;2016-02-01 11:03;320
ITEM032;RNO TLE;2016-02-02 05:43;320
ITEM032;RNO TLE;2016-02-01 02:03;320
ITEM032;RNO... (2 Replies)
Discussion started by: vadim-bzh
2 Replies
JOIN(1) User Commands JOIN(1)
NAME
join - join lines of two files on a common field
SYNOPSIS
join [OPTION]... FILE1 FILE2
DESCRIPTION
For each pair of input lines with identical join fields, write a line to standard output. The default join field is the first, delimited
by whitespace. When FILE1 or FILE2 (not both) is -, read standard input.
-a FILENUM
print unpairable lines coming from file FILENUM, where FILENUM is 1 or 2, corresponding to FILE1 or FILE2
-e EMPTY
replace missing input fields with EMPTY
-i, --ignore-case
ignore differences in case when comparing fields
-j FIELD
equivalent to `-1 FIELD -2 FIELD'
-o FORMAT
obey FORMAT while constructing output line
-t CHAR
use CHAR as input and output field separator
-v FILENUM
like -a FILENUM, but suppress joined output lines
-1 FIELD
join on this FIELD of file 1
-2 FIELD
join on this FIELD of file 2
--check-order
check that the input is correctly sorted, even if all input lines are pairable
--nocheck-order
do not check that the input is correctly sorted
--help display this help and exit
--version
output version information and exit
Unless -t CHAR is given, leading blanks separate fields and are ignored, else fields are separated by CHAR. Any FIELD is a field number
counted from 1. FORMAT is one or more comma or blank separated specifications, each being `FILENUM.FIELD' or `0'. Default FORMAT outputs
the join field, the remaining fields from FILE1, the remaining fields from FILE2, all separated by CHAR.
Important: FILE1 and FILE2 must be sorted on the join fields. E.g., use `sort -k 1b,1' if `join' has no options. If the input is not
sorted and some lines cannot be joined, a warning message will be given.
AUTHOR
Written by Mike Haertel.
REPORTING BUGS
Report join bugs to bug-coreutils@gnu.org
GNU coreutils home page: <http://www.gnu.org/software/coreutils/>
General help using GNU software: <http://www.gnu.org/gethelp/>
COPYRIGHT
Copyright (C) 2009 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>.
This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law.
SEE ALSO
The full documentation for join is maintained as a Texinfo manual. If the info and join programs are properly installed at your site, the
command
info coreutils 'join invocation'
should give you access to the complete manual.
GNU coreutils 7.1 July 2010 JOIN(1)