find uniq lines in file, using the first field of line


 
Thread Tools Search this Thread
Top Forums UNIX for Dummies Questions & Answers find uniq lines in file, using the first field of line
# 1  
Old 10-20-2009
find uniq lines in file, using the first field of line

Hello all, new to unix and have just found the forum.
I think I will be here quite often, and hope that in time i will be able to provide soem help, role on not being a newbie anymore Smilie


I have a question which iI am hoping someone could help me with.

If i have a file with lines in in thus ...

purple 2 color
purple 3 color
purple 4 color
purple 5 color
blue 1 color
blue 2 color
blue 3 color

how do i sort the list so only uniq instances of field 1 show with the highest value of field 2 ?

so in this case i would want to see ....

purple 5 color
blue 3 color

any help would be most welcome Smilie

cheers
grom
# 2  
Old 10-20-2009
Hi.

You can use awk alone for this, or something like:

Code:
sort -nrk2,1 file1 | awk 'C != $1 { C = $1; print }'

# 3  
Old 10-20-2009
many thanks!

works like a charm scottn, very much appreciated.
I have a lot to learn Image

Cheers
# 4  
Old 10-20-2009
So do I, because actually, it doesnt!!

Add:

Code:
purple 27 color

and you'll see why!

Code:
sort -nrk2 file1 | awk '!C[$1] { C[$1]=1; print }'

# 5  
Old 10-20-2009
See what you mean. I am glad your on the ball, I had only tried it on the example I posted, just tried again with your revised solution (and added numbers in there too) and all is ok.

could I trouble you and ask for a brief breakdown on how it works?

I have ordered the sed & awk book from amazon, hope i understand it when it comes LOL

cheers
# 6  
Old 10-20-2009
sort is not a command I use often (evidently!).

My understanding was that by using -nrk2,1 the fields would be sorted (reverse) numerically by field 2, then by field 1 (presumably alphabetically - which was wrong).

If you try
Code:
sort -nrk2 file1

Then that's OK (numerically), but the rest (-nrk2,1) doesn't sort the first field after that as I thought.

Sort is a powerful command if you can master it - something I have neither the time nor inclination to do!

Given that the records are sorted reverse numerically in field 2 with "sort -nrk2"...
Code:

sort -nrk2 file1 | awk '!C[$1] { C[$1]=1; print }'

the awk says:
if I don't have a color (where $1 = purple, or whatever) in my array (C[purple]), then define something (anything) for purple (C[$1]=1) so that I do, and then print the line. If I do have something already defined, then do nothing (thus printing only the first line with each (sorted) color).

Last edited by Scott; 10-20-2009 at 03:40 PM.. Reason: added the word "reverse" in a couple of places, for clarity
# 7  
Old 10-20-2009
thanks, that has helped me make a bit of sense out of what, to me at the moment, seems like voodoo magic Smilie

I look forward to the point where I may be able to help someone on these forums, although I had better not hold my breath, i think i have a lot to learn.

once again, many thanks for your kind help, it's appreciated.

Cheers
 
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Find all lines in file such that each word on that line appears in at least n lines of the file

I have a file where every line includes four expressions with a caret in the middle (plus some other "words" or fields, always separated by spaces). I would like to extract from this file, all those lines such that each of the four expressions containing a caret appears in at least four different... (9 Replies)
Discussion started by: uncleMonty
9 Replies

2. UNIX for Advanced & Expert Users

How to find a string in a line in UNIX file and delete that line and previous 3 lines ?

Hi , i have a file with data as below.This is same file. But actual file contains to many rows. i want to search for a string "Field 039 00" and delete that line and previous 3 lines in that file.. Can some body suggested me how can i do using either sed or awk command ? Field 004... (7 Replies)
Discussion started by: vadlamudy
7 Replies

3. Shell Programming and Scripting

Trying to find the distinct lines using uniq command

Platform :Oracle Linux 6.4 Shell : bash The below file has 7 lines , some of them are duplicates. There are only 3 distinct lines. But why is the uniq command still showing 7 ? I just want the distinct lines to be returned. $ cat test.txt SELECT FC.COORD_SET_ID FROM OM_ORDER_FLOW F, -... (2 Replies)
Discussion started by: kraljic
2 Replies

4. Shell Programming and Scripting

Printing uniq first field with the the highest second field

Hi All, I am searching for a script which will produce an output file with the uniq first field with the second field having highest value among all the duplicates.. The output file will produce only the uniqs which are duplicate 3 times.. Input file X 9 B 5 A 1 Z 9 T 4 C 9 A 4... (13 Replies)
Discussion started by: ailnilanjan
13 Replies

5. Shell Programming and Scripting

Unix help to find blank lines in a file and print numbers on that line

Hi, I would like to know how to solve one of my problems using expert unix commands. I have a file with occasional blank lines; for example; dertu frthu fghtu frtty frtgy frgtui frgtu ghrye frhutp frjuf I need to edit the file so that the file looks like this; (10 Replies)
Discussion started by: Lucky Ali
10 Replies

6. Shell Programming and Scripting

Find 5 lines and replace with 18 line in sql file where it contains multiple blocks.

My sql file xyz_abc.sql in this file there are multiple sql block in this block I need to find the following block rem Subset Rows (&&tempName.*) CREATE VIEW &&tempName.* AS SELECT * FROM &&tempName.* WHERE f is not null and replace with following code rem Subset Rows... (9 Replies)
Discussion started by: Zaheer.mic
9 Replies

7. Shell Programming and Scripting

Find lines in text file with certain data in first field

Hi all, Sorry for the title, I was unsure how to word my issue. I'll get right to the issue. In my text file, I need to find all lines with the same data in the first field. Then I need to create a file with the matching lines merged into one. So my original file will look something like... (4 Replies)
Discussion started by: rstev39147
4 Replies

8. UNIX for Dummies Questions & Answers

Sort and uniq lines of a file while keeping a header line

So, I have a file that has some duplicate lines. The file has a header line that I would like to keep at the top. I could do this by extracting the header from the file, 'sort -u' the remaining lines, and recombine them. But they are quite big, so if there is a way to do it with a single... (1 Reply)
Discussion started by: Digby
1 Replies

9. Shell Programming and Scripting

shellscript to find a line in between a particular set of lines of a text file

i have a file a.txt and following is only one portion. I want to search <branch value="/dev36/AREA/" include="yes"></branch> present in between <template_file name="Approve External" path="core/approve/bin" and </template_file> where the no of lines containing "<branch value= " is increasing ... (2 Replies)
Discussion started by: millan
2 Replies

10. UNIX for Dummies Questions & Answers

How to uniq third field in a file

Hi ; I have a question regarding the uniq command in unix How do I uniq 3rd field in a file ? original file : zoom coord 39 18652 39 18652 zoom coord 39 18653 39 18653 zoom coord 39 18818 39 18818 zoom coord 39 18840 39 18840 zoom coord 41 15096 41 15096 zoom... (1 Reply)
Discussion started by: babycakes
1 Replies
Login or Register to Ask a Question