Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

sed working as uniq 1st word only

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting sed working as uniq 1st word only
# 1  
Old 04-24-2012
sed working as uniq 1st word only
Hello, everyone.
I am having trouble figuring out sed command which emulates uniq. The task I want to do is that 2 consecutive lines in file should be considered the same using the first word only. Example:
Code:
cat tmp.txt
ddd eee
aaa bbb ccc
ddd eee fff
asd fdd
asd fdd bbb
aaa bbb
asd fgh yyy
asd fgh

followed by:
Code:
sort -n |sed '$!N; /^\(.*\)\n\1$/!P; D'

produces output:
Code:
aaa bbb
aaa bbb ccc
asd fdd
asd fdd bbb
asd fgh
asd fgh yyy
ddd eee
ddd eee fff

But I need only one of the "same lines" (preferably the 2nd)
I couldn't figure out the sed magic to get it working, any ideas anyone?
Thank you very much.
# 2  
Old 04-24-2012
With awk:
Code:
sort -n tmp.txt | awk 'a[$1]++'

Last edited by Franklin52; 04-25-2012 at 03:43 AM.. Reason: correcting code
This User Gave Thanks to Franklin52 For This Post:
motorcek
# 3  
Old 04-24-2012
The awk command produces the following error:
Code:
$ sort -n tmp.txt |awk '[$1]++'
awk: syntax error at source line 1
 context is
         >>> [ <<<

I have tried installing gawk but there was no difference. I am using bash shell with FreeBSD. The closest I got with the command:
Code:
sort -n tmp.txt |awk '!a[$3]++'
aaa bbb
aaa bbb ccc
asd fdd bbb
asd fgh yyy
ddd eee fff

The desired output would be:
Code:
aaa bbb ccc
asd fdd bbb
asd fgh yyy
ddd eee fff

It is not every other line, as it seems in this example.
Maybe any other idea? Thanks
# 4  
Old 04-24-2012
You would need $1 instead of $3:

Code:
$ sort -n infile | awk '!A[$1]++'
aaa bbb
asd fdd
ddd eee

You could reverse sort to get the last one
Code:
$ sort -rn infile | awk '!A[$1]++'
ddd eee fff
asd fgh yyy
aaa bbb ccc

Is there any reason you use sort -n since that is numerical sort, while the input does not seem to be so?

--
BTW: Your desired output seems to differ from your requirement to use the first field only. To use the first two fields you could try this:
Code:
$ sort -rn infile | awk '!A[$1,$2]++'
ddd eee fff
asd fgh yyy
asd fdd bbb
aaa bbb ccc

Last edited by Scrutinizer; 04-25-2012 at 04:14 AM..
This User Gave Thanks to Scrutinizer For This Post:
motorcek
# 5  
Old 04-25-2012
Great! That did it, obviously my knowledge of awk needs improvements Smilie
I use numerical sort only because of the fact that the first word is in hex format (without the 0x prefix) and it also produces a neatly arranged list Smilie
Thanks again.
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

awk word boundaries not working

Hi, I am trying below code but the word boundaries not seem to be working. What am I doing incorrectly? echo " ECHO " | awk '{ q="ECHO" ; if ( $0 ~ /\bq\b/) print "HELLO" ; }' OR echo " ECHO " | awk '{ q="ECHO" ; if ( $0 ~ /\b'$q'\b/) print "HELLO" ; }' Or echo " ECHO " | awk... (6 Replies)
Discussion started by: ahmedwaseem2000
6 Replies

2. Shell Programming and Scripting

awk - why this is not working? trying next word!

Hi Experts, Can you please advise , why I am not able to make it work, or why this is not working: I spent quite a lot of time on this figuring out , but not working, file : This is a test file thanks for your reply This is another file again Have a nice day this is a small file... (3 Replies)
Discussion started by: rveri
3 Replies

3. Shell Programming and Scripting

Replacing a particular word with another word in all the xml's under a particular directory with sed

Hi Folks, Could you please advise what will be the SED command to replace a word in all xml's under a particular directory for example let say I rite now at the following below location $ cd /ter/rap/config now under config directory there will be lots of xml file , now my objective is to... (1 Reply)
Discussion started by: punpun66
1 Replies

4. Shell Programming and Scripting

Filtering data using uniq and sed

Hello, Does anyone know an easy way to filter this type of file? I want to get everything that has score (column 2) 100.00 and get rid of duplicates (for example gi|332198263|gb|EGK18963.1| below), so I guess uniq can be used for this? gi|3379182634|gb|EGK18561.1| 100.00... (6 Replies)
Discussion started by: narachaid
6 Replies

5. UNIX for Dummies Questions & Answers

Find EXACT word in files, just the word: no prefix, no suffix, no 'similar', just the word

I have a file that has the words I want to find in other files (but lets say I just want to find my words in a single file). Those words are IDs, so if my word is ZZZ4, outputs like aaZZZ4, ZZZ4bb, aaZZZ4bb, ZZ4, ZZZ, ZyZ4, ZZZ4.8 (or anything like that) WON'T BE USEFUL. I need the whole word... (6 Replies)
Discussion started by: chicchan
6 Replies

6. UNIX for Dummies Questions & Answers

How to print line starts with specific word and contains specific word using sed?

Hi, I have gone through may posts and dint find exact solution for my requirement. I have file which consists below data and same file have lot of other data. <MAPPING DESCRIPTION ='' ISVALID ='YES' NAME='m_TASK_UPDATE' OBJECTVERSION ='1'> <MAPPING DESCRIPTION ='' ISVALID ='NO'... (11 Replies)
Discussion started by: tmalik79
11 Replies

7. UNIX for Dummies Questions & Answers

grep for word not working

Hi All..I need a help i am trying to find a word using below script whereas the word exists in my file nitin.txt as a directory but still i am getting "word not found" output..Your suggestions welcomed.: #to check for existence of nitin #!/bin/bash cd /apps/uat1/deploy/app ls -lrt >... (4 Replies)
Discussion started by: nattynitin
4 Replies

8. Shell Programming and Scripting

uniq not working, so any alternate?

Hi, I have these two files . file1 /home/prog/bug/perl /home/prog/bug/ant /home/prog/bug/make /home/prog/bug/gen /home/prog/bug/tiff file2 /home/prog/bug/make /home/prog/bug/gen i want a output file which should contain file1-file2 (2 Replies)
Discussion started by: debu182
2 Replies

9. Shell Programming and Scripting

Whole word with sed?

Hi experts, need a help in SED file=counter.c module=abcd i=http://svn.company.com/svn/${module}/trunk/counter/${file} When i do echo ${i}| sed "s|http://svn.company.com/svn/${module}/trunk/||g"| sed "s|${file}||g" it results in ounter.c (5 Replies)
Discussion started by: ganga.dharan
5 Replies

10. Shell Programming and Scripting

whole word substitution in SED

I am trying to substitute something with sed and what I want is to substitute a whole word and not part of a word. ie sed 's/class/room/g' filename will substitute both class and classes into room and roomes which is not what i want Grep for instance can use the -w option or <> grep -w... (7 Replies)
Discussion started by: gikay01
7 Replies
Login or Register to Ask a Question