Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Finding unique reocrds at a particular field

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Finding unique reocrds at a particular field
# 1  
Old 12-11-2007
Finding unique reocrds at a particular field
I have a pipe delimited flat file. I want to grep the records that are unique in the 4th field and repeat only once in the file

for e.g.. if the file contains this 3 records

Quote:
00000|05232|25G|LAS VEGAS|SAN DIEGO/LAS VEGAS
00000|05233|25G|LAS VEGAS|SAN DIEGO/LAS VEGAS
00000|11404|25G|UP STATE NY|UPSTATE NY
i want to get the o/p as:
Quote:
00000|11404|25G|UP STATE NY|UPSTATE NY
I just gave a sample here and the file is huge one and i cant just grep from the above example.

can some body let me know how i can do this?
# 2  
Old 12-11-2007
Try this:

Code:
awk '
BEGIN {FS="|"}
{cnt[$4]++;line[$4]=$0}
END {
  for(i in cnt) {
    if(cnt[i]==1) {
      print line[i]
    }
  }
}' file

Regards
Last edited by Franklin52; 12-11-2007 at 03:44 PM.. Reason: typo
# 3  
Old 12-11-2007
I am getting the below error.

awk: syntax error near line 3
awk: illegal statement near line 3
# 4  
Old 12-11-2007
There was a typo on line 3.Smilie

Regards
# 5  
Old 12-11-2007
If you are on Solaris use nawk instead of awk...can't see any typo on line 3.
# 6  
Old 12-11-2007
How can i redirect that to a new file???
# 7  
Old 12-11-2007
You can redirect the output of any command with ">":

Code:
ps >  data.out         # overwrites "data.out"
ps >> data.out         # appends to "data.out"
ps >> data.out 2>&1    # appends, adding both stdout and stderr
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Count of unique lines in field 4

When I use the below awk to count the unique lines in $4 for the input it seems to work. The answer is 3 because $4 is only unique 3 times in all the entries. However, when I use the same on actual data I get 56,536 and I know the answer should be 56,548. My question is there a better way to... (8 Replies)
Discussion started by: cmccabe
8 Replies

2. Shell Programming and Scripting

awk to print unique text in field

I am trying to use awk to print the unique entries in $2 So in the example below there are 3 lines but 2 of the lines match in $2 so only one is used in the output. File.txt chr17:29667512-29667673 NF1:exon.1;NF1:exon.2;NF1:exon.38;NF1:exon.4;NF1:exon.46;NF1:exon.47 703.807... (5 Replies)
Discussion started by: cmccabe
5 Replies

3. Shell Programming and Scripting

Finding a text in files & replacing it with unique strings

Hallo Everyone. I have to admit I'm shell scripting illiterate . I need to find certain strings in several text files and replace each of the string by unique & corresponding text. I prepared a csv file with 3 columns: <filename>;<old_pattern>;<new_pattern> ... (5 Replies)
Discussion started by: gordom
5 Replies

4. Shell Programming and Scripting

Finding unique values in a hash (Perl)

Hi, I have a hash with unique keys associated with some data. my %FINALcontigs = ( 'mira_rep_c765:119reads**', 'ctctactggaagactgac', 'mira_rep_c7454:54reads**', 'atggatactgcgctgttgctaactactgga', 'mira_rep_c6803:12reads**', 'atcgactggatgcagggttgtggtttcta', ... (2 Replies)
Discussion started by: jdilts
2 Replies

5. Shell Programming and Scripting

Unique Field

I have this input file tilenet_test:clar_r5_performance:server_2:4.80762:0%:APM00083103999-009E,APM00083103999-009F tilenet_int:clar_r5_performance:server_2:4.80762:0%:APM00083103999-00C4... (3 Replies)
Discussion started by: greycells
3 Replies

6. Shell Programming and Scripting

Finding the number of unique words in a file

find the number of unique words in a file using sort com- mand. (7 Replies)
Discussion started by: abhikamune
7 Replies

7. Shell Programming and Scripting

Finding unique entries without sorting

Hi Guys, I have two files that I am using: File1 is as follows: wwe khfgv jfo jhgfd hoaha hao lkahe This is like a master file which has entries in the order which I want. (4 Replies)
Discussion started by: npatwardhan
4 Replies

8. Shell Programming and Scripting

Need help with finding unique string in log file

Shell script help Here is 3 sample lines from a log file <date> INFO <java.com.blah> abcd:ID= user login <date> DEBUG <java.com.blah> <nlah bla> abcd:ID=123 user login <date> INFO <java.com.blah> abcd:ID=3243 user login I want to find unique "ID" from this log... (3 Replies)
Discussion started by: gubbu
3 Replies

9. Shell Programming and Scripting

Perl sort unique by one field only

Hi all, I've searched the forum and I can find some code to sort uniquely in perl but not by a single field. I have a file with data such as the following: 1,test,34 1,test2,65 2,test,35, 1,test3,34 2,test,34 What i want to do is sort it uniqely by the first field only so I'd end... (2 Replies)
Discussion started by: Donkey25
2 Replies

10. UNIX for Dummies Questions & Answers

Finding Unique strings which match pattern

I need to grep for a pattern in a file. Files are huge and have several repeated occurances of the strings which match pattern. I just need the strings which contain the pattern in the output. For eg. The contents of my file are as follows. The pattern I want to match by is ABCD ... (5 Replies)
Discussion started by: tektips
5 Replies
Login or Register to Ask a Question