Compare Tab Separated Field with AWK to all and print lines of unique fields.


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Compare Tab Separated Field with AWK to all and print lines of unique fields.
# 1  
Old 05-26-2011
Compare Tab Separated Field with AWK to all and print lines of unique fields.

Hi.

I have a tab separated file that has a couple nearly identical lines. When doing:
Code:
sort file | uniq > file.new

It passes through the nearly identical lines because, well, they still are unique.

a)
I want to look only at field x for uniqueness and if the content in field x is the same as field x in any other line, move all the duplicate lines to a new file called file.duplicates.

b)
I also want to be able to look only at field x for uniqueness and if the content in field x is the same as field x in any other line, remove the following lines with the duplicate field x.

Thanks in advance!
# 2  
Old 05-26-2011
Try this on for size:
Code:
#!/usr/bin/env ksh
if [[ -z $1 ]]  ||  [[ -z $2 ]]
then
   echo "missing parms"
   exit 1
fi

sort -k $1,$1  $2 | awk -v col=$1 -v dup_file=$2.dups '{
    if( last == $(col) )
        print >dup_file;
    else
        print;
    last = $( col );
}'

exit $?

Command line args to the script are the column number (1 based) and the input file name. If your field is numeric, you should use -k ${1}n,$1 instead.

Also, if you use sort -u sort will do the uniq for you without invoking a second process.
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

Print lines based upon unique values in Nth field

For some reason I am having difficulty performing what should be a fairly easy task. I would like to print lines of a file that have a unique value in the first field. For example, I have a large data-set with the following excerpt: PS003,001 MZMWR/ L-DWD// * PS003,001... (4 Replies)
Discussion started by: jvoot
4 Replies

2. Shell Programming and Scripting

awk to print lines based on text in field and value in two additional fields

In the awk below I am trying to print the entire line, along with the header row, if $2 is SNV or MNV or INDEL. If that condition is met or is true, and $3 is less than or equal to 0.05, then in $7 the sub pattern :GMAF= is found and the value after the = sign is checked. If that value is less than... (0 Replies)
Discussion started by: cmccabe
0 Replies

3. Shell Programming and Scripting

awk to print unique text in field before hyphen

Trying to print the unique values in $2 before the -, currently the count is displayed. Hopefully, the below is close. Thank you :). file chr2:46603668-46603902 EPAS1-902|gc=54.3 253.1 chr2:211471445-211471675 CPS1-1205|gc=48.3 264.7 chr19:15291762-15291983 NOTCH3-1003|gc=68.8 195.8... (3 Replies)
Discussion started by: cmccabe
3 Replies

4. Shell Programming and Scripting

awk to print unique text in field

I am trying to use awk to print the unique entries in $2 So in the example below there are 3 lines but 2 of the lines match in $2 so only one is used in the output. File.txt chr17:29667512-29667673 NF1:exon.1;NF1:exon.2;NF1:exon.38;NF1:exon.4;NF1:exon.46;NF1:exon.47 703.807... (5 Replies)
Discussion started by: cmccabe
5 Replies

5. Shell Programming and Scripting

awk print - fields separated with comma's need to ignore inbetween double quotes

I am trying to re-format a .csv file using awk. I have 6 fields in the .csv file. Some of the fields are enclosed in double quotes and contain comma's inside the quotes. awk is breaking this into multiple fields. Sample lines from the .csv file: Device Name,Personnel,Date,Solution... (1 Reply)
Discussion started by: jxrst
1 Replies

6. Shell Programming and Scripting

How to print 1st field and last 2 fields together and the rest of the fields after it using awk?

Hi experts, I need to print the first field first then last two fields should come next and then i need to print rest of the fields. Input : a1,abc,jsd,fhf,fkk,b1,b2 a2,acb,dfg,ghj,b3,c4 a3,djf,wdjg,fkg,dff,ggk,d4,d5 Expected output: a1,b1,b2,abc,jsd,fhf,fkk... (6 Replies)
Discussion started by: 100bees
6 Replies

7. Shell Programming and Scripting

How to Print from nth field to mth fields using awk

Hi, Is there any short method to print from a particular field till another filed using awk? Example File: File1 ==== 1|2|acv|vbc|......|100|342 2|3|afg|nhj|.......|100|346 Expected output: File2 ==== acv|vbc|.....|100 afg|nhj|.....|100 (8 Replies)
Discussion started by: machomaddy
8 Replies

8. Shell Programming and Scripting

Compare multiple files and print unique lines

Hi friends, I have multiple files. For now, let's say I have two of the following style cat 1.txt cat 2.txt output.txt Please note that my files are not sorted and in the output file I need another extra column that says the file from which it is coming. I have more than 100... (19 Replies)
Discussion started by: jacobs.smith
19 Replies

9. Shell Programming and Scripting

awk - print all fields except for last field

How do I print all the fields of a record except for the $(NF) field? (4 Replies)
Discussion started by: locoroco
4 Replies

10. Shell Programming and Scripting

Compare two columns separated by a tab

witam potrzebuje polecenia porownujacego koumny na podstawie n-ostatnich znakow danej linnijki tj mam 2 koumny AiB zawierajace ciag dowolnych znakow (dlugosci w kazdej linijce mga byc rozne wiec uzycie substra odpada) A B ewewewabc nbgujnnabc... (3 Replies)
Discussion started by: Toudi
3 Replies
Login or Register to Ask a Question