Sponsored Content
Top Forums Shell Programming and Scripting Compare Tab Separated Field with AWK to all and print lines of unique fields. Post 302525479 by agama on Thursday 26th of May 2011 08:03:50 PM
Old 05-26-2011
Try this on for size:
Code:
#!/usr/bin/env ksh
if [[ -z $1 ]]  ||  [[ -z $2 ]]
then
   echo "missing parms"
   exit 1
fi

sort -k $1,$1  $2 | awk -v col=$1 -v dup_file=$2.dups '{
    if( last == $(col) )
        print >dup_file;
    else
        print;
    last = $( col );
}'

exit $?

Command line args to the script are the column number (1 based) and the input file name. If your field is numeric, you should use -k ${1}n,$1 instead.

Also, if you use sort -u sort will do the uniq for you without invoking a second process.
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Compare two columns separated by a tab

witam potrzebuje polecenia porownujacego koumny na podstawie n-ostatnich znakow danej linnijki tj mam 2 koumny AiB zawierajace ciag dowolnych znakow (dlugosci w kazdej linijce mga byc rozne wiec uzycie substra odpada) A B ewewewabc nbgujnnabc... (3 Replies)
Discussion started by: Toudi
3 Replies

2. Shell Programming and Scripting

awk - print all fields except for last field

How do I print all the fields of a record except for the $(NF) field? (4 Replies)
Discussion started by: locoroco
4 Replies

3. Shell Programming and Scripting

Compare multiple files and print unique lines

Hi friends, I have multiple files. For now, let's say I have two of the following style cat 1.txt cat 2.txt output.txt Please note that my files are not sorted and in the output file I need another extra column that says the file from which it is coming. I have more than 100... (19 Replies)
Discussion started by: jacobs.smith
19 Replies

4. Shell Programming and Scripting

How to Print from nth field to mth fields using awk

Hi, Is there any short method to print from a particular field till another filed using awk? Example File: File1 ==== 1|2|acv|vbc|......|100|342 2|3|afg|nhj|.......|100|346 Expected output: File2 ==== acv|vbc|.....|100 afg|nhj|.....|100 (8 Replies)
Discussion started by: machomaddy
8 Replies

5. Shell Programming and Scripting

How to print 1st field and last 2 fields together and the rest of the fields after it using awk?

Hi experts, I need to print the first field first then last two fields should come next and then i need to print rest of the fields. Input : a1,abc,jsd,fhf,fkk,b1,b2 a2,acb,dfg,ghj,b3,c4 a3,djf,wdjg,fkg,dff,ggk,d4,d5 Expected output: a1,b1,b2,abc,jsd,fhf,fkk... (6 Replies)
Discussion started by: 100bees
6 Replies

6. Shell Programming and Scripting

awk print - fields separated with comma's need to ignore inbetween double quotes

I am trying to re-format a .csv file using awk. I have 6 fields in the .csv file. Some of the fields are enclosed in double quotes and contain comma's inside the quotes. awk is breaking this into multiple fields. Sample lines from the .csv file: Device Name,Personnel,Date,Solution... (1 Reply)
Discussion started by: jxrst
1 Replies

7. Shell Programming and Scripting

awk to print unique text in field

I am trying to use awk to print the unique entries in $2 So in the example below there are 3 lines but 2 of the lines match in $2 so only one is used in the output. File.txt chr17:29667512-29667673 NF1:exon.1;NF1:exon.2;NF1:exon.38;NF1:exon.4;NF1:exon.46;NF1:exon.47 703.807... (5 Replies)
Discussion started by: cmccabe
5 Replies

8. Shell Programming and Scripting

awk to print unique text in field before hyphen

Trying to print the unique values in $2 before the -, currently the count is displayed. Hopefully, the below is close. Thank you :). file chr2:46603668-46603902 EPAS1-902|gc=54.3 253.1 chr2:211471445-211471675 CPS1-1205|gc=48.3 264.7 chr19:15291762-15291983 NOTCH3-1003|gc=68.8 195.8... (3 Replies)
Discussion started by: cmccabe
3 Replies

9. Shell Programming and Scripting

awk to print lines based on text in field and value in two additional fields

In the awk below I am trying to print the entire line, along with the header row, if $2 is SNV or MNV or INDEL. If that condition is met or is true, and $3 is less than or equal to 0.05, then in $7 the sub pattern :GMAF= is found and the value after the = sign is checked. If that value is less than... (0 Replies)
Discussion started by: cmccabe
0 Replies

10. UNIX for Beginners Questions & Answers

Print lines based upon unique values in Nth field

For some reason I am having difficulty performing what should be a fairly easy task. I would like to print lines of a file that have a unique value in the first field. For example, I have a large data-set with the following excerpt: PS003,001 MZMWR/ L-DWD// * PS003,001... (4 Replies)
Discussion started by: jvoot
4 Replies
TGET(1) 						      General Commands Manual							   TGET(1)

NAME
tget - get termcap values SYNOPSIS
tget [-flag id] [-num id] [-str id] [-goto col line] [[-echo] string] DESCRIPTION
Tget allows shell scripts access to the termcap(3) functions. Flags, numbers and strings can be queried from the termcap database under the entry denoted by the environment variable $TERM. OPTIONS
-flag id Set the exit status to zero if the flag id is set. All other options except -echo set the exit status to id being available or not. The last option sets the final exit code. -num id Print the value of the numeric variable id. -str id Print the value of the string variable id. -goto col line Generates instructions to go to the given column and line if the cm capability exists. -echo string Prints string. Any other argument that does not start with a dash is also printed. EXAMPLE
Try this: tget -str so 'Reverse Video' -str se ENVIRONMENT
TERM Terminal type. TERMCAP Path to the termcap database, by default /etc/termcap:/usr/etc/termcap. SEE ALSO
termcap(3). DIAGNOSTICS
Tget will fail immediately with a descriptive message if the termcap entry can't be found. AUTHOR
Kees J. Bot (kjb@cs.vu.nl) TGET(1)
All times are GMT -4. The time now is 05:34 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy