awk with existence argument?

Login or Register to Ask a Question and Join Our Community

awk with existence argument?

Tags

Login to Discuss or Reply to this Discussion in Our Community

Top Forums Shell Programming and Scripting awk with existence argument?

07-09-2012

Registered User

6, 0

Join Date: Jul 2012

Last Activity: 20 November 2012, 8:34 AM EST

Posts: 6

Thanks Given: 6

Thanked 0 Times in 0 Posts

awk with existence argument?

Greetings, Salutations.

I have 3 column csv files with ~25 million rows and considerable redundancy.

Brief description of data: columns 1 and 2 contain variables, column 3 contains their correlation. For simplicity and discussion, let's use this example:

Code:

a,a,1
a,b,0.8
a,c,0.5
b,a,0.8
b,b,1
b,c,0.2
c,a,0.5
c,b,0.2
c,c,1

I would like to remove the redundant variable combinations with a script (i.e. the correlation of "a" with "b" is the same as the correlation of "b" with "a". I typically use awk for manipulating csv files, but am open to all suggestions.

Currently, I create a new csv where columns 1 and 2 are flipped, cat it and the original file, then remove duplicates using

Code:

awk '!x[$0]++'

However, the inefficiency of this method is problematic with a 25 million row csv file that is then doubled. Does anyone have any suggestions for doing this more cleverly? Something that would check for the existence of a combination of variables, perhaps?

Thanks in advance.

Last edited by R3353; 07-10-2012 at 05:08 PM.. Reason: added CODE tags

R3353

View Public Profile for R3353

Find all posts by R3353

07-09-2012

Registered User

628, 174

Join Date: Oct 2010

Last Activity: 2 December 2017, 5:58 AM EST

Location: Madrid, Spain

Posts: 628

Thanks Given: 8

Thanked 174 Times in 171 Posts

Hi R3353,

One way using perl:

Code:

$ cat script.pl
use warnings;
use strict;

my (%hash);

while ( <> ) { 
        
    chomp;

    my @f = split /,/;

    if ( $f[0] eq $f[1] ) { 
        printf qq|%s\n|, $_; 
        next;
    }   

    my $key = join qq||, sort @f[0,1];

    if ( exists $hash{ $key } ) { 
        delete $hash{ $key };
        next;
    }   

    $hash{ $key } = 1;

    printf qq|%s\n|, $_; 

}
$ cat infile
a,a,1                                                                                                                                                                                                                                        
a,b,0.8                                                                                                                                                                                                                                      
a,c,0.5                                                                                                                                                                                                                                      
b,a,0.8                                                                                                                                                                                                                                      
b,b,1                                                                                                                                                                                                                                        
b,c,0.2                                                                                                                                                                                                                                      
c,a,0.5                                                                                                                                                                                                                                      
c,b,0.2                                                                                                                                                                                                                                      
c,c,1
$ perl script.pl infile
a,a,1                                                                                                                                                                                                                                        
a,b,0.8                                                                                                                                                                                                                                      
a,c,0.5                                                                                                                                                                                                                                      
b,b,1                                                                                                                                                                                                                                        
b,c,0.2                                                                                                                                                                                                                                      
c,c,1

This User Gave Thanks to birei For This Post:

birei

View Public Profile for birei

Find all posts by birei

07-09-2012

Registered User

1,613, 160

Join Date: Oct 2007

Last Activity: 12 February 2019, 12:19 PM EST

Location: USA

Posts: 1,613

Thanks Given: 40

Thanked 160 Times in 150 Posts

Yes awk has an existence operator...

Code:

awk -F, '{if (!($1$2 in x || $2$1 in x)) x[$1$2] = $0} END {for (i in x) print x[i]}' file

This User Gave Thanks to shamrock For This Post:

shamrock

View Public Profile for shamrock

Find all posts by shamrock

07-10-2012

Registered User

6, 0

Join Date: Jul 2012

Last Activity: 20 November 2012, 8:34 AM EST

Posts: 6

Thanks Given: 6

Thanked 0 Times in 0 Posts

Quote:

Originally Posted by shamrock

Yes awk has an existence operator...

Thanks very much shamrock. This works well.

---------- Post updated at 01:28 AM ---------- Previous update was at 01:12 AM ----------

Birei - thank you also. Your perl script also works well.

R3353

View Public Profile for R3353

Find all posts by R3353

Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

[Solved] Argument list with awk

Hello, I want to execute remote command with ssh. For exemple, i have a variable SERVERS=lpar1,lpar2,lpar3 I want to execute some commands like: ssh -q lpar1 ls / ssh -q lpar2 ls / ssh -q lpar3 ls / Can you help me with awk command ? Thank you :)

2. Shell Programming and Scripting

Passing awk variable argument to a script which is being called inside awk

consider the script below sh /opt/hqe/hqapi1-client-5.0.0/bin/hqapi.sh alert list --host=localhost --port=7443 --user=hqadmin --password=hqadmin --secure=true >/tmp/alerts.xml awk -F'' '{for(i=1;i<=NF;i++){ if($i=="Alert id") { if(id!="") if(dt!=""){ cmd="sh someScript.sh...

3. Shell Programming and Scripting

Checking existence of file using awk

Hi, I need to check whether a particular file exists ot not using awk. Can anyone help me please? For Example:script that i am using: awk '{filename =$NF; rc=(system("test -r filename")) print $rc;}' "$1" is not working. Here I am passing a text file as input whose last word contains a...

4. Shell Programming and Scripting

Passing --usage as argument to awk script

I have the awk script below and things go wrong when I do awk -v dsrmx=25 -f ./checkSRDry.awk --usage I basically want to override the usual --usage and --help that awk gives. How do people usually handle this situation when you also want to supply your own usage and help concerning the...

5. Shell Programming and Scripting

Problem passing directory as argument with awk

I'm trying to figure out what's getting passed as the argument when I try to pass a directory as an argument, and I'm getting incredibly strange behavior. For example, from the command line I'm typing: nawk -f ./test.awk ~ test.awk contains the following: { directory = $NF print...

6. Shell Programming and Scripting

passing argument from Cshelll to awk command

Hi all I have got a file digits.data containing the following data 1 3 4 2 4 9 7 3 1 7 3 10 I am writing a script that will pass an argument from C-shell to nawk command. But it seems the values in the nawk comman does not get set. the program does not print no values out. Here is the...

7. Shell Programming and Scripting

Passing non Awk Argument in Awk

Dear Conerned, I am facing a situation where i need to pass an argument which is non-awk variable like day=090319 awk '/TID:R/ && /TTIN:/' transaction.log I want to add this day variable like below awk '/TID:R$day/ && /TTIN:/' transaction.log But it is not working. :confused:

8. Shell Programming and Scripting

awk using env variable as search argument

Hello, I have a file like was123##abcdefg abddef was123##xuzaghg agdfgg was133##CGHAKS DKGJG from the file i need to print the line after ## where the serach value is passed by an env variable called luster (which is currently set to was123): i tried using the below code but it...

9. Shell Programming and Scripting

passing argument into awk

i'm trying to pass a numerical argument with function xyz to print specfic lines of filename, but my 'awk' syntax is incorrect. ie xyx 3 (prints the 3rd line, separated by ':' of filename) function xyz() { arg1=$1 cat filename | awk -F: -v x=$arg1 '{print $x}' } any ideas?

10. UNIX for Dummies Questions & Answers

Passing argument to awk script

I am writing a shell script. Now i need to read in a string and send it to an awk file to compare and search for compatible record. I wrote it like tat: read serial | awk -f generate.awk data.dat p/s: the data file got 6 field. According to an expert, we can write it like tat: read...

Login or Register to Ask a Question