I have 3 column csv files with ~25 million rows and considerable redundancy.
Brief description of data: columns 1 and 2 contain variables, column 3 contains their correlation. For simplicity and discussion, let's use this example:
I would like to remove the redundant variable combinations with a script (i.e. the correlation of "a" with "b" is the same as the correlation of "b" with "a". I typically use awk for manipulating csv files, but am open to all suggestions.
Currently, I create a new csv where columns 1 and 2 are flipped, cat it and the original file, then remove duplicates using
However, the inefficiency of this method is problematic with a 25 million row csv file that is then doubled. Does anyone have any suggestions for doing this more cleverly? Something that would check for the existence of a combination of variables, perhaps?
Thanks in advance.
Last edited by R3353; 07-10-2012 at 05:08 PM..
Reason: added CODE tags
Hello,
I want to execute remote command with ssh.
For exemple, i have a variable
SERVERS=lpar1,lpar2,lpar3
I want to execute some commands like:
ssh -q lpar1 ls /
ssh -q lpar2 ls /
ssh -q lpar3 ls /
Can you help me with awk command ?
Thank you :) (6 Replies)
Hi,
I need to check whether a particular file exists ot not using awk.
Can anyone help me please?
For Example:script that i am using:
awk '{filename =$NF;
rc=(system("test -r filename")) print $rc;}' "$1"
is not working.
Here I am passing a text file as input whose last word contains a... (6 Replies)
I have the awk script below and things go wrong when I do
awk -v dsrmx=25 -f ./checkSRDry.awk --usage
I basically want to override the usual --usage and --help that awk gives.
How do people usually handle this situation when you also want to supply your own usage and help
concerning the... (2 Replies)
I'm trying to figure out what's getting passed as the argument when I try to pass a directory as an argument, and I'm getting incredibly strange behavior. For example, from the command line I'm typing:
nawk -f ./test.awk ~
test.awk contains the following:
{
directory = $NF
print... (13 Replies)
Hi all
I have got a file digits.data containing the following data
1 3 4
2 4 9
7 3 1
7 3 10
I am writing a script that will pass an argument from C-shell to nawk command. But it seems the values in the nawk comman does not get set. the program does not print no values out. Here is the... (1 Reply)
Dear Conerned,
I am facing a situation where i need to pass an argument which is non-awk variable like
day=090319
awk '/TID:R/ && /TTIN:/' transaction.log
I want to add this day variable like below
awk '/TID:R$day/ && /TTIN:/' transaction.log
But it is not working. :confused: (1 Reply)
Hello,
I have a file like
was123##abcdefg abddef
was123##xuzaghg agdfgg
was133##CGHAKS DKGJG
from the file i need to print the line after ## where the serach value is passed by an env variable called luster (which is currently set to was123):
i tried using the below code but it... (7 Replies)
i'm trying to pass a numerical argument with function xyz to print specfic lines of filename, but my 'awk' syntax is incorrect.
ie
xyx 3 (prints the 3rd line, separated by ':' of filename)
function xyz() {
arg1=$1
cat filename | awk -F: -v x=$arg1 '{print $x}'
}
any ideas? (4 Replies)
I am writing a shell script.
Now i need to read in a string and send it to an awk file to compare and search for compatible record.
I wrote it like tat:
read serial | awk -f generate.awk data.dat
p/s: the data file got 6 field.
According to an expert, we can write it like tat:
read... (1 Reply)