Shell Programming and Scripting

Greetings,

I have about 3000 files that I want to search. The first column in all of these 3000 files has a unique serial number on each line. The subsequent columns have lots of data.
I have another masterfile with three columns to help me find all the data I need in a moments notice:

col 1 col2 col3
serial.1 row# file1
serial.2 row# file345
serial.3 row# file1023


what I want to do take list of serial numbers, get the file name and row number for where the data sits, and then spit out the data into one file. I figured awk getline might help call up the filename and row name to go searching, but I am really confused with this command.
Another thought was to make a shell script to go down each row in the masterfile, create a variable for "row number variable" and one for "filenamevariable" and then awk '(NF=rownumbervariable){print $0}' filenamevariable >>outputfile

but cant get this working either.


ANy suggestions?

Thanks,
jeeplou

Welcome to the Forum,

Can you please post the sample input and expected output ( might be you can place two or three files ).

1. large datafiles to be queried called "file.dat*" (column one is serial number, others are data)


file.dat1

rs10001 900 900 100 100
rs10002 800 300 200 100

file.dat2
rs10003 222 111 333 444
rs10004 999 121 232 434



2. small masterlist/cheatsheet called "master.list" with three columns: serial#,row#,filename

rs10001 1 file.dat1
rs10002 2 file.dat1
rs10003 1 file.dat2
rs10004 2 file.dat2


3. If I want to recover rs10001 and rs10004 data, I grep from my masterlist to create a subset of insterest called subset.txt

subset.txt
rs10001 1 file.dat1
rs10004 2 file.dat2


Now, I need to recover the original data for each serial number with the expected output:

rs10001 900 900 100 100
rs10004 999 121 232 434

Thanks,
jeeplou

Try this:

Code:

awk 'NR==FNR{a[$1]=$1;next}a[$1]' subset.txt file.dat*

thanks for the speedy reply. I understand each individual element in your code, but since I'm not sure what it is "saying", I'm not sure how to set this up for my data....can you elaborate briefly please?


also, is it not helpful to use the file name and row number to help awk figure out where to look?

Have you tried the command?

Place file.dat1, file.dat2 and subset.txt in a directory, run the command to see what happens.

it most definitely works!
what I understand from the code is that you are creating an array that puts all of the serial numbers together and then testing each line of the array as a pattern in all of the files.

My concern is the length of time this will take over 64GB of data...I havent tried this yet, but will get the LSF farm going to see what happens. Is there a way to use the file name and row number to speed things up?


thanks again.
jonah