Shell Programming and Scripting

Hi,
I have a list of 70 files in a directory and I need to merge the content of each file into one big matrix file (71 columns x 3060 rows).

Each file has the following format only two columns per file:

Code:

unique    identifier1
randomtext1    randomtext1
a    5
b    3
c    6
d    3
e    2
f    9

Things that need to consider:
1) The first column, 'unique' is a unique field among each file, it's the second column that is different and what I need merged into one data matrix.

2) The randomtext line (second line of each file) I wish I could ignore in my final file...

3) Note, not all files are sorted as the example I posted above.

Final merged file, should be

Code:

unique    identifier1    indentifer2 . . . . . 
a    5    4 . . . . . .
b    3    2 . . . . . .
c    6    6 . . . . . .
d    3    7 . . . . . .
e    2    9 . . . . . .
f    9    4 . . . . . .

Obviously, I need 71 columns in the final file, with the first column as the 'unique' field and the other 70 columns to be the associated value from the 70 files in the common directory.

Can someone help me? Thanks in advance.

Code:

a 1
b 2
c 3

Code:

a 10
b 20
c 30

Code:

join f1 f2

Code:

a 1 10
b 2 20
c 3 30

Thank you for the reply but if I do it that way, it would only join 2 files, how can i do it for all 70 files in one directory?

I just tried that, and I forgot to mention that I would like the output to be tab deliminated.

I think you've been around here long enough to have had a go at this yourself, labrazil. :-)

Try using awk to build an array indexed by the first field. As you process each file you can just append the new values to the existing values stored in the array. Then you can just print out the entire array in your END { } clause. I'm presuming that the same number of unique indices appears in all of the files, otherwise it woudl become more complicated because you'd have to cater for the ones that were missing.

Thanks Anni,

I came across this code

Code:

awk -v OFS='\t' 'NR == FNR {
   _[$1] = $2
 next
}
$1 in _ {
$(NF+1) = _[$1] 
}
 1' 1.txt 2.txt > 3.txt

But two problems with this.
how can I incorporate it to run through all 70 files at once without doing it one by one, and two, how can remove the second line from each file as I'm merging them?

Any ideas?

That solution will be no good for your problem (unless you ran it 70 times!) because the additional field is "forgotten" as each line is processed.

Try implementing what I described rather than just searching for code.

You can avoid processing of each file the second line using FNR != 2 { ... }.