Shell Programming and Scripting

Hello,

I have several handreds of text files. The format of file looks like:

column1 column2 column3 column4 column5
id1 definition1 name1 fieldid comm1
id2 definition2 name2 fieldid comm2
id3 definition3 name3 fieldid comm3
id4 definition4 name2 fieldid comm2
id5 definition5 name1 fieldid comm1
id6 definition6 name4 fieldid comm4
...
...

What I need to do is to extract all of the column3 out, remove the duplicates and transpose column3 to form a new tab dilimited file, each row should inherit the original file name. The new file should look like:

file1 name1 name2 name3 ....
file2 name-a name-b name-c ....
file3 name11 name21 name-x ...
...
...

I am new to shell scripting, awk, sed and etc. Could anyone help me out with this probem?

Thanks in advance.

Steve

Code:

for file in input*; do
   awk '{val[$3]=1} END{ printf FILENAME"\t"; for (i in val) {printf i"\t"} print}' $file >> newfile
done

Thanks so much. It works well if original file doesn't have space in the contents. But how to specify a tab or comma delimited file as input?

Thanks again.

I modified code slightly by putting FS="\t" ahead of val and it worked! As I executed code I got two more problems. First, the row header in eash file should be omitted. Second, there are some records like "---" which shouldn't be included. Can these be solved? Thanks.

or all in awk:

nawk -f shen.awk input*

shen.awk:

Code:

BEGIN {
  OFS="\t"
}
FNR > 1 && $3 != "-" {
   arr[FILENAME , $3]
}
END {
  for(i in arr) {
    split(i, tmp, SUBSEP)
    printf("%s", tmp[1] )
    for(cell in arr) {
      split(cell, ctmp, SUBSEP)
      if ( ctmp[1] == tmp[1] ) {
        printf("%s%s", OFS,ctmp[2])
        delete arr[cell]
      }
    }
    printf "\n"
  }
}