Shell Programming and Scripting

Hi all,
I would be very grateful for some advice on the following.

I have several text files. The files are experiment results with columns of data separated by white space.

The files begin with several lines of header which are all preceeded by a comment character '#'.

Each file has a record number column as its first column and then several columns of values relating to that record. The record numbers go from 1-N in each experiment file

I would like to compare a particular column across several experiment files.

How to I take two of these experiment files and create a new file such that the new file contains:
- none of the header information
- the first column is the record column
- the next column is ,say column 2 from the first file
- the next column is the same column 2 from the second file

I've looked into awk and join but am not sure how to use them in combination to achieve what I want.

Thanks,
Enda

Sample file
File1.dat:
################
# Header info
################
1 10 20 30
2 13 50 13
3 ....
etc

You can work out the desired details for the printout - I just made it: [record#:column#]
nawk -f io.awk file1 file2 fileN

io.awk:

Code:

!/^[#]/ {
  for(i=2; i <= NF; i++) {
    idx = $1 SUBSEP i
    a[idx] = ( idx in a ) ? a[idx] OFS $i : $i
  }
}
END {
  for( rec in a ) {
     split(rec, idxA, SUBSEP)
     printf("[%d:%d]%s%s\n", idxA[1], idxA[2], OFS, a[rec])
  }
}

Thanks very much for your help. I ran the script with gawk and it worked fine. It produces an output with a record for each row:column where the values of the fields are the values of the corresponding row:column in each of the input files.

This is a good starting point from which to combine the input files in many different ways.

How would I post-process the output file, to leave me with files of the following format:

record # column_value_(File1) column_value_(File2) column_value_(FileN)
1
2
3

where there is a file for each of the column values. That is col1 across all input files, column 2 across all input files etc

Thanks very much.

something like this [if I understand your formatting correctly]:

Code:

!/^[#]/ {
  for(i=2; i <= NF; i++) {
    idx = $1 SUBSEP i
    val = i "_" $i "_(" FILENAME ")"
    a[idx] = ( idx in a ) ? a[idx] OFS val : val
  }
}
END {
  for( rec in a ) {
     split(rec, idxA, SUBSEP)
     #printf("[%d:%d]%s%s\n", idxA[1], idxA[2], OFS, a[rec])
     printf("%d%s%s\n", idxA[1], OFS, a[rec])
  }
}

Hi again,

I'm sorry I'm not being clear. I've attached a bash script that does what I need but it's messy.
- it assumes that all files with a certain name pattern in a directory are to be included
- the column to extract is hard-coded as I couldn't pass it to awk within the script

Scenario: each file is the result of an experiment. Each column is a particular value measured in the experiment. Each row is the index of the measurement taken.
I want to a create summary file across all experiment result files such that a summary file joins a given colum from each experiment file.

Example problem:
File_A.dat
#################
# Comments
# Header
#################
1 val_A12 val_A13
2 val_A22 val_A23
3 val_A32 val_A33

File_B.dat
#################
# Comments
# Header
#################
1 val_B12 val_B13
2 val_B22 val_B23
3 val_B32 val_B33

Files might have up to 2500 rows and 30 columns.

Want to pass to a script 'Give me all column 2's from File_A.dat File_B.dat ... File_30.dat

Desired result file (for column 2's):
Result.dat:
1 val_A12 val_B12
2 val_A22 val_B22
2 val_A32 val_B32
etc

Like I say, my script does the job but hopefully you can think of a neater way to do this.

Regards,
Éanna

to get the SECOND column - change 'c=2' to the column you'd like.

nawk -v c=2 io.awk file1 file2 fileN

Code:

!/^[#]/ {
  if ( c > 0 && c <= NF ) {
    idx = $1 SUBSEP c
    a[idx] = ( idx in a ) ? a[idx] OFS $c : $c
  }
}
END {
  for( rec in a ) {
     split(rec, idxA, SUBSEP)
     printf("%d%s%s\n", idxA[1], OFS, a[rec])
  }
}

Thanks. I really appreciate all your help.

Here are the commands that get me the result I need:

gawk -v c=2 -f io.awk ./result*.dat >unsorted.dat
gawk ' { print | "sort" }' sorted.dat >final.dat