Combine columns from 100 files with same structure Post: 302695333

Sponsored Content

Top Forums UNIX for Dummies Questions & Answers Combine columns from 100 files with same structure Post 302695333 by Don Cragun on Sunday 2nd of September 2012 10:06:35 PM

09-02-2012

Registered User

The solution provided my mirni seems to work OK except that it doesn't print the key at the start of each output line. (It also prints the 0 value as 0.000, but I don't see why that really matters.) It does however invoke awk once for each input file and once for each key, and it invokes grep on each input file once for each key.

I believe that the following provides all of the requested features only invoking awk once and sort once (with no need for grep). However, this script requires that all of the data in the input files be accumulated in awk's address space at once while mirni's script just requires that the entries for a single key be kept in memory at once.

To run this, save the following script in a file (e.g.,combine):

Code:

#!/bin/ksh
if [ $# -lt 2 ]
then
    printf "Usage: %s input_file...\n%s\n" "$(basename "$0")" \
        "    At least two input_file operands must be supplied."
    exit 1
fi
awk 'BEGIN {
    fc = -1
}
FNR==1 {fc++
}
  { if(outc[$1] < fc) {
        # Put in "-" entries for files that did not have a match for $1 in the
        # previous input_file operand.
        for(i = outc[$1]; i < fc; i++)
            out[$1] = out[$1] sprintf(" %-5s %-5s %-5s", "-", "-", "-")
    }
    # Add the entries for this file.
    outc[$1] = fc + 1
    out[$1] = out[$1] sprintf(" %-5.3g %-5.3g %-5.3g", $2, $3, $4)
    key[$1] = $1
    if(length($1) > maxl) maxl = length($1)
}
END{for(i in key) {
        if (outc[key[i]] <= fc) {
            # Put  in "=" entries for files that did not have a match in the
            # last input_file operand.
            for(j = outc[i]; j <= fc; j++)
                out[i] = out[i] sprintf(" %-5s %-5s %-5s", "-", "-", "-")
        }
        printf("%-*s%s\n", maxl, key[i], out[i])
    }
}' "$@" | sort -n

Make it executable (chmod +x combine), and run it with something like the following:

Code:

combine PUR.* > output_file

Although the script above specifies ksh, it will also work with at least bash and sh.

This User Gave Thanks to Don Cragun For This Post:

Don Cragun

View Public Profile for Don Cragun

Find all posts by Don Cragun

10 More Discussions You Might Find Interesting

1. UNIX for Advanced & Expert Users

MV files from one directory structure(multiple level) to other directory structure

Hi, I am trying to write a script that will move all the files from source directory structure(multiple levels might exist) to destination directory structure. If a sub folder is source doesnot exist in destination then I have to skip and goto next level. I also need to delete the files in...

2. Shell Programming and Scripting

Combine multiple columns from multiple files

Hi there, I was wondering if someone can help me with this. I am trying the combine multiple columns from multiple files into one file. Example file 1: c0t0d0 c0t2d0 # hostname vgname c0t0d1 c0t2d1 # hostname vgname c0t0d2 c0t2d2 # hostname vgname c0t1d0 c0t3d0 # hostname vgname1...

3. UNIX for Dummies Questions & Answers

combine the values from the first two columns within a file

Hello everybody, I have a text file containing 10,000 rows and 5000 columns. The values are separated by a tab. Ex. file_ex.ped 1 mike 0 0 2 1 A A G G C T A G 1 jack 0 0 2 2 T A G T C A A C 1 Mary 0 0 1 2 A T G C A T G C ... I would like a out put file 1 mike 0 0 2 1 AA GG CT AG 1...

4. Shell Programming and Scripting

How to combine 2 files into 1 file with 2 columns

Hi Guys, I want to combine 2 files and and put together in 1 file and make two columns out it. See below desired output. Any help will be much appreciated. inputfile1.txt 12345 67890 24580 inputfile2.txt AAAAA BBBBB CCCCC

5. UNIX for Dummies Questions & Answers

How to combine 2 files with 6 columns?

This may seem obvious but I am having problems doing this as columns get converted to rows when i try to write a script. I have 2 files text1.txt and text2.txt each of which have 6 columns of numbers separated by a space. I need to combine the 2 files so that the output file text3.txt maintains...

6. Shell Programming and Scripting

Combine columns from multiple files

Can anybody help on the script to combine/concatenate columns from multiple files input1 4 135 5 185 6 85 11 30 16 72 17 30 21 52 22 76 input2 2 50 4 50 6 33 8 62 10 25 12 46 14 42 15 46output

7. Shell Programming and Scripting

Combine columns from many files but keep them aligned in columns-shorter left column issue

Hello everyone, I searched the forum looking for answers to this but I could not pinpoint exactly what I need as I keep having trouble. I have many files each having two columns and hundreds of rows. first column is a string (can have many words) and the second column is a number.The files are...

8. Shell Programming and Scripting

Combine columns - awk

Need some help with this ... please 60644,NJ090237_0263_GRP,NJ090237_0263_VIEW,NJ090237_0263_PSGRP,NJ090237_0263_GOLD_CSGRP,,06E:0_08E:0_09E:0_11E:0,0CE5,TDEV,34,VP_TIER...

9. Shell Programming and Scripting

Combine Columns

Input NJ090237_0263_GRP,NJ090237_0263_VIEW,NJ090237_0263_PSGRP,NJ090237_0263_GOLD_CSGRP,06E:0_08E:0_09E:0_11E:0,0CE5 NJ090237_0264_GRP,NJ090237_0263_VIEW,NJ090237_0264_PSGRP,NJ090237_0263_GOLD_CSGRP,06E:0_08E:0_09E:0_11E:0,0CE5...

10. UNIX for Beginners Questions & Answers

Automate splitting of files , scp files as each split completes and combine files on target server

i use the split command to split a one terabyte backup file into 10 chunks of 100 GB each. The files are split one after the other. While the files is being split, I will like to scp the files one after the other as soon as the previous one completes, from server A to Server B. Then on server B ,...

LEARN ABOUT REDHAT

igawk

IGAWK(1)							 Utility Commands							  IGAWK(1)

NAME

       igawk - gawk with include files

SYNOPSIS

       igawk [ all gawk options ] -f program-file [ -- ] file ...
       igawk [ all gawk options ] [ -- ] program-text file ...

DESCRIPTION

       Igawk is a simple shell script that adds the ability to have ``include files'' to gawk(1).

       AWK programs for igawk are the same as for gawk, except that, in addition, you may have lines like

	      @include getopt.awk

       in your program to include the file getopt.awk from either the current directory or one of the other directories in the search path.

OPTIONS

       See gawk(1) for a full description of the AWK language and the options that gawk supports.

EXAMPLES

       cat << EOF > test.awk
       @include getopt.awk

       BEGIN {
	    while (getopt(ARGC, ARGV, "am:q") != -1)
		 ...
       }
       EOF

       igawk -f test.awk

SEE ALSO

       gawk(1)

       Effective AWK Programming, Edition 1.0, published by the Free Software Foundation, 1995.

AUTHOR

       Arnold Robbins (arnold@skeeve.com).

Free Software Foundation					    Nov 3 1999								  IGAWK(1)