Shell Programming and Scripting

Hi,

I am pretty new to Linux and I have a question.

I have 3 tab delimited text files which look like this:

FileA:


PROTEINID DESCRIPTION PEPTIDES FRAMES

GB://115298678 _gi_115298678_ref_NP_000055.2_ complement C3 precursor [Homo sapiens] 45 55
GB://4502027 _gi_4502027_ref_NP_000468.1_ serum albumin preproprotein [Homo sapiens] 34 73
Entrez://strain 11128 / EHEC _tr_C8UFA3_C8UFA3_ECO1A Conserved predicted protein OS=Escherichia coli O111:H_ (strain 11128 / EHEC) GN=ygfY 26 31
GB://296080754 _gi_296080754_ref_NP_001171670.1_ fibrinogen beta chain isoform 2 preproprotein [Homo sapiens] 23 30
GB://4557871 _gi_4557871_ref_NP_001054.1_ serotransferrin precursor [Homo sapiens] 16 23
GB://70906439 _gi_70906439_ref_NP_068656.2_ fibrinogen gamma chain isoform gamma_B precursor [Homo sapiens] 16 20
GB://66932947 _gi_66932947_ref_NP_000005.2_ alpha_2_macroglobulin precursor [Homo sapiens] 15 17


FileB:

PROTEINID DESCRIPTION PEPTIDES FRAMES

GB://115298678 _gi_115298678_ref_NP_000055.2_ complement C3 precursor [Homo sapiens] 43 52
GB://4502027 _gi_4502027_ref_NP_000468.1_ serum albumin preproprotein [Homo sapiens] 33 71
Entrez://strain 11128 / EHEC _tr_C8UL96_C8UL96_ECO1A HCP oxidoreductase_ NADH_dependent OS=Escherichia coli O111:H_ (strain 11128 / EHEC) GN=hcr 22 24
GB://296080754 _gi_296080754_ref_NP_001171670.1_ fibrinogen beta chain isoform 2 preproprotein [Homo sapiens] 21 24
GB://4557871 _gi_4557871_ref_NP_001054.1_ serotransferrin precursor [Homo sapiens] 16 24
GB://66932947 _gi_66932947_ref_NP_000005.2_ alpha_2_macroglobulin precursor [Homo sapiens] 15 16
GB://70906439 _gi_70906439_ref_NP_068656.2_ fibrinogen gamma chain isoform gamma_B precursor [Homo sapiens] 14 18

FileC:

GB://115298678 _gi_115298678_ref_NP_000055.2_ complement C3 precursor [Homo sapiens] 43 55
GB://4502027 _gi_4502027_ref_NP_000468.1_ serum albumin preproprotein [Homo sapiens] 30 67
GB://296080754 _gi_296080754_ref_NP_001171670.1_ fibrinogen beta chain isoform 2 preproprotein [Homo sapiens] 25 28
Entrez://strain 11128 / EHEC _tr_C8UF29_C8UF29_ECO1A Protease III OS=Escherichia coli O111:H_ (strain 11128 / EHEC) GN=ptr 24 28
GB://4557871 _gi_4557871_ref_NP_001054.1_ serotransferrin precursor [Homo sapiens] 16 23
GB://70906439 _gi_70906439_ref_NP_068656.2_ fibrinogen gamma chain isoform gamma_B precursor [Homo sapiens] 15 20
GB://4557485 _gi_4557485_ref_NP_000087.1_ ceruloplasmin precursor [Homo sapiens] 15 19


Explanation of the format:
PROTEINID: GB://4557485
DESCRIPTION: _gi_4557485_ref_NP_000087.1_ ceruloplasmin precursor [Homo sapiens]
PEPTIDES: 15
FRAMES: 19


I have actually 6 such files which have thousands of entries and I want to output the ones that are only common to the three files using the 1st column as the match criterion and display everything that follows the matched entries in every file separated by tab. Can you please help me with it.

I know how to do this with just two files. Multiple files is something I haven't tried and I really need help with that!

Thanks.

If order is not important, then this should work:

(You may need to list each data file on the command line; I assumed something like data_files.* would list all 6)

Code:

sort data_files.* | awk -v expect=6 '
    function p(     x )   # print them if there was one in each file. print 1/line
    {
            if( length( buffer ) == expect )
                for( x in buffer )
                    printf( "%s\n", buffer[x] )
                printf( "\n" );          # blank line between groups
    }

    {
        if( last && $1 != last )          # next group; print last group if in all files
        {
            p();
            idx = 0;
            split( "", buffer, "." );       # poor mans delete buffer on older awk versions
        }

        buffer[++idx] = $0;
        last = $1;
    }
    END { p(); }
'

Hi,

Thank you so much.. This works perfect.

But I was looking to see whether I can get all the group information side by side instead of them in a new line? I know that'll go on to generate a lot of columns. That way I can eliminate some of the information that I do not require easily instead of eliminating that from the rows.

Foreg, I want something which looks like this:

PROTEINID DESCRIPTION PEPTIDES FRAMES PROTEINID DESCRIPTION PEPTIDES FRAMES PROTEINID DESCRIPTION PEPTIDES FRAMES
and not in the new line instead the same group are separated by tabs and the new group is separated by new line.

Can you please help me with it?

Assumption is first column in unique within each file (ie no duplicate readings), and you want to print Ids that appear in 3 or more of the 6 files (ie if ID appears in file 1,2 and 6 it should be printed).

Code:

awk -F"\t" '{A[$1]++}
{O[$1]=(($1 in O)?O[$1] FS:$1 FS)$2 FS$3 FS$4}
END{for(i in O) if(A[i]>2) print O[i];} ' data_files.*

Hi,

Thanks for your reply. yes, this is what I am looking for.. but I want only those entries that are present in all the 3 files or the 6 files that I will be working with.

I do not want print those entries that are only found in 1,2 and 6 files.

OK this version will only print if ID is in ALL supplied files:

Code:

awk -F"\t" 'FNR==1{F++}{A[$1]++}
{O[$1]=(($1 in O)?O[$1] FS:$1 FS)$2 FS$3 FS$4}
END{for(i in O) if(A[i]==F) print O[i]} ' data_files.*

It works great!.. thanks a lot!