Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Merge specific columns of two files

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Merge specific columns of two files
# 1  
Old 09-01-2014
Merge specific columns of two files
Hello,

I have two tab delimited text files. Both files have the same number of rows but not necessarily the same number of columns. The column headers look like,

File 1:
Code:
f0order	CVorder	Name	f0	RI_9	E99	E199	E299	E399	E499	E599	E699	E799	E899	E999

File 2:
Code:
f0order	CVorder	Name	f0	RI_9	E399	E499	E599	E699	E799	E899	E999	E1099	E1199	E1299	E1399	E1499

I have a pair of values like E299 and E399. For those values I would want the columns up to E299 from the first file and then all the columns from E399 to the end from the second file.

desired result
Code:
f0order	CVorder	Name	f0	RI_9	E99	E199	E299	E399	E499	E599	E699	E799	E899	E999	E1099	E1199	E1299	E1399	E1499

I have seen some code that looks like,

paste file1 file2 | awk '{print $1,$2,$3,$5}'

but I'm not sure which output files are from which input files or how to specify a range of columns. I also don't know how to associates my E299 style values with the awk tokens $1,$2, etc.

Am I on the right track with this or is there some other method I should be looking at?

LMHmedchem
# 2  
Old 09-01-2014
You could use awk like this:

Code:
awk -v cs="E229" -v ce="E399" '
FNR==1 {
  if (NR>1) {
    for(s=1;s<=NF&&$s!=ce;s++);
    e=NF
  }
  else for(s=e=1;e<=NF&&$e!=cs;e++);
}
{ for(i=s;i<=e;i++) D[FNR]=D[FNR]"\t"$i }
END{ for(i=1;i<=FNR;i++) print substr(D[i],2) }' file1 file2

This User Gave Thanks to Chubler_XL For This Post:
LMHmedchem
# 3  
Old 09-01-2014
Thanks for the suggestion.

I will need to use bash variables for the values. Can I do something like,

Code:
FILE1_VALUE="E229"
FILE2_VALUE="E329"

awk -v cs=$FILE1_VALUE -v ce=$FILE2_VALUE '
FNR==1 {
  if (NR>1) {
    for(s=1;s<=NF&&$s!=ce;s++);
    e=NF
  }
  else for(s=e=1;e<=NF&&$e!=cs;e++);
}
{ for(i=s;i<=e;i++) D[FNR]=D[FNR]"\t"$i }
END{ for(i=1;i<=FNR;i++) print substr(D[i],2) }' file1 file2

or do I need a different syntax?

LMHmedchem
# 4  
Old 09-01-2014
That should be fine as long as the values don't contain white space, probably best to quote them anyway like this:

Code:
awk -v cs="$FILE1_VALUE" -v ce="$FILE2_VALUE" '

# 5  
Old 09-01-2014
One more question, I don't see the file name of the result file. Do I just add a redirect after file1 file2?

file1 file2 > my_new_filename

LMHmedchem
# 6  
Old 09-01-2014
Yes, just redirect to a new file name as you proposed.
# 7  
Old 09-02-2014
Not sure if I understood your requirement correctly, but give this a try (using adapted FILEn_VALUES as those provided do not show up in your sample):
Code:
FILE1_VALUE="E299"
FILE2_VALUE="E399"
awk     'NR==1          {DL=""
                         for (i=1; i<=NF; i++) {
                                 printf "%s%s", DL, $i
                                 DL="\t"
                                 if ($i == E1) break
                                }
                         getline < F2
                         for (j=1; j<=NF && $j != S2; j++) ;
                         for (k=j; k<=NF; k++) printf "%s%s", DL, $k
                         printf "\n"
                         next
                        }
                        {DL=""
                         for (k=1; k<=i; k++) { printf "%s%s", DL, $k; DL="\t" }
                         getline < F2
                         for (k=j; k<=NF; k++) printf "%s%s", DL, $k
                         printf "\n"
                        }
        ' FS="\t" E1="$FILE1_VALUE" S2="$FILE2_VALUE" F2="file2" file1
f0order CVorder Name    f0      RI_9    E99     E199    E299    E399    E499    E599    E699    E799    E899    E999    E1099   E1199   E1299
01      02      03      04      05      06      07      08      26      27      28      29      30      31      32      33      34      35

with file1
Code:
f0order CVorder Name    f0      RI_9    E99     E199    E299    E399    E499    E599    E699    E799    E899    E999
01      02      03      04      05      06      07      08      09      10      11      12      13      14      15

and file2
Code:
f0order CVorder Name    f0      RI_9    E399    E499    E599    E699    E799    E899    E999    E1099   E1199   E1299   E1399   E1499
21      22      23      24      25      26      27      28      29      30      31      32      33      34      35      36      37

This User Gave Thanks to RudiC For This Post:
LMHmedchem
Login or Register to Ask a Question

Previous Thread | Next Thread

9 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Merge files based on columns

011111123444 1234 1 20000 011111123444 1235 1 30000 011111123446 1234 3 40000 011111123447 1234 4 50000 011111123448 1234 3 50000 File2: 011111123444,Rsttponrfgtrgtrkrfrgtrgrer 011111123446,Rsttponrfgtrgtr 011111123447,Rsttponrfgtrguii 011111123448,Rsttponrfgtrgtjiiu I have 2 files... (4 Replies)
Discussion started by: vinus
4 Replies

2. Shell Programming and Scripting

Merge columns from multiple files

Hello and Good day I have a lot of files with same number of rows and columns.$2 and $3 are the same in all files . I need to merge $2,$3,$6 from first file and $6 from another files. File1: $1 $2 $3 $4 $5 $6... (8 Replies)
Discussion started by: ali.seifaddini
8 Replies

3. Shell Programming and Scripting

Merge columns on different files

Hello, I have two files that have this format: file 1 86.82 0.00 86.82 43.61 86.84 0.00 86.84 43.61 86.86 0.00 86.86 43.61 86.88 0.00 86.88 43.61 file 2 86.82 0.22 86.84 0.22 86.86 0.22 86.88 0.22 I would like to merge these two files such that the final file looks like... (5 Replies)
Discussion started by: kayak
5 Replies

4. UNIX for Dummies Questions & Answers

Merge columns from multiple files

Hi all, I've searched the web for a long time trying to figure out how to merge columns from multiple files. I know paste will append columns like so: paste file1 file2 file3 file4 file5 ... But this becomes inconvenient when you want to append a large number of files into a single file. ... (2 Replies)
Discussion started by: torchij
2 Replies

5. UNIX for Dummies Questions & Answers

How to count specific columns and merge with unique ones?

Hi. I am not sure the title gives an optimal description of what I want to do. I have several text files that contain data in many columns. All the files are organized the same way, but the data in the columns might differ. I want to count the number of times data occur in specific columns,... (0 Replies)
Discussion started by: JamesT
0 Replies

6. Shell Programming and Scripting

Merge columns of different files

Hi, I have tab limited file 1 and tab limited file 2 The output should contain common first column vales and corresponding 2nd column values; AND also unique first column value with corresponding 2nd column value of the file that contains it and 0 for the second file. the output should... (10 Replies)
Discussion started by: polsum
10 Replies

7. Shell Programming and Scripting

merge the two files which has contain columns

Hi may i ask how to accomplish this task: I have 2 files which has multiple columns first file 1 a 2 b 3 c 4 d second file 14 a 9 .... 13 b 10.... 12 c 11... 11 d 12... I want to merge the second file to first file that will looks like this ... (2 Replies)
Discussion started by: jao_madn
2 Replies

8. Shell Programming and Scripting

Merge 2 columns/remove specific spaces

Hi, I have a requirement to remove certain spaces from a table of information, but I'm unsure where to start. A typical table will be like this: ABCDE 1 Elton John 25 12 15 9 3 ABCDE 2 Oasis 29 13 4 6 9 ABCDE 3 The Rolling Stones 55 19 3 8 6The goal is to remove only the spaces between... (11 Replies)
Discussion started by: danhodges99
11 Replies

9. Shell Programming and Scripting

merge columns into one line after a specific pattern

Hi all, im a linux newbie, plz help! I have a file - box -------- Fox-2 -------- UF29 zip42 -------- zf-CW SNF2_N Heli_Z -------- Fox -------- Kel_1 box (3 Replies)
Discussion started by: sam_2921
3 Replies
Login or Register to Ask a Question