Multiple file merge by column | Unix Linux Forums | Shell Programming and Scripting

  Go Back    


Shell Programming and Scripting Post questions about KSH, CSH, SH, BASH, PERL, PHP, SED, AWK and OTHER shell scripts and shell scripting languages here.

Multiple file merge by column

Shell Programming and Scripting


Closed Thread    
 
Thread Tools Search this Thread Display Modes
    #1  
Old 02-11-2013
atzounis atzounis is offline
Registered User
 
Join Date: Feb 2013
Last Activity: 14 February 2013, 4:33 AM EST
Posts: 8
Thanks: 3
Thanked 0 Times in 0 Posts
Ubuntu Multiple file merge by column

Hello all,
I am quite new in linux shell scripting and I have this issue.
I ve got some files including measurements taken every 10minutes for a whole day.
File name format is:
Code:
00.00, 00.10, 00.20,....23.50

File structure is:
Code:
x   |  y  |  temperature

x and y is the same in all files (same sampling places). Only temperatures change.
What i want to do is merge all files in one file that will be like:


Code:
x |  y  |  temperature-filename 00.00 |  temerature-filename 00.10 | ... | temperature-filename 23.50

In other words, for fixed X and Y, to put all the temperatures one after the other incrementally and export everything in one file (not in shell because system may crash due to the number of files and data).
I tried awk, sed and grep scripts. What i find a bit tricky is to pass the file name in the script and create a loop that increases in [0-9][0-9].[0-9] [0-9] mode.
Any help would be highly appreciated. Thank you in advance.
Regards.
A.

Last edited by Scrutinizer; 02-11-2013 at 06:18 AM.. Reason: code tags
Sponsored Links
    #2  
Old 02-11-2013
pamu pamu is offline
Registered User
 
Join Date: Mar 2012
Last Activity: 15 September 2014, 8:06 AM EDT
Posts: 1,649
Thanks: 58
Thanked 478 Times in 474 Posts
Try sth like this..


Code:
 
awk -F "|" '{X[$1,$2]=X[$1,$2]?X[$1,$2] FS $3 "-" FILENAME : $0 "-" FILENAME}END{for(i in X){print X[i]}}' *.txt

As *.txt give all your files as input.
The Following User Says Thank You to pamu For This Useful Post:
atzounis (02-11-2013)
Sponsored Links
    #3  
Old 02-11-2013
atzounis atzounis is offline
Registered User
 
Join Date: Feb 2013
Last Activity: 14 February 2013, 4:33 AM EST
Posts: 8
Thanks: 3
Thanked 0 Times in 0 Posts
Hello and thanks a lot for your immediate reply.
It was my mistake to use "|" as separator because the actual separator is " " (single space) and files are .dat
Guess the code that does the work and has a file as an output is:


Code:
awk -F " " '{X[$1,$2]=X[$1,$2]?X[$1,$2] FS $3 "-" FILENAME : $0 "-" FILENAME}END{for(i in X){print X[i]}}' *.dat >output.dat

Right?

---------- Post updated at 01:49 PM ---------- Previous update was at 01:43 PM ----------

Just tried it. It really works, apart from one thing: This code puts one file at the end of the other.
Example:
file1-00.00
x y temp
1 1 10

and file2-00.10
x y temp
1 1 12

Become file output
x y temp
file1-00.00
1 1 10
file2-00.10
1 1 12

I would prefer the output to be like
x y temp-file1-00.00 temp-file2-00.10 temp-fileXX-23.50
1 1 10 12 zz
    #4  
Old 02-11-2013
pamu pamu is offline
Registered User
 
Join Date: Mar 2012
Last Activity: 15 September 2014, 8:06 AM EDT
Posts: 1,649
Thanks: 58
Thanked 478 Times in 474 Posts
just you need small modification...

tried on two files, hope it shoud work fo all...


Code:
 
$ awk 'FNR==1{X[$1,$2,FNR]=X[$1,$2,FNR]?X[$1,$2,FNR] FS $3 "-" FILENAME : $0 "-" FILENAME}
FNR>1{X[$1,$2,FNR]=X[$1,$2,FNR]?X[$1,$2,FNR] FS $3 : $0}
{Z[$1,$2]++}
END{for(j=1;j<=2;j++){for(i in Z){if(X[i,j]){print X[i,j]}}}}' file1 file2

x y temp-file1 temp-file2
1 1 10 12


Last edited by pamu; 02-11-2013 at 07:02 AM..
The Following User Says Thank You to pamu For This Useful Post:
atzounis (02-11-2013)
Sponsored Links
    #5  
Old 02-11-2013
atzounis atzounis is offline
Registered User
 
Join Date: Feb 2013
Last Activity: 14 February 2013, 4:33 AM EST
Posts: 8
Thanks: 3
Thanked 0 Times in 0 Posts
Nope... it is breaking somewhere... :-(
I changed the file names in an easier pattern. Now they are all in 00.00.dat (00.00.dat, 00.10.dat, ... , 23.50.dat)
I attach a sample so that you can have a better view.
Thank you once again for your time!
Kind regards
A.
Attached Files
File Type: gz files.tar.gz (64.7 KB, 13 views)
Sponsored Links
    #6  
Old 02-13-2013
atzounis atzounis is offline
Registered User
 
Join Date: Feb 2013
Last Activity: 14 February 2013, 4:33 AM EST
Posts: 8
Thanks: 3
Thanked 0 Times in 0 Posts
There has to be something wrong about this code. I am still trying...
For the attached files. The output is:

Code:
 
1.00 163.00 292.3690
-00.10.dat 292.0391
-00.20.dat 291.8038
-00.30.dat
3.00 163.00 292.3675
 292.0370
 291.8015

In other words the script is not printing the x/y values and it only prints only the first price of each file (i took a sample of 3 files) in one column not in one row.
The output, according to these sample files should be like:

Code:
 x(m)   y(m) Pot. Temperature (K)-00.10.dat Pot. Temperature (K)-00.20.dat Pot. Temperature (K)-00.30.dat 
1.00 163.00 292.369 292.0391 291.8038
3.00 163.00 292.3675 292.037 291.8015
5.00 163.00 292.3661 292.0351 291.7992
7.00 163.00 292.3647 292.0332 291.7971
9.00 163.00 292.3634 292.0314 291.7951
11.00 163.00 292.3622 292.0297	291.7931
13.00 163.00 292.361 292.028 291.7911
15.00 163.00 292.3598 292.0264	291.7893
17.00 163.00 292.3587 292.0248	291.7874
19.00 163.00 292.3576 292.0232	291.7856
21.00 163.00 292.3565 292.0217	291.7838
23.00 163.00 292.3554 292.0201	291.782
25.00 163.00 292.3543 292.0186	291.7801
27.00 163.00 292.3532 292.0169	291.7783
29.00 163.00 292.3521 292.0153	291.7764
31.00 163.00 292.3508 292.0136	291.7744
33.00 163.00 292.3495 292.0118	291.7724

Attached Files
File Type: gz files.tar.gz (396 Bytes, 10 views)

Last edited by atzounis; 02-13-2013 at 10:52 AM..
Sponsored Links
    #7  
Old 02-13-2013
atzounis atzounis is offline
Registered User
 
Join Date: Feb 2013
Last Activity: 14 February 2013, 4:33 AM EST
Posts: 8
Thanks: 3
Thanked 0 Times in 0 Posts
I came up with this code:


Code:
 awk '{a[FNR]=a[FNR] FS $X;t=(FNR>T)?FNR:t}END {for (i=1;i<=t;i++) print a[i]}' *.dat > output.txt

It works perfectly for extracting the first and second columns ( X=1 or =2) but for X = 3 it puts again the column one at the end of the other, not side by side. What am I missing?
Sponsored Links
Closed Thread

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

More UNIX and Linux Forum Topics You Might Find Helpful
Thread Thread Starter Forum Replies Last Post
How do I merge multiple columns into one column? Vnguyen UNIX for Dummies Questions & Answers 4 12-19-2012 03:52 AM
How to merge multiple rows into single row if first column matches ? cbm_000 Shell Programming and Scripting 3 05-29-2012 03:48 AM
merge two column multiple files into one mary271 UNIX for Advanced & Expert Users 8 01-21-2012 03:08 PM
Writing a loop to merge multiple files by common column evelibertine UNIX for Dummies Questions & Answers 1 10-07-2011 11:20 AM
merge two two txt files into one file based on one column techmoris Shell Programming and Scripting 5 08-28-2009 06:21 PM



All times are GMT -4. The time now is 12:45 PM.