Multiple file merge by column


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Multiple file merge by column
# 1  
Old 02-11-2013
Ubuntu Multiple file merge by column

Hello all,
I am quite new in linux shell scripting and I have this issue.
I ve got some files including measurements taken every 10minutes for a whole day.
File name format is:
Code:
00.00, 00.10, 00.20,....23.50

File structure is:
Code:
x   |  y  |  temperature

x and y is the same in all files (same sampling places). Only temperatures change.
What i want to do is merge all files in one file that will be like:

Code:
x |  y  |  temperature-filename 00.00 |  temerature-filename 00.10 | ... | temperature-filename 23.50

In other words, for fixed X and Y, to put all the temperatures one after the other incrementally and export everything in one file (not in shell because system may crash due to the number of files and data).
I tried awk, sed and grep scripts. What i find a bit tricky is to pass the file name in the script and create a loop that increases in [0-9][0-9].[0-9] [0-9] mode.
Any help would be highly appreciated. Thank you in advance.
Regards.
A.

Last edited by Scrutinizer; 02-11-2013 at 07:18 AM.. Reason: code tags
# 2  
Old 02-11-2013
Try sth like this..

Code:
 
awk -F "|" '{X[$1,$2]=X[$1,$2]?X[$1,$2] FS $3 "-" FILENAME : $0 "-" FILENAME}END{for(i in X){print X[i]}}' *.txt

As *.txtgive all your files as input.
This User Gave Thanks to pamu For This Post:
# 3  
Old 02-11-2013
Hello and thanks a lot for your immediate reply.
It was my mistake to use "|" as separator because the actual separator is " " (single space) and files are .dat
Guess the code that does the work and has a file as an output is:

Code:
awk -F " " '{X[$1,$2]=X[$1,$2]?X[$1,$2] FS $3 "-" FILENAME : $0 "-" FILENAME}END{for(i in X){print X[i]}}' *.dat >output.dat

Right?

---------- Post updated at 01:49 PM ---------- Previous update was at 01:43 PM ----------

Just tried it. It really works, apart from one thing: This code puts one file at the end of the other.
Example:
file1-00.00
x y temp
1 1 10

and file2-00.10
x y temp
1 1 12

Become file output
x y temp
file1-00.00
1 1 10
file2-00.10
1 1 12

I would prefer the output to be like
x y temp-file1-00.00 temp-file2-00.10 temp-fileXX-23.50
1 1 10 12 zz
# 4  
Old 02-11-2013
just you need small modification...

tried on two files, hope it shoud work fo all...

Code:
 
$ awk 'FNR==1{X[$1,$2,FNR]=X[$1,$2,FNR]?X[$1,$2,FNR] FS $3 "-" FILENAME : $0 "-" FILENAME}
FNR>1{X[$1,$2,FNR]=X[$1,$2,FNR]?X[$1,$2,FNR] FS $3 : $0}
{Z[$1,$2]++}
END{for(j=1;j<=2;j++){for(i in Z){if(X[i,j]){print X[i,j]}}}}' file1 file2

x y temp-file1 temp-file2
1 1 10 12


Last edited by pamu; 02-11-2013 at 08:02 AM..
This User Gave Thanks to pamu For This Post:
# 5  
Old 02-11-2013
Nope... it is breaking somewhere... :-(
I changed the file names in an easier pattern. Now they are all in 00.00.dat (00.00.dat, 00.10.dat, ... , 23.50.dat)
I attach a sample so that you can have a better view.
Thank you once again for your time!
Kind regards
A.
# 6  
Old 02-13-2013
There has to be something wrong about this code. I am still trying...
For the attached files. The output is:
Code:
 
1.00 163.00 292.3690
-00.10.dat 292.0391
-00.20.dat 291.8038
-00.30.dat
3.00 163.00 292.3675
 292.0370
 291.8015

In other words the script is not printing the x/y values and it only prints only the first price of each file (i took a sample of 3 files) in one column not in one row.
The output, according to these sample files should be like:
Code:
 x(m)   y(m) Pot. Temperature (K)-00.10.dat Pot. Temperature (K)-00.20.dat Pot. Temperature (K)-00.30.dat 
1.00 163.00 292.369 292.0391 291.8038
3.00 163.00 292.3675 292.037 291.8015
5.00 163.00 292.3661 292.0351 291.7992
7.00 163.00 292.3647 292.0332 291.7971
9.00 163.00 292.3634 292.0314 291.7951
11.00 163.00 292.3622 292.0297	291.7931
13.00 163.00 292.361 292.028 291.7911
15.00 163.00 292.3598 292.0264	291.7893
17.00 163.00 292.3587 292.0248	291.7874
19.00 163.00 292.3576 292.0232	291.7856
21.00 163.00 292.3565 292.0217	291.7838
23.00 163.00 292.3554 292.0201	291.782
25.00 163.00 292.3543 292.0186	291.7801
27.00 163.00 292.3532 292.0169	291.7783
29.00 163.00 292.3521 292.0153	291.7764
31.00 163.00 292.3508 292.0136	291.7744
33.00 163.00 292.3495 292.0118	291.7724


Last edited by atzounis; 02-13-2013 at 11:52 AM..
# 7  
Old 02-13-2013
I came up with this code:

Code:
 awk '{a[FNR]=a[FNR] FS $X;t=(FNR>T)?FNR:t}END {for (i=1;i<=t;i++) print a[i]}' *.dat > output.txt

It works perfectly for extracting the first and second columns ( X=1 or =2) but for X = 3 it puts again the column one at the end of the other, not side by side. What am I missing?
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

Cut specific column from 2 file and merge

Hi ALL, I have two file. I need to combine these two file based on a layout. I used the below code and able to extract the record. But now able to insert that to a 3'rd file in between the extract FILE 1 CAID NUMBER 1-20 TID NUMBER 21-22 LABEL CHAR 23-44 BASE 45-60... (5 Replies)
Discussion started by: arunkumar_mca
5 Replies

2. Shell Programming and Scripting

Merge column file

Hi All, I have on file1 and file2 some, $cat file1 aaa bbb ccc ddd eee fff ggg hhh iii jjj with line blank, and (12 Replies)
Discussion started by: aav1307
12 Replies

3. Shell Programming and Scripting

Seperated by columns, merge in a file, sort them on common column

Hi All, I have 4 files in below format. I took them as an example. File 1: Cut from position 1-4 then 6-7 then 8-14 then rest left and make them as columns in one new file. Inserting character H to the initial of all line like HCTOT. CTOT 456787897 Low fever CTOR 556712345 High fever... (2 Replies)
Discussion started by: Mannu2525
2 Replies

4. Shell Programming and Scripting

Merge the multiple text files into one file

Hi All, I am trying to merge all the text files into one file using below snippet cat /home/Temp/Test/Log/*.txt >> all.txt But it seems it is not working. I have multiple files like Output_ServerName1.txt, Output_ServreName2.txt I want to merge each file into one single file and... (6 Replies)
Discussion started by: sharsour
6 Replies

5. UNIX for Dummies Questions & Answers

How do I merge multiple columns into one column?

Hi all, I'm looking for a way to merge multiple columns (from one file) into a single column in an output file. The file I have looks somewhat like this: @HWI-ST212 1:N:0 AGTCCTACCGGGAGT + @@@DDDDDHHHHHII @HWI-ST212 1:N:0 CGTTTAAAAATTTCT + @;@B;DDDDH?:F;F... (4 Replies)
Discussion started by: Vnguyen
4 Replies

6. Shell Programming and Scripting

How to merge multiple rows into single row if first column matches ?

Hi, Can anyone suggest quick way to get desired output? Sample input file content: A 12 9 A -0.3 2.3 B 1.0 -4 C 34 1000 C -111 900 C 99 0.09 Output required: A 12 9 -0.3 2.3 B 1.0 -4 C 34 1000 -111 900 99 0.09 Thanks (3 Replies)
Discussion started by: cbm_000
3 Replies

7. UNIX for Advanced & Expert Users

merge two column multiple files into one

Hi I have multiple files each with two columns and I need to combine all those file into a tab delimited file. (multiple entry with same name separated by a comma) The content of the files are as follows: --- file1.txt: name var1 aaa xx aaa gg bbb yy ddd zz --- file2.txt ... (8 Replies)
Discussion started by: mary271
8 Replies

8. UNIX for Dummies Questions & Answers

Writing a loop to merge multiple files by common column

I have 100 data files labelled 250.1.txt through 250.100.txt. The second column of the data files partially match (there is about %90 overlap). Each data file has 4 columns. I want the merge all these text files by the matching values in the second column. In the output, the first column should... (1 Reply)
Discussion started by: evelibertine
1 Replies

9. Shell Programming and Scripting

Help with merge two file based on similar column content

Input file 1: A1BG A1BG A1BG A1CF A1CF BCAS BCAS A2LD1 A2M A2M HAT . . Input file 2: A1BG All A1CF TEMP (5 Replies)
Discussion started by: perl_beginner
5 Replies

10. Shell Programming and Scripting

merge two two txt files into one file based on one column

Hi, I have file1.txt and file2.txt and would like to create file3.txt based on one column in UNIX Eg: file1.txt 17328756,0000786623.pdf,0000786623 20115537,0000793892.pdf,0000793892 file2.txt 12521_74_4.zip,0000786623.pdf 12521_15_5.zip,0000793892.pdf Desired Output ... (5 Replies)
Discussion started by: techmoris
5 Replies
Login or Register to Ask a Question