Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Combining certain columns of multiple files into one file

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Combining certain columns of multiple files into one file
# 1  
Old 09-07-2017
Question Combining certain columns of multiple files into one file
Hello Unix gurus,

I have a large number of files (say X) each containing two columns of data and the same number of rows.

I would like to combine these files to create a unique merged file containing X columns corresponding to the second column of each file (with a bonus of having the first column of the first file only as a first column of the merged file).

I have only started recently to use bash scripting to try and do this, so I am quite new to this, but I have been trying "paste", or even "pr" as below:
Code:
Not the actual code ("..." signify a long list of file names or columns):

pr -m -t -s\  file1 file2 file3 [...] fileX | gawk '{print $1,$2,$4,$6,$8,[...],$X }' > merged.file

I don't really know what I am doing wrong, but when I don't get an error message ("pr: page width too narrow") it is insanely slow, even on a machine with loads of RAM. Would there be a better way to do this basic transformation?

Many thanks for your help and your time!

KS
# 2  
Old 09-07-2017
Quote:
Originally Posted by ksennin
Hello Unix gurus,

I have a large number of files (say X) each containing two columns of data and the same number of rows.

I would like to combine these files to create a unique merged file containing X columns corresponding to the second column of each file (with a bonus of having the first column of the first file only as a first column of the merged file).

I have only started recently to use bash scripting to try and do this, so I am quite new to this, but I have been trying "paste", or even "pr" as below:
Code:
Not the actual code ("..." signify a long list of file names or columns):

pr -m -t -s\  file1 file2 file3 [...] fileX | gawk '{print $1,$2,$4,$6,$8,[...],$X }' > merged.file

I don't really know what I am doing wrong, but when I don't get an error message ("pr: page width too narrow") it is insanely slow, even on a machine with loads of RAM. Would there be a better way to do this basic transformation?

Many thanks for your help and your time!

KS
read the man pages for the cut and paste commands.

---------- Post updated at 10:05 AM ---------- Previous update was at 10:01 AM ----------

Quote:
Originally Posted by jgt
read the man pages for the cut and paste commands.
use the cut command to retrieve only the second field in all but the first file. then paste all the resultant files to the first.
This User Gave Thanks to jgt For This Post:
ksennin
# 3  
Old 09-07-2017
With a recent bash that provides "process substitution", try
Code:
join <(join file[12]) <(join file[34])

This User Gave Thanks to RudiC For This Post:
ksennin
# 4  
Old 09-07-2017
Thank you very much to both for helpful replies
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

Combining multiple files into one

Hello Everyone, I have 4 different files (one column in each) that I'm trying to combine into 1 file with four columns. Having issues trying to get the columns to format properly. I have tried the following: paste file1 file2 file3 file4 | column -s $'\t' -t > results.txt paste file1 file2... (1 Reply)
Discussion started by: malk71
1 Replies

2. Shell Programming and Scripting

Join two files combining multiple columns and produce mix and match output

I would like to join two files when two columns in each file matches with each other and then produce an output when taking multiple columns. Like I have file A 1234,ABCD,23,JOHN,NJ,USA 2345,ABCD,24,SAM,NY,USA 5678,GHIJ,24,TOM,NY,USA 5678,WXYZ,27,MAT,NJ,USA and file B ... (2 Replies)
Discussion started by: mady135
2 Replies

3. Linux

[Solved] Combining columns from different files

Hey Guys & Gals, I am stuck with the following ; I have 2 text files, each containing 2 columns. My goal is to have a column from the 2nd file placed inbetween the columns in the first file. Basically the idea is, each address has a different name (but 1 name per address) but 1 address... (6 Replies)
Discussion started by: TAPE
6 Replies

4. Shell Programming and Scripting

Combining columns from multiple files into one single output file

Hi, I have 3 files with one column value as shown File: a.txt ------------ Data_a1 Data_a2 File2: b.txt ------------ Data_b1 Data_b2 Data_b3 Data_b4 File3: c.txt ------------ Data_c1 Data_c2 Data_c3 Data_c4 Data_c5 (6 Replies)
Discussion started by: vfrg
6 Replies

5. Shell Programming and Scripting

Combining multiple files

I have 2 files. each having 3 coloums 1st field date as 20130322 2nd field time as 05:55 3rd field numberic value File 2 has entries missing for some date time. FILE1 20130322 05:35 2219 20130322 05:40 1809 20130322 05:45 1617 20130322 05:50 ... (2 Replies)
Discussion started by: sandeepkmehra
2 Replies

6. Shell Programming and Scripting

Combining multiple column files into one with file name as first row

Hello All, I have several column files like this $cat a_b_s1.xls 1wert 2tg 3asd 4asdf 5asdf $cat c_d_s2.xls 1wert 2tg 3asd 4asdf 5asdf desired put put $cat combined.txt s1 s2 (2 Replies)
Discussion started by: avatar_007
2 Replies

7. Shell Programming and Scripting

Combining columns from multiple files to one file

I'm trying to combine colums from multiple file to a single file but having some issues, appreciate your help. The filenames are the same except for the extension, path1.m0 --------- a b c d e f g h i path1.m1 --------- m n o p q r s t u File names are path1.m The... (3 Replies)
Discussion started by: rkmca
3 Replies

8. UNIX for Dummies Questions & Answers

Combining two text files as columns?

I have one space delimited file with multiple columns and one tab delimited file with multiple columns (They have the same number of rows). I want to basically combine these two text files into a new text file by column. How would I go about doing that? (1 Reply)
Discussion started by: evelibertine
1 Replies

9. Shell Programming and Scripting

Combining columns from different files

I have two files I need to combine. The problem I'm having is I need to only combine data from the second file in the empty spaces of the first. For example: file1 Data Field Data Field Data Field Data Field file2 a - Insert Data b - Insert Data c - Insert Data d - Insert Data... (10 Replies)
Discussion started by: handband2
10 Replies

10. Shell Programming and Scripting

combining columns from different files

Hi all, I would be very grateful for some advice on the following. I have several text files. The files are experiment results with columns of data separated by white space. The files begin with several lines of header which are all preceeded by a comment character '#'. Each file has a... (10 Replies)
Discussion started by: iomaire
10 Replies
Login or Register to Ask a Question