Sponsored Content
Top Forums Shell Programming and Scripting Combining certain columns of multiple files into one file Post 303003062 by ksennin on Thursday 7th of September 2017 08:51:31 AM
Old 09-07-2017
Question Combining certain columns of multiple files into one file

Hello Unix gurus,

I have a large number of files (say X) each containing two columns of data and the same number of rows.

I would like to combine these files to create a unique merged file containing X columns corresponding to the second column of each file (with a bonus of having the first column of the first file only as a first column of the merged file).

I have only started recently to use bash scripting to try and do this, so I am quite new to this, but I have been trying "paste", or even "pr" as below:
Code:
Not the actual code ("..." signify a long list of file names or columns):

pr -m -t -s\  file1 file2 file3 [...] fileX | gawk '{print $1,$2,$4,$6,$8,[...],$X }' > merged.file

I don't really know what I am doing wrong, but when I don't get an error message ("pr: page width too narrow") it is insanely slow, even on a machine with loads of RAM. Would there be a better way to do this basic transformation?

Many thanks for your help and your time!

KS
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

combining columns from different files

Hi all, I would be very grateful for some advice on the following. I have several text files. The files are experiment results with columns of data separated by white space. The files begin with several lines of header which are all preceeded by a comment character '#'. Each file has a... (10 Replies)
Discussion started by: iomaire
10 Replies

2. Shell Programming and Scripting

Combining columns from different files

I have two files I need to combine. The problem I'm having is I need to only combine data from the second file in the empty spaces of the first. For example: file1 Data Field Data Field Data Field Data Field file2 a - Insert Data b - Insert Data c - Insert Data d - Insert Data... (10 Replies)
Discussion started by: handband2
10 Replies

3. UNIX for Dummies Questions & Answers

Combining two text files as columns?

I have one space delimited file with multiple columns and one tab delimited file with multiple columns (They have the same number of rows). I want to basically combine these two text files into a new text file by column. How would I go about doing that? (1 Reply)
Discussion started by: evelibertine
1 Replies

4. Shell Programming and Scripting

Combining columns from multiple files to one file

I'm trying to combine colums from multiple file to a single file but having some issues, appreciate your help. The filenames are the same except for the extension, path1.m0 --------- a b c d e f g h i path1.m1 --------- m n o p q r s t u File names are path1.m The... (3 Replies)
Discussion started by: rkmca
3 Replies

5. Shell Programming and Scripting

Combining multiple column files into one with file name as first row

Hello All, I have several column files like this $cat a_b_s1.xls 1wert 2tg 3asd 4asdf 5asdf $cat c_d_s2.xls 1wert 2tg 3asd 4asdf 5asdf desired put put $cat combined.txt s1 s2 (2 Replies)
Discussion started by: avatar_007
2 Replies

6. Shell Programming and Scripting

Combining multiple files

I have 2 files. each having 3 coloums 1st field date as 20130322 2nd field time as 05:55 3rd field numberic value File 2 has entries missing for some date time. FILE1 20130322 05:35 2219 20130322 05:40 1809 20130322 05:45 1617 20130322 05:50 ... (2 Replies)
Discussion started by: sandeepkmehra
2 Replies

7. Shell Programming and Scripting

Combining columns from multiple files into one single output file

Hi, I have 3 files with one column value as shown File: a.txt ------------ Data_a1 Data_a2 File2: b.txt ------------ Data_b1 Data_b2 Data_b3 Data_b4 File3: c.txt ------------ Data_c1 Data_c2 Data_c3 Data_c4 Data_c5 (6 Replies)
Discussion started by: vfrg
6 Replies

8. Linux

[Solved] Combining columns from different files

Hey Guys & Gals, I am stuck with the following ; I have 2 text files, each containing 2 columns. My goal is to have a column from the 2nd file placed inbetween the columns in the first file. Basically the idea is, each address has a different name (but 1 name per address) but 1 address... (6 Replies)
Discussion started by: TAPE
6 Replies

9. Shell Programming and Scripting

Join two files combining multiple columns and produce mix and match output

I would like to join two files when two columns in each file matches with each other and then produce an output when taking multiple columns. Like I have file A 1234,ABCD,23,JOHN,NJ,USA 2345,ABCD,24,SAM,NY,USA 5678,GHIJ,24,TOM,NY,USA 5678,WXYZ,27,MAT,NJ,USA and file B ... (2 Replies)
Discussion started by: mady135
2 Replies

10. UNIX for Beginners Questions & Answers

Combining multiple files into one

Hello Everyone, I have 4 different files (one column in each) that I'm trying to combine into 1 file with four columns. Having issues trying to get the columns to format properly. I have tried the following: paste file1 file2 file3 file4 | column -s $'\t' -t > results.txt paste file1 file2... (1 Reply)
Discussion started by: malk71
1 Replies
paste(1)						      General Commands Manual							  paste(1)

Name
       paste - merge file data

Syntax
       paste file1 file2...
       paste -dlist file1 file2...
       paste -s [-dlist] file1 file2...

Description
       In  the	first  two forms, concatenates corresponding lines of the given input files file1, file2, etc.	It treats each file as a column or
       columns of a table and pastes them together horizontally (parallel merging).

       In the last form, the command combines subsequent lines of the input file (serial merging).

       In all cases, lines are glued together with the tab character, or with characters from an optionally specified  list.   Output  is  to  the
       standard output, so it can be used as the start of a pipe, or as a filter, if - is used in place of a file name.

Options
       -       Used in place of any file name, to read a line from the standard input.	(There is no prompting).

       -dlist  Replaces  characters  of  all but last file with nontabs characters (default tab).  One or more characters immediately following -d
	       replace the default tab as the line concatenation character.  The list is used circularly, i. e. when exhausted, it is reused.	In
	       parallel  merging  (i. e. no -s option), the lines from the last file are always terminated with a new-line character, not from the
	       list.  The list may contain the special escape sequences: 
 (new-line), 	 (tab), \ (backslash), and  (empty string, not a null
	       character).   Quoting  may  be  necessary,  if characters have special meaning to the shell (for example, to get one backslash, use
	       -d"\\" ).
	       Without this option, the new-line characters of each but the last file (or last line in case of the -s option) are  replaced  by  a
	       tab character.  This option allows replacing the tab character by one or more alternate characters (see below).

       -s      Merges  subsequent  lines  rather  than	one  from  each input file.  Use tab for concatenation, unless a list is specified with -d
	       option.	Regardless of the list, the very last character of the file is forced to be a new-line.

Examples
       ls | paste -d" " -
       list directory in one column
       ls | paste - - - -
       list directory in four columns
       paste -s -d"	
" file
       combine pairs of lines into lines

Diagnostics
       line too long
		 Output lines are restricted to 511 characters.

       too many files
		 Except for -s option, no more than 12 input files may be specified.

See Also
       cut(1), grep(1), pr(1)

																	  paste(1)
All times are GMT -4. The time now is 11:55 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy