Sponsored Content
Top Forums Shell Programming and Scripting Join multiple files by column with awk Post 302427218 by ikki on Friday 4th of June 2010 08:19:25 AM
Old 06-04-2010
awk is not necessary for this task

If you don't have to use awk, the result you ask for can perhaps be achieved using basic shell tools and sed.

Eg. if the files are called file1, file2 and file3 and have the headers stripped off you could achieve the desired result this way:

Code:
$ cat file1
cnvi0000001 5 164388439 -0.4241 0.0097
cnvi0000002 5 165771245 0.4448 1
cnvi0000003 5 165772271 0.4321 0
cnvi0000004 5 166325838 0.0403 0.9971
cnvi0000005 5 166710354 0.2355 0
$ cat file2
cnvi0000001 5 164388439 0.0736 0
cnvi0000002 5 165771245 0.1811 1
cnvi0000003 5 165772271 0.2955 0.0042
cnvi0000004 5 166325838 -0.118 0.9883
$ cat file3
cnvi0000001 5 164388439 0.2449 0
cnvi0000002 5 165771245 -0.0163 1
cnvi0000003 5 165772271 0.3361 0
cnvi0000004 5 166325838 0.0307 0.9867
cnvi0000005 5 166710354 0.1529 0
$ paste file* | sed -e 's/\t\t/\t     /g;s/\t/ /g;s/ /\t/g' | cut  -f 2,3,4,9,14
5       164388439       -0.4241 0.0736  0.2449
5       165771245       0.4448  0.1811  -0.0163
5       165772271       0.4321  0.2955  0.3361
5       166325838       0.0403  -0.118  0.0307
5       166710354       0.2355          0.1529

This assumes that the fields in file1, file2 and file3 are separated with spaces. If they are sepatared by tabs, the sed command has to be modified so it prints tabs instead of spaces, but I didn't test it.

EDIT: You should post more details of the data you need to manipulate. For example, if more than one files are shorter than the others, the above will not work reliably. Also, is it possible that some records are skipped? For example a line starting with cnvi0000004 is immediately followed by a line starting with cnvi0000006. The best solution in my opinion would be to preprocess the files so they are of equal length of lines, and insert "empty" data for missing fields, eg. "-".

EDIT2: A more robust sed command handling possible consecutive empty records:
Code:
$ paste file* file2 file2 file3 | sed -e 's/\([^\t]\)\t/\1 /g;s/\t/     /g;s/\t/ /g;s/ /\t/g' | cut  -f 2,3,4,9,14,19,24,29
5       164388439       -0.4241 0.0736  0.2449  0.0736  0.0736  0.2449
5       165771245       0.4448  0.1811  -0.0163 0.1811  0.1811  -0.0163
5       165772271       0.4321  0.2955  0.3361  0.2955  0.2955  0.3361
5       166325838       0.0403  -0.118  0.0307  -0.118  -0.118  0.0307
5       166710354       0.2355          0.1529                  0.1529


Last edited by ikki; 06-04-2010 at 10:11 AM.. Reason: addemdum
 

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Join 2 files with multiple columns: awk/grep/join?

Hello, My apologies if this has been posted elsewhere, I have had a look at several threads but I am still confused how to use these functions. I have two files, each with 5 columns: File A: (tab-delimited) PDB CHAIN Start End Fragment 1avq A 171 176 awyfan 1avq A 172 177 wyfany 1c7k A 2 7... (3 Replies)
Discussion started by: InfoSeeker
3 Replies

2. Shell Programming and Scripting

Join multiple files based on 1 common column

I have n files (for ex:64 files) with one similar column. Is it possible to combine them all based on that column ? file1 ax100 20 30 40 ax200 22 33 44 file2 ax100 10 20 40 ax200 12 13 44 file2 ax100 0 0 4 ax200 2 3 4 (9 Replies)
Discussion started by: quincyjones
9 Replies

3. Shell Programming and Scripting

Join and awk max column

Hi Friends, I have a file1 with 3400 records that are tab separated and I have a file2 with 6220 records. I want to merge both these files. I tried using join file1 and file2 after sorting. But, the records should be (3400*6220 = 21148000). Instead, I get only around 11133567. Is there anything... (13 Replies)
Discussion started by: jacobs.smith
13 Replies

4. Shell Programming and Scripting

Awk - join multiple files

Is it possible to join all the files with input1 based on 1st column? input1 a b c d e f input2 a b input3 a e input4 c (2 Replies)
Discussion started by: quincyjones
2 Replies

5. UNIX for Dummies Questions & Answers

How to use the the join command to join multiple files by a common column

Hi, I have 20 tab delimited text files that have a common column (column 1). The files are named GSM1.txt through GSM20.txt. Each file has 3 columns (2 other columns in addition to the first common column). I want to write a script to join the files by the first common column so that in the... (5 Replies)
Discussion started by: evelibertine
5 Replies

6. Shell Programming and Scripting

Awk: Multiple Replace In Column From Two Different Files

Master_1.txt 2372,MTS,AP 919821,Airtel,DL 0819,MTS,MUM 919849788001,Airtel,AP 1430,Aircel MP,20 405899143999999,MTS,KRL USSDLIKE,MTS,DEL Master_2.txt 919136,DL 9664,RAJ 919143,KOL 9888,PUN Input File: (4 Replies)
Discussion started by: siramitsharma
4 Replies

7. UNIX for Dummies Questions & Answers

Join with awk different column

hi guys, i need help I need to join file2 to file1 when column 3 in my file1 and column 1 in my file2 in the same string file1 AA|RR|ESKIM RE|DD|RED WE|WW|SUPSS file2 ESKIM|ES SUPSS|SS Output AA|RR|ESKIM|ES RE|DD|RED| WE|WW|SUPSS|SS (3 Replies)
Discussion started by: radius
3 Replies

8. Shell Programming and Scripting

Join 2nd column of multiple files

Dear All, I have many files formatted like this: file1.txt: 1/2-SBSRNA4 18 A1BG 3 A1BG-AS1 6 A1CF 0 A2LD1 1 A2M 1160 file2.txt 1/2-SBSRNA4 53 A1BG 1 A1BG-AS1 7 A1CF 0 A2LD1 3 A2M 2780 (5 Replies)
Discussion started by: paolo.kunder
5 Replies

9. Shell Programming and Scripting

Join columns across multiple lines in a Text based on common column using BASH

Hello, I have a file with 2 columns ( tableName , ColumnName) delimited by a Pipe like below . File is sorted by ColumnName. Table1|Column1 Table2|Column1 Table5|Column1 Table3|Column2 Table2|Column2 Table4|Column3 Table2|Column3 Table2|Column4 Table5|Column4 Table2|Column5 From... (6 Replies)
Discussion started by: nv186000
6 Replies

10. Shell Programming and Scripting

Join, merge, fill NULL the void columns of multiples files like sql "LEFT JOIN" by using awk

Hello, This post is already here but want to do this with another way Merge multiples files with multiples duplicates keys by filling "NULL" the void columns for anothers joinning files file1.csv: 1|abc 1|def 2|ghi 2|jkl 3|mno 3|pqr file2.csv: 1|123|jojo 1|NULL|bibi... (2 Replies)
Discussion started by: yjacknewton
2 Replies
CAT(1)							    BSD General Commands Manual 						    CAT(1)

NAME
cat -- concatenate and print files SYNOPSIS
cat [-benstuv] [file ...] DESCRIPTION
The cat utility reads files sequentially, writing them to the standard output. The file operands are processed in command-line order. If file is a single dash ('-') or absent, cat reads from the standard input. If file is a UNIX domain socket, cat connects to it and then reads it until EOF. This complements the UNIX domain binding capability available in inetd(8). The options are as follows: -b Number the non-blank output lines, starting at 1. -e Display non-printing characters (see the -v option), and display a dollar sign ('$') at the end of each line. -n Number the output lines, starting at 1. -s Squeeze multiple adjacent empty lines, causing the output to be single spaced. -t Display non-printing characters (see the -v option), and display tab characters as '^I'. -u Disable output buffering. -v Display non-printing characters so they are visible. Control characters print as '^X' for control-X; the delete character (octal 0177) prints as '^?'. Non-ASCII characters (with the high bit set) are printed as 'M-' (for meta) followed by the character for the low 7 bits. EXIT STATUS
The cat utility exits 0 on success, and >0 if an error occurs. EXAMPLES
The command: cat file1 will print the contents of file1 to the standard output. The command: cat file1 file2 > file3 will sequentially print the contents of file1 and file2 to the file file3, truncating file3 if it already exists. See the manual page for your shell (i.e., sh(1)) for more information on redirection. The command: cat file1 - file2 - file3 will print the contents of file1, print data it receives from the standard input until it receives an EOF ('^D') character, print the con- tents of file2, read and output contents of the standard input again, then finally output the contents of file3. Note that if the standard input referred to a file, the second dash on the command-line would have no effect, since the entire contents of the file would have already been read and printed by cat when it encountered the first '-' operand. SEE ALSO
head(1), more(1), pr(1), sh(1), tail(1), vis(1), zcat(1), setbuf(3) Rob Pike, "UNIX Style, or cat -v Considered Harmful", USENIX Summer Conference Proceedings, 1983. STANDARDS
The cat utility is compliant with the IEEE Std 1003.2-1992 (``POSIX.2'') specification. The flags [-benstv] are extensions to the specification. HISTORY
A cat utility appeared in Version 1 AT&T UNIX. Dennis Ritchie designed and wrote the first man page. It appears to have been cat(1). BUGS
Because of the shell language mechanism used to perform output redirection, the command ``cat file1 file2 > file1'' will cause the original data in file1 to be destroyed! The cat utility does not recognize multibyte characters when the -t or -v option is in effect. BSD
March 21, 2004 BSD
All times are GMT -4. The time now is 05:23 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy