Sponsored Content
Full Discussion: Merge files based on columns
Top Forums Shell Programming and Scripting Merge files based on columns Post 302908919 by Michael Stora on Friday 11th of July 2014 01:53:40 PM
Old 07-11-2014
Join command is useful and the standard answer you will get but depending on the type of join one or both files must be sorted.

Often when I want to do this one file is changing constantly and huge, the other is relatively small and static or rarely changed. In that case, I usually read the smaller file into an associative array and match that way.

I've done this in both Bash and AWK (AWK is faster): For example in an AWK BEGIN block:
Code:
BEGIN { while (getline < "'"$indexFile"'" ) { split($0,myIndex,","); indexArr[imyIndex[1]]=myIndex[2] }
                     close("'"$indexFile"'") }

Read up on associative arrays as they are very powerful features of high level languages.

Mike

PS. I am just a Padawan compared to maser Yoda. I passed a Bash variable, Yoda did not. NR=FNR with two files indeed . . . Smilie

Last edited by Michael Stora; 07-11-2014 at 04:49 PM.. Reason: index is a reserved word in AWK so I changed to myIndex
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Compare two files and merge columns in a third

Hi, I'm working with snmp, with a little script I'm able to obtain from a switch a list with a couple of values with this format Port Mac 1 00:0A:0B:0C:0D:0E .... (hundred of entries) Now with a simple arp on a router I am able to obtain another list 00:0A:0B:0C:0D:0E... (20 Replies)
Discussion started by: CM64
20 Replies

2. Shell Programming and Scripting

merge the two files which has contain columns

Hi may i ask how to accomplish this task: I have 2 files which has multiple columns first file 1 a 2 b 3 c 4 d second file 14 a 9 .... 13 b 10.... 12 c 11... 11 d 12... I want to merge the second file to first file that will looks like this ... (2 Replies)
Discussion started by: jao_madn
2 Replies

3. Shell Programming and Scripting

Merge columns of different files

Hi, I have tab limited file 1 and tab limited file 2 The output should contain common first column vales and corresponding 2nd column values; AND also unique first column value with corresponding 2nd column value of the file that contains it and 0 for the second file. the output should... (10 Replies)
Discussion started by: polsum
10 Replies

4. Shell Programming and Scripting

file merge based on common columns

I have two files 1.txt 34, ABC, 7, 8, 0.9 35, CDE, 6.5, -2, 0.01 2.txt 34, ABC, 9, 6, -1.9 35, CDE, 8.5, -2.3, 5.01 So in both files common columns are 1 and 2 so final o/p should look like 34, ABC, 7, 8, 0.9, 9, 6, -1.9 35, CDE, 6.5, -2, 0.01, 8.5, -2.3, 5.01 I tried using... (3 Replies)
Discussion started by: manas_ranjan
3 Replies

5. Shell Programming and Scripting

Merge two files matching columns

Hi! I need to merge two files when col1 (x:x:x) matching and adds second column from file1.txt. # cat 1.txt aaa;a12 bbb;b13 ccc;c33 ddd;d55 eee;e11 # cat 2.txt bbb;b55;34444;d55 aaa;a15;35666;a44 I try with this awk and I get succesfully first column from 1.txt: # awk -F";"... (2 Replies)
Discussion started by: fhluque
2 Replies

6. Shell Programming and Scripting

Merge columns on different files

Hello, I have two files that have this format: file 1 86.82 0.00 86.82 43.61 86.84 0.00 86.84 43.61 86.86 0.00 86.86 43.61 86.88 0.00 86.88 43.61 file 2 86.82 0.22 86.84 0.22 86.86 0.22 86.88 0.22 I would like to merge these two files such that the final file looks like... (5 Replies)
Discussion started by: kayak
5 Replies

7. Shell Programming and Scripting

Merge 2 files with one reference columns

Hi All Source1 servername1,patchid1 servername1,patchid2 servername1,patchid3 servername2,patchid1 servername2,patchid2 servername3,patchid4 servername3,patchid5 Source2 servername1,appname1 servername1,appname2 servername1,appname3 servername2,appname1 servername2,appname2... (13 Replies)
Discussion started by: mv_mv
13 Replies

8. Shell Programming and Scripting

Merge records based on multiple columns

Hi, I have a file with 16 columns and out of these 16 columns 14 are key columns, 15 th is order column and 16th column is having information. I need to concate the 16th column based on value of 1-14th column as key in order of 15th column. Here are the example file Input File (multiple... (3 Replies)
Discussion started by: Ravi Agrawal
3 Replies

9. Shell Programming and Scripting

Merge specific columns of two files

Hello, I have two tab delimited text files. Both files have the same number of rows but not necessarily the same number of columns. The column headers look like, File 1: f0order CVorder Name f0 RI_9 E99 E199 E299 E399 E499 E599 E699 E799 E899 E999 File 2:... (9 Replies)
Discussion started by: LMHmedchem
9 Replies

10. UNIX for Dummies Questions & Answers

Merge selective columns from files based on common key

Hi, I am trying to selectively merge two files based on keys reported in the 1st column. File1: #file1-header1 file1-header2 111 qwe rtz uio 198 asd fgh jkl 165 yxc 789 poi uzt rew 89 lkj File2: #file2-header2 file2-header2 165 ghz nko2 ... (2 Replies)
Discussion started by: dovah
2 Replies
App::ClusterSSH::Host(3pm)				User Contributed Perl Documentation				App::ClusterSSH::Host(3pm)

NAME
ClusterSSH::Host - Object representing a host. SYNOPSIS
use ClusterSSH::Host; my $host = ClusterSSH::Host->new({ hostname => 'hostname', }); my $host = ClusterSSH::Host->parse_host_string('username@hostname:1234'); DESCRIPTION
Object representing a host. Include details to contact the host such as hostname/ipaddress, username and port. METHODS
$host=ClusterSSH::Host->new ({ hostname => 'hostname' }) Create a new host object. 'hostname' is a required arg, 'username' and 'port' are optional. Raises exception if an error occurs. $host->get_hostname $host->get_username $host->get_port $host->get_master Return specific details about the host $host->set_username $host->set_port $host->set_master Set specific details about the host after its been created. get_realname If the server name provided is not an IP address (either IPv4 or IPv6) attempt to resolve it and retun the discovered names. get_givenname Alias to get_hostname, for use when " get_realname " might return something different parse_host_string Given a host string, returns a host object. Parses hosts such as check_ssh_hostname Check the objects hostname to see whether or not it may be configured within the users $HOME/.ssh/config configuration file host 192.168.0.1 user@host user@192.168.0.1 host:port [1234:1234:1234::4567]:port 1234:1234:1234::4567 and so on. Cope with IPv4 and IPv6 addresses - raises a warning if the IPv6 address is ambiguous (i.e. in the last example, is the 4567 part of the IPv6 address or a port definition?) and assumes it is part of address. Use brackets to avoid seeing warning. AUTHOR
Duncan Ferguson, "<duncan_j_ferguson at yahoo.co.uk>" LICENSE AND COPYRIGHT
Copyright 1999-2010 Duncan Ferguson. This program is free software; you can redistribute it and/or modify it under the terms of either: the GNU General Public License as published by the Free Software Foundation; or the Artistic License. See http://dev.perl.org/licenses/ for more information. perl v5.14.2 2012-06-24 App::ClusterSSH::Host(3pm)
All times are GMT -4. The time now is 06:36 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy