Sponsored Content
Top Forums Shell Programming and Scripting Merge two columns from two files into one if another column matches Post 302723087 by pbluescript on Monday 29th of October 2012 11:20:44 AM
Old 10-29-2012
Merge two columns from two files into one if another column matches

I have two text files that look something like this:

Code:
A:B:C 123
D:E:F 234
G:H:I 345
J:K:L 123
M:N:O 456
P:Q:R 567

Code:
A:B:C 456
D:E:F 567
G:H:I 678
J:K:L 456
M:N:O 789
P:Q:R 890

I want to find the line where the first column matches and then combine the second columns into a single column so the output looks like this:

Code:
A:B:C 123456
D:E:F 234567
G:H:I 345678
J:K:L 123456
M:N:O 456789
P:Q:R 567890

After that, I want to search through the second columns and remove duplicate ones. In this case, that would mean the A:B:C line and the J:K:L line have matching second columns, so I would remove the J:K:L line.

Code:
A:B:C 123456
D:E:F 234567
G:H:I 345678
M:N:O 456789
P:Q:R 567890

I've tried some awk, but I am not coming anywhere close with this one. Smilie The files can also get pretty big with hundreds of millions of lines, so a speedier option would be best. Thanks!
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Joining columns from two files, if the key matches

I am trying to join/paste columns from two files for the rows with matching first field. Any help will be appreciated. Files can not be sorted and may not have all rows in both files. Thanks. File1 aaa 111 bbb 222 ccc 333 File2 aaa sss mmmm ccc kkkk llll ddd xxx yyy Want to... (1 Reply)
Discussion started by: sk_sd
1 Replies

2. Shell Programming and Scripting

merge the two files which has contain columns

Hi may i ask how to accomplish this task: I have 2 files which has multiple columns first file 1 a 2 b 3 c 4 d second file 14 a 9 .... 13 b 10.... 12 c 11... 11 d 12... I want to merge the second file to first file that will looks like this ... (2 Replies)
Discussion started by: jao_madn
2 Replies

3. Shell Programming and Scripting

Merge columns of different files

Hi, I have tab limited file 1 and tab limited file 2 The output should contain common first column vales and corresponding 2nd column values; AND also unique first column value with corresponding 2nd column value of the file that contains it and 0 for the second file. the output should... (10 Replies)
Discussion started by: polsum
10 Replies

4. Shell Programming and Scripting

How to merge multiple rows into single row if first column matches ?

Hi, Can anyone suggest quick way to get desired output? Sample input file content: A 12 9 A -0.3 2.3 B 1.0 -4 C 34 1000 C -111 900 C 99 0.09 Output required: A 12 9 -0.3 2.3 B 1.0 -4 C 34 1000 -111 900 99 0.09 Thanks (3 Replies)
Discussion started by: cbm_000
3 Replies

5. UNIX for Dummies Questions & Answers

How do I merge multiple columns into one column?

Hi all, I'm looking for a way to merge multiple columns (from one file) into a single column in an output file. The file I have looks somewhat like this: @HWI-ST212 1:N:0 AGTCCTACCGGGAGT + @@@DDDDDHHHHHII @HWI-ST212 1:N:0 CGTTTAAAAATTTCT + @;@B;DDDDH?:F;F... (4 Replies)
Discussion started by: Vnguyen
4 Replies

6. Shell Programming and Scripting

How to get difference of the same column between two files when other column matches?

File 1: 20130416,235800,10.78.25.104,BR2-loc,60.0,1624,50.0,0,50.0,0 20130416,235800,10.78.25.104,BR1-LOC,70.0,10,50.0,0,70.0,0 20130416,235800,10.78.25.104,Hub_None,60.0,15,60.0,0,50.0,0 File 2: 20130417,000200,10.78.25.104,BR2-loc,60.0,1626,50.0,0,50.0,0... (3 Replies)
Discussion started by: Lakshmikumari
3 Replies

7. Shell Programming and Scripting

Merge columns on different files

Hello, I have two files that have this format: file 1 86.82 0.00 86.82 43.61 86.84 0.00 86.84 43.61 86.86 0.00 86.86 43.61 86.88 0.00 86.88 43.61 file 2 86.82 0.22 86.84 0.22 86.86 0.22 86.88 0.22 I would like to merge these two files such that the final file looks like... (5 Replies)
Discussion started by: kayak
5 Replies

8. Shell Programming and Scripting

Seperated by columns, merge in a file, sort them on common column

Hi All, I have 4 files in below format. I took them as an example. File 1: Cut from position 1-4 then 6-7 then 8-14 then rest left and make them as columns in one new file. Inserting character H to the initial of all line like HCTOT. CTOT 456787897 Low fever CTOR 556712345 High fever... (2 Replies)
Discussion started by: Mannu2525
2 Replies

9. UNIX for Beginners Questions & Answers

Matches columns from two different files in shell script

Hi friends, i want to compare first columns from two different files ,if equal print the file2's second column else print the zero.Please help me... file1: a b c d efile2: a 1 c 20 e 30 desired output: 1 0 20 0 30 Please use CODE tags as required by forum rules! Please post in... (1 Reply)
Discussion started by: bhaskar illa
1 Replies

10. Shell Programming and Scripting

awk add all columns if column 1 name matches

Hi - I want to add all columns if column1 name matches. TOPIC1 5 1 4 TOPIC2 3 2 1 TOPIC3 7 2 5 TOPIC1 6 3 3 TOPIC2 4 1 3 TOPIC3 9 5 4 . . . . . . . . . . . . Result should look like TOPIC1 11 4 7 TOPIC2 7 3 4 (1 Reply)
Discussion started by: oraclermanpt
1 Replies
Regexp::Common::number(3)				User Contributed Perl Documentation				 Regexp::Common::number(3)

NAME
Regexp::Common::number -- provide regexes for numbers SYNOPSIS
use Regexp::Common qw /number/; while (<>) { /^$RE{num}{int}$/ and print "Integer "; /^$RE{num}{real}$/ and print "Real "; /^$RE{num}{real}{-base => 16}$/ and print "Hexadecimal real "; } DESCRIPTION
Please consult the manual of Regexp::Common for a general description of the works of this interface. Do not use this module directly, but load it via Regexp::Common. $RE{num}{int}{-base}{-sep}{-group}{-places}{-sign} Returns a pattern that matches an integer. If "-base => B" is specified, the integer is in base B, with "2 <= B <= 36". For bases larger than 10, upper case letters are used. The default base is 10. If "-sep => P" is specified, the pattern P is required as a grouping marker within the number. If this option is not given, no grouping marker is used. If "-group => N" is specified, digits between grouping markers must be grouped in sequences of exactly N digits. The default value of N is 3. If "-group => N,M" is specified, digits between grouping markers must be grouped in sequences of at least N digits, and at most M digits. This option is ignored unless the "-sep" option is used. If "-places => N" is specified, the integer recognized must be exactly N digits wide. If "-places => N,M" is specified, the integer must be at least N wide, and at most M characters. There is no default, which means that integers are unlimited in size. This option is ignored if the "-sep" option is used. If "-sign => P" is used, it's a pattern the leading sign has to match. This defaults to "[-+]?", which means the number is optionally preceded by a minus or a plus. If you want to match unsigned integers, use $RE{num}{int}{-sign => ''}. For example: $RE{num}{int} # match 1234567 $RE{num}{int}{-sep=>','} # match 1,234,567 $RE{num}{int}{-sep=>',?'} # match 1234567 or 1,234,567 $RE{num}{int}{-sep=>'.'}{-group=>4} # match 1.2345.6789 Under "-keep" (see Regexp::Common): $1 captures the entire number $2 captures the optional sign of the number $3 captures the complete set of digits $RE{num}{real}{-base}{-radix}{-places}{-sep}{-group}{-expon} Returns a pattern that matches a floating-point number. If "-base=N" is specified, the number is assumed to be in that base (with A..Z representing the digits for 11..36). By default, the base is 10. If "-radix=P" is specified, the pattern P is used as the radix point for the number (i.e. the "decimal point" in base 10). The default is "qr/[.]/". If "-places=N" is specified, the number is assumed to have exactly N places after the radix point. If "-places=M,N" is specified, the number is assumed to have between M and N places after the radix point. By default, the number of places is unrestricted. If "-sep=P" specified, the pattern P is required as a grouping marker within the pre-radix section of the number. By default, no separator is allowed. If "-group=N" is specified, digits between grouping separators must be grouped in sequences of exactly N characters. The default value of N is 3. If "-expon=P" is specified, the pattern P is used as the exponential marker. The default value of P is "qr/[Ee]/". If "-sign=P" is specified, the pattern P is used to match the leading sign (and the sign of the exponent). This defaults to "[-+]?", means means that an optional plus or minus sign can be used. For example: $RE{num}{real} # matches 123.456 or -0.1234567 $RE{num}{real}{-places=>2} # matches 123.45 or -0.12 $RE{num}{real}{-places=>'0,3'} # matches 123.456 or 0 or 9.8 $RE{num}{real}{-sep=>'[,.]?'} # matches 123,456 or 123.456 $RE{num}{real}{-base=>3'} # matches 121.102 Under "-keep": $1 captures the entire match $2 captures the optional sign of the number $3 captures the complete mantissa $4 captures the whole number portion of the mantissa $5 captures the radix point $6 captures the fractional portion of the mantissa $7 captures the optional exponent marker $8 captures the entire exponent value $9 captures the optional sign of the exponent $10 captures the digits of the exponent $RE{num}{dec}{-radix}{-places}{-sep}{-group}{-expon} A synonym for $RE{num}{real}{-base=>10}{...} $RE{num}{oct}{-radix}{-places}{-sep}{-group}{-expon} A synonym for $RE{num}{real}{-base=>8}{...} $RE{num}{bin}{-radix}{-places}{-sep}{-group}{-expon} A synonym for $RE{num}{real}{-base=>2}{...} $RE{num}{hex}{-radix}{-places}{-sep}{-group}{-expon} A synonym for $RE{num}{real}{-base=>16}{...} $RE{num}{decimal}{-base}{-radix}{-places}{-sep}{-group} The same as $RE{num}{real}, except that an exponent isn't allowed. Hence, this returns a pattern matching decimal numbers. If "-base=N" is specified, the number is assumed to be in that base (with A..Z representing the digits for 11..36). By default, the base is 10. If "-radix=P" is specified, the pattern P is used as the radix point for the number (i.e. the "decimal point" in base 10). The default is "qr/[.]/". If "-places=N" is specified, the number is assumed to have exactly N places after the radix point. If "-places=M,N" is specified, the number is assumed to have between M and N places after the radix point. By default, the number of places is unrestricted. If "-sep=P" specified, the pattern P is required as a grouping marker within the pre-radix section of the number. By default, no separator is allowed. If "-group=N" is specified, digits between grouping separators must be grouped in sequences of exactly N characters. The default value of N is 3. For example: $RE{num}{decimal} # matches 123.456 or -0.1234567 $RE{num}{decimal}{-places=>2} # matches 123.45 or -0.12 $RE{num}{decimal}{-places=>'0,3'} # matches 123.456 or 0 or 9.8 $RE{num}{decimal}{-sep=>'[,.]?'} # matches 123,456 or 123.456 $RE{num}{decimal}{-base=>3'} # matches 121.102 Under "-keep": $1 captures the entire match $2 captures the optional sign of the number $3 captures the complete mantissa $4 captures the whole number portion of the mantissa $5 captures the radix point $6 captures the fractional portion of the mantissa $RE{num}{square} Returns a pattern that matches a (decimal) square. Because Perl's arithmetic is lossy when using integers over about 53 bits, this pattern only recognizes numbers less than 9000000000000000, if one uses a Perl that is configured to use 64 bit integers. Otherwise, the limit is 2147483647. These restrictions were introduced in versions 2.116 and 2.117 of Regexp::Common. Regardless whether "-keep" was set, the matched number will be returned in $1. This pattern is available for version 5.008 and up. $RE{num}{roman} Returns a pattern that matches an integer written in Roman numbers. Case doesn't matter. Only the more modern style, that is, no more than three repetitions of a letter, is recognized. The largest number matched is MMMCMXCIX, or 3999. Larger numbers cannot be expressed using ASCII characters. A future version will be able to deal with the Unicode symbols to match larger Roman numbers. Under "-keep", the number will be captured in $1. SEE ALSO
Regexp::Common for a general description of how to use this interface. AUTHOR
Damian Conway (damian@conway.org) MAINTAINANCE
This package is maintained by Abigail (regexp-common@abigail.be). BUGS AND IRRITATIONS
Bound to be plenty. For a start, there are many common regexes missing. Send them in to regexp-common@abigail.be. LICENSE and COPYRIGHT This software is Copyright (c) 2001 - 2013, Damian Conway and Abigail. This module is free software, and maybe used under any of the following licenses: 1) The Perl Artistic License. See the file COPYRIGHT.AL. 2) The Perl Artistic License 2.0. See the file COPYRIGHT.AL2. 3) The BSD Licence. See the file COPYRIGHT.BSD. 4) The MIT Licence. See the file COPYRIGHT.MIT. perl v5.18.2 2013-03-11 Regexp::Common::number(3)
All times are GMT -4. The time now is 01:08 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy