Visit Our UNIX and Linux User Community


Merge two rows using awk or python


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Merge two rows using awk or python
# 1  
Old 08-28-2009
Merge two rows using awk or python

Hi,
Suppose I have a space delimited file like this:

Serial# 1970 1971 1972 1973 1974
193532 21 2 X X X
200201 20 30 X X 40
200201 X X 13 15 X
393666 66 3 X X 5
393666 77 X X X X

First, I want to check the serial#, if any two lines have the same serial#,(in this case line 2+3, and line 4+5 qualify), then merge these two lines by replacing X with the value of the other line.

Also, when there is a conflict, in this case line 4+5 have the first column as 66,77(rather than having X in either line or in both lines), then do not merge even though they have the same serial#, but flag both lines with FLAGGED on the CONFLICT_FLAG column.

The result would be:

Serial# 1970 1971 1972 1973 1974 CONFLICT_FLAG
193532 21 2 X X X
200201 20 30 13 15 40
393666 66 3 X X 5 FLAGGED
393666 77 X X X X FLAGGED

Is it possible to do this in either python or awk? Thank you.
# 2  
Old 08-28-2009
below perl should help you some.
Code:
open FH,"<a";
while(<FH>){
 my @tmp = split;
 my @t1=@tmp;
 if (not exists $hash{$tmp[0]}){
   $hash{$tmp[0]}=[@tmp[1..5]];
 }
 else{
   $hash{$tmp[0]}=[@{$hash{$tmp[0]}},@tmp[1..5]];
 }
}
foreach my $key ( keys %hash ){
 my $flag=0;
 my @tmp = @{$hash{$key}};
 if($#tmp < 6){
   print $key," ";
   print join " ", @{$hash{$key}};
   print "\n";
 }
 else{
   my @t = @tmp;
   map { s/X/0/ } @t;
   for(my $i=0;$i<=4;$i++){
     if($t[$i]!=0 && $t[$i+5]!=0){
	$flag=1;	
     }
   }
   if($flag==1){
	print $key," ";
	print join " ", @tmp[0..4];
	print " FLAGGED\n";
	print $key," ";
	print join " ", @tmp[5..9];
	print " FLAGGED\n";
   }
   else{
	print $key," ";
	for(my $i=0;$i<=4;$i++){
	  print $t[$i]+$t[$i+5]," ";
	}
        print "\n";
   }
 }
}

# 3  
Old 08-28-2009
thanks a lot, unfortunately, perl is not installed on site, nor do i have permission to.
is it possible to do it in python or awk?

Previous Thread | Next Thread
Test Your Knowledge in Computers #600
Difficulty: Medium
Functions in C do not need be declared before they can be used.
True or False?

10 More Discussions You Might Find Interesting

1. Programming

[python] merge pdf files.

First off I am very new to python but not to scripting I have done a lot of bash scripting. I need to create a python script for work that will combine multiple pdf files into one pdf file and archive both the combined file and the original pdf files. So we receive zip files from a client... (6 Replies)
Discussion started by: SaltCityScripts
6 Replies

2. UNIX for Dummies Questions & Answers

Merge rows into one if first 2 columns match

Hi, I wanted to merge the content and below is input and required output info. Input: /hello,a,r /hello,a,L /hello,a,X /hi,b,v /hi,b,c O/p: /hello,a,r:L:X /hi,v,:v:c Use code tags, thanks. (6 Replies)
Discussion started by: ankitas
6 Replies

3. Shell Programming and Scripting

Merge files and remove duplicated rows

In a folder I'll several times daily receive new files that I want to combine into one big file, without any duplicate rows. The file name in the folder will look like e.q: MissingData_2014-08-25_09-30-18.txt MissingData_2014-08-25_09-30-14.txt MissingData_2014-08-26_09-30-12.txt The content... (9 Replies)
Discussion started by: Bergans
9 Replies

4. Shell Programming and Scripting

Merge matching rows

Hello, I need this output. thank you very much. input: Code: ***table***wood ***snack***top ***table***garfield ***big***zen ***table***cars output: Code: ***table***wood2345garfield2345cars ***snack***top ***big***zen (7 Replies)
Discussion started by: tara123
7 Replies

5. Shell Programming and Scripting

Merge files based on both common and uncommon rows

Hi, I have two files A (2190 rows) and file B (1100 rows). I want to merge the contents of two files based on common field, also I need the unmatched rows from file A file A: ABC XYZ PQR file B: >LMN|chr1:11000-12456: >ABC|chr15:176578-187678: >PQR|chr3:14567-15866: output... (3 Replies)
Discussion started by: Diya123
3 Replies

6. UNIX for Dummies Questions & Answers

Merge rows in bid data file

Dear all, Please help me ,,,, if I have input file like this A_AA960715 leucine-rich repeat-containing protein GO:0006952 defense response P A_AA960715 leucine-rich repeat-containing protein GO:0008152 metabolic process P A_AA960715 leucine-rich... (5 Replies)
Discussion started by: AAWT
5 Replies

7. UNIX for Dummies Questions & Answers

Merge rows with common column

Dear all I have big file with two columns A_AA960715 GO:0006952 A_AA960715 GO:0008152 A_AA960715 GO:0016491 A_AA960715 GO:0007165 A_AA960715 GO:0005618 A_AA960716 GO:0006952 A_AA960716 GO:0005618 A_AA960716... (15 Replies)
Discussion started by: AAWT
15 Replies

8. Shell Programming and Scripting

merge similar rows

I have a large file (10M lines) that contains two columns: a frequency and a string, ex: 3 aaaaa 4 bbbbb 2 ccccc 5 aaaaa 1 ddddd 4 ccccc I need to merge the lines whose string part is the same, while updating the frequency. The output should look like this: 8 aaaaa 4 bbbbb 5 ccccc... (2 Replies)
Discussion started by: tootles564
2 Replies

9. Shell Programming and Scripting

How to merge rows into columns ????

Hi guz I want to merge multiple rows into a multiple columns based on the first column. The file has symbol // I want to break the symbool // and I nedd exactlynew column at that point the output will be like this please guyz help in this isssue!!!!! merging rows into columns ... (4 Replies)
Discussion started by: bogu0001
4 Replies

10. Shell Programming and Scripting

merge rows based on a common column

Hi guys, Please guide me if you have a solution to this problem. I have tried paste -s but it's not giving the desired output. I have a file with the following content- A123 box1 B345 bat2 C431 my_id A123 service C431 box1 A123 my_id I need two different outputs- OUTPUT1 A123... (6 Replies)
Discussion started by: smriti_shridhar
6 Replies

Featured Tech Videos