Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Awk: adding fields after matching $1

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Awk: adding fields after matching $1
# 1  
Old 11-03-2013
Awk: adding fields after matching $1
Dear AWK-experts!

I did get stuck in the task of combining files after matching fields, so I'm still awkward with learning AWK.

There are 2 files: one containing 3 columns with ID, coding status, and score for long noncoding RNAs:

file1 (1.txt) (>5000 lines)

Code:
TCONS_00017358;noncoding;0.968717
TCONS_00024657;noncoding;0.995485
TCONS_00005907;noncoding;0.808883 
TCONS_00013445;noncoding;0.998353 
TCONS_00001277;noncoding;0.934182 
TCONS_00047071;noncoding;0.919281 
TCONS_00040152;noncoding;0.988357

The second file contains the same ID ($1 == $1) and a corresponding Gene name in field $2:

file2 (2.txt)
Code:
TCONS_00024657;KCNC2 
TCONS_00005907;CDH11 
TCONS_00013445;TAF12 
TCONS_00047071;MYO19 
TCONS_00040152;KCNC2 
TCONS_00017358;PAIP1 
TCONS_00001277;CHAD4

I would like to add the Gene name ($2 from file2) as an additional column to file1 in a newly generated file3. Every Gene name is located somewhere in file2. Because of asymmetric distributed duplicates in file2 a simple sort after ID and cat doesn't work. At the moment I'm trying.

Code:
awk -F ";" > 3.txt 'NR == FNR {   
               _[$1] = $1; _[$2] = $2   
               next 
               } 
                  if (_[$1] == $1)  {   
                  print $0, _[$2]   
              }' 2.txt 1.txt

Any help would be awesome!!!
# 2  
Old 11-03-2013
Try:
Code:
awk -F";" 'NR==FNR{a[$1]=$2;next}{$4=a[$1]}1' OFS=";" 2.txt 1.txt

This User Gave Thanks to bartus11 For This Post:
kben
# 3  
Old 11-03-2013
Many thanks! It's perfect, going to study the magic of the working code.

Code:
TCONS_00017358;noncoding;0.968717;PAIP1 
TCONS_00024657;noncoding;0.995485;KCNC2 
TCONS_00005907;noncoding;0.808883;CDH11 
TCONS_00013445;noncoding;0.998353;TAF12 
TCONS_00001277;noncoding;0.934182;CHAD4
TCONS_00047071;noncoding;0.919281;MYO19 
TCONS_00040152;noncoding;0.988357;KCNC2

---------- Post updated at 03:21 PM ---------- Previous update was at 02:18 PM ----------

The duplicates seem to mess it upSmilie The script perfectly works on my example (now deemed to be incomplete), but if it get's to this:

file1 (ID + coding status + score)

Code:
TCONS_00000001;noncoding;0.833737
TCONS_00000002;coding;0.00522615
TCONS_00000004;coding;0.0284272
TCONS_00000006;noncoding;0.982945

file2 (ID + gen names)

Code:
TCONS_00000001;DDX11L1
TCONS_00000001;DDX11L1
TCONS_00000001;DDX11L1
TCONS_00000002;OR4F5
TCONS_00000004;LOC100133331
TCONS_00000004;LOC100133331
TCONS_00000004;LOC100133331
TCONS_00000004;LOC100133331
TCONS_00000003;LOC100132062
TCONS_00000003;LOC100132062
TCONS_00000003;LOC100132062
TCONS_00000005;OR4F3
TCONS_00000006;LOC643837
TCONS_00000006;LOC643837
TCONS_00000006;LOC643837
TCONS_00000010;LOC643837
TCONS_00000010;LOC643837
TCONS_00000010;LOC643837
TCONS_00000010;LOC643837
TCONS_00000010;LOC643837
TCONS_00000010;LOC643837
TCONS_00000010;LOC643837
TCONS_00000012;LOC643837
TCONS_00000012;LOC643837
TCONS_00000012;LOC643837
TCONS_00000012;LOC643837
TCONS_00000012;LOC643837
TCONS_00000012;LOC643837
TCONS_00000014;LOC643837
TCONS_00000014;LOC643837
TCONS_00000014;LOC643837
TCONS_00000014;LOC643837
TCONS_00000014;LOC643837
TCONS_00000016;LOC643837
TCONS_00000016;LOC643837
TCONS_00000016;LOC643837
TCONS_00000016;LOC643837
TCONS_00000017;LOC643837
TCONS_00000017;LOC643837
TCONS_00000017;LOC643837
TCONS_00000017;LOC643837
TCONS_00000015;LOC643837
TCONS_00000015;LOC643837
TCONS_00000015;LOC643837
TCONS_00000015;LOC643837
TCONS_00000015;LOC643837
TCONS_00000015;LOC643837
TCONS_00000013;LOC643837
TCONS_00000013;LOC643837
TCONS_00000013;LOC643837
TCONS_00000013;LOC643837
TCONS_00000013;LOC643837
TCONS_00000013;LOC643837
TCONS_00000013;LOC643837
TCONS_00000011;LOC643837
TCONS_00000011;LOC643837
TCONS_00000011;LOC643837
TCONS_00000011;LOC643837
TCONS_00000011;LOC643837
TCONS_00000011;LOC643837
TCONS_00000011;LOC643837
TCONS_00000011;LOC643837
TCONS_00000009;LOC643837
TCONS_00000009;LOC643837
TCONS_00000009;LOC643837
TCONS_00000009;LOC643837
TCONS_00000009;LOC643837
TCONS_00000009;LOC643837
TCONS_00000009;LOC643837
TCONS_00000009;LOC643837
TCONS_00000008;LOC643837
TCONS_00000008;LOC643837
TCONS_00000008;LOC643837
TCONS_00000008;LOC643837

the script output changes to:

Code:
TCONS_00000001;noncoding;0.833737   
TCONS_00000002;coding;0.00522615   
TCONS_00000004;coding;0.0284272   
TCONS_00000006;noncoding;0.982945

Last edited by Scrutinizer; 11-03-2013 at 03:40 PM..
# 4  
Old 11-03-2013
That should not be - it should use the last occurrence of $1 for the replacement. In your above case, the $2 to the respective $1 are identical, so this would not matter anyhow. Applying Bartus11's proposal to your two files yields
Code:
TCONS_00000001;noncoding;0.833737;DDX11L1
TCONS_00000002;coding;0.00522615;OR4F5
TCONS_00000004;coding;0.0284272;LOC100133331
TCONS_00000006;noncoding;0.982945;LOC643837

- which is fine from my perspective...
# 5  
Old 11-03-2013
Thanks! I'm checking for format issues in the input files!

---------- Post updated at 08:22 PM ---------- Previous update was at 04:18 PM ----------

Still can't get it workingSmilie With 40 lines things run smoothly but with 45000 lines - invain.

---------- Post updated at 08:27 PM ---------- Previous update was at 08:22 PM ----------

Could there be a size limit with AWK? (file1 1.5 MB, file2 10 MB)
# 6  
Old 11-04-2013
Unlikely, 45000 lines is nothing. What is your OS and version? What isn't working? Do you get output like you described in post #3, or is it different ? Does it work when you download your own samples in post #3 and run Bartus11' suggestion on them?
# 7  
Old 11-04-2013
kben, post following output (using your target, "big" files):
Code:
head 1.txt | cat -ev
head 2.txt | cat -ev

This User Gave Thanks to bartus11 For This Post:
kben
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

awk for matching fields between files with repeated records

Hello all, I am having trouble with what should be an easy task, but seem to be missing something fundamental. I have two files, with File 1 consisting of a single field of many thousands of records. I also have File 2 with two fields and many thousands of records. My goal is that when $1 of... (2 Replies)
Discussion started by: jvoot
2 Replies

2. UNIX for Beginners Questions & Answers

Continued trouble matching fields in different files and selective field printing ([g]awk)

I apologize in advance, but I continue to have trouble searching for matches between two files and then printing portions of each to output in awk and would very much appreciate some help. I have data as follows: File1 PS012,002 PRQ 0 1 1 17 1 0 -1 3 2 1 2 -1 ... (7 Replies)
Discussion started by: jvoot
7 Replies

3. Shell Programming and Scripting

awk to print fields that match using conditions and a default value for non-matching in two files

Trying to use awk to match the contents of each line in file1 with $5 in file2. Both files are tab-delimited and there may be a space or special character in the name being matched in file2, for example in file1 the name is BRCA1 but in file2 the name is BRCA 1 or in file1 name is BCR but in file2... (6 Replies)
Discussion started by: cmccabe
6 Replies

4. UNIX for Beginners Questions & Answers

Awk: matching multiple fields between 2 files

Hi, I have 2 tab-delimited input files as follows. file1.tab: green A apple red B apple file2.tab: apple - A;Z Objective: Return $1 of file1 if, . $1 of file2 matches $3 of file1 and, . any single element (separated by ";") in $3 of file2 is present in $2 of file1 In order to... (3 Replies)
Discussion started by: beca123456
3 Replies

5. UNIX for Advanced & Expert Users

awk print all fields except matching regex

grep -v will exclude matching lines, but I want something that will print all lines but exclude a matching field. The pattern that I want excluded is '/mnt/svn' If there is a better solution than awk I am happy to hear about it, but I would like to see this done in awk as well. I know I can... (11 Replies)
Discussion started by: glev2005
11 Replies

6. Shell Programming and Scripting

awk question ? set 2 variables by matching fields

Hello, I'm trying to get the TOP and BASE numbers printed out File looks like this: 2300 CAR # 2300 is the TOP 2310 CAR 2335 CAR 2455 CAR # 2455 is the BASE 1000 MOTOR # 2455 will become this TOP 2000 MOTOR 3000 MOTOR 4000 MOTOR # 4000 is the BASE 2345 BIKE # 4000... (8 Replies)
Discussion started by: charlieglen
8 Replies

7. Shell Programming and Scripting

Awk - Script assistance on identifying non matching fields

Hoping for some assistance. my source file consists of: os, ip, username win7, 123.56.78, john win7, 123.56.78, paul win7, 10.1.1.1, john win7, 10.2.2.3, joe I've been trying to run a script that will only return ip and username where the IP address is the same and the username is... (3 Replies)
Discussion started by: tekvaio
3 Replies

8. Shell Programming and Scripting

AWK : Add Fields of lines with matching field

Dear All, I would like to add values of a field, if the lines match in a certain field. Then I would like to divide the sum though the number of lines that have a matched field. This is the Input: Input: Test1 5 Test1 10 Test2 2 Test2 5 Test2 13 Test3 4 Output: Test1 7.5 Test1 7.5... (6 Replies)
Discussion started by: DerSeb
6 Replies

9. Shell Programming and Scripting

AWK- delimiting the strings and matching the fields

Hello, I am newbie in awk. I have just started learning it. 1) I have input file which looks like: {4812 4009 1602 2756 306} {4814 4010 1603 2757 309} {8116 9362 10779 } {10779 10121 9193 10963 10908} {1602 2756 306 957 1025} {1603 2757 307} and so on..... 2) In output: a)... (10 Replies)
Discussion started by: kajolo
10 Replies

10. Shell Programming and Scripting

AWK Matching Fields and Combining Files

Hello! I am writing a program to run through two large lists of data (~300,000 rows), find where rows in one file match another, and combine them based on matching fields. Due to the large file sizes, I'm guessing AWK will be the most efficient way to do this. Overall, the input and output I'm... (5 Replies)
Discussion started by: Michelangelo
5 Replies
Login or Register to Ask a Question