Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Join two files with matching columns

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Join two files with matching columns
# 1  
Old 06-24-2013
Join two files with matching columns
Hi,

I need to join two files together with one common value in a column. I think I can use awk or join or a combination but I can't quite get it.

Basically my data looks like this, with the TICKER columns matching up in each file

File1
Code:
TICKER,column 1, column, 2, column, 3, column 4

File2.
Code:
TICKER,STATUS

I need them to join up like this into file3.

Code:
TICKER,STATUS,column 1, column, 2, column, 3, column 4

thank you
Last edited by Scrutinizer; 06-25-2013 at 01:17 AM.. Reason: code tags
# 2  
Old 06-24-2013
This type of question is one of frequently asked. Did you search this forum before posting?
# 3  
Old 06-24-2013
Yes I did. I am read through several threads and am currently trying this:

Code:
join -t\, file1 file2

which works, but it only prints out 5 lines for some reason, even though the first column for each file is identical.
Last edited by Scrutinizer; 06-25-2013 at 01:18 AM.. Reason: code tags
# 4  
Old 06-24-2013
You could try using awk:
Code:
awk -F, 'NR==FNR{A[$1]=$0;next}$1 in A{$0=$0","A[$1];print}' file1 file2

# 5  
Old 06-24-2013
That is very helpful, thank you!

Can you help me understand what is being matched up with that line of code, though? For example if I wanted file2 to be matched with column 2 instead, but only print column 1 to the output, what would I have to change around?

thanks!!
# 6  
Old 06-25-2013
Quote:
Originally Posted by unkleruckus
Yes I did. I am read through several threads and am currently trying this:

Code:
join -t\, file1 file2

which works, but it only prints out 5 lines for some reason, even though the first column for each file is identical.
The join command would work too, but it requires that the lines in the inout files are in sorted order..


Quote:
Originally Posted by unkleruckus
That is very helpful, thank you!

Can you help me understand what is being matched up with that line of code, though? For example if I wanted file2 to be matched with column 2 instead, but only print column 1 to the output, what would I have to change around?

thanks!!
Then you could try something like this:
Code:
awk 'NR==FNR{A[$2]; next} $2 in A{print $1}' file1 file2

If that does not work, you would need to provide a better data sample..
Last edited by Scrutinizer; 06-25-2013 at 01:27 AM..
# 7  
Old 06-25-2013
Hey guys

thanks for the help with this, I was able to accomplish my goal by doing the following:

Code:
sort -t "," file1.csv > temp1.csv
sort -t "," -k 2 file2.csv > temp2.csv
join -t\, -1 1 -2 2 -o0 1.2,2.1,1.3,1.4,1.5,1.6 temp1.csv temp2.csv > OUTPUT.csv

and all my data goes to the correct spot! I am seeing trailing commas at the end of my OUTPUT.csv, though. Any idea of why that happens? It's probably not a big deal but I'd like my script to output the data as cleanly as possible.

thanks again for the help with this!
Last edited by Scrutinizer; 06-26-2013 at 01:12 AM..
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

Data match 2 files based on first 2 columns matching only and join if match

Hi, i have 2 files , the data i need to match is in masterfile and i need to pull out column 3 from master if column 1 and 2 match and output entire row to new file I have tried with join and awk and i keep getting blank outputs or same file is there an easier way than what i am... (4 Replies)
Discussion started by: axis88
4 Replies

2. Shell Programming and Scripting

Join, merge, fill NULL the void columns of multiples files like sql "LEFT JOIN" by using awk

Hello, This post is already here but want to do this with another way Merge multiples files with multiples duplicates keys by filling "NULL" the void columns for anothers joinning files file1.csv: 1|abc 1|def 2|ghi 2|jkl 3|mno 3|pqr file2.csv: 1|123|jojo 1|NULL|bibi... (2 Replies)
Discussion started by: yjacknewton
2 Replies

3. Shell Programming and Scripting

Join and merge multiple files with duplicate key and fill void columns

Join and merge multiple files with duplicate key and fill void columns Hi guys, I have many files that I want to merge: file1.csv: 1|abc 1|def 2|ghi 2|jkl 3|mno 3|pqr file2.csv: (5 Replies)
Discussion started by: yjacknewton
5 Replies

4. Shell Programming and Scripting

Join two files combining multiple columns and produce mix and match output

I would like to join two files when two columns in each file matches with each other and then produce an output when taking multiple columns. Like I have file A 1234,ABCD,23,JOHN,NJ,USA 2345,ABCD,24,SAM,NY,USA 5678,GHIJ,24,TOM,NY,USA 5678,WXYZ,27,MAT,NJ,USA and file B ... (2 Replies)
Discussion started by: mady135
2 Replies

5. Shell Programming and Scripting

Join 4 files on first three columns

Hi, Can someone suggest me on how to join 4 files by comparing the first three columns? ---------- Post updated at 03:56 PM ---------- Previous update was at 03:42 PM ---------- Hope it helps someone. I was looking online for a solution and on stackoverflow, I found a solution and tried... (6 Replies)
Discussion started by: jacobs.smith
6 Replies

6. Shell Programming and Scripting

Matching the substring and join two files

Hi I had two files like below. file-1 101001234567890 101001234567891 101001234567892 101001234567893 101001234567894 101001234567895 101001234567896 101001234567897 101001234567898 101001234567899 file-2 (6 Replies)
Discussion started by: p_sai_ias
6 Replies

7. Shell Programming and Scripting

Join 3 or more files using matching column

Dear Forum, Full title of the topic would be: "Join 3 or more files using matching column without full list in any of these columns" I have several, typically 3 or 4 files which I need to join, something like FULL JOIN in slq scripts, all combinations of matches should be printed into an... (3 Replies)
Discussion started by: cyz700
3 Replies

8. Shell Programming and Scripting

matching columns from two files

Hey, I have two files that have exactly the same format. They are both tab-delimited and contain 12 columns. However the # of rows vary. What I want to do is match columns # 5,6 and 7 between the two files. If they do match exactly (based on numbers) then I want the whole row from file 2 to... (1 Reply)
Discussion started by: phil_heath
1 Replies

9. UNIX for Dummies Questions & Answers

Join 2 files with multiple columns: awk/grep/join?

Hello, My apologies if this has been posted elsewhere, I have had a look at several threads but I am still confused how to use these functions. I have two files, each with 5 columns: File A: (tab-delimited) PDB CHAIN Start End Fragment 1avq A 171 176 awyfan 1avq A 172 177 wyfany 1c7k A 2 7... (3 Replies)
Discussion started by: InfoSeeker
3 Replies

10. Shell Programming and Scripting

Join columns from 2 files

Hello guys. I need to join columns of Start Time and End time of 2 files. The Examples is: File 1 APPLICATION GROUP_NAME JOB_NAME ODATE STATUS START_TIME END_TIME :: MODCMPDA EDPONLINE MC00A1700 071102 Ended OK 20071102 17:00:01 File 2 APPLICATION GROUP_NAME JOB_NAME ODATE... (2 Replies)
Discussion started by: osramos
2 Replies
Login or Register to Ask a Question