Matching two fields in two csv files, create new file and append match


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Matching two fields in two csv files, create new file and append match
# 1  
Old 09-21-2018
Matching two fields in two csv files, create new file and append match

I am trying to parse two csv files and make a match in one column then print the entire file to a new file and append an additional column that gives description from the match to the new file. If a match is not made, I would like to add "NA" to the end of the file

Command that Ive been using but has not returned any matches
Code:
awk -F, 'FNR==NR{f2[$1]=$2;next} FNR==1{print $0, "VLAN Name";next} {print $0,($5 in f2)?f2[$5]:"NA"}' OFS=, file2 file1

file1 - (Print entire file to new file, match vlan fields)

Code:
Hostname,Port,Name,Status,Vlan,Duplex,Speed,Type
shintstat,Te1/1 ,	Trunk- ,	connected ,	11 ,	full ,	10G ,	10GBase-CU 3M,
shintstat,Te1/2 ,	Trunk ,	connected ,	17 ,	full ,	10G ,	10GBase-CU 3M,
shintstat,Te1/3 ,	Trunk ,	connected ,	trunk ,	full ,	10G ,	10GBase-CU 3M,
shintstat,Te1/4 ,	Trunk ,	connected ,	trunk ,	full ,	10G ,	10GBase-CU 3M,


file2: (Match VLAN field in file2 with Vlan field in file1 then append Name field to new file / file3 with the heading Vlan Name)

Code:
VLAN,Name,Status,Ports,Device
1,default,active,Te1-7 Te1-8 Te1-10 Te1-11 Te2-2 Te2-4 Te2-5 Te2-7 Te2-8 Te3-2 Te3-3 Te3-4 Te4-2 Te4-3 Te4-4 Gi5-8 Gi5-9 Gi5-10 Gi5-11 Gi5-14 Gi5-15 Gi5-16 Gi5-18 Gi5-19 Gi5-20 Gi5-22 Gi5-23 Gi5-25 Gi5-26 Gi5-27 Gi5-28 Gi5-29 Gi5-30 Gi5-31 Gi5-32 Gi5-33 Gi5-34 Gi5-35 Gi5-36 Gi5-37 Gi5-38 Gi5-39 Gi5-40 Gi5-41 Gi5-42 Gi5-43 Gi5-44 Gi5-45 Gi5-46 Gi6-3 Gi6-4 Gi6-5 Gi6-6 Gi6-7 Gi6-8 Gi6-9 Gi6-10 Gi6-11 Gi6-12 Gi6-13 Gi6-14 Gi6-15 Gi6-16 Gi6-17 Gi6-18 Gi6-19 Gi6-20 Gi6-26 Gi6-27 Gi6-28 Gi6-30 Gi6-32 Gi6-33 Gi6-34 Gi6-35 Gi6-36 Gi6-38 Gi6-39 Gi6-40 Gi6-47 Gi7-1 Gi7-16 Gi7-17 Gi7-19 Gi7-20 Gi7-21 Gi7-23 Gi7-24 Gi7-28 Gi7-29 Gi7-30 Gi7-40,./CoreTech.logshvlan.txt
10,Data,active,Gi5-12 Gi7-8 Gi7-10 Gi7-11 Gi7-12,./CoreTech.logshvlan.txt
11,VLAN0011,active,,./CoreTech.logshvlan.txt
16,stuff,active,,./CoreTech.logshvlan.txt
17,morestuff,active,,./CoreTech.logshvlan.txt
19,morestuff,active,,



Desired output (file3)

Code:
Hostname,Port,Name,Status,Vlan,Duplex,Speed,Type,Vlan Name
shintstat,Te1/1 ,	Trunk- ,	connected ,	11 ,	full ,	10G ,	10GBase-CU 3M,
shintstat,Te1/2 ,	Trunk ,	connected ,	17 ,	full ,	10G ,	10GBase-CU 3M,


Last edited by Scott; 09-21-2018 at 09:29 AM.. Reason: Added code tags to data
# 2  
Old 09-21-2018
Having added code tags to the data as well, I see that file1 has a lot of whitespace (spaces and/or tabs) in the fields. If you're using -F to specify a very specific field separator (,), then the whitespace in those fields will become part of the fields themselves, so an exact match with fields in the other file which do not have the exact same whitespace won't work.

Assuming the logic works, a simple option would be to remove any whitespace directly before, or after a comma in file1.

e.g.
Code:
$ sed 's/[[:space:]]*,[[:space:]]*/,/g' file1 > file1.$$ && awk -F, 'FNR==NR{f2[$1]=$2;next} FNR==1{print $0, "VLAN Name";next} {print $0,($5 in f2)?f2[$5]:"NA"}' OFS=, file2 file1.$$ && rm file1.$$
Hostname,Port,Name,Status,Vlan,Duplex,Speed,Type,VLAN Name
shintstat,Te1/1,Trunk-,connected,11,full,10G,10GBase-CU 3M,,VLAN0011
shintstat,Te1/2,Trunk,connected,17,full,10G,10GBase-CU 3M,,morestuff
shintstat,Te1/3,Trunk,connected,trunk,full,10G,10GBase-CU 3M,,NA
shintstat,Te1/4,Trunk,connected,trunk,full,10G,10GBase-CU 3M,,NA

This User Gave Thanks to Scott For This Post:
# 3  
Old 09-21-2018
Don't double post. What's new compared to your other thread https://www.unix.com/shell-programming-and-scripting/279906-comparing-two-columns-two-files-printing-third-based-match.html except the two (new?) input files? Your (new?) desired output doesn't seem to satisfy your specification, does it?


EDIT: adding
Code:
{gsub (/[ 	]*,[ 	]*/, ",")}

(note the <TAB> chars!) as the first statement in your awk script might help accomplish your task.

Last edited by RudiC; 09-21-2018 at 10:07 AM..
# 4  
Old 09-21-2018
RudiC, sorry. The other post was all over the board and I had a chance to look at this again this morning. I though it was just cleaner to start over. I apologize, and thanks again for your help

------ Post updated at 01:26 PM ------

This works perfect however I am unable to pipe the output to a new file. What am I missing? In my limited experience with SED & AWK, I never had an issue with piping ouput
# 5  
Old 09-21-2018
The redirection is a shell thing, not an awk or sed thing. Put it after the awk command, regardless of which solution you use.

e.g.
Code:
awk ...... file2 file1 > file3

or
Code:
sed .. && awk ... file2 file1 > file3 && rm ...

or, when using multiple commands, you can wrap the whole thing in parenthesis, and redirect that, warts and all. e.g.
Code:
(command1 && command2 && command3) > file3

# 6  
Old 09-21-2018
Perfect, thank you for explaining!! Can you explain the parts of the command? I want to understand the structure, Ive been struggling with the awk command structure
# 7  
Old 09-21-2018
or all in "trimming" awk:
awk -f dis.awk file2 file1
where dis.awk is:
Code:
BEGIN {
  FS=OFS=","
  tab=sprintf("\t")
}
function trim(str)
{
    sub("^([ ]*|" tab "*)", "", str)
    sub("([ ]*|" tab "*)" "$", "", str)
    return str
}

FNR==NR { f2[trim($1)]=trim($2);next }
FNR==1   { print $0, "VLAN Name";next}
{ $5=trim($5);print $0,($5 in f2)?f2[$5]:"NA" }

Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

Match Fields between two files, print portions of each file together when matched in ([g]awk)'

I've written an awk script to compare two fields in two different files and then print portions of each file on the same line when matched. It works reasonably well, but every now and again, I notice some errors and cannot seem to figure out what the issue may be and am turning to you for help. ... (2 Replies)
Discussion started by: jvoot
2 Replies

2. Shell Programming and Scripting

awk to print fields that match using conditions and a default value for non-matching in two files

Trying to use awk to match the contents of each line in file1 with $5 in file2. Both files are tab-delimited and there may be a space or special character in the name being matched in file2, for example in file1 the name is BRCA1 but in file2 the name is BRCA 1 or in file1 name is BCR but in file2... (6 Replies)
Discussion started by: cmccabe
6 Replies

3. UNIX for Beginners Questions & Answers

How to count lines of CSV file where 2 fields match variables?

I'm trying to use awk to count the occurrences of two matching fields of a CSV file. For instance, for data that looks like this... Joe,Blue,Yes,No,High Mike,Blue,Yes,Yes,Low Joe,Red,No,No,Low Joe,Red,Yes,Yes,Low I've been trying to use code like this... countvar=`awk ' $2~/$color/... (4 Replies)
Discussion started by: nmoore2843
4 Replies

4. Shell Programming and Scripting

Compare 2 files of csv file and match column data and create a new csv file of them

Hi, I am newbie in shell script. I need your help to solve my problem. Firstly, I have 2 files of csv and i want to compare of the contents then the output will be written in a new csv file. File1: SourceFile,DateTimeOriginal /home/intannf/foto/IMG_0713.JPG,2015:02:17 11:14:07... (8 Replies)
Discussion started by: refrain
8 Replies

5. Shell Programming and Scripting

Match columns from two csv files and update field in one of the csv file

Hi, I have a file of csv data, which looks like this: file1: 1AA,LGV_PONCEY_LES_ATHEE,1,\N,1,00020460E1,0,\N,\N,\N,\N,2,00.22335321,0.00466628 2BB,LES_POUGES_ASF,\N,200,200,00006298G1,0,\N,\N,\N,\N,1,00.30887539,0.00050312... (10 Replies)
Discussion started by: djoseph
10 Replies

6. Shell Programming and Scripting

awk help: Match data fields from 2 files & output results from both into 1 file

I need to take 2 input files and create 1 output based on matches from each file. I am looking to match field #1 in both files (Userid) and create an output file that will be a combination of fields from both file1 and file2 if there are any differences in the fields 2,3,4,5,or 6. Below is an... (5 Replies)
Discussion started by: ambroze
5 Replies

7. Shell Programming and Scripting

Matching and Merging csv data fields based on a common field

Dear List, I have a file of csv data which has a different line per compliance check per host. I do not want any omissions from this csv data file which looks like this: date,hostname,status,color,check 02-03-2012,COMP1,FAIL,Yellow,auth_pass_change... (3 Replies)
Discussion started by: landossa
3 Replies

8. UNIX for Dummies Questions & Answers

Match values from 2 files and append certain fields

Hi, I need help on appending certain field in my file1.txt based on matched patterns in file2.txt using awk or sed. The blue color need to match with one of the data in field $2 in file2.txt. If match, BEGIN and FINISHED value in red will have a new value from field $3 and $4 accordingly. ... (1 Reply)
Discussion started by: redse171
1 Replies

9. Shell Programming and Scripting

Match data based on two fields, and append to a line

I need to write a program to do something like a 'vlookup' in excel. I want to match data from file2 based on two fields (where both match) in file1, and for matching lines, add the data from two of the fields from file2 to file1. If anyone knows something in perl or awk that can do this, I'd be... (20 Replies)
Discussion started by: jamessmith01
20 Replies

10. Shell Programming and Scripting

How to create a CSV File by reading fields from separate files

SHELL SCRIPT Hi, I have 3 separate files within a folder. Every File contains data in a single column like File1 contains data mayank sushant dheeraj File2 contains DSA_AT MG_AT FLAT_09 File3 contains data 123123 232323 (2 Replies)
Discussion started by: mayanksargoch
2 Replies
Login or Register to Ask a Question