Visit Our UNIX and Linux User Community


Merging 2 .CSV files in Unix


 
Thread Tools Search this Thread
Top Forums UNIX for Dummies Questions & Answers Merging 2 .CSV files in Unix
# 1  
Old 05-02-2008
Merging 2 .CSV files in Unix

I need a little help as I am a complete novice at scripting in unix. However, i am posed with an issue...Smilie i have two csv files in the following format@

FILE1.CSV:
HEADER
HEADER
Header
, , HEADER
001X ,,200
002X ,,300
003X ,,300
004X ,,300



FILE2.CSV:
HEADER
HEADER
Header
, , HEADER
001X ,,300
003X ,,500
005X ,,600

006X ,,100


I need the above to files to merged into on final file as below:

FINAL_MERGED.CSV:
HEADER
HEADER
Header
, , HEADER
001X ,,300
002X ,,300
003X ,,500
004X ,,300
005X ,,600
006X ,,100



I have tried using the join command but that only lets me to join the two files but as you can see from the above i need the ability to be able to overwrite values from file 1 whith those in file 2.

Any assistance would be greatly appreciated.
Thanks

Last edited by chachabronson; 05-06-2008 at 05:58 AM..
# 2  
Old 05-02-2008
anybody outhere with a clue???
# 3  
Old 05-04-2008
Could you clarify the logic?
Why you want 005X to be 100 and not 600?
# 4  
Old 05-06-2008
Quote:
Originally Posted by radoulov
Could you clarify the logic?
Why you want 005X to be 100 and not 600?
Sorry my mistake that was due to a typo. I have edited and amended above as required.

The logic behing it is that the first csv (FILE1.CSV) file is produced but may have incorrect figures associated with it. Once the information is readily available the second csv (FILE2.CSV) is produced with the correct figures and any missing codes.

Once this has been done file 1 and file 2 need to be merged to have the correct figures from file 2 and also to include any missing codes that are on file 2 but not file 1.

I need to do this in Unix, my initial thoughts were to use the join command but not sure if this is possible but may be some use of lookup could be done... again i am a programming and unix newbie so would appreciate any assistance.

Thanks
# 5  
Old 05-06-2008
Use nawk or /usr/xpg4/bn/awk on Solaris.
You may need to modify the code, if the format of your input doesn't match exactly the samples you posted.

Code:
awk 'END {
for (i=5; i<=n; i++)
  if (f2[_[i]])
    print f2[_[i]]
}
NR == FNR {
f2[_[n = FNR] = $1] = $0
next
}
$1 in f2 { 
$0 = f2[$1] 
delete f2[$1]
}1' FS=, OFS=, file2.csv file1.csv


Last edited by radoulov; 05-06-2008 at 09:37 AM..
# 6  
Old 05-09-2008
Thanks for your help, however i am unable to get teh code above to work.
I have had some assistance with the above from anothe forumite who gave me the following code:

#!/bin/ksh
cat file1.csv file2.csv >> temp
grep X temp | cut -d"," -f1 | sort -u > final2
while read line
do
line1=`grep $line file2.csv`
if [ $? -eq 0 ]
then
echo $line1 >> final_out
else
line2=`grep $line file1.csv`
echo $line2 >> final_out
fi

done < final2


If anyone could explain in detail as to what is happening in the loop that would be great... i have tested the above and it works like a dream but i want to learn rather than just use code. I am a complet novice so would like a description in a basic level which will help me understand for future.

Thanks to everyone who has helped so far
# 7  
Old 08-05-2008
Join csv

Hi .. as written I tried to implement the logic.....I could successfully do the concatenation....but my file "final_2" is blank.....
 

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Merging fields in CSV

Hi experts, I have a csv file which has one field (ID) repeated multiple times with corresponding other field values. I need to convert this file in a format where for a ID all other values has to be present in single field. For Eg : Here in below file ID 1 is repeated 3 times with different... (7 Replies)
Discussion started by: bharathbangalor
7 Replies

2. Shell Programming and Scripting

Merging two files in UNIX

Hi Experts, Need urgent solution for a problem. I have two files file1 and file2. file1 is tab separated and file2 is comma separated. I need to merge both the files into single file based on CUST_ID by retaining the headers of file1 Matching CUST_IDs should be placed one below the other in... (11 Replies)
Discussion started by: bharathbangalor
11 Replies

3. Shell Programming and Scripting

Merging Files in UNIX shell script

I have the urge to merge some files using unix shell script but I'm very new using this language and I haven't succeeded yet. The requirement is to merge the header, body and footer into one file with the name "ANY-NAME" in below example. To identify which files should be merged, I have flagged... (9 Replies)
Discussion started by: black_soul
9 Replies

4. Shell Programming and Scripting

Help with merging two CSV files

Hi, I have following 2 CSV files file1.txt A1,B1,C1,D1,E1 A2,B2,C2,D2,E2 A3,B3,C3,D3,E3 .... file2.txt A1,B1,P1,Q1,R1,S1,T1,U1 A1,B1,P2,Q2,R2,S2,T2,U2 A1,B1,P3,Q3,R3,S3,T3,U3 A2,B2,X1,Y1,Z1,I1,J1,K1 A2,B2,X2,Y2,Z2,I2,J2,K2 A2,B2,X3,Y3,Z3,I3,J3,K3 A2,B2,X4,Y4,Z4,I4,J4,K4... (2 Replies)
Discussion started by: learnoutmore99
2 Replies

5. Shell Programming and Scripting

Merging Very large CSV files in Unix

Hi, I have two very large CSV files, which I want to merge (equi-join) based on a key (column). One of the file (say F1) would have ~30 MM records and 700 columns. The other file (~f2) would have same # of records and lesser columns (say 50). I want to create an output file joining on a... (3 Replies)
Discussion started by: student_007
3 Replies

6. Shell Programming and Scripting

Merging all (48) CSV files from a directory

I have 48 csv files in my directory that all have this form: Storm Speed (mph),43.0410781151 Storm motion direction (degrees),261.580774982 MLCAPE,2450.54098661 MLCIN,-9.85040520279 MLLCL,230 MLLFC,1070.39871 MLEL,207.194689294 MLCT,Not enough data Sbcape,2203.97617778... (3 Replies)
Discussion started by: RissaR
3 Replies

7. Shell Programming and Scripting

Matching lines across multiple csv files and merging a particular field

I have about 20 CSV's that all look like this: "","","","","","","","","","","","","","","",""What I've been told I need to produce is the exact same thing, but with each file now containing the start_code from every other file where the email matches. It doesn't matter if any of the other... (1 Reply)
Discussion started by: Demosthenes
1 Replies

8. UNIX for Dummies Questions & Answers

Merging two CSV files by 3 primary keys (columns)

Hi there! I have the following problem: I have a set of files called rates_op_yyyyddmm with the format below (which corresponds to the file rates_op_20090130) 30-JAN-2009,ED,FEB09,C,96.375,,,0,,,,,,2.375,,,,,, 30-JAN-2009,ED,FEB09,C,96.5,,,0,,,,,,2.25,,,,,,... (2 Replies)
Discussion started by: Pep Puigvert
2 Replies

9. Shell Programming and Scripting

Merging files to create CSV file

Hi, I have different files of the same type, as: Time: 100 snr: 88 perf: 10 other: 222 Each of these files are created periodically. What I need to do is to merge all of them into one but having the following form: (2 Replies)
Discussion started by: Ravendark
2 Replies

10. UNIX for Dummies Questions & Answers

Merging Non-Empty Columns within a CSV

I am trying to place all my data in a single row (order doesn't matter). Note I am a Unix novice, so please go easy on me. Here is an example Raw data: row# (1) 45 64 23 (2) 32 1 6 56 (3) 32 45 Needs to be like this: row# (1) 45 (2) 32 (3) 32 ... (2 Replies)
Discussion started by: mmann1123
2 Replies

Featured Tech Videos