Deduping file


 
Thread Tools Search this Thread
Top Forums UNIX for Dummies Questions & Answers Deduping file
# 8  
Old 04-13-2008
Code:
awk 'END { for (k in x) 
print k, x[k], y[k] }
{ x[$1FS$2FS$3] = x[$1FS$2FS$3] ? x[$1FS$2FS$3] ";" $4 : $4
y[$1FS$2FS$3] = $5 } ' FS=\| OFS=\| Test_File.dat

Code:
x[$1FS$2FS$3] = x[$1FS$2FS$3] ? x[$1FS$2FS$3] ";" $4 : $4

x is an associative array with keys based on your logical primary key $1, $2 and $3 and values/elements based on the value of the fourth filed.
The if expression ? then : else statement is composing your elements in the desired format (if array[key] exists -> then append $4 to its value).

Code:
y[$1FS$2FS$3] = $5

Another associative array with the same keys and elements based on the last field.


Code:
END { for (k in x) 
print k, x[k], y[k] }

After reading the entire input for every key in the x array,
print the desired values: the x key, the x value and the y value.

Hope this helps.
 
Login or Register to Ask a Question

Previous Thread | Next Thread

3 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Shell script (sh file) logic to compare contents of one file with another file and output to file

Shell script logic Hi I have 2 input files like with file 1 content as (file1) "BRGTEST-242" a.txt "BRGTEST-240" a.txt "BRGTEST-219" e.txt File 2 contents as fle(2) "BRGTEST-244" a.txt "BRGTEST-244" b.txt "BRGTEST-231" c.txt "BRGTEST-231" d.txt "BRGTEST-221" e.txt I want to get... (22 Replies)
Discussion started by: pottic
22 Replies

2. Shell Programming and Scripting

Compare 2 text file with 1 column in each file and write mismatch data to 3rd file

Hi, I need to compare 2 text files with around 60000 rows and 1 column. I need to compare these and write the mismatch data to 3rd file. File1 - file2 = file3 wc -l file1.txt 58112 wc -l file2.txt 55260 head -5 file1.txt 101214200123 101214700300 101250030067 101214100500... (10 Replies)
Discussion started by: Divya Nochiyil
10 Replies

3. Shell Programming and Scripting

Match list of strings in File A and compare with File B, C and write to a output file in CSV format

Hi Friends, I'm a great fan of this forum... it has helped me tone my skills in shell scripting. I have a challenge here, which I'm sure you guys would help me in achieving... File A has a list of job ids and I need to compare this with the File B (*.log) and File C (extend *.log) and copy... (6 Replies)
Discussion started by: asnandhakumar
6 Replies
Login or Register to Ask a Question