concatenate all duplicate line in a file.


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting concatenate all duplicate line in a file.
# 15  
Old 08-28-2008
Thanks All....
it is working....
but i am facing another problem

while i am trying to change the format

like

794051400123|99.98|COM|24|MOM|62|RNO|73|SOM|25

it gives me result like

794051400123|99.98
|COM|24|MOM|62|RNO|73|SOM|25

i.e brake 1 line into 2 line...
# 16  
Old 08-28-2008
Please post a sample of input lines (preferably less than five lines) which allows us to reproduce this problem. With the input you have posted, the code works. We can't know what the special cases are if you don't provide representative input samples.
# 17  
Old 08-28-2008
here i past my code:-

awk -F '|' 'NF==0 { next } $1 == prev {collect=collect","$2"="$3; next }
{ if (collect) print collect; prev = $1; collect = $1","$5","$6","$7","$2"="$3; next }
END { if (collect) print collect}'

Input:-

794051400123|COM|24|0|BD|R|99.98

794051413727|COM|11|0|BD|R|28.99

794051415622|COM|23|0|BD|R|28.99

883929004676|COM|0|0|BD|R|28.99
794051400123|MOM|62|0|BD|R|99.98

794051413727|MOM|4|0|BD|R|28.99

794051415622|MOM|80|0|BD|R|28.99

883929004676|MOM|0|0|BD|R|28.99

883929017164|MOM|0|0|BD|R|39.99
794051400123|RNO|73|0|BD|R|99.98

Corrent Out put:-


794051400123,BD,R,99.98,COM=24,MOM=62,SOM=25,RNO=73
794051413727,BD,R,28.99,COM=11,MOM=4,RNO=8
.....

But my out put(Wrong):-

794051400123,BD,R,99.98
,COM=24,MOM=62,SOM=25,RNO=73
794051413727,BD,R,28.99
,COM=11,MOM=4,RNO=8
.....

i.e breaks line from 99.98..Which is wrong.
# 18  
Old 08-28-2008
With the input you posted, I get the following output.

Code:
794051400123,BD,R,99.98,COM=24
794051413727,BD,R,28.99,COM=11
794051415622,BD,R,28.99,COM=23
883929004676,BD,R,28.99,COM=0
794051400123,BD,R,99.98,MOM=62
794051413727,BD,R,28.99,MOM=4
794051415622,BD,R,28.99,MOM=80
883929004676,BD,R,28.99,MOM=0
883929017164,BD,R,39.99,MOM=0
794051400123,BD,R,99.98,RNO=73

There is not a SOM in sight in your sample input data, so I don't understand where that would come from.

If I sort the input before feeding it to awk, I get the following result.

Code:
794051400123,BD,R,99.98,COM=24,MOM=62,RNO=73
794051413727,BD,R,28.99,COM=11,MOM=4
794051415622,BD,R,28.99,COM=23,MOM=80
883929004676,BD,R,28.99,COM=0,MOM=0
883929017164,BD,R,39.99,MOM=0

If I understand your requirements, this is correct.

In other words, I cannot reproduce your problem. Again, we need input and output which we can reproduce the problem with.
# 19  
Old 08-28-2008
yes...

794051400123,BD,R,99.98,COM=24,MOM=62,RNO=73
794051413727,BD,R,28.99,COM=11,MOM=4
794051415622,BD,R,28.99,COM=23,MOM=80
883929004676,BD,R,28.99,COM=0,MOM=0
883929017164,BD,R,39.99,MOM=0

is correct

but when i run my shell script all lines brakes from 99.98,28.99...etc.

i.e
794051400123,BD,R,99.98
,COM=24,MOM=62,RNO=73
794051413727,BD,R
,28.99,COM=11,MOM=4

i don't understand why it breaks the line? Will there any may to merge 2 line into one line.I have tried it but falied...Can u pls paste your code?
# 20  
Old 08-28-2008
Code:
sort inputfile | awk ... your awk script here ...

# 21  
Old 08-28-2008
If you have DOS carriage returns in the file, that could perhaps also explain the breaks.
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Log file - Delete duplicate line & keep last date

Hello All ! I need your help on this case, I have a csv file with this: ITEM105;ARI FSR;2016-02-01 08:02;243 ITEM101;ARI FSR;2016-02-01 06:02;240 ITEM032;RNO TLE;2016-02-01 11:03;320 ITEM032;RNO TLE;2016-02-02 05:43;320 ITEM032;RNO TLE;2016-02-01 02:03;320 ITEM032;RNO... (2 Replies)
Discussion started by: vadim-bzh
2 Replies

2. Shell Programming and Scripting

Help with concatenate multiple line into one line

Hi, Do anybody experience how to concatenate multiple line into one line by using awk or perl command? Input file: >set1 QAWEQRQ@EWQEASED ASDAEQW QAWEQRQTQ ASRFQWRGWQ From the above Input file, it got 5 lines Desired output file: >set1... (6 Replies)
Discussion started by: perl_beginner
6 Replies

3. Shell Programming and Scripting

Honey, I broke awk! (duplicate line removal in 30M line 3.7GB csv file)

I have a script that builds a database ~30 million lines, ~3.7 GB .cvs file. After multiple optimzations It takes about 62 min to bring in and parse all the files and used to take 10 min to remove duplicates until I was requested to add another column. I am using the highly optimized awk code: awk... (34 Replies)
Discussion started by: Michael Stora
34 Replies

4. Shell Programming and Scripting

Concatenate small line with next line perl script

Hello to all, I'm new to perl, I have input file that contains the string below: 315350535ff450000014534130101ff4500ff45453779ff450ff45545f01ff45ff453245341ff4500000545000This string has as line separator "ff45". So, I want to print each line but the code below is not working. perl -pe '... (2 Replies)
Discussion started by: Ophiuchus
2 Replies

5. Shell Programming and Scripting

Delete Duplicate line (not really) from the file

I need help in figuring out hoe to delete lines in a data file. The data file is huge. I am currently using "vi" to search and delete the lines - which is cumbersome since it takes lots of time to save that file (due to its huge size). Here is the issue. I have a data file with the following... (4 Replies)
Discussion started by: GosarJunk
4 Replies

6. Shell Programming and Scripting

awk concatenate every line of a file in a single line

I have several hundreds of tiny files which need to be concatenated into one single line and all those in a single file. Some files have several blank lines. Tried to use this script but failed on it. awk 'END { print r } r && !/^/ { print FILENAME, r; r = "" }{ r = r ? r $0 : $0 }' *.txt... (8 Replies)
Discussion started by: sdf
8 Replies

7. Shell Programming and Scripting

remove of duplicate line from a file

I have a file a.txt having content like deepak ram sham deepram sita kumar I Want to delete the first line containing "deep" ... I tried using... grep -i 'deep' a.txt It gives me 2 rows...I want to delete the first one.. + need to know the command to delete the line from... (5 Replies)
Discussion started by: saluja.deepak
5 Replies

8. Shell Programming and Scripting

How to find duplicate line in log file?

Hi guys, I'm really happy to find this forum I have a log file, and I have to find all lines that have "error" word, and then save this output in file, the output file has to have just only one line to any Duplicated lines and counter that show how many time this lines duplicated? I already... (2 Replies)
Discussion started by: wax_light
2 Replies

9. Shell Programming and Scripting

Concatenate strings line by line

Hi, I have a noob question . Can someone help me how to concatenate line by line using this variables? var1: Apple| Banana| var2: Red Yellow then how can I concatenate both line by line? in which the result would be: Apple|Red Banana|Yellow just to generate a row result i was... (6 Replies)
Discussion started by: hagdanan
6 Replies

10. Shell Programming and Scripting

duplicate line in a text file

i would like to scan file in for duplicate lines, and print the duplicates to another file, oh and it has to be case insensitive. example line1 line2 line2 line3 line4 line4 outputfile: line2 line4 any ideas (5 Replies)
Discussion started by: nixguy
5 Replies
Login or Register to Ask a Question