Shell Programming and Scripting

Hi, I have file like this -

ABC 123
ABC 456
ABC 321
CDE 789
CDE 345
FGH 111
FGH 222
FGH 333
XYZ 678

I need the output like this:

ABC 123,456
CDE 789,345
FGH 111, 222
XYZ 678

Meaning I want to concat the lines that have the same first column, but I only need the first two lines. If the key field (like above ABC) has more than 3 lines, I only want to concat the first two lines.

Any idea how to do this?

Thanks!!

Code:

awk '{if (b[$1]<2) {a[$1]=a[$1] FS $2;b[$1]++}} END {for (i in a) print i,a[i]|"sort"}' urfile

Hi Rdcwayx,

Thanks for the quick reply. My question is, I am dealing with large amout of data. Probably about 10 million. Will this have performance issue?

Thanks

Are the lines sorted by the first column?

You should just run it and find out if performance and resource consumption is acceptable. What does and doesn't have performance issues depends on the person, situation, priority of the job, the hardware, etc.

Only thing that can be stated by looking at the awk/sort code, is that it will require something on the order of 2 x 10 x avg_line_length megabytes of memory for a 10 million line data set, since both awk and sort will require full copies of the data in memory (worst case scenario). If the average line length is 50 characters, it could approach 1 gigabyte or ram required.

Regards,
Alister

If the data is ordered by the 1e column, this should work fine:

Code:

awk '$1==k{s=s "," $2; next}
s{print s}
{s=$0;k=$1}
END{print s}' file

Hello, Franklin52:

Actually, that code isn't quite right. The original problem statement requires that at most two values are matched per key. This solution will continue to assimilate as many as found. Changing

Code:

{print s}

to

Code:

s{match(s,/^[^ ]* [^,]*(,[^,]*)?/); print substr(s, RSTART, RLENGTH)}

works around that by discarding unwanted values at print time. A counter is probably a nicer fix, though:

Code:

awk '$1==k {if (++i<3) s=s "," $2; next}
s{print s}
{s=$0;k=$1;i=1}
END{print s}' file

Regards,
Alister