Shell Programming and Scripting

Hi Masters,

Here is the code someone posted days ago, but it didn't work well because it did't omit the newline character when $2 conbines. Can someone help me out?

awk '{if($1==prev) printf(" /// "$2); else {printf("\n"$1" " $2);prev=$1}}'

Basically, this code is trying to remove the redundent line in a two column spreadsheet file based on the field one, but merge field 2 if field one is redundent.

Thanks in advance.

Need the sample text again please.
I think your thread was removed in violation of Rule Number 6.

Thanks! You are right. So I am fraid of violating rules again. But the basic idea is,

H82421 RDH11
H82421 DKFZP564M1462
H82435 BAGE4
H82435 MLL3
H82527 C20orf17
H82532 -
H82706 ID2
H82801 -
H82812 TNRC6B
H82872 UACA

What I need,

H82421 RDH11 /// DKFZP564M1462
H82435 BAGE4 /// MLL3
H82527 C20orf17
H82532 -
H82706 ID2
H82801 -
H82812 TNRC6B
H82872 UACA

Your code gave the line break in between $2,
H82421 RDH11^M /// DKFZP564M1462
H82435 BAGE4^M /// MLL3
H82527 C20orf17
H82532 -
H82706 ID2
H82801 -
H82812 TNRC6B
H82872 UACA

I could use some text editor to remove line breaks, but thought the complete code could be helpful for other people too. In addition, if the redundent lines are not sorted, what code should be? Thanks so much.

Are you by any chance viewing the output file in DOS/Windows?
"^M" normally appears when you do so..as these OSes do not recognize new lines(UNIX) properly

awk 'BEGIN{x="";i=0} NR == 1 {x=$1;y=$0} NR > 1 && x == $1 {printf("%s /// %s\n",y,$2);i=NR+1} NR == i {x=$1;y=$0} NR > i && x != $1 {printf("%s\n",y);x=$1;y=$0} END {printf("%s\n",y);}'

Thank you all. Both of your code generated almost same output in terms of line breaks, second code didn't work with multiple redundent lines.

First output,
AA001897 SPTA1
AA001918 MI-ER1^M /// ER1^M /// KIAA1610^M /// DKFZp781G0451
AA001924 Cep63^M /// FLJ13386
AA001950 KIF13A
AA001952 -
AA001970 -
AA001976 -

Second output,
AA001897 SPTA1
AA001918 MI-ER1^M /// ER1
AA001918 MI-ER1^M /// KIAA1610
AA001918 MI-ER1^M /// DKFZp781G0451
AA001924 Cep63^M /// FLJ13386
AA001950 KIF13A
AA001952 -
AA001970 -
AA001976 -
AA002006 -

I just wonder if this is my os problem or not. I am using Mac OS X, and bash shell.

It probably could be a OS problem. You could check by looking for special charcters by using "cat -vet" on the output file. Regarding your question on what do we do if the input is unsorted wrt to the first column, we sort it first using:

sort -t' ' -k 1 testfile.txt

and pipe the output to the awk statement you used.