Shell Programming and Scripting

Hi,

If i have a file with xml format, i would like to remove duplicated records and save to a new file. Is it possible...to write script to do it?

Try

Code:

uniq inputfile

I don't know if it's possible in shell or not, but it's possible in Perl. Do consider that option if you can.

Is the Perl can run under ksh Unix?

Also, the record is a bit difference...it look like

record1:
this is testing
my id is 2001
end:
record2:
this is testing2
my id is 2002
end:
record3:
this is testing
my id is 2002
end:
record4:
this is testing2
my id is 2002
end:

For the above, record 2 and 4 is duplicated. Because of the "id" and "testing2" is the same. if only one line is the same which is not called duplicated..

So Perl or any friend can help for the script?

I havent tested this, but please check it

Code:

paste -s -d"\t\t\t\n" filename|sort -u |tr "\t" "\n"

You can try to use awk.
Create the following awk script uniq.awk :

Code:

/^end:/ {
   if (! (Record in Records)) {
      Records[Record];
      print RecordLabel ":";
      print Record;
      print $0;  
      Record = "";
   }
   next;
}
$1 ~ /^.*:/ {
   sub(/:.*/, "", $1);
   RecordLabel = $1;
   next;
}
{
   Record = (Record ? Record "\n" : "") $0;
}

and execute it :

Code:

$ awk -f uniq.awk filename
record1:
this is testing
my id is 2001
end:
record2:
this is testing2
my id is 2002
end:
record3:
this is testing
my id is 2002
end:
$

jean-Pierre.

Dear Sir,

It would be great help if you can describe the code below in detail, I have just started to learn about awk and I can say that understanding of following code in a clear way would help me a lot in future.
Thanks in advance.