remove duplicated xml record in a file under unix

happyv · #1 (**permalink**) 09-20-2006

Hi,

If i have a file with xml format, i would like to remove duplicated records and save to a new file. Is it possible...to write script to do it?

tayyabq8 · #2 (**permalink**) 09-20-2006

Try

Code:

uniq inputfile

Yogesh Sawant · #3 (**permalink**) 09-20-2006

I don't know if it's possible in shell or not, but it's possible in Perl. Do consider that option if you can.

happyv · #4 (**permalink**) 09-20-2006

Is the Perl can run under ksh Unix?

Also, the record is a bit difference...it look like

record1:
this is testing
my id is 2001
end:
record2:
this is testing2
my id is 2002
end:
record3:
this is testing
my id is 2002
end:
record4:
this is testing2
my id is 2002
end:

For the above, record 2 and 4 is duplicated. Because of the "id" and "testing2" is the same. if only one line is the same which is not called duplicated..

So Perl or any friend can help for the script?

ranj@chn · #5 (**permalink**) 09-20-2006

I havent tested this, but please check it

Code:

paste -s -d"\t\t\t\n" filename|sort -u |tr "\t" "\n"

aigles · #6 (**permalink**) 09-20-2006

You can try to use awk.
Create the following awk script uniq.awk :

Code:

/^end:/ {
   if (! (Record in Records)) {
      Records[Record];
      print RecordLabel ":";
      print Record;
      print $0;  
      Record = "";
   }
   next;
}
$1 ~ /^.*:/ {
   sub(/:.*/, "", $1);
   RecordLabel = $1;
   next;
}
{
   Record = (Record ? Record "\n" : "") $0;
}

and execute it :

Code:

$ awk -f uniq.awk filename
record1:
this is testing
my id is 2001
end:
record2:
this is testing2
my id is 2002
end:
record3:
this is testing
my id is 2002
end:
$

jean-Pierre.

nervous · #7 (**permalink**) 09-20-2006

Dear Sir,

It would be great help if you can describe the code below in detail, I have just started to learn about awk and I can say that understanding of following code in a clear way would help me a lot in future.

Quote:

/^end:/ {
if (! (Record in Records)) {
Records[Record];
print RecordLabel ":";
print Record;
print $0;
Record = "";
}
next;
}
$1 ~ /^.*:/ {
sub(/:.*/, "", $1);
RecordLabel = $1;
next;
}
{
Record = (Record ? Record "\n" : "") $0;
}

Thanks in advance.

More UNIX and Linux Forum Topics You Might Find Helpful
Thread	Thread Starter	Forum	Replies	Last Post
how to read record by record from a file in unix	raoscb	UNIX for Dummies Questions & Answers	1	05-16-2008 06:30 AM
remove duplicated columns	kamel.seg	Shell Programming and Scripting	6	02-21-2008 07:36 AM
Remove First and Last Record from a file	ravikuc	UNIX for Dummies Questions & Answers	1	10-11-2007 03:35 AM
remove duplicated lines without sort	lalelle	Shell Programming and Scripting	6	08-21-2007 07:44 AM
command to remove last record on file	mheinen	UNIX for Dummies Questions & Answers	4	01-09-2007 04:39 PM

Thread Tools	Search this Thread
Show Printable Version Email this Page	Search this Thread: Advanced Search
Display Modes	Rate This Thread
Linear Mode Switch to Hybrid Mode Switch to Threaded Mode	Rate This Thread: