Shell Programming and Scripting

Hi,
I'm having trouble in achieving the following scenario. There is a txt file with thousands of lines and few lines are repeated, which needs to be removed using a script.

File.txt

Code:

20140522121432,0,12,ram
Loc=India
From=ram@xxx.com, To=ravi@yyy.com,,
1
2
3
4
.
.
30
20140523121432,0,12,roger
Loc=redmont
From=roger@xxx.com, To=david@yyy.com,,
20140523121432,0,12,roger
Loc=redmont
From=roger@xxx.com, To=david@yyy.com,,
1
2
3
.
.
30
......
....
......
...

So as you see in the file

Code:

20140523121432,0,12,roger
Loc=redmont
From=roger@xxx.com, To=david@yyy.com,,

This line is repeating, so these are to be removed via a script.

Anybody having any idea on this

How to identify the lines? I am pretty sure you don't want to just remove the duplicates from the file?

If I am wrong,

Code:

awk '!a[$0]++' file

This will remove the duplicate lines without changing the order of the lines. (unlike sort -u)

Ya that was my concern..

Your code will remove all duplicate lines, but what i need is to remove only duplicate in headers like this:
I thought of using any pattern matching kinda logic but not able to move with it..

How is your script supposed to know where a set of headers starts and where a set of headers ends?

As given in the file.txt the header is of 3 lines only for the entire file, so how about we find "To" or "From" in the pattern matching in AWK and if it matches the same in next 3 lines then from that point it should remove the previous 3 lines.

will this work?.. or is there any other approach to do this..

In your sample, the three line header to be removed immediately follows the header it duplicates. Do you want to remove any matching three line header anywhere in the remainder of the file, or do you only want to remove three line headers that are adjacent to duplicate three line headers?

Does the third line in a header have to start with From= and contain , To=, or can From= and To= appear anywhere on a line?

Do the two lines before the line containing From= have to be in any particular format for the three lines ending with the line containing From= to be considered to be a three line header?

Don, the thing is if it immediately follows one header it will be the replica of the same so we have no issues on it.

Just need to remove one of those replica headers ( adjacent ones) by any means..