Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Remove duplicate text

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Remove duplicate text
# 1  
Old 06-07-2008
Remove duplicate text
Hello,

I have a log file which is generated by a script which looks like this:

Code:
userid: 7
starttime: Sat May 24 23:24:13 CEST 2008
endtime: Sat May 24 23:26:57 CEST 2008
total time spent: 2.73072 minutes / 163.843 seconds
date: Sat Jun 7 16:09:03 CEST 2008

userid: 8
starttime: Sun May 25 00:14:30 CEST 2008
endtime: Sun May 25 00:14:32 CEST 2008
total time spent: 0.0304667 minutes / 1.828 seconds
date: Sat Jun 7 16:10:02 CEST 2008

userid: 9
starttime: Sun May 25 00:14:30 CEST 2008
endtime: Sun May 25 00:14:32 CEST 2008
total time spent: 0.0304667 minutes / 1.828 seconds
date: Sat Jun 7 16:11:01 CEST 2008

Everytime when I run the script, it will increase the userid by one and adds information(start time, end time etc).
Anyone knows if there is an efficient way to remove the whole last block of text when its starttime or endtime or both is duplicate of the previous block?

Thanks in advance.
# 2  
Old 06-07-2008
Try this:

Code:
awk 'BEGIN{RS="";FS="\n"}
t!=$2$3{t=$2$3;print $0,"\n"}
' "file"

Regards
# 3  
Old 06-07-2008
Doesn't work Smilie , looks like there's no change when I execute the script.
Can you also explain what the code does? I'm new at using awk.
# 4  
Old 06-07-2008
Does your file has a different format as you posted? Is every block seperated by an empty line?
I've copy and paste your example and it works fine for me.

This changes the default record and field seperator so you can treat every line as a field and every block as a record. The 2nd field is the start time end the 3th the end time:

Code:
awk 'BEGIN{RS="";FS="\n"}

Variable t is a reminder, print only records if the start and end time is different from the previous block:

Code:
t!=$2$3{t=$2$3;print $0,"\n"}

Regards
# 5  
Old 06-07-2008
The original file format looks like this:

Code:
UserID: 7
Start Time: Sat May 24 23:24:13 CEST 2008
End Time: Sat May 24 23:26:57 CEST 2008
Total time spent: 2.73072 minutes / 163.843 seconds
Date: Sat Jun 7 16:09:03 CEST 2008

UserID: 8
Start Time: Sun May 25 00:14:30 CEST 2008
End Time: Sun May 25 00:14:32 CEST 2008
Total time spent: 0.0304667 minutes / 1.828 seconds
Date: Sat Jun 7 16:10:02 CEST 2008

UserID: 9
Start Time: Sun May 25 00:14:30 CEST 2008
End Time: Sun May 25 00:14:32 CEST 2008
Total time spent: 0.0304667 minutes / 1.828 seconds
Date: Sat Jun 7 16:11:01 CEST 2008

I forgot to mention that there's an extra line at the beginning of the file followed by an empty line. Which says something about my log file. Thought it would be possible to get awk to read from the end of the file instead of from the top.

[edit2]

My fault, it's also in html format, not plain text in a log file, so when awk reads the file, it reads multiple lines in the .html file instead of what you see in the output of the log in a browser. There are a lot of lines in the file, should I just paste them here or can I send you the file?
Last edited by dejavu88; 06-07-2008 at 07:24 PM..
# 6  
Old 06-08-2008
You should strip the html tags before processing the log file. Different solutions here:

- PHP function strip_tags()
- lynx text browser with option -dump
- html2text utility.
- using sed to get rid of the tags

Can you post an extract of your html file. Or attach it to your reply.
# 7  
Old 06-08-2008
I only stripped the last two html tags with a bit of code Franklin52 wrote for me in another post. So my script is able to add new log information and then put back the last two html tags after the log information over and over again. I'd rather not strip the html tags before processing the log, cause it's gonna be a hassle to put them back in the exact place where they were.

I've put the log up on pastebin:
http://pastebin.com/m2ba76b50
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Remove duplicate occurrences of text pattern

Hi folks! I have a file which contains a 1000 lines. On each line i have multiple occurrences ( 26 to be exact ) of pattern folder#/folder#. # is depicting the line number in the file some text here folder1/folder1 some text here folder1/folder1 some text here folder1/folder1 some text... (7 Replies)
Discussion started by: martinsmith
7 Replies

2. Shell Programming and Scripting

How to remove duplicate text blocks from a file?

Hi All I have a list of files which will have duplicate list of blocks of text. Following is a sample of the file, I have removed the sensitive information from the file. All the code samples starts from <TR BGCOLOR="white"> and Ends with IP address and two html tags like this. 10.14.22.22... (3 Replies)
Discussion started by: mahasona
3 Replies

3. Windows & DOS: Issues & Discussions

Remove duplicate lines from text files.

So, I have text files, one "fail.txt" And one "color.txt" I now want to use a command line (DOS) to remove ANY line that is PRESENT IN BOTH from each text file. Afterwards there shall be no duplicate lines. (1 Reply)
Discussion started by: pasc
1 Replies

4. UNIX for Dummies Questions & Answers

Remove duplicate

Hi, How can I replace || with space and then remove duplicate from following text? T111||T222||T444||T222||T555 Thanks in advance (10 Replies)
Discussion started by: tinku981
10 Replies

5. Shell Programming and Scripting

Remove duplicate

Hi , I have a pipe seperated file repo.psv where i need to remove duplicates based on the 1st column only. Can anyone help with a Unix script ? Input: 15277105||Common Stick|ESHR||Common Stock|CYRO AB 15277105||Common Stick|ESHR||Common Stock|CYRO AB 16111278||Common Stick|ESHR||Common... (12 Replies)
Discussion started by: samrat dutta
12 Replies

6. Shell Programming and Scripting

Filter or remove duplicate block of text without distinguishing marks or fields

Hello, Although I have found similar questions, I could not find advice that could help with our problem. The issue: We have several hundreds text files containing repeated blocks of text (I guess back at the time they were prepared like that to optmize printing). The block of texts... (13 Replies)
Discussion started by: samask
13 Replies

7. Shell Programming and Scripting

remove duplicate

Hi, I am tryung to use shell or perl to remove duplicate characters for example , if I have " I love google" it will become I love ggle" or even "I loveggle" if removing duplicate white space Thanks CC (6 Replies)
Discussion started by: ccp
6 Replies

8. Shell Programming and Scripting

Remove duplicate files based on text string?

Hi I have been struggling with a script for removing duplicate messages from a shared mailbox. I would like to search for duplicate messages based on the “Message-ID” string within the messages files. I have managed to find the duplicate “Message-ID” strings and (if I would like) delete... (1 Reply)
Discussion started by: spangberg
1 Replies

9. Shell Programming and Scripting

Remove duplicate

Hi all, I have a text file fileA.txt DXRV|02/28/2006 11:36:49.049|SAC||||CDxAcct=2420991350 DXRV|02/28/2006 11:37:06.404|SAC||||CDxAcct=6070970034 DXRV|02/28/2006 11:37:25.740|SAC||||CDxAcct=2420991350 DXRV|02/28/2006 11:38:32.633|SAC||||CDxAcct=6070970034 DXRV|02/28/2006... (2 Replies)
Discussion started by: sabercats
2 Replies

10. Shell Programming and Scripting

Remove duplicate ???

Hi all, I have a out.log file CARR|02/26/2006 10:58:30.107|CDxAcct=1405157051 CARR|02/26/2006 11:11:30.107|CDxAcct=1405157051 CARR|02/26/2006 11:18:30.107|CDxAcct=7659579782 CARR|02/26/2006 11:28:30.107|CDxAcct=9534922327 CARR|02/26/2006 11:38:30.107|CDxAcct=9534922327 CARR|02/26/2006... (3 Replies)
Discussion started by: sabercats
3 Replies
Login or Register to Ask a Question