Sponsored Content
Full Discussion: Remove duplicate text
Top Forums Shell Programming and Scripting Remove duplicate text Post 302203290 by Franklin52 on Saturday 7th of June 2008 05:35:58 PM
Old 06-07-2008
Does your file has a different format as you posted? Is every block seperated by an empty line?
I've copy and paste your example and it works fine for me.

This changes the default record and field seperator so you can treat every line as a field and every block as a record. The 2nd field is the start time end the 3th the end time:

Code:
awk 'BEGIN{RS="";FS="\n"}

Variable t is a reminder, print only records if the start and end time is different from the previous block:

Code:
t!=$2$3{t=$2$3;print $0,"\n"}

Regards
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Remove duplicate ???

Hi all, I have a out.log file CARR|02/26/2006 10:58:30.107|CDxAcct=1405157051 CARR|02/26/2006 11:11:30.107|CDxAcct=1405157051 CARR|02/26/2006 11:18:30.107|CDxAcct=7659579782 CARR|02/26/2006 11:28:30.107|CDxAcct=9534922327 CARR|02/26/2006 11:38:30.107|CDxAcct=9534922327 CARR|02/26/2006... (3 Replies)
Discussion started by: sabercats
3 Replies

2. Shell Programming and Scripting

Remove duplicate

Hi all, I have a text file fileA.txt DXRV|02/28/2006 11:36:49.049|SAC||||CDxAcct=2420991350 DXRV|02/28/2006 11:37:06.404|SAC||||CDxAcct=6070970034 DXRV|02/28/2006 11:37:25.740|SAC||||CDxAcct=2420991350 DXRV|02/28/2006 11:38:32.633|SAC||||CDxAcct=6070970034 DXRV|02/28/2006... (2 Replies)
Discussion started by: sabercats
2 Replies

3. Shell Programming and Scripting

Remove duplicate files based on text string?

Hi I have been struggling with a script for removing duplicate messages from a shared mailbox. I would like to search for duplicate messages based on the “Message-ID” string within the messages files. I have managed to find the duplicate “Message-ID” strings and (if I would like) delete... (1 Reply)
Discussion started by: spangberg
1 Replies

4. Shell Programming and Scripting

remove duplicate

Hi, I am tryung to use shell or perl to remove duplicate characters for example , if I have " I love google" it will become I love ggle" or even "I loveggle" if removing duplicate white space Thanks CC (6 Replies)
Discussion started by: ccp
6 Replies

5. Shell Programming and Scripting

Filter or remove duplicate block of text without distinguishing marks or fields

Hello, Although I have found similar questions, I could not find advice that could help with our problem. The issue: We have several hundreds text files containing repeated blocks of text (I guess back at the time they were prepared like that to optmize printing). The block of texts... (13 Replies)
Discussion started by: samask
13 Replies

6. Shell Programming and Scripting

Remove duplicate

Hi , I have a pipe seperated file repo.psv where i need to remove duplicates based on the 1st column only. Can anyone help with a Unix script ? Input: 15277105||Common Stick|ESHR||Common Stock|CYRO AB 15277105||Common Stick|ESHR||Common Stock|CYRO AB 16111278||Common Stick|ESHR||Common... (12 Replies)
Discussion started by: samrat dutta
12 Replies

7. UNIX for Dummies Questions & Answers

Remove duplicate

Hi, How can I replace || with space and then remove duplicate from following text? T111||T222||T444||T222||T555 Thanks in advance (10 Replies)
Discussion started by: tinku981
10 Replies

8. Windows & DOS: Issues & Discussions

Remove duplicate lines from text files.

So, I have text files, one "fail.txt" And one "color.txt" I now want to use a command line (DOS) to remove ANY line that is PRESENT IN BOTH from each text file. Afterwards there shall be no duplicate lines. (1 Reply)
Discussion started by: pasc
1 Replies

9. Shell Programming and Scripting

How to remove duplicate text blocks from a file?

Hi All I have a list of files which will have duplicate list of blocks of text. Following is a sample of the file, I have removed the sensitive information from the file. All the code samples starts from <TR BGCOLOR="white"> and Ends with IP address and two html tags like this. 10.14.22.22... (3 Replies)
Discussion started by: mahasona
3 Replies

10. Shell Programming and Scripting

Remove duplicate occurrences of text pattern

Hi folks! I have a file which contains a 1000 lines. On each line i have multiple occurrences ( 26 to be exact ) of pattern folder#/folder#. # is depicting the line number in the file some text here folder1/folder1 some text here folder1/folder1 some text here folder1/folder1 some text... (7 Replies)
Discussion started by: martinsmith
7 Replies
xfs_logprint(8) 					      System Manager's Manual						   xfs_logprint(8)

NAME
xfs_logprint - print the log of an XFS filesystem SYNOPSIS
xfs_logprint [ options ] device DESCRIPTION
xfs_logprint prints the log of an XFS filesystem (see xfs(5)). The device argument is the pathname of the partition or logical volume con- taining the filesystem. The device can be a regular file if the -f option is used. The contents of the filesystem remain undisturbed. There are two major modes of operation in xfs_logprint. One mode is better for filesystem operation debugging. It is called the transactional view and is enabled through the -t option. The transactional view prints only the portion of the log that pertains to recovery. In other words, it prints out complete transactions between the tail and the head. This view tries to display each transaction without regard to how they are split across log records. The second mode starts printing out information from the beginning of the log. Some error blocks might print out in the beginning because the last log record usually overlaps the oldest log record. A message is printed when the physical end of the log is reached and when the logical end of the log is reached. A log record view is displayed one record at a time. Transactions that span log records may not be decoded fully. OPTIONS
-b Extract and print buffer information. Only used in transactional view. -c Attempt to continue when an error is detected. -C filename Copy the log from the filesystem to the file filename. The log itself is not printed. -d Dump the log from front to end, printing where each log record is located on disk. -D Do not decode anything; just print data. -e Exit when an error is found in the log. Normally, xfs_logprint tries to continue and unwind from bad logs. However, sometimes it just dies in bad ways. Using this option prevents core dumps. -f Specifies that the filesystem image to be processed is stored in a regular file at device (see the mkfs.xfs(8) -d file option). This might happen if an image copy of a filesystem has been made into an ordinary file with xfs_copy(8). -l logdev External log device. Only for those filesystems which use an external log. -i Extract and print inode information. Only used in transactional view. -q Extract and print quota information. Only used in transactional view. -n Do not try and interpret log data; just interpret log header information. -o Also print buffer data in hex. Normally, buffer data is just decoded, so better information can be printed. -s start-block Override any notion of where to start printing. -t Print out the transactional view. SEE ALSO
mkfs.xfs(8), mount(8). xfs_logprint(8)
All times are GMT -4. The time now is 04:25 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy