Remove groups of repeating lines


 
Thread Tools Search this Thread
Top Forums UNIX for Dummies Questions & Answers Remove groups of repeating lines
# 1  
Old 02-07-2011
Remove groups of repeating lines

I know uniq exists, but am not sure how to remove repeating lines when they are groups of two different lines repeating themselves, without using sort. I need them to be sorted in the original order, just to remove repeats.

Code:
 cd /media/AUDIO/WAVE/9780743518673/mp3
~/Desktop/mp3-to-m4b 9780743518673-GOOD GERMAN "GOOD GERMAN" "Joseph Kanon" 2004
 n
 cd /media/AUDIO/WAVE/9780743518673/mp3
~/Desktop/mp3-to-m4b 9780743518673-GOOD GERMAN "GOOD GERMAN" "Joseph Kanon" 2004
 n
 cd /media/AUDIO/WAVE/9780743518673/mp3
~/Desktop/mp3-to-m4b 9780743518673-GOOD GERMAN "GOOD GERMAN" "Joseph Kanon" 2004
 n
 cd /media/AUDIO/WAVE/9780743518673/mp3
~/Desktop/mp3-to-m4b 9780743518673-GOOD GERMAN "GOOD GERMAN" "Joseph Kanon" 2004
 n
 cd /media/AUDIO/WAVE/9780743518802/mp3
~/Desktop/mp3-to-m4b 9780743518802-NEVER CHANGE "NEVER CHANGE" "Elizabeth Berg" 2004
 n
 cd /media/AUDIO/WAVE/9780743518802/mp3
~/Desktop/mp3-to-m4b 9780743518802-NEVER CHANGE "NEVER CHANGE" "Elizabeth Berg" 2004
 n
 cd /media/AUDIO/WAVE/9780743518802/mp3
~/Desktop/mp3-to-m4b 9780743518802-NEVER CHANGE "NEVER CHANGE" "Elizabeth Berg" 2004
 n
 cd /media/AUDIO/WAVE/9780743518802/mp3
~/Desktop/mp3-to-m4b 9780743518802-NEVER CHANGE "NEVER CHANGE" "Elizabeth Berg" 2004
 n
 cd /media/AUDIO/WAVE/9780743518802/mp3
~/Desktop/mp3-to-m4b 9780743518802-NEVER CHANGE "NEVER CHANGE" "Elizabeth Berg" 2004
 n
 cd /media/AUDIO/WAVE/9780743518840/mp3
~/Desktop/mp3-to-m4b 9780743518840-SQUANDERING AIMLESSLY "SQUANDERING AIMLESSLY" "David Brancaccio" 2004
 n
 cd /media/AUDIO/WAVE/9780743518840/mp3
~/Desktop/mp3-to-m4b 9780743518840-SQUANDERING AIMLESSLY "SQUANDERING AIMLESSLY" "David Brancaccio" 2004
 n
 cd /media/AUDIO/WAVE/9780743518840/mp3
~/Desktop/mp3-to-m4b 9780743518840-SQUANDERING AIMLESSLY "SQUANDERING AIMLESSLY" "David Brancaccio" 2004
 n
 cd /media/AUDIO/WAVE/9780743518840/mp3
~/Desktop/mp3-to-m4b 9780743518840-SQUANDERING AIMLESSLY "SQUANDERING AIMLESSLY" "David Brancaccio" 2004
 n

Please help to remove. I colored the repeats to show what I want deleted.
# 2  
Old 02-07-2011
This is the one of the most frequently asked questions Smilie

Code:
awk '!_[$0]++' infile

This User Gave Thanks to radoulov For This Post:
 
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Bash repeating lines for some files but not all

The bash below executes and seems to work fine on those files in which . However on those files where there is no additional CNV detected that line repeats multiple times instead of only once. I tried adding an END as all lines are printed but that doesn't help. I can not seem to solve this... (5 Replies)
Discussion started by: cmccabe
5 Replies

2. Shell Programming and Scripting

Compare last 90 logs and print repeating lines with >20

*log files are in date order sample logs... ciscoresets_20120314 ciscoresets_20120313 ciscoresets_20120312 ciscoresets_20120311 ciscoresets_20120310 cat ciscoresets_20120314 SYDGRE04,10,9 SYDGRE04,10,10 SYDGRE04,10,11 SYDGRE04,10,12 SYDGRE04,10,13 SYDGRE04,10,14 SYDGRE04,10,15... (2 Replies)
Discussion started by: slashbash
2 Replies

3. Shell Programming and Scripting

Printing the lines which are repeating in a files

Hi, I need to find the lines which are repeating in a file cat file1 abcdef 23-1 abcdef 24-1 bcdeff 25-0 ttdcfg 26-0 ttdcfg 20-0 bcdef1 25-0 bcdef2 25-0 bcdef3 25-0 bcdef4 25-0 bcdef4 00-0any help is greatly appreciated. Thanks in advance. In need to find which one are... (3 Replies)
Discussion started by: jpkumar10
3 Replies

4. Shell Programming and Scripting

How to remove a semi-repeating character in position 1 of a file

My file is in a good column format but several lines in the file begin with a zero. I'm in KSH and looking for a command to remove this zero and keep the text next to it. I don't want any of the zeros in the other columns removed. Below is a snip from the file... all I need to do is remove that... (2 Replies)
Discussion started by: right_coaster
2 Replies

5. UNIX for Dummies Questions & Answers

Using sed command to remove multiple instances of repeating headers in one file?

Hi, I have catenated multiple output files (from a monte carlo run) into one big output file. Each individual file has it's own two line header. So when I catenate, there are multiple two line headers (of the same wording) within the big file. How do I use the sed command to search for the... (1 Reply)
Discussion started by: rebazon
1 Replies

6. Shell Programming and Scripting

Merging non-repeating columns of lines

Hello, I have file to work with. It has 5 columns. The first three, altogether, constitutes the position. The 4th column contains some values for downstream analysis and the fifth column contains some values that I want to add to 4th column (only if they happen to be in the same position). My... (5 Replies)
Discussion started by: menenuh
5 Replies

7. Shell Programming and Scripting

Remove repeating pattern from beginning of file names.

I want a shell script that will traverse a file system starting at specific path. And look at all file names for repeating sequences of and remove them from the file name. The portion of the name that gets removed has to be a repeating sequence of the same characters. So the script would... (3 Replies)
Discussion started by: z399y
3 Replies

8. Shell Programming and Scripting

Repeating groups problem

Hi I am trying to locate all strings seperated by the string -CR-, the string will have an unknown number of these -CR- strings, I've used the regex: ^(?:(.*?)-CR-)+$ As shown in the test code: my ($myString)="This is the first line-CR-second line-CR-third line-CR-fourth... (3 Replies)
Discussion started by: joncoop
3 Replies

9. Shell Programming and Scripting

merge 2 files (without repeating any lines)

I need to add the content of file1 to file2 - all lines but not those existing in file2 already, so the "cat file1 >> file2" doesn't work. For example, file1: 100 xxxxxx str1 102 xxxxxx str2 File2: 50 xxxxxxx xxx 30 xxxxxxxxxxx 102 xxxxxx str2 xxxx ...... the result: 50 xxxxxxx... (9 Replies)
Discussion started by: bluemoon1
9 Replies

10. UNIX for Dummies Questions & Answers

Omit repeating lines

Can someone help me with the following 2 objectives? 1) The following command is just an example. It gets a list of all print jobs. From there I am trying to extract the printer name. It works with the following command: lpstat -W "completed" -o | awk -F- '{ print $1}' Problem is, I want... (6 Replies)
Discussion started by: TheCrunge
6 Replies
Login or Register to Ask a Question