Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Remove duplicate consecutive lines with specific string

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Remove duplicate consecutive lines with specific string
# 1  
Old 03-21-2017
Remove duplicate consecutive lines with specific string
Hello,

I'm trying to remove the duplicate consecutive lines with specific string "WARNING".

File.txt
Code:
abc;
WARNING 2345
WARNING 2345
WARNING 2345
WARNING 2345
WARNING 2345
bcd;
abc;
123
123
123
WARNING 1234
WARNING 2345
WARNING 2345
efgh;
1234
1234
12
134
WARNING 2345
WARNING 1234
WARNING 1234

Output should be something like this -
out.txt

Code:
abc;
WARNING 2345
bcd;
abc;
123
123
123
WARNING 1234
WARNING 2345
efgh;
1234
1234
12
134
WARNING 2345
WARNING 1234

I have tried sed but didn't accomplish what I'm trying to achieve -

Code:
sed -r 'N; /(WARNING)[^\n]*\n\1/ s/\n.*//; P; D' File.txt
abc;
WARNING 2345
WARNING 2345
WARNING 2345
bcd;
abc;
123
123
123
WARNING 1234
WARNING 2345
efgh;
1234
1234
12
134
WARNING 2345
WARNING 1234

Please provide your suggestions to do that using sed or any other method.

Thank you.
Last edited by Mannu2525; 03-21-2017 at 04:47 AM.. Reason: changed the file name in sed command
# 2  
Old 03-21-2017
Hello Mannu2525,

Could you please try following and let me know if this helps you.
Code:
awk 'NR==1{print;last=$0;next} ($0 != last){print}  {last=$0}'   Input_file

EDIT: Adding one more solution on same too now.
Code:
awk '{printf("%s",NR==1?$0 RS:(($0 != last)?$0 RS:""));last=$0}'

EDIT2: If you want to remove sequential only duplicates of lines which have string WARNING then following may help you in same too(with GNU awk).
Code:
awk '/WARNING/ && !a[$0]++{print;next} !/WARNING/{delete a;print}'   Input_file

Thanks,
R. Singh
Last edited by RavinderSingh13; 03-21-2017 at 06:06 AM.. Reason: Adding one more solution on same successfully too now.
# 3  
Old 03-21-2017
Able to do it using
Code:
sed '/WARNING/N;/^\(.*\)\n\1$/!P; D' File.txt

This User Gave Thanks to Mannu2525 For This Post:
vbe
# 4  
Old 03-21-2017
Thanks Mannu, we appreciate when members let us know they have solved or found a solution and even more when showing us how, for the good of all the community
# 5  
Old 03-21-2017
Another awk solution
Code:
awk '{if($1=="WARNING" && $0==dup){next}} {print;dup=$0}' file

# 6  
Old 03-21-2017
Code:
awk '$0!=dup; {dup=_} /WARNING/{dup=$0}' file

This User Gave Thanks to RudiC For This Post:
andy391791
# 7  
Old 07-26-2017
Quote:
Originally Posted by Mannu2525
Able to do it using
Code:
sed '/WARNING/N;/^\(.*\)\n\1$/!P; D' File.txt

This is really tricky.
With a Unix sed you also have the strange behavior that N exits without a default print, if it's on the last line (i.e. fails); the usual work-around is $!N.
Also many Unix sed need { and } (and lables) on separate lines or separate -e arguments.
Code:
sed -e '/^WARNING/{' -e '$!N;/^\(.*\)\n\1$/!P;D' -e '}' File.txt

Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Check/print missing number in a consecutive range and remove duplicate numbers

Hi, In an ideal scenario, I will have a listing of db transaction log that gets copied to a DR site and if I have them all, they will be numbered consecutively like below. 1_79811_01234567.arc 1_79812_01234567.arc 1_79813_01234567.arc 1_79814_01234567.arc 1_79815_01234567.arc... (3 Replies)
Discussion started by: newbie_01
3 Replies

2. Shell Programming and Scripting

Grep three consecutive lines if each lines contains certain string

say we have : 2914 | REQUEST | whatever 2914 | RESPONSE | whatever 2914 | SUCCESS | whatever 2985 | RESPONSE | whatever 2986 | REQUEST | whatever 2990 | REQUEST | whatever 2985 | RESPONSE | whatever 2996 | REQUEST | whatever 2010 | SUCCESS | whatever 2013 | REQUEST | whatever 2013 |... (7 Replies)
Discussion started by: Saumitra Pandey
7 Replies

3. Shell Programming and Scripting

Remove not only the duplicate string but also the keyword of the string in Perl

Hi Perl users, I have another problem with text processing in Perl. I have a file below: Linux Unix Linux Windows SUN MACOS SUN SUN HP-AUX I want the result below: Unix Windows SUN MACOS HP-AUX so the duplicate string will be removed and also the keyword of the string on... (2 Replies)
Discussion started by: askari
2 Replies

4. UNIX for Dummies Questions & Answers

Remove Duplicate Lines

Hi I need this output. Thanks. Input: TAZ YET FOO FOO VAK TAZ BAR Output: YET VAK BAR (10 Replies)
Discussion started by: tara123
10 Replies

5. Shell Programming and Scripting

Grep couple of consecutive lines if each lines contains certain string

Hello, I want to extract from a file like : 20120530025502914 | REQUEST | whatever 20120530025502968 | RESPONSE | whatever 20120530025502985 | RESPONSE | whatever 20120530025502996 | REQUEST | whatever 20120530025503013 | REQUEST | whatever 20120530025503045 | RESPONSE | whatever I want... (14 Replies)
Discussion started by: black_fender
14 Replies

6. Shell Programming and Scripting

remove consecutive duplicate rows

I have some data that looks like, 1 3300665.mol 3300665 5177008 102.093 2 3300665.mol 3300665 5177008 102.093 3 3294015.mol 3294015 5131552 102.114 4 3294015.mol 3294015 5131552 102.114 5 3293734.mol 3293734 5129625 104.152 6 3293734.mol ... (13 Replies)
Discussion started by: LMHmedchem
13 Replies

7. Shell Programming and Scripting

remove html tags,consecutive duplicate lines

I need help with a script that will remove all HTML tags from an HTML document and remove any consecutive duplicate lines, and save it as a text document. The user should have the option of including the name of an html file as an argument for the script, but if none is provided, then the script... (7 Replies)
Discussion started by: clicstic
7 Replies

8. Shell Programming and Scripting

Merge two non-consecutive lines based on line number or string

This is a variation of an earlier post found here: unixcom/shell-programming-scripting/159821-merge-two-non-consecutive-lines.html User Bartus11 was kind enough to solve that example. Previously, I needed help combining two lines that are non-consecutive in a file. Now I need to do the... (7 Replies)
Discussion started by: munkee
7 Replies

9. Shell Programming and Scripting

Remove duplicate lines

Hi, I have a huge file which is about 50GB. There are many lines. The file format likes 21 rs885550 0 9887804 C C T C C C C C C C 21 rs210498 0 9928860 0 0 C C 0 0 0 0 0 0 21 rs303304 0 9941889 A A A A A A A A A A 22 rs303304 0 9941890 0 A A A A A A A A A The question is that there are a few... (4 Replies)
Discussion started by: zhshqzyc
4 Replies

10. Shell Programming and Scripting

merging of 2 consecutive lines in a file for a specific pattern

Hi , I'm looking for a way to merge two lines only for a given pattern / condition. Input : abcd/dad + -49.201 2.09 -49.5 34 ewrew rewtre * fdsgfds/dsgf/sdfdsfasdd + -4.30 0.62 -49.5 45 sdfdsf cvbbv * sdfds/retret/asdsaddsa + ... (1 Reply)
Discussion started by: novice_man
1 Replies
Login or Register to Ask a Question