Login or Register to Ask a Question and Join Our Community

Sponsored Content
Full Discussion: How to remove repetitive lines in a file with sed?
Top Forums Shell Programming and Scripting How to remove repetitive lines in a file with sed? Post 303022141 by RudiC on Sunday 26th of August 2018 07:26:55 AM
Old 08-26-2018
You wanted sed? Try this; it works on your sample file, but I'm unsure how it would deal with above mentioned large file:
Code:
sed -rn '1h; 1!H; s/$/./; G; s/^(([^.]*)\.\n(.*\2.*))\2$/\3/; s/^[^.]*\.\n//; s/\n\n/\n/; h;  $p' file
Hello World
Welcome
Welcome Back
This is my test

 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Unix Remove repetitive alphabets

Hi, I am trying to write a script that will take 2 or more instances of repetitive alphabets (ZZ) to be removed from a field. This should only happen from beginning and end of a field. For Example : Input File a) ZZZIBM Corporation b) ZZZIBM Corporation ZZZZZ b) IBM ZZZ... (26 Replies)
Discussion started by: msalam65
26 Replies

2. Shell Programming and Scripting

using sed to remove lines

Can somebody explain why my sed command is not working. I do the folloinwg: Generates a binary file to /tmp/x1.out /usr/lib/sa/sa2 -s 4:00 -e 8:00 -i 3600 -A -o /tmp/x1.out decodes the file (no problem so far) sar -f /tmp/x1.out When I do this it does not appear to delete the... (4 Replies)
Discussion started by: BeefStu
4 Replies

3. Shell Programming and Scripting

grep/sed to remove lines in file

Hi, I have a file with values, file1: BELL-1180-1180-81|577:1017| BELL-1180-1180-81|jm10i-auto-stub1/577:102| BELL-1180-1180-81|jm10i-auto-stub1/577:101| BELL-1180-1180-81|jm10i-auto-stub1/577:1700| BELL-1180-1180-81|jm10i-auto-stub1/577:1699| I need to remove the lines which has... (9 Replies)
Discussion started by: giri_luck
9 Replies

4. Shell Programming and Scripting

Print lines between two repetitive patterns

Hi users I have one file which has number of occurrence of one pattern examples Adjustmenttype,11 xyz 10 dwe 9 abd 13 def 14 Adjustmenttype,11 xyz 24 dwe 34 abd 35 def 11 nmb 12 Adjustmenttype, not eleven .... ... ... (2 Replies)
Discussion started by: eranmoh
2 Replies

5. Shell Programming and Scripting

Remove a range of lines from a file using sed

Hi I am having some issue editing a file in sed. What I want to do is, in a loop pass a variable to a sed command. Sed should then search a file for a line that matches that variable, then remove all lines below until it reaches a line starting with a constant. I have managed to write a... (14 Replies)
Discussion started by: Andy82
14 Replies

6. UNIX for Dummies Questions & Answers

How to remove certain lines using sed?

Hi I have the following kind of line sin my file . print ' this is first'. print ' this is firs and next ' ' line continuous '. -- this is entire print line. print ' this is first and next ' ' line continuous and' 'still there now over'. -- this 3lines together a single print line. ... (5 Replies)
Discussion started by: Sivajee
5 Replies

7. Shell Programming and Scripting

How to remove certain lines using sed?

Hi, I am new to unix and i started some scripting recently. Please go through the following script i wrote. #!/bin/sh file='path../tfile' file1='path../tfile1' rmfile='path../test2' C1=1 C2=1 exec 3< $file1 while read LINE1; do read LINE2 <&3 a=$LINE1 b=`expr $LINE2 - 1` ... (1 Reply)
Discussion started by: Subbu123
1 Replies

8. Shell Programming and Scripting

Using sed in a loop/to remove lines contained in variable from file

I've tried numerous commands, but I am not sure how to use sed in a loop. This is what I have: VARZ contains CARD_FILE_LIST and it also contains CARD_FILE_LIST2 so echo "$VARZ" CARD_FILE_LIST CARD_FILE_LIST2 I have a file with 60 lines in /tmp/testfile it and I want those lines deleted... (3 Replies)
Discussion started by: newbie2010
3 Replies

9. Shell Programming and Scripting

I need help to removing repetitive lines

Hello SuperUsers! First i wanna say my english sucks.. Don't hate me for that. :rolleyes: I need to make a Bash for removing smilar lines from an output file. My output file always same. Line 1 & 2 Stays. And others similar to this lines needs to be delete. </UsageData><?xml version="1.0"... (4 Replies)
Discussion started by: morphin
4 Replies

10. Shell Programming and Scripting

sed to remove all lines in file that are not .vcf.gz extention

I am trying to use sed to remove all lines in a file that are nor vcf.gz. The sed below runs but returns all the files with vcf.gz in them, rather then just the ones that end in only that extention. Thank you :). file ... (9 Replies)
Discussion started by: cmccabe
9 Replies

LEARN ABOUT SUSE

pom2

POM2(1) 						User Contributed Perl Documentation						   POM2(1)

NAME

       pom2 - convert POD to Text, HTML, etc., with Pod::POM

SYNOPSIS

	   pom2 text MyFile.pm > MyFile.txt
	   pom2 html MyFile.pm > MyFile.html
	   pom2 pod  MyFile.pm > Myfile.pod

	   pom2 format [options] MyFile.pm > Myfile.xyz

DESCRIPTION

       This script uses "Pod::POM" to convert a Pod document into text, HTML, back into Pod (e.g. to normalise a document to fix any markup
       errors), or any other format for which you have a view module.

       If the viewer is not one of the viewers bundled with "Pod::POM", the script searches for an installed "Pod::POM::View" module that matches
       the specified format.  For example if you have "Pod::POM::View::DocBook" installed then you could say:

	   pod2 docbook MyFile.pm > MyFile.xml

       If any options other than "--help" are specified then they are passed to the constructor method of the view class.  For example:

	   pod2 docbook --root=chapter --titlecasing MyFile.pm > MyFile.xml

       would convert the Pod document to a DocBook chapter document with the titlecasing option enabled.  Note that any string prefixed by ""--""
       is taken as a valid option and passed to the constructor; if no value is specified then a value of 1 is passed in.

AUTHOR

       Andy Wardley <abw@kfs.org>

       extended by Andrew Ford <A.Ford@ford-mason.co.uk>

VERSION

       This is version 0.3 of pom2.

COPYRIGHT

       Copyright (C) 2000, 2001 Andy Wardley.  All Rights Reserved.

       Copyright (C) 2009 Andrew Ford.	All Rights Reserved.

       This module is free software; you can redistribute it and/or modify it under the same terms as Perl itself.

SEE ALSO

       For further information please see Pod::POM.

perl v5.12.1							    2009-03-19								   POM2(1)
All times are GMT -4. The time now is 08:50 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy