Remove duplicate occurrences of text pattern

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Remove comments like pattern from text

Hi , We need to remove comment like pattern from a code text. The possible comment expressions are as follows. Input BizComment : Special/*@ Name:bzt_53_3aea640a_51783afa_5d64_0 BizHidden:true @*/ /* lookup Disease Category Therapuetic Class */ a=b;...

2. Shell Programming and Scripting

How to remove duplicate text blocks from a file?

Hi All I have a list of files which will have duplicate list of blocks of text. Following is a sample of the file, I have removed the sensitive information from the file. All the code samples starts from <TR BGCOLOR="white"> and Ends with IP address and two html tags like this. 10.14.22.22...

3. Windows & DOS: Issues & Discussions

Remove duplicate lines from text files.

So, I have text files, one "fail.txt" And one "color.txt" I now want to use a command line (DOS) to remove ANY line that is PRESENT IN BOTH from each text file. Afterwards there shall be no duplicate lines.

4. Shell Programming and Scripting

Remove duplicate line starting with a pattern

HI, I have the below input file /* ----------------- cmdsDlyStartFWJ -----------------*/ UNIX_JOB CMDS065J RUN ANY CMDNAME sleep 5 AGENT CMDSHP USER proddata RUN MON,TUE,WED,THU,FRI DELAYSUB 02:00 /* "Triggers daily file watcher jobs" */ ENVAR...

5. Shell Programming and Scripting

Filter or remove duplicate block of text without distinguishing marks or fields

Hello, Although I have found similar questions, I could not find advice that could help with our problem. The issue: We have several hundreds text files containing repeated blocks of text (I guess back at the time they were prepared like that to optmize printing). The block of texts...

6. Shell Programming and Scripting

Help with remove last text of a file that have specific pattern

Input file matrix-remodelling_associated_8_ aurora_interacting_1_ L20 von_factor_A_domain_1 ATP_containing_3B_ . . Output file matrix-remodelling_associated_8 aurora_interacting_1 L20 von_factor_A_domain_1 ATP_containing_3B . .

7. Shell Programming and Scripting

How to remove all text except pattern

i have nasty html file with 2000+ simbols in 1 row...i need to remove whole the code except title="Some title..." and store those into file with titles (the whole text is in variable text) i've tried something like this: echo $text | sed 's/.*$title=\".+\"$.*/\1/' > titles.html BUT it does...

8. Shell Programming and Scripting

Count the number of occurrences of a pattern between each occurrence of a different pattern

I need to count the number of occurrences of a pattern, say 'key', between each occurrence of a different pattern, say 'lu'. Here's a portion of the text I'm trying to parse: lu S1234L_149_m1_vg.6, part-att 1, vdp-att 1 p-reserver IID 0xdb registrations: key 4156 4353 0000 0000 ...

9. Shell Programming and Scripting

Remove duplicate files based on text string?

Hi I have been struggling with a script for removing duplicate messages from a shared mailbox. I would like to search for duplicate messages based on the “Message-ID” string within the messages files. I have managed to find the duplicate “Message-ID” strings and (if I would like) delete...

10. Shell Programming and Scripting

Remove duplicate text

Hello, I have a log file which is generated by a script which looks like this: userid: 7 starttime: Sat May 24 23:24:13 CEST 2008 endtime: Sat May 24 23:26:57 CEST 2008 total time spent: 2.73072 minutes / 163.843 seconds date: Sat Jun 7 16:09:03 CEST 2008 userid: 8 starttime: Sun May...

LEARN ABOUT DEBIAN

plan9-grep

GREP(1) 						      General Commands Manual							   GREP(1)

NAME

       grep, g - search a file for a pattern

SYNOPSIS

       grep [ option ...  ] pattern [ file ...	]

       g [ option ...  ] pattern [ file ...  ]

DESCRIPTION

       Grep  searches the input files (standard input default) for lines that match the pattern, a regular expression as defined in regexp(7) with
       the addition of a newline character as an alternative (substitute for |) with lowest precedence.  Normally, each line matching the  pattern
       is `selected', and each selected line is copied to the standard output.	The options are

       -c     Print only a count of matching lines.
       -h     Do not print file name tags (headers) with output lines.
       -e     The  following  argument	is taken as a pattern.	This option makes it easy to specify patterns that might confuse argument parsing,
	      such as -n.
       -i     Ignore alphabetic case distinctions.  The implementation folds into lower case all letters in the pattern and input before interpre-
	      tation.  Matched lines are printed in their original form.
       -l     (ell) Print the names of files with selected lines; don't print the lines.
       -L     Print the names of files with no selected lines; the converse of -l.
       -n     Mark each printed line with its line number counted in its file.
       -s     Produce no output, but return status.
       -v     Reverse: print lines that do not match the pattern.
       -f     The pattern argument is the name of a file containing regular expressions one per line.
       -b     Don't buffer the output: write each output line as soon as it is discovered.

       Output  lines  are  tagged  by  file name when there is more than one input file.  (To force this tagging, include /dev/null as a file name
       argument.)

       Care should be taken when using the shell metacharacters $*[^|()= and newline in pattern; it is safest to enclose the entire expression in
       single quotes '...'.  An expression starting with '*' will treat the rest of the expression as literal characters.

       G invokes grep with -n and forces tagging of output lines by file name.	If no files are listed, it searches all files matching

	      *.C *.b *.c *.h *.m *.cc *.java *.cgi *.pl *.py *.tex *.ms

SOURCE

       /src/cmd/grep
       /bin/g

SEE ALSO

       ed(1), awk(1), sed(1), sam(1), regexp(7)

DIAGNOSTICS

       Exit status is null if any lines are selected, or non-null when no lines are selected or an error occurs.

																	   GREP(1)

Shell Programming and Scripting