sed to delete items in an array from a file Post: 302976035

Sponsored Content

Top Forums Shell Programming and Scripting sed to delete items in an array from a file Post 302976035 by Don Cragun on Wednesday 22nd of June 2016 08:32:17 PM

06-22-2016

Registered User

You say you want to remove useless repeated data, but the code you are using replaces every copy of the data (not just repeated data) with a <space> character (even if the data you want to "remove" is at the start, at the end, or in the middle of a longer string.

Please be very clear about:

whether you want to replace occurrences of the strings you find in a file with a <space> character or want to remove occurrences of those strings,
whether you want to replace or remove all occurrences you find or just want to remove duplicate occurrences,
whether you want to replace or remove occurrences even if they are in the middle of larger "words", only if they are separate "words", or only if they are the complete contents of an input line of text, and
whether you want to remove lines that have been turned into blank lines or empty lines by the changes made above or keep those blank or empty lines in your updated file.

If you want to completely remove complete lines of matched text (as in your example), consider using something more like:

Code:

grep -Fvx -f file_of_lines_to_remove file_to_be_updated > updated_file

to do the entire job in one pass instead of one pass per word "removed". If you aren't removing complete lines, consider using awk to process each input file once instead of using sed to process each input file once per word to be "removed".

This User Gave Thanks to Don Cragun For This Post:

Don Cragun

View Public Profile for Don Cragun

Find all posts by Don Cragun

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

delete line in file with sed

sed "/$titlesearch/d" movielist will delete any line in the file movielist that matches $titlesearch, but this is only a screen print correct ? how can I have sed to actually delete the line from the file so that it doesnt appear next time I open the file ? thanks Steffen

2. Shell Programming and Scripting

Delete particular value from file using 'sed'

Hi, I have two files test1,test2. If the number in test1 file exist in test2 then i want to remove that from test2 file. Ex: File- test1 12 13 14 15 ============== File- test2 1A~12 2B~13 3C~33 4D~22 I want to remove row which contains 12,13 from test2. I am using this sed...

3. Shell Programming and Scripting

awk between items including items

OS=HP-UX ksh The following works, except I want to include the <start> and <end> in the output. awk -F '<start>' 'BEGIN{RS="<end>"; OFS="\n"; ORS=""} {print $2} somefile.log' The following work in bash but not in ksh sed -n '/^<start>/,/^<end>/{/LABEL$/!p}' somefile.log

4. Shell Programming and Scripting

how to delete \n in a large file with sed

Hello i have a big file with a specific format and delimiter is "�" : �field1�$field2�$field3�$field4�$field5�$field6�$field� in this file we have a field which are very long (more than 20000 chars !!!!) so through vi i cant manipulate them. despite this i managed to suppress lines that...

5. UNIX for Dummies Questions & Answers

sed how to delete between two words within a file

I'm hoping someone could help me out please :) I have several .txt files with several hundred lines in each that look like this: 10241;</td><td>10241</td><td class="b">x2801;</td><td>2801</td><td>TEXT-1</td></tr> 10242;</td><td>10242</td><td...

6. Shell Programming and Scripting

sed - delete everything from the first IP in the file

Hi, I have a text file with content as follows: servers list ips 10.10.10.1 test 2 3 4 desktop station 10.10.10.3 4 test 4 laptops quesy 2013 2012 10.100.18.0 test 4 6 8 all I need is: servers list ips desktop station laptops quesy 2013 2012All I need is the correct command or script that...

7. Shell Programming and Scripting

Add items to ksh array

Hi I need a code help put below items in to array and display in order like this Thanks, Ashan 0AD4 R2 03A1 000198701333 0AD4 R1 270B 000198701616 ------ 0AD4 R2 03A1 000198701333 0AD4 R1 270B 000198701616

8. UNIX for Advanced & Expert Users

Sed: couldn't write 1378 items to stdout: No space left on device

Hi , In file first line start with "",when trying to remove using sed i am getting the below error .Please advise sed -e 's///g' -e 's/$]//g' segment1.txt >>segment_m1 sed: couldn't write 1378 items to stdout: No space left on device Thanks, Mohan

9. Shell Programming and Scripting

Removing duplicate items from an array

Hello Experts, I am trying to write a shell script to find duplicate items in an array, this is what i have tried : #!/bin/bash echo "This is another sample Progg to remove duplicate items from an array" echo "How many number do you want to insert" read n for (( i=0; i<$n; i++ )) do ...

10. Shell Programming and Scripting

Pass an array to awk to sequentially look for a list of items in a file

Hello, I need to collect some statistical results from a series of files that are being generated by other software. The files are tab delimited. There are 4 different sets of statistics in each file where there is a line indicating what the statistic set is, followed by 5 lines of values. It...

LEARN ABOUT REDHAT

prunehistory

PRUNEHISTORY(8) 					      System Manager's Manual						   PRUNEHISTORY(8)

NAME

       prunehistory - remove file names from Usenet history file

SYNOPSIS

       prunehistory [ -f filename ] [ -p ]

DESCRIPTION

       Prunehistory  modifies  the  history(5)	text file to ``remove'' a set of filenames from it.  The filenames are removed by overwriting them
       with spaces, so that the size and position of any following entries does not change.

       Prunehistory reads the standard input.  The input is taken as a set of lines.  Blank lines and lines starting with a  number  sign  (``#'')
       are ignored.  All other lines are should consist of a Message-ID followed by zero or more filenames.

       The  Message-ID	is  used as the dbz(3) key to get an offset into the text file.  If no filenames are mentioned on the input line, then all
       filenames in the text are ``removed.''  If any filenames are mentioned, they are converted into the history file notation.  If they  appear
       in the line for the specified Message-ID then they are removed.

       Since innd(8) only appends to the text file, prunehistory does not need to have any interaction with it.

OPTIONS

       -p     Prunehistory will normally complain about lines that do not follow the correct format.  If the ``-p'' flag is used, then the program
	      will silently print any invalid lines on its standard output.  (Blank lines and comment lines are also passed through.)  This can be
	      useful when prunehistory is used as a filter for other programs such as reap.

       -f     The default name of the history file is <pathdb in inn.conf>/history; to specify a different name, use the ``-f'' flag.

HISTORY

       Written by Rich $alz <rsalz@uunet.uu.net> for InterNetNews.  This is revision 1.7, dated 1998/04/09.

SEE ALSO

       dbz(3), history(5), inn.conf(5), innd(8).

																   PRUNEHISTORY(8)

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

delete line in file with sed

Discussion started by: forever_49ers

2. Shell Programming and Scripting

Delete particular value from file using 'sed'

Discussion started by: sai_nj

3. Shell Programming and Scripting

awk between items including items

Discussion started by: Ikon

4. Shell Programming and Scripting

how to delete \n in a large file with sed

Discussion started by: ade05fr