sed to delete items in an array from a file


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting sed to delete items in an array from a file
# 1  
Old 06-22-2016
sed to delete items in an array from a file

I need to create a shell script to delete multiple items (Strings) at a time from a file.

I need to iterate through a list of strings.

My plan is to create an array and then iterate through the array.
My code is not working
Code:
#!/bin/bash -x


declare -a array=(one, two, three, four)

for element in "${array[@]}"
do
    sed "s/"$element"//g" input
    #awk "{gsub("$element", "");print}" input
done

the file input contains
one
two
three
four
# 2  
Old 06-22-2016
Why do you want an array?
Code:
for element in one two three four
do
...

# 3  
Old 06-22-2016
input file contains
one
two
three
four

I moved away from the array to an input file. The script hangs as shown here.
Code:
#!/bin/bash -x


for i in `cat input`
#while true
do
    sed -e 's/'"$i"'/'" "'/g'
#    echo $i
done > output.remove

shell$ ./remove-these.sh

++ cat input
+ for i in '`cat input`'
+ sed -e 's/one/ /g'

Code:
#!/bin/bash -x

for i in `cat input`
do
    sed -e 's/'"$i"'/'" "'/g' input > output.remove
#done < input > outfile1
done

# 4  
Old 06-22-2016
Where are the items to delete,
and what is the file you want to delete the items from?

If your items are from the file "input" then it does not make sense to delete the items from the file "input". It will become empty. Or do you want to prove just that?
# 5  
Old 06-22-2016
The items to delete are in the input file. This script is just to prove the loop works. input is the file I want to delete from. I could use sed edit in place or not. Some internet posts said the sed -i option was not working in a loop. I did not verify.

I have an 18MB data file with a lot of redundant repeated lines that need to be deleted. Having an outside file containing a list of those lines to be deleted is best for this script. I want to read into the loop the source file and output to a second file. The source file has about 15 MB of repeated useless data to remove so looping over the file seems the best choice. I dont really care what construct I use but I am not having luck with those I have tested.
# 6  
Old 06-22-2016
You say you want to remove useless repeated data, but the code you are using replaces every copy of the data (not just repeated data) with a <space> character (even if the data you want to "remove" is at the start, at the end, or in the middle of a longer string.

Please be very clear about:
  1. whether you want to replace occurrences of the strings you find in a file with a <space> character or want to remove occurrences of those strings,
  2. whether you want to replace or remove all occurrences you find or just want to remove duplicate occurrences,
  3. whether you want to replace or remove occurrences even if they are in the middle of larger "words", only if they are separate "words", or only if they are the complete contents of an input line of text, and
  4. whether you want to remove lines that have been turned into blank lines or empty lines by the changes made above or keep those blank or empty lines in your updated file.

If you want to completely remove complete lines of matched text (as in your example), consider using something more like:
Code:
grep -Fvx -f file_of_lines_to_remove file_to_be_updated > updated_file

to do the entire job in one pass instead of one pass per word "removed". If you aren't removing complete lines, consider using awk to process each input file once instead of using sed to process each input file once per word to be "removed".
This User Gave Thanks to Don Cragun For This Post:
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Pass an array to awk to sequentially look for a list of items in a file

Hello, I need to collect some statistical results from a series of files that are being generated by other software. The files are tab delimited. There are 4 different sets of statistics in each file where there is a line indicating what the statistic set is, followed by 5 lines of values. It... (8 Replies)
Discussion started by: LMHmedchem
8 Replies

2. Shell Programming and Scripting

Removing duplicate items from an array

Hello Experts, I am trying to write a shell script to find duplicate items in an array, this is what i have tried : #!/bin/bash echo "This is another sample Progg to remove duplicate items from an array" echo "How many number do you want to insert" read n for (( i=0; i<$n; i++ )) do ... (5 Replies)
Discussion started by: mukulverma2408
5 Replies

3. UNIX for Advanced & Expert Users

Sed: couldn't write 1378 items to stdout: No space left on device

Hi , In file first line start with "",when trying to remove using sed i am getting the below error .Please advise sed -e 's///g' -e 's/$]//g' segment1.txt >>segment_m1 sed: couldn't write 1378 items to stdout: No space left on device Thanks, Mohan (1 Reply)
Discussion started by: mohan705
1 Replies

4. Shell Programming and Scripting

Add items to ksh array

Hi I need a code help put below items in to array and display in order like this Thanks, Ashan 0AD4 R2 03A1 000198701333 0AD4 R1 270B 000198701616 ------ 0AD4 R2 03A1 000198701333 0AD4 R1 270B 000198701616 (1 Reply)
Discussion started by: ashanabey
1 Replies

5. Shell Programming and Scripting

sed - delete everything from the first IP in the file

Hi, I have a text file with content as follows: servers list ips 10.10.10.1 test 2 3 4 desktop station 10.10.10.3 4 test 4 laptops quesy 2013 2012 10.100.18.0 test 4 6 8 all I need is: servers list ips desktop station laptops quesy 2013 2012All I need is the correct command or script that... (4 Replies)
Discussion started by: e_mikey_2000
4 Replies

6. UNIX for Dummies Questions & Answers

sed how to delete between two words within a file

I'm hoping someone could help me out please :) I have several .txt files with several hundred lines in each that look like this: 10241;</td><td>10241</td><td class="b">x2801;</td><td>2801</td><td>TEXT-1</td></tr> 10242;</td><td>10242</td><td... (4 Replies)
Discussion started by: martinsmith
4 Replies

7. Shell Programming and Scripting

how to delete \n in a large file with sed

Hello i have a big file with a specific format and delimiter is "§" : §field1§$field2§$field3§$field4§$field5§$field6§$field§ in this file we have a field which are very long (more than 20000 chars !!!!) so through vi i cant manipulate them. despite this i managed to suppress lines that... (11 Replies)
Discussion started by: ade05fr
11 Replies

8. Shell Programming and Scripting

awk between items including items

OS=HP-UX ksh The following works, except I want to include the <start> and <end> in the output. awk -F '<start>' 'BEGIN{RS="<end>"; OFS="\n"; ORS=""} {print $2} somefile.log' The following work in bash but not in ksh sed -n '/^<start>/,/^<end>/{/LABEL$/!p}' somefile.log (4 Replies)
Discussion started by: Ikon
4 Replies

9. Shell Programming and Scripting

Delete particular value from file using 'sed'

Hi, I have two files test1,test2. If the number in test1 file exist in test2 then i want to remove that from test2 file. Ex: File- test1 12 13 14 15 ============== File- test2 1A~12 2B~13 3C~33 4D~22 I want to remove row which contains 12,13 from test2. I am using this sed... (2 Replies)
Discussion started by: sai_nj
2 Replies

10. Shell Programming and Scripting

delete line in file with sed

sed "/$titlesearch/d" movielist will delete any line in the file movielist that matches $titlesearch, but this is only a screen print correct ? how can I have sed to actually delete the line from the file so that it doesnt appear next time I open the file ? thanks Steffen (8 Replies)
Discussion started by: forever_49ers
8 Replies
Login or Register to Ask a Question