Using sed in a loop/to remove lines contained in variable from file


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Using sed in a loop/to remove lines contained in variable from file
# 1  
Old 12-04-2014
Using sed in a loop/to remove lines contained in variable from file

I've tried numerous commands, but I am not sure how to use sed in a loop. This is what I have:

VARZ contains CARD_FILE_LIST and it also contains CARD_FILE_LIST2
so
echo "$VARZ"
CARD_FILE_LIST
CARD_FILE_LIST2

I have a file with 60 lines in /tmp/testfile it and I want those lines deleted from the file. Note that VARZ may not always contain CARD_FILE_LIST or CARD_FILE_LIST2. It may contain another line. I want the command to delete whatever values are contained by VARZ preferably with sed or awk.

I have tried:

Code:
for i in `cat /tmp/testfile`;do echo $i |sed "s/$VARZ//g"`;done

Code:
sed "/$VARZ/d" /tmp/testfile

Code:
sed '/'"${VARX}"'/d /tmp/testfile

I have the old version of sed so I can't use sed -i to make changes but obviously I could output it to another file and rename that one if I need to. I just want to get the command/loop working.

Any help is appreciated.
# 2  
Old 12-04-2014
That won't work, because the substitute command in sed cannot take multiple lines..

If you are using bash and your OS supports it, try something like:
Code:
grep -vFf <(echo "$VARZ") /tmp/testfile

Otherwise you could try:
Try
Code:
echo "$VARZ" | grep -vfF - /tmp/testfile

Note that partial matches will also remove the entry, so if you do not want that it will need to be further refined..

Last edited by Scrutinizer; 12-04-2014 at 03:49 PM..
# 3  
Old 12-04-2014
Running sed 60 times to remove 60 lines is like making 60 telephone calls to say 60 words.
This User Gave Thanks to Corona688 For This Post:
# 4  
Old 12-04-2014
It might be good to turn the short file into a script of piped together sed's or whatever, so one pas on the long file is all that is needed. This can be done in a file or env var, using eval.

Really, all that is needed is a generated sed script for one sed, unless you need more speed. 'sed' is not a shell command, but a different script interpreter.
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

How to remove repetitive lines in a file with sed?

Hello, My goal is the make all x times repeated lines into a single line. I need to attain the expected output with sed -i , I need to overwrite the MyFile MyFile: Hello World Welcome Hello World Welcome Back This is my test Expected output: Hello World Welcome Welcome Back This is... (6 Replies)
Discussion started by: baris35
6 Replies

2. Shell Programming and Scripting

sed to remove all lines in file that are not .vcf.gz extention

I am trying to use sed to remove all lines in a file that are nor vcf.gz. The sed below runs but returns all the files with vcf.gz in them, rather then just the ones that end in only that extention. Thank you :). file ... (9 Replies)
Discussion started by: cmccabe
9 Replies

3. Shell Programming and Scripting

ksh Loop through file one, remove lines from file two

Good Afternoon, I start with a file named biglist.txt. I have another file smallerlist. txt I want to remove the lines from smallerlist.txt from biglist.txt and leave those lines that do not reside in smallerlist.txt. Thanks !! (2 Replies)
Discussion started by: popeye
2 Replies

4. Shell Programming and Scripting

How to remove certain lines using sed?

Hi, I am new to unix and i started some scripting recently. Please go through the following script i wrote. #!/bin/sh file='path../tfile' file1='path../tfile1' rmfile='path../test2' C1=1 C2=1 exec 3< $file1 while read LINE1; do read LINE2 <&3 a=$LINE1 b=`expr $LINE2 - 1` ... (1 Reply)
Discussion started by: Subbu123
1 Replies

5. UNIX for Dummies Questions & Answers

How to remove certain lines using sed?

Hi I have the following kind of line sin my file . print ' this is first'. print ' this is firs and next ' ' line continuous '. -- this is entire print line. print ' this is first and next ' ' line continuous and' 'still there now over'. -- this 3lines together a single print line. ... (5 Replies)
Discussion started by: Sivajee
5 Replies

6. Shell Programming and Scripting

Remove a range of lines from a file using sed

Hi I am having some issue editing a file in sed. What I want to do is, in a loop pass a variable to a sed command. Sed should then search a file for a line that matches that variable, then remove all lines below until it reaches a line starting with a constant. I have managed to write a... (14 Replies)
Discussion started by: Andy82
14 Replies

7. Shell Programming and Scripting

grep/sed to remove lines in file

Hi, I have a file with values, file1: BELL-1180-1180-81|577:1017| BELL-1180-1180-81|jm10i-auto-stub1/577:102| BELL-1180-1180-81|jm10i-auto-stub1/577:101| BELL-1180-1180-81|jm10i-auto-stub1/577:1700| BELL-1180-1180-81|jm10i-auto-stub1/577:1699| I need to remove the lines which has... (9 Replies)
Discussion started by: giri_luck
9 Replies

8. Shell Programming and Scripting

Having sed to read lines of a file with the use of a variable..Possible?

This i will print the second line of a file sed -n '2p' test2 The use of a variable is impossible here. a=1 while ; do line=`sed -n '$a p' test2` # do some things here with the line variable a=`expr $a + 1` done But the uotput of sed command is 'p'..... What can i do to use a variable... (2 Replies)
Discussion started by: hakermania
2 Replies

9. Shell Programming and Scripting

using sed to remove lines

Can somebody explain why my sed command is not working. I do the folloinwg: Generates a binary file to /tmp/x1.out /usr/lib/sa/sa2 -s 4:00 -e 8:00 -i 3600 -A -o /tmp/x1.out decodes the file (no problem so far) sar -f /tmp/x1.out When I do this it does not appear to delete the... (4 Replies)
Discussion started by: BeefStu
4 Replies

10. UNIX for Advanced & Expert Users

Redirect contained in variable not expanded as expected

Hi all, I have a selection of files that contain several commands that write to a file and are started as background processes similar to below. command_to_run_simulator -simulator_arguments > log_file 2>&1 & command_to_run_simulator -simulator_arguments > log_file 2>&1 &... (2 Replies)
Discussion started by: noose
2 Replies
Login or Register to Ask a Question