Need help with how to search a file for a variable string and delete that line


Login or Register for Dates, Times and to Reply

 
Thread Tools Search this Thread
Top Forums UNIX for Beginners Questions & Answers Need help with how to search a file for a variable string and delete that line
# 1  
Need help with how to search a file for a variable string and delete that line

Hi,

I have a working script.

It does what I am intending it to but a bit confused whether the sed part is supposed to be working or not. Further down is the script with the sed part that should have been working but not and the grep -v part which is the workaround that I am using at the moment.

I am posting this to get some help on how to use sed to get it to work. I've run out of things to try. OS is Solaris 5.9. Not sure how to check the sed version. I can't use sed -i, it gives error.

list.all is the input file that we want to search and delete line. Below is just an example. The original file is 10K lines long, it contains the absolute file path plus some other commands of sorts in between.

Code:
  
  '/one/one/one/one',
  '/two/two/two/two',
  '/three/three/three/three',
  '/four/four/four/four',
  '/five/five/five/five',
  '/six/six/six/six',
  '/seven/seven/seven/seven',
  '/eight/eight/eight/eight',
  '/nine/nine/nine/nine',
  '/ten/ten/ten/ten',

list.err contain the string that we want to search and delete line from list.all. We want to search for the string that is the first field and delete line containing that string.

Code:
/one/one/one/one: No such file or directory
/three/three/three/three: No such file or directory
/five/five/five/five: No such file or directory
/seven/seven/seven/seven: No such file or directory
/nine/nine/nine/nine: No such file or directory

Below is the script that I am using at the moment:

Code:
$: cat fix.ksh
#!/bin/ksh
#

source="list.all"
cp -p ${source}.save ${source}

echo
echo "------------------------------------------"
echo
echo "- This is the list that we want to cleanse"
echo
cat ${source}
echo
echo "------------------------------------------"
echo

echo ""
echo "============================"
echo "- sed not working :("
echo "============================"
echo ""
count=0
while read line
do
   let count=$count+1
   datafile=`echo $line | awk -F":" '{ print $1 }'`
   echo "- $count // Checking $datafile ... "
   grep -i "${datafile}" ${source}
   sed "#${datafile}#d" ${source} > ${source}.tmp
   cp -p ${source}.tmp ${source}
   echo ""
done < list.err
echo
echo "==> Not working :("
echo "Contents of ${source} is as below:"
echo
cat ${source}

echo
echo "============================"
echo "- Using grep -v "
echo "============================"
echo
cp -p ${source}.save ${source}
count=0
while read line
do
   let count=$count+1
   datafile=`echo $line | awk -F":" '{ print $1 }'`
   echo "- $count // Checking $datafile ... "
   grep -v "${datafile}" ${source} > ${source}.tmp
   cp -p ${source}.tmp ${source}
   echo ""
done < list.err
echo
echo "==> Required Working Output:"
echo "Contents of ${source} is as below:"
echo
cat ${source}

echo
echo "============================"
echo "- Using grep and sed"
echo "============================"
echo
cp -p ${source}.save ${source}
count=0
while read line
do
   let count=$count+1
   datafile=`echo $line | awk -F":" '{ print $1 }'`
   echo "- $count // Checking $datafile ... "
   n=`grep -n "${datafile}" ${source} | awk -F":" '{ print $1 }'`
   sed "${n}d" ${source} > ${source}.tmp
   cp -p ${source}.tmp ${source}
   echo ""
done < list.err
echo
echo "==> Required Working Output:"
echo "Contents of ${source} is as below:"
echo
cat ${source}

Below is an example run of the script:

Code:
$: ./fix.ksh

------------------------------------------

- This is the list that we want to cleanse

  '/one/one/one/one',
  '/two/two/two/two',
  '/three/three/three/three',
  '/four/four/four/four',
  '/five/five/five/five',
  '/six/six/six/six',
  '/seven/seven/seven/seven',
  '/eight/eight/eight/eight',
  '/nine/nine/nine/nine',
  '/ten/ten/ten/ten',

------------------------------------------


============================
- sed not working :(
============================

- 1 // Checking /one/one/one/one ...
  '/one/one/one/one',

- 2 // Checking /three/three/three/three ...
  '/three/three/three/three',

- 3 // Checking /five/five/five/five ...
  '/five/five/five/five',

- 4 // Checking /seven/seven/seven/seven ...
  '/seven/seven/seven/seven',

- 5 // Checking /nine/nine/nine/nine ...
  '/nine/nine/nine/nine',


==> Not working :(
Contents of list.all is as below:

  '/one/one/one/one',
  '/two/two/two/two',
  '/three/three/three/three',
  '/four/four/four/four',
  '/five/five/five/five',
  '/six/six/six/six',
  '/seven/seven/seven/seven',
  '/eight/eight/eight/eight',
  '/nine/nine/nine/nine',
  '/ten/ten/ten/ten',

============================
- Using grep -v
============================

- 1 // Checking /one/one/one/one ...

- 2 // Checking /three/three/three/three ...

- 3 // Checking /five/five/five/five ...

- 4 // Checking /seven/seven/seven/seven ...

- 5 // Checking /nine/nine/nine/nine ...


==> Required Working Output:
Contents of list.all is as below:

  '/two/two/two/two',
  '/four/four/four/four',
  '/six/six/six/six',
  '/eight/eight/eight/eight',
  '/ten/ten/ten/ten',

============================
- Using grep and sed
============================

- 1 // Checking /one/one/one/one ...

- 2 // Checking /three/three/three/three ...

- 3 // Checking /five/five/five/five ...

- 4 // Checking /seven/seven/seven/seven ...

- 5 // Checking /nine/nine/nine/nine ...


==> Required Working Output:
Contents of list.all is as below:

  '/two/two/two/two',
  '/four/four/four/four',
  '/six/six/six/six',
  '/eight/eight/eight/eight',
  '/ten/ten/ten/ten',

I used
Code:
sed "#${datafile}#d" ${source}

instead of
Code:
sed "/${datafile}/d" ${source}

because the latter gives the error
Code:
First RE may not be null

For the moment, I stick with the grep -v option because the grep -n and sed "${n}d" option doesn't work if there is multiple match of the variable string. I've tried some of the ones posted to the forum that has a similar question and they doesn't work for me. The sed that I have doesn't allow the sed -i option.

Any advise much appreciated. Thanks in advance.
# 2  
You're so close, just off by one character. When you use a character other than / as the delimiter in a BRE search pattern, you have to escape the first use of that character with a backslash (i.e., \). So, if you change:
Code:
   sed "#${datafile}#d" ${source} > ${source}.tmp

to:
Code:
   sed "\#${datafile}#d" ${source} > ${source}.tmp

it works.

Note, however, that there is no need to use awk to split fields from the lines you read from list.err. There are several things you can do to get the same results without needing to use a command substitution invoking awk (which is a rather costly and slow way of doing what you want to do). One easy way is to let read do the field splitting for you since you're already calling it. For example, if you change:
Code:
while read line
do
   let count=$count+1
   datafile=`echo $line | awk -F":" '{ print $1 }'`

in all places in your script where it occurs to:
Code:
while IFS=':' read datafile line
do
   let count=$count+1

you'll get the same results.

Last edited by Don Cragun; 02-23-2019 at 04:29 AM.. Reason: Add alternative to awk command substitution.
This User Gave Thanks to Don Cragun For This Post:
# 3  
On top of what Don Cragun already said, your script runs several commands per error list line, which might be considered not too efficient, esp. when dealing with long files. How about condensing the entire script down to a one liner with one grep and one sed only, making use of "process substitution" available in ksh et al.? Like
Code:
grep -vf<(sed -n 's/:.*$//; 1h;1!H; ${g;p}' list.err) list.all
'/two/two/two/two',
'/four/four/four/four',
'/six/six/six/six',
'/eight/eight/eight/eight',
'/ten/ten/ten/ten',


Last edited by RudiC; 02-23-2019 at 07:47 AM..
# 4  
Thanks Don, I feel so small looking at what's missing, I thought when I tried //, that's it so I change it to a # but never thought about the \.

--- Post updated at 10:22 PM ---

Hi RudiC,

I like the one-liner one but it gives me error as below:

Using
Code:
grep -vf<(sed -n 's/:.*$//; 1h;1!H; ${g;p}' list.err) list.all

Gives the following error:

Code:
sed: command garbled: s/:.*$//; 1h;1!H; ${g;p}
grep: illegal option -- f
Usage: grep -hblcnsviw pattern file . . .

Using
Code:
/usr/xpg4/bin/grep -vf<(sed -n 's/:.*$//; 1h;1!H; ${g;p}' list.err) list.all

or
Code:
/usr/xpg4/bin/grep -vf<(/usr/xpg4/bin/sed -n 's/:.*$//; 1h;1!H; ${g;p}' list.err) list.all

Gives the following error:

Code:
sed: command garbled: s/:.*$//; 1h;1!H; ${g;p}

--- Post updated at 10:27 PM ---

Hi RudiC

FYI, the one-liner as expected does work on Linux :-) but on the Solaris one. Smilie
# 5  
Quote:
Originally Posted by newbie_01
Thanks Don, I feel so small looking at what's missing, I thought when I tried //, that's it so I change it to a # but never thought about the \.
Hi newbie_01,
There is no reason to feel small. We're all here to learn. Next time you'll remember what you need to do.

Unfortunately you didn't tell us what operating system you're using. With a ksh with a 1988 vintage from many UNIX systems (including the Solaris system we could now guess that you're using), you can't get process substitution (you need a ksh93 for that). This is why it is so important for you to tell us what operating system and shell (including version numbers) you're using each time you start a thread so we know which operating system and shell extensions have a chance of working in your environment.

Even without process substitution, one could use something like:
Code:
#!/bin/ksh
IAm=${0##*/}
TmpFile=$IAm.$$
trap 'rm -rf "$TmpFile"' 0

sed 's/:.*$//' list.err > "$TmpFile"
/usr/xpg4/bin/grep -vf "$TmpFile" list.all

It isn't a one-liner, but it just invokes grep once, sed once, and rm once at the end to remove the temporary file it creates.

This should work on both Solaris and Linux systems (but you'll need to remove the /usr/xpg4/bin/ on the Linux system).
This User Gave Thanks to Don Cragun For This Post:
# 6  
Hi Don

OS Version and ksh version below:

Code:
$: uname -a
SunOS <hostname> 5.8 Generic_Virtual sun4v sparc sun4v
$: strings /bin/ksh | grep -i version
@(#)Version M-11/16/88i
$: what /bin/ksh
/bin/ksh:
        Version M-11/16/88i
        SunOS 5.8 Generic 110662-26 Mar 2011

Had tried your script and that works superbly. Love the way you use the trap. I think I should start adding that in all of the scripts especially for SIGKILL=9 when we have to do a kill -9.

Thanks a lot again.
# 7  
Quote:
Originally Posted by newbie_01
Hi Don

... ... ...

Had tried your script and that works superbly. Love the way you use the trap. I think I should start adding that in all of the scripts especially for SIGKILL=9 when we have to do a kill -9.

Thanks a lot again.
Hi newbie_01,
I'm glad it is working for you.

Unfortunately, kill -9 can't be caught so a trap won't work in that case. Fortunately, that is the only case that won't work. You can use trap to clean up on normal exit, SIGINT (kill -2 or ctl-c), and SIGQUIT (kill -3 or ctl-\). The SIGINT and SIGQUIT cases are significant since they have keyboard shortcuts as well as kill commands that can be used to generate them. The control C and control \ are the usual defaults on American keyboards, but you can change the characters used to generate both of those signals using the stty utility if you want different keyboard shortcuts.

If you use SIGKILL (kill -9) to terminate a job (any job), you'll have to manually go in and delete any temporary files that job may have created. Always try using SIGINT first to gives jobs a chance to clean up after themselves and use SIGKILL only as a last resort.
This User Gave Thanks to Don Cragun For This Post:
Login or Register for Dates, Times and to Reply

Previous Thread | Next Thread
Thread Tools Search this Thread
Search this Thread:
Advanced Search

Test Your Knowledge in Computers #183
Difficulty: Easy
Over 40% of UNIX.com users voted that bash was their most useful shell.
True or False?

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Search for a string,delete the line and replace with new string in a file

Hi Everyone, I have a requirement in ksh where i have a set of files in a directory. I need to search each and every file if a particular string is present in the file, delete that line and replace that line with another string expression in the same file. I am very new to unix. Kindly help... (10 Replies)
Discussion started by: Pradhikshan
10 Replies

2. Shell Programming and Scripting

Search string within a file and list common words from the line having the search string

Hi, Need your help for this scripting issue I have. I am not really good at this, so seeking your help. I have a file looking similar to this: Hello, i am human and name=ABCD. How are you? Hello, i am human and name=PQRS. I am good. Hello, i am human and name=ABCD. Good bye. Hello, i... (12 Replies)
Discussion started by: royzlife
12 Replies

3. Shell Programming and Scripting

Search a string in a text file and add another string at the end of line

Dear All I am having a text file which is having more than 200 lines. EX: 001010122 12000 BIB 12000 11200 1200003 001010122 2000 AND 12000 11200 1200003 001010122 12000 KVB 12000 11200 1200003 In the above file i want to search for string KVB... (5 Replies)
Discussion started by: suryanarayana
5 Replies

4. Shell Programming and Scripting

Sed find exact string and delete line with variable

All, I am trying to read in a variable and search a file then delete based on that string, but i want to match exact word. This works but it matches all, i don't want to match anthing that contains the string, just the exact string. sed -i "/$feedname/d" file I tried sed... (1 Reply)
Discussion started by: markdjones82
1 Replies

5. UNIX for Dummies Questions & Answers

search for a string and delete it from the file

Hi , I am breaking my head from past one day ...to delete lines from a file which match to the string pattern.:wall: I am storing the search string in a variable and search if the file exists in the folder,if not delete that entry from the file. I am having problem to delete that line from... (2 Replies)
Discussion started by: rashmisb
2 Replies

6. Shell Programming and Scripting

Grep a string from input file and delete next three lines including the line contains string in xml

Hi, 1_strings file contains $ cat 1_strings /home/$USER/Src /home/Valid /home/Review$ cat myxml <projected value="some string" path="/home/$USER/Src"> <input 1/> <estimate value/> <somestring/> </projected> <few more lines > <projected value="some string" path="/home/$USER/check">... (4 Replies)
Discussion started by: greet_sed
4 Replies

7. Shell Programming and Scripting

Search for a line, delete a string in it

let me start out by saying i have ZERO exp with any kind of scripting, so sorry if this is really basic stuff..... For example, I need to have a script that will search a file and find this line in the file: *.cat;dog;kennel;house;barn;horse;hay;coat hat and remove the "coat" from the... (12 Replies)
Discussion started by: skunky
12 Replies

8. Shell Programming and Scripting

search a string in a line and save it in a variable

Hi I want to read a file line by line and search for a particular string in each line(say for example string containing @ )and save that string into a variable. Can someone suggest me the way to implement it.I am using K- shell Thanks Ishita (5 Replies)
Discussion started by: Ishita
5 Replies

9. Shell Programming and Scripting

search string and delete the line

Hi All, I have a file from Mainframe which has one of the lines with so many words... i tried to fold, format to 80 chararcter.. stil did not work. So i have decided to search for a string in that line Ex.FLIGHT PLAN and once if it is found i want to delete the entire line. Please help... (2 Replies)
Discussion started by: digitalrg
2 Replies

10. Shell Programming and Scripting

Search for string in a file and extract another string to a variable

Hi, guys. I have one question: I need to search for a string in a file, and then extract another string from the file and assign it to a variable. For example: the contents of the file (group) is below: ... ftp:x:23: mail:x:34 ... testing:x:2001 sales:x:2002 development:x:2003 ...... (6 Replies)
Discussion started by: daikeyang
6 Replies

Featured Tech Videos