Need help with how to search a file for a variable string and delete that line
Hi,
I have a working script.
It does what I am intending it to but a bit confused whether the sed part is supposed to be working or not. Further down is the script with the sed part that should have been working but not and the grep -v part which is the workaround that I am using at the moment.
I am posting this to get some help on how to use sed to get it to work. I've run out of things to try. OS is Solaris 5.9. Not sure how to check the sed version. I can't use sed -i, it gives error.
list.all is the input file that we want to search and delete line. Below is just an example. The original file is 10K lines long, it contains the absolute file path plus some other commands of sorts in between.
list.err contain the string that we want to search and delete line from list.all. We want to search for the string that is the first field and delete line containing that string.
Code:
/one/one/one/one: No such file or directory
/three/three/three/three: No such file or directory
/five/five/five/five: No such file or directory
/seven/seven/seven/seven: No such file or directory
/nine/nine/nine/nine: No such file or directory
Below is the script that I am using at the moment:
Code:
$: cat fix.ksh
#!/bin/ksh
#
source="list.all"
cp -p ${source}.save ${source}
echo
echo "------------------------------------------"
echo
echo "- This is the list that we want to cleanse"
echo
cat ${source}
echo
echo "------------------------------------------"
echo
echo ""
echo "============================"
echo "- sed not working :("
echo "============================"
echo ""
count=0
while read line
do
let count=$count+1
datafile=`echo $line | awk -F":" '{ print $1 }'`
echo "- $count // Checking $datafile ... "
grep -i "${datafile}" ${source}
sed "#${datafile}#d" ${source} > ${source}.tmp
cp -p ${source}.tmp ${source}
echo ""
done < list.err
echo
echo "==> Not working :("
echo "Contents of ${source} is as below:"
echo
cat ${source}
echo
echo "============================"
echo "- Using grep -v "
echo "============================"
echo
cp -p ${source}.save ${source}
count=0
while read line
do
let count=$count+1
datafile=`echo $line | awk -F":" '{ print $1 }'`
echo "- $count // Checking $datafile ... "
grep -v "${datafile}" ${source} > ${source}.tmp
cp -p ${source}.tmp ${source}
echo ""
done < list.err
echo
echo "==> Required Working Output:"
echo "Contents of ${source} is as below:"
echo
cat ${source}
echo
echo "============================"
echo "- Using grep and sed"
echo "============================"
echo
cp -p ${source}.save ${source}
count=0
while read line
do
let count=$count+1
datafile=`echo $line | awk -F":" '{ print $1 }'`
echo "- $count // Checking $datafile ... "
n=`grep -n "${datafile}" ${source} | awk -F":" '{ print $1 }'`
sed "${n}d" ${source} > ${source}.tmp
cp -p ${source}.tmp ${source}
echo ""
done < list.err
echo
echo "==> Required Working Output:"
echo "Contents of ${source} is as below:"
echo
cat ${source}
Below is an example run of the script:
Code:
$: ./fix.ksh
------------------------------------------
- This is the list that we want to cleanse
'/one/one/one/one',
'/two/two/two/two',
'/three/three/three/three',
'/four/four/four/four',
'/five/five/five/five',
'/six/six/six/six',
'/seven/seven/seven/seven',
'/eight/eight/eight/eight',
'/nine/nine/nine/nine',
'/ten/ten/ten/ten',
------------------------------------------
============================
- sed not working :(
============================
- 1 // Checking /one/one/one/one ...
'/one/one/one/one',
- 2 // Checking /three/three/three/three ...
'/three/three/three/three',
- 3 // Checking /five/five/five/five ...
'/five/five/five/five',
- 4 // Checking /seven/seven/seven/seven ...
'/seven/seven/seven/seven',
- 5 // Checking /nine/nine/nine/nine ...
'/nine/nine/nine/nine',
==> Not working :(
Contents of list.all is as below:
'/one/one/one/one',
'/two/two/two/two',
'/three/three/three/three',
'/four/four/four/four',
'/five/five/five/five',
'/six/six/six/six',
'/seven/seven/seven/seven',
'/eight/eight/eight/eight',
'/nine/nine/nine/nine',
'/ten/ten/ten/ten',
============================
- Using grep -v
============================
- 1 // Checking /one/one/one/one ...
- 2 // Checking /three/three/three/three ...
- 3 // Checking /five/five/five/five ...
- 4 // Checking /seven/seven/seven/seven ...
- 5 // Checking /nine/nine/nine/nine ...
==> Required Working Output:
Contents of list.all is as below:
'/two/two/two/two',
'/four/four/four/four',
'/six/six/six/six',
'/eight/eight/eight/eight',
'/ten/ten/ten/ten',
============================
- Using grep and sed
============================
- 1 // Checking /one/one/one/one ...
- 2 // Checking /three/three/three/three ...
- 3 // Checking /five/five/five/five ...
- 4 // Checking /seven/seven/seven/seven ...
- 5 // Checking /nine/nine/nine/nine ...
==> Required Working Output:
Contents of list.all is as below:
'/two/two/two/two',
'/four/four/four/four',
'/six/six/six/six',
'/eight/eight/eight/eight',
'/ten/ten/ten/ten',
I used
Code:
sed "#${datafile}#d" ${source}
instead of
Code:
sed "/${datafile}/d" ${source}
because the latter gives the error
Code:
First RE may not be null
For the moment, I stick with the grep -v option because the grep -n and sed "${n}d" option doesn't work if there is multiple match of the variable string. I've tried some of the ones posted to the forum that has a similar question and they doesn't work for me. The sed that I have doesn't allow the sed -i option.
You're so close, just off by one character. When you use a character other than / as the delimiter in a BRE search pattern, you have to escape the first use of that character with a backslash (i.e., \). So, if you change:
Code:
sed "#${datafile}#d" ${source} > ${source}.tmp
to:
Code:
sed "\#${datafile}#d" ${source} > ${source}.tmp
it works.
Note, however, that there is no need to use awk to split fields from the lines you read from list.err. There are several things you can do to get the same results without needing to use a command substitution invoking awk (which is a rather costly and slow way of doing what you want to do). One easy way is to let read do the field splitting for you since you're already calling it. For example, if you change:
Code:
while read line
do
let count=$count+1
datafile=`echo $line | awk -F":" '{ print $1 }'`
in all places in your script where it occurs to:
Code:
while IFS=':' read datafile line
do
let count=$count+1
you'll get the same results.
Last edited by Don Cragun; 02-23-2019 at 03:29 AM..
Reason: Add alternative to awk command substitution.
This User Gave Thanks to Don Cragun For This Post:
On top of what Don Cragun already said, your script runs several commands per error list line, which might be considered not too efficient, esp. when dealing with long files. How about condensing the entire script down to a one liner with one grep and one sed only, making use of "process substitution" available in ksh et al.? Like
Thanks Don, I feel so small looking at what's missing, I thought when I tried //, that's it so I change it to a # but never thought about the \.
Hi newbie_01,
There is no reason to feel small. We're all here to learn. Next time you'll remember what you need to do.
Unfortunately you didn't tell us what operating system you're using. With a ksh with a 1988 vintage from many UNIX systems (including the Solaris system we could now guess that you're using), you can't get process substitution (you need a ksh93 for that). This is why it is so important for you to tell us what operating system and shell (including version numbers) you're using each time you start a thread so we know which operating system and shell extensions have a chance of working in your environment.
Even without process substitution, one could use something like:
$: uname -a
SunOS <hostname> 5.8 Generic_Virtual sun4v sparc sun4v
$: strings /bin/ksh | grep -i version
@(#)Version M-11/16/88i
$: what /bin/ksh
/bin/ksh:
Version M-11/16/88i
SunOS 5.8 Generic 110662-26 Mar 2011
Had tried your script and that works superbly. Love the way you use the trap. I think I should start adding that in all of the scripts especially for SIGKILL=9 when we have to do a kill -9.
Had tried your script and that works superbly. Love the way you use the trap. I think I should start adding that in all of the scripts especially for SIGKILL=9 when we have to do a kill -9.
Thanks a lot again.
Hi newbie_01,
I'm glad it is working for you.
Unfortunately, kill -9 can't be caught so a trap won't work in that case. Fortunately, that is the only case that won't work. You can use trap to clean up on normal exit, SIGINT (kill -2 or ctl-c), and SIGQUIT (kill -3 or ctl-\). The SIGINT and SIGQUIT cases are significant since they have keyboard shortcuts as well as kill commands that can be used to generate them. The control C and control \ are the usual defaults on American keyboards, but you can change the characters used to generate both of those signals using the stty utility if you want different keyboard shortcuts.
If you use SIGKILL (kill -9) to terminate a job (any job), you'll have to manually go in and delete any temporary files that job may have created. Always try using SIGINT first to gives jobs a chance to clean up after themselves and use SIGKILL only as a last resort.
This User Gave Thanks to Don Cragun For This Post:
Hi Everyone,
I have a requirement in ksh where i have a set of files in a directory. I need to search each and every file if a particular string is present in the file, delete that line and replace that line with another string expression in the same file.
I am very new to unix. Kindly help... (10 Replies)
Hi,
Need your help for this scripting issue I have. I am not really good at this, so seeking your help.
I have a file looking similar to this:
Hello, i am human and name=ABCD.
How are you?
Hello, i am human and name=PQRS.
I am good.
Hello, i am human and name=ABCD.
Good bye.
Hello, i... (12 Replies)
Dear All
I am having a text file which is having more than 200 lines.
EX:
001010122 12000 BIB 12000 11200 1200003
001010122 2000 AND 12000 11200 1200003
001010122 12000 KVB 12000 11200 1200003
In the above file i want to search for string KVB... (5 Replies)
All,
I am trying to read in a variable and search a file then delete based on that string, but i want to match exact word.
This works but it matches all, i don't want to match anthing that contains the string, just the exact string.
sed -i "/$feedname/d" file
I tried
sed... (1 Reply)
Hi , I am breaking my head from past one day ...to delete lines from a file which match to the string pattern.:wall:
I am storing the search string in a variable and search if the file exists in the folder,if not delete that entry from the file.
I am having problem to delete that line from... (2 Replies)
let me start out by saying i have ZERO exp with any kind of scripting, so sorry if this is really basic stuff.....
For example, I need to have a script that will search a file and find this line in the file:
*.cat;dog;kennel;house;barn;horse;hay;coat hat
and remove the "coat" from the... (12 Replies)
Hi
I want to read a file line by line and search for a particular string in each line(say for example string containing @ )and save that string into a variable.
Can someone suggest me the way to implement it.I am using K- shell
Thanks
Ishita (5 Replies)
Hi All,
I have a file from Mainframe which has one of the lines with so many words... i tried to fold, format to 80 chararcter.. stil did not work. So i have decided to search for a string in that line Ex.FLIGHT PLAN and once if it is found i want to delete the entire line.
Please help... (2 Replies)
Hi, guys. I have one question:
I need to search for a string in a file, and then extract another string from the file and assign it to a variable.
For example:
the contents of the file (group) is below:
...
ftp:x:23:
mail:x:34
...
testing:x:2001
sales:x:2002
development:x:2003
...... (6 Replies)
02-23-2019
