Using sed to edit multiple files


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Using sed to edit multiple files
# 1  
Old 01-04-2019
Using sed to edit multiple files

Created a shell script to invoke sed to edit multiple files, but am missing something.
Here's the shell script:
Code:
oracle:$ cat edit_scripts.sh
#!/bin/sh
#------------------------------------------------------------------------------
# edit_scripts.sh
#
# This script executes sed to make global edits to the template files
#------------------------------------------------------------------------------
for i in *.sh; do
  sed -f script.sed < $i
done
#
for i in *.sql; do
  sed -f script.sed < $i
done

As you can see, I'm using a script file 'script.sed' to supply the actual editing commands:
Code:
oracle:$ cat script.sed
s/@yymmdd@/181016/g
s/@gi patch num@/12345678/g
s/@jvm patch num@/87654321/g

And for testing purposes, I have one .sh script to be edited:
Code:
oracle:$ ll *.sh
-rwxr-x--- 1 oracle oinstall 359 Jan  3 14:30 edit_scripts.sh
-rw-r--r-- 1 oracle oinstall 163 Jan  3 14:46 sed_sample.sh

oracle:$ cat sed_sample.sh
#-------------------
#  This should be the date: @yymmdd@
#  This should be the gi patch num:  @gi patch num@
#  This should be the jvm patch num: @jvm patch num@

But when I execute edit_scripts.sh, sed_sample.sh is not modified:
Code:
oracle:$ ./edit_scripts.sh
#!/bin/sh
#------------------------------------------------------------------------------
# edit_scripts.sh
#
# This script executes sed to make global edits to the template files
#------------------------------------------------------------------------------
for i in *.sh; do
  sed -f script.sed < $i
done
#
for i in *.sql; do
  sed -f script.sed < $i
done
#-------------------
#  This should be the date: 181016
#  This should be the gi patch num:  12345678
#  This should be the jvm patch num: 87654321
./edit_scripts.sh: line 12: *.sql: No such file or directory

oracle:$ cat sed_sample.sh
#-------------------
#  This should be the date: @yymmdd@
#  This should be the gi patch num:  @gi patch num@
#  This should be the jvm patch num: @jvm patch num@

# 2  
Old 01-04-2019
Depending on your OS...
either:
Code:
sed -i -f script.sed $i

OR
Code:
{ rm $I; sed -f script.sed > $i; } < $i

OR
other multiple other ways...
This User Gave Thanks to vgersh99 For This Post:
# 3  
Old 01-04-2019
Quote:
Originally Posted by edstevens
Created a shell script to invoke sed to edit multiple files, but am missing something.
Here's the shell script:
Code:
oracle:$ cat edit_scripts.sh
#!/bin/sh
#------------------------------------------------------------------------------
# edit_scripts.sh
#
# This script executes sed to make global edits to the template files
#------------------------------------------------------------------------------
for i in *.sh; do
  sed -f script.sed < $i
done
#
for i in *.sql; do
  sed -f script.sed < $i
done

The problem is in this line:

Code:
sed -f script.sed < $i

First, sed doesn't need a redirection because it can work on files. Notice that i quoted "$i" because otherwise files with blanks in their names would break your script. It is generally a good idea to religiously quote whatever is quotable in shell scripts - it just comes with the impulse to convert the infidels, like i am doing to you right now. ;-)

Code:
sed -f script.sed  "$i"

Second, sed puts it output to <stdout> as per default. Probably it did do the changes but they were shown on your screen, not your file. To make the changes lasting you need to redirect this output into a (new!) file and only then move that file over the old one. As an additional measure of caution only do so if sed returned 0. This guards against i.e. disk-full-conditions where the output cannot be written (fully) to the disk and you move then a corrupt script over a perfectly fine template:

Code:
if sed -f script.sed  "$i" > "$i".tmp ; then
     mv "$i".tmp "$i"
fi

I hope this helps.

bakunin
# 4  
Old 01-04-2019
The first example worked for me, even though the man page for sed on my system does not list the '-i' switch. My system is running Oracle Linux, which is derived from RHEL.

Thanks for the assistance.
# 5  
Old 01-04-2019
Quote:
Originally Posted by edstevens
The first example worked for me, even though the man page for sed on my system does not list the '-i' switch. My system is running Oracle Linux, which is derived from RHEL.
It does so for a reason because it is not advisable to use the "-i" switch at all, even in the few versions of sed not adhering to the standards which have it, because the the standards also do NOT have the switch at all.

The reason is that "behind the curtain" the "-i" switch does the same as i said: creating a temporary file and then copying it over the original. It does NOT really do "in-place-editing" as the documentation says, it does in fact the same every script programmer does. This in itself is not bad but: if you do it yourself you have control over where the temporary file goes and that it is removed afterwards. With the "-i" switch you don't have this control and in case of an error it may well be that it is left stuck somewhere and such a leftover chunk of junk is filling up valuable disk space.

Furthermore it changes file permissions and ownership, which a real in-place-editing wouldn't do. Consider the following example:

Code:
$ id
uid=1000(edstevens) gid=1000(edstevens) groups=1000(edstevens),999(users)

$ ls -l
total 189960
-rw-rw-r-- 1 bakunin users       90 Nov 13 15:16 script.original

That means you can edit script.original because you are in the group "users" which has write privilege.

Now run this - and guess, what:
Code:
$ sed -i '...<whatever, it doesn't matter>...' script.original

$ ls -l
total 189960
-rw-rw-r-- 1 edstevens edstevens       99 Jan 04 18:00 script.original

Did you notice? The ownership is changed! Now, that happens when you create a temporary file too, but there you are aware that you are dealing with a new file. Here you have no reason to suspect because you have been (wrongly) told it will be the same file plus your changes. It isn't. It will have a new inode number, it will be located somewhere else on the disk physically, etc.. This is why you should stay away from this switch. Have i mentioned that it doesn't conform to the standard either? If you want to create portable script which will not work somewhere but mysteriously not so somewhere else you better adhere to the standards slavishly. They define what you can rightfully expect from a system. If you expect more than chances are you will be disappointed sooner or later.

I hope this helps.

bakunin
These 3 Users Gave Thanks to bakunin For This Post:
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

sed parser behaving strange on replacing multiple words in multiple files

I have 4000 files like $cat clus_grp_seq10_g.phy 18 1002 anig_OJJ65951_1 ATGGTTTCGCAGCGTGATAGAGAATTGTTTAGGGATGATATTCGCTCGCGAGGAACGAAGCTCAATGCTGCCGAGCGCGAGAGTCTGCTAAGGCCATATCTGCCAGATCCGTCTGACCTTCCACGCAGGCCACTTCAGCGGCGCAAGAAGGTTCCTCG aver_OOF92921_1 ... (1 Reply)
Discussion started by: sammy777888
1 Replies

2. Shell Programming and Scripting

Help require to edit multiple files

I have 6 different pipe delimiter files. My loads failing due to missing company code. File1: 31 st field is company code. 402660076310|2014-12-10 17:22:39|2280361|MRYKI|1||CA|92507|US||1|1|0|0|0||N|A1|ONT|1001891771660009250700402660076310|WM|0201|RALA |2014-12-12|5|2014-12-12|5||FRI - 12... (4 Replies)
Discussion started by: srikanth38
4 Replies

3. Shell Programming and Scripting

SED command using multiple input files

What is the syntax to use multiple input files in a SED command. i.e. substitute a word with a phrase in every file in a directory. for every file in /usr/include that has the word "date" in the file grep -l '\<date\>' /usr/include/*.h find each occurrence of the word "time" in the file &... (3 Replies)
Discussion started by: sheoguey
3 Replies

4. UNIX for Dummies Questions & Answers

best method of replacing multiple strings in multiple files - sed or awk? most simple preferred :)

Hi guys, say I have a few files in a directory (58 text files or somthing) each one contains mulitple strings that I wish to replace with other strings so in these 58 files I'm looking for say the following strings: JAM (replace with BUTTER) BREAD (replace with CRACKER) SCOOP (replace... (19 Replies)
Discussion started by: rich@ardz
19 Replies

5. Shell Programming and Scripting

Read and edit multiple files using a while loop

Hi all, I would like to simply read a file which lists a number of pathnames and files, then search and replace key strings using a few vi commands: :1,$s/search_str/replace_str/g<return> but I am not sure how to automate the <return> of these vis commands when I am putting this in a... (8 Replies)
Discussion started by: cyberfrog
8 Replies

6. UNIX for Dummies Questions & Answers

SED on multiple files

Hello all, Search & replace works fine using sed on a single file. Ex: sed -i 's/day/night/g' test1.sh There are many *.sh files in my current directory that I would like use sed on. I tried running the sed command using wild card but it did not work. sed -i 's/day/night/g' *.sh ... (7 Replies)
Discussion started by: luft
7 Replies

7. UNIX for Dummies Questions & Answers

sed on multiple files?

Hi, I want to do a search and replace on multiple text files. Can I use sed to do this? i.e. I want to do something like: $ sed *.html -e 's/<\/body>/<\!-- blah -->\n<\/body>/' | grep -1 body ... then pipe/ channel the results back into the same files that were searched. how would... (2 Replies)
Discussion started by: mgrahamnz
2 Replies

8. Shell Programming and Scripting

how to use multiple files in sed with w command

i have a command like : sed -n 's/^* /&/w even' <file if i want to write to multiple files like sed -n 's/^* /&/w zero two three' < file its not working it is taking "zero two three" as a single file i want to write to 3 seperate files . pls can anyone help me (2 Replies)
Discussion started by: santosh1234
2 Replies

9. UNIX for Dummies Questions & Answers

Edit Multiple Files in VI

Here's what I have... $ vi foo1 - open foo1 and work around for a while. I yank a few lines into a buffer and then :w to save. Next I :e foo2 to open foo2 and paste my buffer. I :w to save, but I would like to then be able to go directly back into foo1 where I was before I opened foo2. ... (4 Replies)
Discussion started by: djschmitt
4 Replies

10. UNIX for Dummies Questions & Answers

sed across multiple files

I've got a bunch of files (40 or so) and in each there is a substitution I need to perform. I execuet the following sed command but it just make the changes to the screen without affecting the original file. sed "s/, LA/,LA/g" * (All files in the directory need this update). So I tried... (4 Replies)
Discussion started by: peter.herlihy
4 Replies
Login or Register to Ask a Question