UNIX for Dummies Questions & Answers

I've got a sed shell script I've been using for quite a while to do batch find/replace operations on .xml files containing various types of text entries. This is the script I use:

Code:

#!/bin/bash
while read text_old text_new; do
        sed_script+="s/$text_old/$text_new/g;"
done < trans_old_to_new
find . -type f -name \*.xml -exec sed -i.BAK "$sed_script" '{}' +

Accompanying the shell script is a text or xml file with a simple 2-column structure. The left column contains the text to find, the right column contains the text to replace it with, and the two columns are separated with a space.

Code:

00-11-22-33-44-55 AA-BB-CC-DD-EE-FF
11-22-33-44-55-66 BB-CC-DD-EE-FF-GG

Is there a way to modify this shell script or the format of the 2-column replacement matrix to accommodate for text strings containing spaces? For example:

Code:

Replace this text containing spaces SEPARATOR With this text containing spaces

Change the delimter in file trans_old_to_new to any character which is not present in your search or replacement string

Code:

#!/bin/bash
IFS="|" # Set IFS to delimiter used in trans_old_to_new
while read text_old text_new; do
        sed_script+="s/$text_old/$text_new/g;"
done < trans_old_to_new
find . -type f -name \*.xml -exec sed -i.BAK "$sed_script" '{}' +

Excellent, thank you. I've set it up to use a pipe and it works for plain text entries.
One thing I should have included in my original post was that text_old and text_new may contain forward- and/or back-slashes for paths, so text_old_to_new might contain something like this:

Code:

\some\old\path|\a\new\path

and the actual xml file might contain a line similar to this:

Code:

<Result path="S:\output\series\some\old\path\2014-10-01.txt" /><Source value="S:\input\fieldset\list.txt />

The pink text is what needs to be changed, but is not. I tried escaping the slashes in the trans_old_to_new file, like so:

Code:

\\some\\old\\path|\\a\\new\\path

but then I get the following error when running the script:

Code:

sed: -e expression #1, char 146: unterminated `s' command

Is there some different way of escaping the slashes?

This problem is related but different than the first one: the first problem (spaces) was that the character you wanted to use was special in some say to the shell - the solution was to move that special meaning around to another character (pipe instead of space) by redefining the "IFS" (internal field separator) char. Note that - once you might need the pipe char in your expressions - you can move the special meaning somewhere else by using even another character you will not need otherwise at the moment to fill that role.

Back to your problem at hand: the backslash (and many other) character(s) have an equally special meaning to sed as the space char has to the shell, but in this case you cannot shift these around. All these are used in "regular expressions" and are called "metacharacters". Here is a list of the most oftenly used ones: ".*&\[{]}/"

The backslash is used to strip their special meaning from these characters and make them ordinary characters again. For example the the dot "." is used to denote one character, regardless of which it is. The sequence "\." instead means a single dot char. The same way "\x" means a literal "x" (even though "x" would mean that anyway) and "\\" means a literal backslash. The easiest way therefore is to just escape any character on the list prior to using it in an expression. The only character we leave out is the "|" which will be consumed by the shell anyway:

Code:

#! /bin/bash
set -o lastpipe

local sed_script=""
local text_old=""
local text_new=""

sed 's/[^|]/\\&/g' /path/to/trans_old_to_new |\
while IFS="|" read text_old text_new junk ; do
     sed_script+="s/${text_old}/${text_old}/g;"
done

find . -type f -name \*.xml -exec sed -i.BAK "$sed_script" {} +

Notice, that i used a pipeline and set "lastpipe" for this so that "$sed_script" is not local to the while-loop. This is only possible for bash upwards of some version i do not know for sure (IIRC 4.2), but i find this way of writing a loop fed by some datastream easier to read.

I also introduced a third variable "junk" into the read-statement to filter out unwanted artefacts that might be on some input lines.

I hope this helps.

bakunin

Or use a plain-text substitution.
This is not possible with sed, but with perl and with awk:

Code:

#!/bin/bash
find . -type f -name \*.xml |
while IFS="" read -r targetfile
do
 cp -p "$targetfile" "$targetfile".BAK &&
 awk -F"|" '
NR==FNR {map[$1]=$2; next} {for (m in map) {for (li=0; (i=index($0,m)) && i>li; li=i) {$0=substr($0,1,i-1) map[m] substr($0,i+length(m))}} print}
' trans_old_to_new "$targetfile".BAK > "$targetfile"
done

Bakunin, that script throws a bunch of errors:

Code:

/path/to/translate.sh: line 2: set: lastpipe: invalid option name
/path/to/translate.sh: line 4: local: can only be used in a function
/path/to/translate.sh: line 5: local: can only be used in a function
/path/to/translate.sh: line 6: local: can only be used in a function

Also, I think in your script you may have duplicated the column reference here:

Code:

sed_script+="s/${text_old}/${text_old}/g;"

Just to be clear, there is no file named text_old nor text_new. There are 3 files in total:
1. the xml file containing the text to be replaced
2. the translate.sh script containing the sed command
3. the trans_old_to_new file, containing two columns: old text and new text.

Sorry, copied that off from the terminal without double-checking: "text_old" should be "text_new" of course, and "local" is an alias in my environment - use "typeset" instead.

This, i am afraid, i can do nothing about: as i told you only relatively recent bash-versions know this option, yours might be not recent enough. As i explained, i used it only to avoid the ugly bottom-steered while-loop, which you seem to have to use in this case.

I hope this helps.

bakunin