Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

sed replacing specific characters and control characters by escaping

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting sed replacing specific characters and control characters by escaping
# 1  
Old 04-27-2012
sed replacing specific characters and control characters by escaping
Code:
sed -e "s/[\.\/\%'=:\(\)\`_\"\@\;\+\*\#\$\?\!]/ /g" old.txt > new.txt

While I do know some control characters need to be escaped, can normal characters also be escaped and still work the same way? Basically I do not know all control characters that have a special meaning, for example, ?, ., % have a meaning and have to be escaped to be replaced.

Does the above sed work as expected to replace all of the below characters with a space?
Code:
 ,
  .
  @
  /
  #
  :
  (
  )
  '
  *
  %
  $
  +
  ?
  _
  =
  "
  !
  ;

Last edited by Franklin52; 04-27-2012 at 04:45 AM.. Reason: Please use code tags
# 2  
Old 04-27-2012
The easiest way to find out is to test it.

Code:
echo "@" | sed ...

# 3  
Old 04-27-2012
But the concern is if it removed other characters or did something you did not expect to, there was no way to test this Smilie

---------- Post updated at 06:42 AM ---------- Previous update was at 06:30 AM ----------

ok thanks, may be that is true this regexp is correct and we can excape normal and control characters both to be safe Smilie

Code:
echo "abc- cde fg.hi" | sed -e "s/[\.\/\%'=:\(\)\`_\"\@\;\+\*\#\$\?\!]/ /g"
echo "abc- cde fg / hi" | sed -e "s/[\.\/\%'=:\(\)\`_\"\@\;\+\*\#\$\?\!]/ /g"
echo "abc- cde fg/hi" | sed -e "s/[\.\/\%'=:\(\)\`_\"\@\;\+\*\#\$\?\!]/ /g"
echo "abc- cde fg \ hi" | sed -e "s/[\.\/\%'=:\(\)\`_\"\@\;\+\*\#\$\?\!]/ /g"
echo "abc- cde fg\hi" | sed -e "s/[\.\/\%'=:\(\)\`_\"\@\;\+\*\#\$\?\!]/ /g"
echo "abc- cde fg'hi" | sed -e "s/[\.\/\%'=:\(\)\`_\"\@\;\+\*\#\$\?\!]/ /g"
echo "abc- cde fg ' hi" | sed -e "s/[\.\/\%'=:\(\)\`_\"\@\;\+\*\#\$\?\!]/ /g"
echo "abc- cde fg % hi" | sed -e "s/[\.\/\%'=:\(\)\`_\"\@\;\+\*\#\$\?\!]/ /g"
echo "abc- cde fg % hi" | sed -e "s/[\.\/\%'=:\(\)\`_\"\@\;\+\*\#\$\?\!]/ /g"
echo "abc- cde fg = hi" | sed -e "s/[\.\/\%'=:\(\)\`_\"\@\;\+\*\#\$\?\!]/ /g"
echo "abc- cde fg _ hi" | sed -e "s/[\.\/\%'=:\(\)\`_\"\@\;\+\*\#\$\?\!]/ /g"
echo "abc- cde fg_hi" | sed -e "s/[\.\/\%'=:\(\)\`_\"\@\;\+\*\#\$\?\!]/ /g"
echo "abc- cde fg , hi" | sed -e "s/[\.\/\%'=:\(\)\`_\"\@\;\+\*\#\$\?\!]/ /g"
echo "abc- cde fg,hi" | sed -e "s/[\.\/\%'=:\(\)\`_\"\@\;\+\*\#\$\?\!]/ /g"
echo "abc- cde fg : hi" | sed -e "s/[\.\/\%'=:\(\)\`_\"\@\;\+\*\#\$\?\!]/ /g"
echo "abc- cde fg:hi" | sed -e "s/[\.\/\%'=:\(\)\`_\"\@\;\+\*\#\$\?\!]/ /g"
echo "abc- cde fg ( hi" | sed -e "s/[\.\/\%'=:\(\)\`_\"\@\;\+\*\#\$\?\!]/ /g"
echo "abc- cde fg(hi" | sed -e "s/[\.\/\%'=:\(\)\`_\"\@\;\+\*\#\$\?\!]/ /g"
echo "abc- cde fg ) hi" | sed -e "s/[\.\/\%'=:\(\)\`_\"\@\;\+\*\#\$\?\!]/ /g"
echo "abc- cde fg ? hi" | sed -e "s/[\.\/\%'=:\(\)\`_\"\@\;\+\*\#\$\?\!]/ /g"
echo "abc- cde fg @ hi" | sed -e "s/[\.\/\%'=:\(\)\`_\"\@\;\+\*\#\$\?\!]/ /g"
echo "abc- cde fg@hi" | sed -e "s/[\.\/\%'=:\(\)\`_\"\@\;\+\*\#\$\?\!]/ /g"
echo "abc- cde fg + hi" | sed -e "s/[\.\/\%'=:\(\)\`_\"\@\;\+\*\#\$\?\!]/ /g"
echo "abc- cde fg+hi" | sed -e "s/[\.\/\%'=:\(\)\`_\"\@\;\+\*\#\$\?\!]/ /g"
echo "abc- cde fg * hi" | sed -e "s/[\.\/\%'=:\(\)\`_\"\@\;\+\*\#\$\?\!]/ /g"
echo "abc- cde fg*hi" | sed -e "s/[\.\/\%'=:\(\)\`_\"\@\;\+\*\#\$\?\!]/ /g"
echo "abc- cde fg $ hi" | sed -e "s/[\.\/\%'=:\(\)\`_\"\@\;\+\*\#\$\?\!]/ /g"
echo "abc- cde fg$hi" | sed -e "s/[\.\/\%'=:\(\)\`_\"\@\;\+\*\#\$\?\!]/ /g"
echo "abc- cde fg ! hi" | sed -e "s/[\.\/\%'=:\(\)\`_\"\@\;\+\*\#\$\?\!]/ /g"
echo "abc- cde fg!hi" | sed -e "s/[\.\/\%'=:\(\)\`_\"\@\;\+\*\#\$\?\!]/ /g"


Moderator's Comments:
Mod Comment Welcome to the UNIX and Linux Forums. Please use [code][/code] tags. Video tutorial on how to use them
Last edited by Scrutinizer; 04-27-2012 at 11:49 AM..
# 4  
Old 04-27-2012
Hi why don't you try it yourself? But it would be preferable to use single quotes instead of double quotes, to prevent the shell from nibbling away at the back slashes in front of some characters ( $ ` " \ and <newline> ), so sed will not see them.

Code:
echo "\\\`\$\ 
\""

produces:
Code:
\`$"

Also by putting \ in front of some of the normal characters, you are introducing a special meaning:

Code:
$ echo ntrv | sed -n "/\n\t\r\v/p"
$ echo ntrv | sed -n "/ntrv/p"
ntrv

# 5  
Old 05-02-2012
Single quotes seem to fail in following cases:
Code:
echo "abc- cde fg ' hi" | sed -e 's/[\.\/\%'=:\(\)\`_\'\@\;\+\*\#\$\?\!]/ /g'
echo "abc- cde fg ' hi" | sed -e 's/[\.\/\%'=:\(\)\`_"'"\@\;\+\*\#\$\?\!]/ /g'

As single quotes cannot be escaped with a backslash and double quotes like to evaluate, looks like double quotes is a best bet in my case.

Still investigating if we use sed double quoted "" escape normal character such as number 1 as \1 inside the sed, will it replace the \ and 1 as well, I know it \$ evaluates to $?

---------- Post updated at 01:08 PM ---------- Previous update was at 01:02 PM ----------

Quote:
Originally Posted by ijustneeda
Single quotes seem to fail in following cases:
Code:
echo "abc- cde fg ' hi" | sed -e 's/[\.\/\%'=:\(\)\`_\'\@\;\+\*\#\$\?\!]/ /g'
echo "abc- cde fg ' hi" | sed -e 's/[\.\/\%'=:\(\)\`_"'"\@\;\+\*\#\$\?\!]/ /g'

As single quotes cannot be escaped with a backslash and double quotes like to evaluate, looks like double quotes is a best bet in my case.

Still investigating if we use sed double quoted "" escape normal character such as number 1 as \1 inside the sed, will it replace the \ and 1 as well, I know it \$ evaluates to $?
Bad example of \1 as it is back reference Smilie, let me use \A
echo "abc-A \ cde fg A \ hi" | sed -e "s/\A/ /g"
abc- \ cde fg \ hi
does not replace '\'
# 6  
Old 05-02-2012
If it's really that simple, you could just use tr and stop worrying about regex special characters, since tr doesn't have regular expressions at all, just ranges and a few character classes.

The odd '"'"' bits are to get single-quote characters inside a string that's otherwise single-quotes, since you can't escape anything in single-quotes. The colors show what's actually happening -- the single quotes end, double quotes containing a single quote begin, then it goes back to single quotes.

Code:
$ echo 'Hello ,.@/#:()'"'"'*%$+?_="!; Goodbye' | tr '[,.@/#:()'"'"'*%$+?_="!;]' '_'

Hello ___________________ Goodbye

$ echo 'Hello ,.@/#:()'"'"'*%$+?_="!; Goodbye' | tr -s '[,.@/#:()'"'"'*%$+?_="!;]' '_'

Hello _ Goodbye

$

You'd need a double-backslash \\ inside that range to get a backslash, and would need to escape [ ] - characters inside the range, but that's pretty much it for specials.
Last edited by Corona688; 05-02-2012 at 06:17 PM..
# 7  
Old 05-02-2012
Wanted to continue to use sed in script as it HAD been accepted as working in my team. Smilie
Wonder how fast tr is compared to sed on GB of text files?
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. HP-UX

Replacing control characters in HPUX

Hello dears, Please tell me how can I replace control characters with normal string. I have a file that contains normal string and ^A ^C control characters. I tried to use sed and awk without any luck. sed 's/^A/foo/g' text > text1 //not worked sed 's/\x01/foo/g' text > text1 ... (6 Replies)
Discussion started by: sembii
6 Replies

2. Shell Programming and Scripting

Replacing specific characters using sed

Hi, I have a text file which is output from a server and it lists all the files in a specific volume. However, the volume name appears as volume_name:. I would like to replace this with \\volume_name\volume_name. This is not a problem in itself as I can use sed to globally look for the... (8 Replies)
Discussion started by: vnayak
8 Replies

3. Shell Programming and Scripting

How delete characters of specific line with sed?

Hi, I have a text file with some lines like this: /MEDIA/DISK1/23568742.MOV /MEDIA/DISK1/87456321.AVI /MEDIA/DISK2/PART1/45753131.AVI /IMPORT/44452.WAV ... I want to remove the last 12 characters in each line that it ends "AVI". Should look like this: /MEDIA/DISK1/23568742.MOV... (12 Replies)
Discussion started by: inaki
12 Replies

4. Shell Programming and Scripting

Sed - merge lines bw 2 specific characters

Hi, I have a bash script and I am looking for a command that will merge specific lines together. Sample Data: registration time = 1300890272 Id = 1 setd = 0 tagunt = 26 tagId=6, length=8, value= tagId=9, length=5, value= tagId=7, length=2, value= tagId=16, length=2, value= tagId=32,... (8 Replies)
Discussion started by: Winsarc
8 Replies

5. UNIX for Dummies Questions & Answers

sed replacing in vi, / characters in the middle

I tried to replace the following in vi: old: 'e/thesis/pp/zones/zones' new: 'd/so162/fix/pp' For that, I used: :%s/e/thesis/pp/zones/zones/d/so162/fix/pp/g but doesn't work, a trailing character error message appeared. How can I get it? Thanks in advance (3 Replies)
Discussion started by: Gery
3 Replies

6. UNIX for Dummies Questions & Answers

Escaping non-readable characters using grep, sed or awk

I'm trying to parse out DNS logs from dozens of different domain controllers over a large period of time. The logs are rolled up into individual text files by size, which may contain only a portion of a day's activity or several day's worth (depending on amount of activity). I'm splitting them by... (4 Replies)
Discussion started by: seanwpaul
4 Replies

7. Shell Programming and Scripting

Escaping Special characters

I want to append the following line to /var/spool/cron/root: */7 * * * * /root/'Linux CPU (EDF).sh' > /dev/null 2>&1 How to accomplish this using echo? ---------- Post updated at 04:09 PM ---------- Previous update was at 04:07 PM ---------- "Linux CPU (EDF)" is actually stored in a... (11 Replies)
Discussion started by: proactiveaditya
11 Replies

8. Shell Programming and Scripting

help on sed replacing special characters

Hello, I have a file with many lines with below format: \abc\\1234 jkl\\567 def\\345 \pqr\\567 \xyz\\234 Here, i need to do 2 things. 1. replace \\ with \ 2. remove starting \ so output to be as below: (11 Replies)
Discussion started by: prvnrk
11 Replies

9. Shell Programming and Scripting

replacing specific characters in a string

Hi Friends, Following is an output of a script OPWQERIUTYKLSADFJHGXZNMCVBWQOPERIUTYKLSADFJHGXZNMCVB I want to replace above string's 11 to 17 character by ******* (7 stars) How can it be done? Please somebody guide me. (6 Replies)
Discussion started by: anushree.a
6 Replies

10. Shell Programming and Scripting

how to replace control characters using sed?

How can I use sed to replace a ctrl character such as 'new line' (\0a) to something else? Or any other good command can do this job? Thanks, Hillxy (5 Replies)
Discussion started by: hillxy
5 Replies
Login or Register to Ask a Question