Special characters in command arguments


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Special characters in command arguments
# 1  
Old 03-02-2012
Special characters in command arguments

Hey Guys,

I have a program that populates a database based on input I feed it.

so it would look like:
cmd arg1 arg2 ... argN

Now some of the arguments have special characters in them so I wrote a small script to pull the args and wrap them in quotes ('arg'). The problem is sometimes the characters can be quotes. So for example the text "don't" should become " 'don'\''t' " so I'm trying to write a small script that will read in a file that looks like:
cmd1 arg1 arg2 ... argN
...
cmdM arg1 arg2 ... argN

and for each one wrap the args. So far I have:
Code:
 
#!/usr/bin/env ksh
#set the internal field seperator for KSH to space
IFS=" "
#read in the first argument assuming it's a file
file=$1
#loop at each line of the file
while read line ; do
# build array of space seperated arguments
  set -A lineargs $line
  for arg in ${lineargs[*]}; do
    arg=`echo $arg | sed 's/'\''/'\'\\\'\'/g'`
    arg=\'${arg}\'
    echo $arg
  done
  echo "new cmd"
done <"$file"

The problem I'm having is with the SED replace command it breaks down to the following:

Code:
#   replace    '     with       '\''          globally/everywhere
     's/'     \'     '/'        \'\\\'\'         '\g'

The part that I'm trying to replace with ( '\'' ) just doesn't seem to work. I get either just 3 quote marks or sometimes when I play with it 2 quote marks and the backslash.

If anyone has any recomendations I would really appreciate it because as far as I can tell this should work. Would make my life much easier to not have to copy and paste my populate commands one at a time into the shell.Smilie
# 2  
Old 03-02-2012
You are attempting to put single quotes inside single quotes, which just doesn't work, because single quotes don't let you escape anything. Try double quotes.

Code:
"s/'/\\'/g"

I'm not positive that's what you want however -- you showed the output you don't want, but not any output you do want, just a regex which didn't do what you want...
# 3  
Old 03-02-2012
Thanks for responding.

The reason I'm using single quotes is because double quotes will try to expand things that look like shell variables correct? If one arugment is "hel$lo" wouldn't I get "hel" with an error saying couldn't find shell variable $lo?

An example could be this:

I have the following line in the file:
cmd hey there erTS#$%^)'werk""""''' how are you doing?

I want to get out
cmd 'hey' 'there' 'erTS#$%^)'\''werk""""'\'\'\' 'how' 'are' 'you' 'doing?'
That is probably too difficult so in my script I'm settling for the following:
cmd 'hey' 'there' 'erTS#$%^)'\''werk""""'\'''\'''\''' 'how' 'are' 'you' 'doing?'
Which will be equivalent to the first one as the extra '' inserted will result in no characters.

---------- Post updated at 02:09 PM ---------- Previous update was at 02:05 PM ----------

I just realized that is impossible to read. So for clarity I'll space it out in the chucks I believe the shell will read:

first one (optimal):
'erTS#$%^)' \' 'werk""""' \' \' \'

2nd one (settling):
'erTS#$%^)' \' 'werk""""' \' '' \' '' \' ''
# 4  
Old 03-02-2012
Unless you disable pathname expansion (globbing), your attempt to quote with sed is too little too late.

Code:
$ ls
d1      d2      f1      f2
$ line='cmd * d? f[12]'
$ set -A lineargs $line
$ echo "${lineargs[@]}"
cmd d1 d2 f1 f2 d1 d2 f1 f2
$ echo $line
cmd d1 d2 f1 f2 d1 d2 f1 f2
$ set -o noglob
$ echo $line
cmd * d? f[12]

Why exactly you're trying to do this is unclear, since the code you provided culminates with an uninformative echo statement. It's possible you're jumping through these hopes for no reason. It would be helpful if you shared the code that is actually used to do the work.

Regards,
Alister
# 5  
Old 03-02-2012
Thanks for responding.

I think you're misunderstanding what I'm trying to do.

First lets start with this:

Code:
me@server dir $ls
file1  file2  file3  file4
me@server dir $echo '*'
*
me@server dir $echo *
file1 file2 file3 file4
me@server dir $temp='*'
me@server dir $echo $temp
file1 file2 file3 file4
a7me@server dir $echo "$temp"
*
me@server dir $

I don't see how it's too little too late. It's doing exactly what the shell should do. Single quotes don't expand anything and double quotes allow expansion. If you set a variable to '*' then it literally equals *, however if instead you set it to just * then glob comes into play and you get an array containing all of the files/directories in your current directory.

I'm currectly writing a program to do this I did post all the code I had in the first post. Which works almost as it should. If you give it a file that is newline seperated such as:

Code:
cmd1 opt1 opt2 opt3 opt'4 opt5
cmd2 opt1 opt2 opt3 'opt4 opt5
cmd3 opt1 opt2 opt3 opt4' opt5
cmd4 opt'1 'opt2 opt3' opt4' opt5

It currently loops through and prints the following (I'm only going to show the last line):
Code:
'cmd4'
'opt'\1'
''\opt2'
'opt3'\'
'opt4'\'
'opt5'
new cmd

So for the argument opt4' I expect to see 'opt4'\'''. In this case I'm getting 'opt4'\' which means the part that replacess all occurences of ' with '\'' isn't working correctly. It also isn't consistent as for arg 2 ' is replaced with just '\

I think the code makes it clear that I'm simply replacing all occurences of ' with '\'' and then appending ' to the begining and end of each string/arg. This because everything in between the two single quotes will not be expanded and will be read literally. However the shell isn't capaple of allowing me to escape a ' in the middle of a single quoted string. (can't do: 'don't' can do 'don'\''t')

The final version would of course execute command and pass the args, however I want to see what is going to the command before I call it so I can verify if it's working which is why it currently just prints the args line per line.
# 6  
Old 03-02-2012
Quote:
Originally Posted by aaron0001
I don't see how it's too little too late. It's doing exactly what the shell should do. Single quotes don't expand anything and double quotes allow expansion.
Single quotes don't expand variables, either Smilie
Quote:
If you set a variable to '*' then it literally equals *, however if instead you set it to just * then glob comes into play and you get an array containing all of the files/directories in your current directory.
Not true.

Code:
VAR=*
$ echo $VAR
file1 file2 file3 ...
$ echo "$VAR"
*
$

Substitution happens when the variable is expanded. It does not happen when the variable is assigned.

It doesn't become an array, either. It becomes an argument list. * makes no sense anywhere but arguments.

Quote:
I'm currectly writing a program to do this I did post all the code I had in the first post. Which works almost as it should. If you give it a file that is newline seperated such as:

Code:
cmd1 opt1 opt2 opt3 opt'4 opt5
cmd2 opt1 opt2 opt3 'opt4 opt5
cmd3 opt1 opt2 opt3 opt4' opt5
cmd4 opt'1 'opt2 opt3' opt4' opt5

It currently loops through and prints the following (I'm only going to show the last line):
Code:
'cmd4'
'opt'\1'
''\opt2'
'opt3'\'
'opt4'\'
'opt5'
new cmd

Is this meant to be shell code? Escapes don't work inside single quotes! You get a literal backslash.
Code:
$ echo '\'
\
$ echo '\\'
\\
$

Meaning, 'opt4\'' is a syntax error.

What are you actually trying to do, here? Not just the manner you're hellbent on accomplishing it with? There are likely better ways than kludging escapes into everything.

---------- Post updated at 01:38 PM ---------- Previous update was at 01:31 PM ----------

I have a new idea. Working on it.

---------- Post updated at 01:52 PM ---------- Previous update was at 01:38 PM ----------

Code:
$ cat superquote.awk
BEGIN { OFS="' '"; }
{
        # Do a global substitution for all things we haven't explicitly allowed.
        # & becomes the match, so \& becomes \[match], except we need to
        # escape the escaping so it becomes a real backslash.
        # Yes, four backslashes are needed to make one backslash.  kill me
        gsub(/[^a-zA-Z0-9 \t]/, "\\\\&");
        $1=$1   # do a substitution so awk auto-substitutes ' ' between all parameters

        # Print all arguments for debugging, to stderr for easy filtering
        for(N=1; N<=NF; N++) print "'" $N "'" >"/dev/stderr"

        print "'" $0 "'"

}

$ awk -f superquote.awk data
'cmd1'
'opt1'
'opt2'
'opt3'
'opt\'4'
'opt5'
'cmd1' 'opt1' 'opt2' 'opt3' 'opt\'4' 'opt5'
'cmd2'
'opt1'
'opt2'
'opt3'
'\'opt4'
'opt5'
'cmd2' 'opt1' 'opt2' 'opt3' '\'opt4' 'opt5'
'cmd3'
'opt1'
'opt2'
'opt3'
'opt4\''
'opt5'
'cmd3' 'opt1' 'opt2' 'opt3' 'opt4\'' 'opt5'
'cmd4'
'opt\'1'
'\'opt2'
'opt3\''
'opt4\''
'opt5'
'cmd4' 'opt\'1' '\'opt2' 'opt3\'' 'opt4\'' 'opt5'

# ...but the separate ones are printed direct to terminal, not to stdout.
# They won't flow into a pipe, and can be ignored as you please by simple
# redirection.
$ awk -f superquote.awk data 2>/dev/null
'cmd1' 'opt1' 'opt2' 'opt3' 'opt\'4' 'opt5'
'cmd2' 'opt1' 'opt2' 'opt3' '\'opt4' 'opt5'
'cmd3' 'opt1' 'opt2' 'opt3' 'opt4\'' 'opt5'
'cmd4' 'opt\'1' '\'opt2' 'opt3\'' 'opt4\'' 'opt5'

$

# 7  
Old 03-02-2012
I think we are still speaking different languages. Your first few sentences agree with me while saying you don' agree =/.

Next, at no point do I have a \ inside single quotes like '\', if you look I'm concatenating 3 strings like such 'don' '\ 't'. The part I'm doing in the middle '\'' the first ' is a closing quote on the first string the \' is the actual quote and the final ' starts the next string.

So I'm splitting one string into several and then concatenating them back essentially.

don't becomes 'don' \' 't' .which if echoed prints don't as expected.

awk is obviously a swiss army knife. But at this point I would like to know why I can't accomplish a replace of ' with ''\' with sed.
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Using CURL command with special characters in URL

Hi. I 'm trying to hit a REST api and retrieve a JSON feed, and the URL has special characters in it. Something like: Example Domain The below curl command is failing curl -X GET https://www.example.com/?sample=name&id=1001 saying bad command at id=1001 I am going to use this as part... (3 Replies)
Discussion started by: kumarjt
3 Replies

2. Shell Programming and Scripting

Tcsh script - sed command for special characters

Hi, I am using sed command to replace following line in gz file- sed -n 's/""COS_12_TM_1" ( 1101110101001001010011110100000010110100010010000000100000000010XX010000000 )"/""COS_12_TM_1" ( 110111010100100101001111MM00000010110100010010000000100000000010XX010000000 )"/g' filename.gz $x=... (4 Replies)
Discussion started by: Preeti Chandra
4 Replies

3. Shell Programming and Scripting

How to escape all special characters?

I have an application which I am integrating with that accepts the password via a CLI. I am running in to issues with passwords that contain special characters. I tried to escape them all, but I ran in to an issue where I cannot escape the characters ' ] My attempt is as follows: $... (2 Replies)
Discussion started by: AMG1978
2 Replies

4. Shell Programming and Scripting

A test command parameter is not valid, when special characters are tried to match

Hi I have a scenario where hyphen(-) from file should be ignored I used the following code if && ; then if ; then pow=$LINE echo $pow > history.txt flag=1 fi fi I get the following output ./valid.sh: -: 0403-012 A test... (7 Replies)
Discussion started by: Priya Amaresh
7 Replies

5. Shell Programming and Scripting

Need help in replacing special characters

I am writing a ksh script. I need to replace a set of characters in an xml file. FROM="ÀÁÂÃÄÅÆÇÈÉÊËÌÍÎÏÐÑÒÓÔÕÖØÙÛÚÜÝßàáâãäåçèéêëìíîïðñòóôõö¿¶ø®"; TO="AAAAAAACEEEEIIIIDNOOOOOOUUUUYSaaaaaaceeeeiiiionooooo N R" I have used the code- sed 's/$FROM/$TO/g'<abc.xml But its not working. Can anyone tell me the code to do this? (3 Replies)
Discussion started by: saga20
3 Replies

6. Shell Programming and Scripting

Replace special characters with Escape characters?

i need to replace the any special characters with escape characters like below. test!=123-> test\!\=123 !@#$%^&*()-= to be replaced by \!\@\#\$\%\^\&\*\(\)\-\= (8 Replies)
Discussion started by: laknar
8 Replies

7. Shell Programming and Scripting

special characters

Hey guys, I'm trying to replace "]Facebook" from the text but sed 's/]Facebook/Johan/g' is not working could you please help me with that? (6 Replies)
Discussion started by: Johanni
6 Replies

8. UNIX for Dummies Questions & Answers

How to see special characters?

Hi all, I was wondering how can i see the special characters like \t, \n or anything else in a file by using Nano or any other linux command like less, more etc (6 Replies)
Discussion started by: gvj
6 Replies

9. Shell Programming and Scripting

Special characters

When I open a file in vi, I see the following characters: \302\240 Can someone explain what these characters mean. Is it ASCII format? I need to trim those characters from a file. I am doing the following: tr -d '\302\240' ---------- Post updated at 08:35 PM ---------- Previous... (1 Reply)
Discussion started by: sid1982
1 Replies

10. UNIX for Dummies Questions & Answers

special characters

I have one file which is named ^? ( the DEL character ) I'd like to know how to rename or copy the file by using its i-node number TYIA (2 Replies)
Discussion started by: nawnaw
2 Replies
Login or Register to Ask a Question