URL encoding


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting URL encoding
# 1  
Old 04-06-2008
Question URL encoding

Hi All,

I want to do URL encoding using shell script in my project. I decided that the sed is the correct tool to do this. But I am unable achieve what I wanted using sed. kindly help me to get rid of this.

My requirement is , there will be one URL with all special character, spaces etc...

For ex. https://www.xxxxxx.com/change&$ ^this to?%checkthe@-functionality..I want to do URL encoding only after ? mark. Final result should be https://www.xxxxxx.com/change&$ ^this to?<encoded 2nd part>

Thanks in advance

Regards
Vichu
# 2  
Old 04-06-2008
There is nothing after the ? which requires encoding except the percent sign. Perhaps you should pick a more detailed example.

It's not going to be very elegant to do this in sed because it requires a loop, and loops are kind of tricky in sed. Basically, stash away the part you don't want to encode, loop over the remaining part, moving away everything you have already encoded by appending it to the stash.

Maybe something like this, instead?

Code:
perl -ple 's/\?(.*)//;
my $tail = $1;
$tail =~ s/([%? +&!<>()])/sprintf "%%%02x", ord($1) /ge;
s/$/$tail/'

The list of characters which require or might benefit from escaping is quite probably not complete. This assumes you have nothing after the URL which is not part of the URL, and that the first question mark separates the tail which requires encoding from the base URL.
# 3  
Old 04-06-2008
Seperating URL into two parts (? is the delimiter before ? is the first part and after ? is the second part) and I can do encoding for the second part using awk..But I don't want to split the line. Just skipping first part of URL and encoding the second part.
# 4  
Old 04-06-2008
Did you try the code I posted?
# 5  
Old 04-07-2008
Hi era,

Thanks for your promptness..But I don't want to do it in perl...My requirement is in shell..
# 6  
Old 04-07-2008
Use 2 files:
urlencode.sed
Code:
s/%/%25/g
s/ /%20/g
s/ /%09/g
s/!/%21/g
s/"/%22/g
s/#/%23/g
s/\$/%24/g
s/\&/%26/g
s/'\''/%27/g
s/(/%28/g
s/)/%29/g
s/\*/%2a/g
s/+/%2b/g
s/,/%2c/g
s/-/%2d/g
s/\./%2e/g
s/\//%2f/g
s/:/%3a/g
s/;/%3b/g
s//%3e/g
s/?/%3f/g
s/@/%40/g
s/\[/%5b/g
s/\\/%5c/g
s/\]/%5d/g
s/\^/%5e/g
s/_/%5f/g
s/`/%60/g
s/{/%7b/g
s/|/%7c/g
s/}/%7d/g
s/~/%7e/g
s/      /%09/g

urlencode.sh
Code:
#!/bin/ksh

STR1=$(echo "https://www.xxxxxx.com/change&$ ^this to?%checkthe@-functionality" | cut -d\? -f1)
STR2=$(echo "https://www.xxxxxx.com/change&$ ^this to?%checkthe@-functionality" | cut -d\? -f2)

OUT2=$(echo "$STR2" | sed -f urlencode.sed)

echo "$STR1?$OUT2"

Result:
Code:
./urlencode.sh
https://www.xxxxxx.com/change&$ ^this to?%25checkthe%40%2dfunctionality

OK for you ? Smilie
# 7  
Old 04-07-2008
I proposed the same procedure,whatever you have given here, using awk.But my team didnot accept it. They dont want to use extra varialbes.They would like to do encoding (IInd part) on the same variable.

My proposed steps
==============
str1=`echo "https://www.xxxxxx.com/change&$ ^this to?%checkthe@-functionality" | awk -F? '{print $1}'`

str2=`echo "https://www.xxxxxx.com/change&$ ^this to?%checkthe@-functionality" | awk -F? '{print $2}'`

str3=`echo $str2 | sed -f ./seq.sed `

echo $str?$str3

Do you have any idea?
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Url encoding a string using sed

Hi I was hoping some one would know if it is possible to url encode a string using sed? My problem is I have extracted some key value pairs from a text file with sed, and will be inserting these pairs as source variables into a curl script to automatically download some xml from our server. My... (5 Replies)
Discussion started by: Paul Walker
5 Replies

2. Solaris

View file encoding then change encoding.

Hi all!! Im using command file -i myfile.xml to validate XML file encoding, but it is just saying regular file . Im expecting / looking an output as UTF8 or ANSI / ASCII Is there command to display the files encoding? Thank you! (2 Replies)
Discussion started by: mrreds
2 Replies

3. Shell Programming and Scripting

Reading URL using Mechanize and dump all the contents of the URL to a file

Hello, Am very new to perl , please help me here !! I need help in reading a URL from command line using PERL:: Mechanize and needs all the contents from the URL to get into a file. below is the script which i have written so far , #!/usr/bin/perl use LWP::UserAgent; use... (2 Replies)
Discussion started by: scott_cog
2 Replies

4. Shell Programming and Scripting

Use curl to send a static xml file using url encoding to a web page using pos

Hi I am try to use curl to send a static xml file using url encoding to a web page using post. This has to go through a particular port on our firewall as well. This is my first exposure to curl and am not having much success, so any help you can supply, or point me in the right direction would be... (1 Reply)
Discussion started by: Paul Walker
1 Replies

5. Shell Programming and Scripting

URL/HTML encoding

Hey guys, looking for a way to encode a string into URL and HTML in a bash script that I'm making to encode strings in various different digests etc. Can't find anything on it anywhere else on the forums. Any help much appreciated, still very new to bash and programming etc. (4 Replies)
Discussion started by: 3therk1ll
4 Replies

6. UNIX for Dummies Questions & Answers

Awk: print all URL addresses between iframe tags without repeating an already printed URL

Here is what I have so far: find . -name "*php*" -or -name "*htm*" | xargs grep -i iframe | awk -F'"' '/<iframe*/{gsub(/.\*iframe>/,"\"");print $2}' Here is an example content of a PHP or HTM(HTML) file: <iframe src="http://ADDRESS_1/?click=5BBB08\" width=1 height=1... (18 Replies)
Discussion started by: striker4o
18 Replies

7. Shell Programming and Scripting

How to achieve UTF-8 encoding & URL escape in an xml file?

Is there any i can achieve entity escaping, URL escaping & UTF-8 encoded for the xml generated through shell script? #! /bin/bash echo "<path>" >> file.xml for x in `ls filename*` do echo -e "\t<dir>" >> file.xml echo -e "\t\t<file>$x</file>" >> file.xml... (0 Replies)
Discussion started by: vel4ever
0 Replies

8. UNIX for Advanced & Expert Users

.htaccess url encoding

Hi, I tried hard to find out solution but no success. I have put together this code in .htaccess: Rewritecond %{THE_REQUEST} ^{3,9}\ /index\.php\?(+)query=(.*?)&(+)start=(.*?)&(+)\ HTTP/ Rewriterule ^index\.php$ http://subdomain.domain.com/%2/%4? RewriteRule ^(+)/(*)$... (0 Replies)
Discussion started by: LukasB
0 Replies

9. Shell Programming and Scripting

How to find the file encoding and updating the file encoding?

Hi, I am beginner to Unix. My requirement is to validate the encoding used in the incoming file(csv,txt).If it is encoded with UTF-8 format,then the file should remain as such otherwise i need to chnage the encoding to UTF-8. Please advice me how to proceed on this. (7 Replies)
Discussion started by: cnraja
7 Replies

10. Shell Programming and Scripting

url calling and parameter passing to url in script

Hi all, I need to write a unix script in which need to call a url. Then need to pass parameters to that url. please help. Regards, gander_ss (1 Reply)
Discussion started by: gander_ss
1 Replies
Login or Register to Ask a Question

Featured Tech Videos