Url encoding a string using sed

Login or Register for Dates, Times and to Reply

Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Url encoding a string using sed
# 1  
Url encoding a string using sed

Hi I was hoping some one would know if it is possible to url encode a string using sed?
My problem is I have extracted some key value pairs from a text file with sed, and will be inserting these pairs as source variables into a curl script to automatically download some xml from our server.
My problem is that one of the value pairs needs to be url encoded. The "j_password=0NRx5KpiquE32LrlXApzrA==" needs to be url encoded
Below are my key value pairs

These pairs will be read into a curl script, Below is part of the line from my curl script that the key value pairs will be read in as variables;
 curl ''  lastLoggedInCompany=public;~~ JSESSIONID=$JSESSIONID0' -H 'Origin:' -H~~'Referer:' -H 'Connection: keep-alive' --data 'j_username=ticorder&j_password=$j_password&lastLoggedInCompany=public&iv=$iv&salt=$salt&hash=' --compressed

I had read online that curl can be used to url encode. Although I am not sure how I would use this function in curl to encode only part of the curl command that I need url encoded?
The script I am using to extract the value pairs is below
sed  -i ' -e ' -n '/POST/,$p' /WebLoads/MainGo/CaptureFile-0
sed -i '' -e $'s/\t/\ /g' /WebLoads/MainGo/CaptureFile-0
sed -i '' -e $'s/\ \ /\ /g' /WebLoads/MainGo/CaptureFile-0
sed -i '' -e $'s/\ GET.*//g' /WebLoads/MainGo/CaptureFile-0
sed -i '' -e $'s/.*_uc\=/_uc\=/g' /WebLoads/MainGo/CaptureFile-0
sed -i ' -e ' -n '/.*/,// G; s/\n/&&/; /^\([ -~]*\n\).*\n\1/d; s/\n//; h; P' /WebLoads/MainGo/CaptureFile-0
sed -i '' -e $'s/\t/\ /g' /WebLoads/MainGo/CaptureFile-0
sed -i '' -e s'/.*\=public\;\ JSESSIONID\=/JSESSIONID0\=/g' /WebLoads/MainGo/CaptureFile-0
sed -i '' -e s'/Form item\:\ /\'$'\n/g' /WebLoads/MainGo/CaptureFile-0
sed -i '' -e s'/\ POST.*//g' /WebLoads/MainGo/CaptureFile-0
sed -i '' -e s'/.*j_username.*//g' /WebLoads/MainGo/CaptureFile-0
sed -i '' -e s'/.*companySelection.*//g' /WebLoads/MainGo/CaptureFile-0
sed -i '' -e s'/.*Login.*//g' /WebLoads/MainGo/CaptureFile-0
sed -i '' -e s'/.*lastLoggedInCompany.*//g' /WebLoads/MainGo/CaptureFile-0
sed -i '' -e s'/.*hash.*//g' /WebLoads/MainGo/CaptureFile-0
sed -i '' -e s'/\;\ /\'$'\n/g' /WebLoads/MainGo/CaptureFile-0
sed -i '' -e s'/\"\ =\ \"/\=/g' /WebLoads/MainGo/CaptureFile-0
sed -i '' -e s'/\"\,//g' /WebLoads/MainGo/CaptureFile-0
sed -i '' -e s'/^\"//g' /WebLoads/MainGo/CaptureFile-0
sed -i ' -e ' -n '/.*/,// G; s/\n/&&/; /^\([ -~]*\n\).*\n\1/d; s/\n//; h; P' /WebLoads/MainGo/CaptureFile-0
sed -i '' -e '/^\s*$/d' /WebLoads/MainGo/CaptureFile-0

I've looked online and it does not seem very hopeful to be able to accomplish this with sed, but It would be my first choice since my understanding of sed is better than some of the other available programs. If there is another way of accomplishing this with another program I would be more than thankful if some one was able to show me how this could be done.
Thank you
# 2  
You could use a shell function to encode a passed string into a variable like this:


url_var() {
  # URL encode string and optionally store in shell variable
  # usage: url_var <string> [var]

  local LC_ALL=C
  local encoded=""
  local i c

  for (( i=0 ; i<${#1} ; i++ )); do
     case "$c" in
        [a-zA-Z0-9/_.~-] ) encoded="${encoded}$c" ;;
        * ) printf -v encoded "%s%%%02x" "$encoded" "'${c}" ;;
  [ $# -gt 1 ] &&
     printf -v "$2" "%s" "${encoded}" ||
     printf "%s\n" "${encoded}"

url_var j_password=0NRx5KpiquE32LrlXApzrA== PASS
printf "PASS=%s\n" "$PASS"

Last edited by Chubler_XL; 02-18-2018 at 09:32 PM..
# 3  
I'm not sure your curl command does what you intend it to do, as e.g. the pairs of single quotes don't match up. Once you sourced your "key / value pair" file so all the needed shell variables are defined, use Chubler_XL's function (in a "command substitution"!) to encode the variables in question:
curl '  lastLoggedInCompany=public;~~ JSESSIONID='$JSESSIONID0' -H Origin: -H~~Referer: -H Connection: keep-alive --data j_username=ticorder&j_password='$(url_var $j_password)'&lastLoggedInCompany=public&iv='$iv'&salt='$salt'&hash= --compressed'

Not sure if I entirely and correctly understood your request, nor if the other variables used wouldn't need a URL-encoding as well.
# 4  
Hi thank you for helping me with my problem, I appreciate the time you've taken very much.
The curl command I am using is one that I have used before to download our needed xml from our server. It has stopped working due to an upgrade on the server it was logging into.
The server we were extracting the xml from is now using more dynamic security that requires a new password and supporting cookies every twenty four hours, I was able to extract the key value pairs using tshark from network trafic and had hoped to insert these into my curl command as variables, it does seem strange that the only one that requires url encoding is the j_pasword, I am just following the format of the existing command which url encodes this for some reason.
I has hoping you could go into some more detail on how to use Chubler_XL shell function and how it works as I would like to learn for myself what it is doing.
Should I save this as a file and make a call to it?
Thank you
# 5  
You can take above as an example and encode every single variable that you use in your command. like $(url_var $JSESSIONID0) etc. Give it a try and report back.
This User Gave Thanks to RudiC For This Post:
# 6  
Hi thanks for getting me going with this it seems to be working for me a little better for now.
I had never used a shell function before and this is my first exposure to one.
Ill try and puzzle out any remaining problems .

Thanks for all the help!!
Login or Register for Dates, Times and to Reply

Previous Thread | Next Thread
Thread Tools Search this Thread
Search this Thread:
Advanced Search

Test Your Knowledge in Computers #523
Difficulty: Medium
Some programming languages implement strings by defining arrays of characters.
True or False?

10 More Discussions You Might Find Interesting

1. Solaris

View file encoding then change encoding.

Hi all!! Im using command file -i myfile.xml to validate XML file encoding, but it is just saying regular file . Im expecting / looking an output as UTF8 or ANSI / ASCII Is there command to display the files encoding? Thank you! (2 Replies)
Discussion started by: mrreds
2 Replies

2. Shell Programming and Scripting

Use curl to send a static xml file using url encoding to a web page using pos

Hi I am try to use curl to send a static xml file using url encoding to a web page using post. This has to go through a particular port on our firewall as well. This is my first exposure to curl and am not having much success, so any help you can supply, or point me in the right direction would be... (1 Reply)
Discussion started by: Paul Walker
1 Replies

3. Shell Programming and Scripting

Replace URL using sed

Original Line {background-image:url('http://www.myoldhost.com/images/scds/tsp3.png');} Expected {background-image:url('http://www.mynewhost.com/nndn/hddh/ccdcd.png');} I am using following syntax STATIC_HOST_TEMP="http://myhost.com/temp/xyx.png" $sed -e... (1 Reply)
Discussion started by: 8055
1 Replies

4. Shell Programming and Scripting

URL/HTML encoding

Hey guys, looking for a way to encode a string into URL and HTML in a bash script that I'm making to encode strings in various different digests etc. Can't find anything on it anywhere else on the forums. Any help much appreciated, still very new to bash and programming etc. (4 Replies)
Discussion started by: 3therk1ll
4 Replies

5. Shell Programming and Scripting

How to check string encoding?

I want to check if the string is WINDOWS-1251 or UTF-8 can you help me to find the string encoding??? or maybe to get URL Content-Type charset with wget? this is my function on PHP function check_utf8($str) { $len = strlen($str); for($i = 0; $i < $len; $i++){ $c =... (2 Replies)
Discussion started by: sanantonio7777
2 Replies

6. Shell Programming and Scripting

How to achieve UTF-8 encoding & URL escape in an xml file?

Is there any i can achieve entity escaping, URL escaping & UTF-8 encoded for the xml generated through shell script? #! /bin/bash echo "<path>" >> file.xml for x in `ls filename*` do echo -e "\t<dir>" >> file.xml echo -e "\t\t<file>$x</file>" >> file.xml... (0 Replies)
Discussion started by: vel4ever
0 Replies

7. Shell Programming and Scripting

python - string encoding error

I'm trying to pull a google calendar (successful) and then put the contents into a mysql db (almost successful). On one of the field I keep getting an encode error: #!/usr/bin/python from xml.etree import ElementTree import gdata.calendar.data import gdata.calendar.client import... (12 Replies)
Discussion started by: unclecameron
12 Replies

8. UNIX for Advanced & Expert Users

.htaccess url encoding

Hi, I tried hard to find out solution but no success. I have put together this code in .htaccess: Rewritecond %{THE_REQUEST} ^{3,9}\ /index\.php\?(+)query=(.*?)&(+)start=(.*?)&(+)\ HTTP/ Rewriterule ^index\.php$ http://subdomain.domain.com/%2/%4? RewriteRule ^(+)/(*)$... (0 Replies)
Discussion started by: LukasB
0 Replies

9. Shell Programming and Scripting

URL encoding

Hi All, I want to do URL encoding using shell script in my project. I decided that the sed is the correct tool to do this. But I am unable achieve what I wanted using sed. kindly help me to get rid of this. My requirement is , there will be one URL with all special character, spaces etc ... (8 Replies)
Discussion started by: Vichu
8 Replies

10. Shell Programming and Scripting

parsing url string

Hi, I have a url like this:- http://resource.ibab.ac.in/cgi-bin/pubmed_abstract/y.cgi?pmid=+1.10529272+&pmid=+3.8379586 I want to parse the url string.I want to get the values 1.10529272 3.8379586 How do i do that? (5 Replies)
Discussion started by: vanitham
5 Replies

Featured Tech Videos