Visit Our UNIX and Linux User Community

Splitting URL request string including escape characters

Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Splitting URL request string including escape characters
# 1  
Old 09-03-2009
Question Splitting URL request string including escape characters

Hi All,
I have a text file containing two fields, the first field holds the website name and the second one holds the rest of the URL (Request part). My requirement is to split the request part based on the characters "=" and "&". The process should also consider the URL escape characters, for example if the URL contains any of the escape code listed below it should be replaced with the corresponding character. A sample is shown below:
INPUT (Data in the text file, can contain 1 million rows)

Note in the below output %3A has been replaced by ":" and %2F with "/"
WEBISTENAME;KEY;VALUE; I_CMDTypeOrig;NULL; I_OrderReceiptTime;10:35:10; verifySO;Y; I_Acc ;ABC373; loadEvaluated;false; I_OrderReceiptDate;09/02/2009; I_SecId;JED

URL Escape Characters
Character    Escape Code
SPACE => %20
[ => %5B
< => %3C
] => %5D
> => %3E
` => %60
# => %23
;  => %3B
% => %25
/  => %2F
{  => %7B
?  => %3F
}  => %7D
:  => %3A
|  => %7C
@ => %40
\  => %5C
=  => %3D
^ => %5E
&  => %26
~ => %7E
$  => %24

Any pointers to arrive at the above output will be greatly appreciated.

Thanks in advance.

Last edited by john2022; 09-03-2009 at 02:55 PM.. Reason: code tags added
# 2  
Old 09-03-2009
To keep the forums high quality for all users, please take the time to format your posts correctly.

First of all, use Code Tags when you post any code or data samples so others can easily read your code. You can easily do this by highlighting your code and then clicking on the # in the editing menu. (You can also type code tags [code] and [/code] by hand.)

Second, avoid adding color or different fonts and font size to your posts. Selective use of color to highlight a single word or phrase can be useful at times, but using color, in general, makes the forums harder to read, especially bright colors like red.

Third, be careful when you cut-and-paste, edit any odd characters and make sure all links are working property.

Thank You.

The UNIX and Linux Forums
# 3  
Old 09-03-2009
One pointer for the first part:

awk 'BEGIN{FS="[;=&]";OFS=";"}{for (i=2;i<=NF;i+=($(i+1)?1:2)) print $1,$i,$(i+1)?$(i+1):"NULL"}'  input.file

For the replacement part, either you work with the awk string functions like sub(), gsub() or gensub() or first process your input file with sed that I think will be more efficient in string replacement.

Last edited by ripat; 09-03-2009 at 05:10 PM..
# 4  
Old 09-04-2009
thank you

[quote=ripat;302350395]One pointer for the first part:

Thanks for the quick repsonse Ripat. I will work based on your suggestion.
# 5  
Old 09-04-2009
To urldecode in awk, I found this:

Previous Thread | Next Thread
Test Your Knowledge in Computers #5
Difficulty: Easy
The UNIX trademark is a "certification mark".
True or False?

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Escape characters

i am executing script from A server which will execute the script in B server , as below. ssh A 'ssh B echo 'select * from testing where name ='test'' i am getting the below output. select * from testing where name=test but i need the output where clause with quotes , tried with... (3 Replies)
Discussion started by: expert
3 Replies

2. Shell Programming and Scripting

Need HELP with AWK split. Need to check for "special characters" in string before splitting the file

Hi Experts. I'm stuck with the below AWK code where i'm trying to move the records containing any special characters in the last field to a bad file. awk -F, '{if ($NF ~ /^|^/) print >"goodfile";else print >"badfile"}' filename sample data 1,abc,def,1234,A * 2,bed,dec,342,* A ... (6 Replies)
Discussion started by: shell_boy23
6 Replies

3. Shell Programming and Scripting

Replace special characters with Escape characters?

i need to replace the any special characters with escape characters like below. test!=123-> test\!\=123 !@#$%^&*()-= to be replaced by \!\@\#\$\%\^\&\*\(\)\-\= (8 Replies)
Discussion started by: laknar
8 Replies

4. Shell Programming and Scripting

How to achieve UTF-8 encoding & URL escape in an xml file?

Is there any i can achieve entity escaping, URL escaping & UTF-8 encoded for the xml generated through shell script? #! /bin/bash echo "<path>" >> file.xml for x in `ls filename*` do echo -e "\t<dir>" >> file.xml echo -e "\t\t<file>$x</file>" >> file.xml... (0 Replies)
Discussion started by: vel4ever
0 Replies

5. Shell Programming and Scripting

Grep a string from input file and delete next three lines including the line contains string in xml

Hi, 1_strings file contains $ cat 1_strings /home/$USER/Src /home/Valid /home/Review$ cat myxml <projected value="some string" path="/home/$USER/Src"> <input 1/> <estimate value/> <somestring/> </projected> <few more lines > <projected value="some string" path="/home/$USER/check">... (4 Replies)
Discussion started by: greet_sed
4 Replies

6. Shell Programming and Scripting

search string in a file and retrieve 10 lines including string line

Hi Guys, I am trying to write a perl script to search a string "Name" in the file "FILE" and also want to create a new file and push the searched string Name line along with 10 lines following the same. can anyone of you please let me know how to go about it ? (8 Replies)
Discussion started by: sukrish
8 Replies

7. Shell Programming and Scripting

bash curl escape & in url

I'm running a curl command in bash, but the & in the middle causes the second half of the line to run in the background, here's what I'm trying to do: lat="37.451" lon="-122.18" url="$lat&lng=$lon" curl -s "$url" I tried escaping the & with \&,... (4 Replies)
Discussion started by: unclecameron
4 Replies

8. UNIX for Advanced & Expert Users

Remove escape characters from string

Hello all, I have a string var which contains formatting characters at the end, it is a string with EScape sequences at the end of it. How can I remove them so that I only keep the 'real' text? I tried : var1=${var1%%\033[0m} does not seem to do the job .... Please help Thanks (2 Replies)
Discussion started by: gio001
2 Replies

9. Shell Programming and Scripting

How cgi(shell script) to receive the URL REQUEST from Front end (visual C++)?

I am facing a problem which is weblogic convert to CGI method. The original process is -> the Front End is using Visual C++, and backend is using WEBLOGIC the based programming is JAVA. The method for Weblogic to receive the request from Front end (Visual C++) thru URL. for example :... (0 Replies)
Discussion started by: ryanW
0 Replies

10. Shell Programming and Scripting

escape characters..

hey i want to know the unix commands to replace all the character escape sequences with their "C" values in a string... thanks in advance..! Regards, Sharanya (9 Replies)
Discussion started by: sharsin2001
9 Replies

Featured Tech Videos