Using awk to split a string


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Using awk to split a string
# 1  
Old 01-10-2013
Using awk to split a string

Hey guys, I've been trying to find an answer to this, and I've been reading up on awk as much as possible, but I'm at a loss at the moment.

I'll start off by saying I'm trying to learn, so forgive me if I ask questions about your answers.

Here is what I'm trying to accomplish. I have a long string of text which I'll link below. I want to be able to go through the text and pull out certain keywords and place them and the text associated with them in a file.

Here is an example string:
Code:
013-01-10 16:21:59,911 SECURE POST Data (examplesite.com):
session%5Busername_or_email%5D=user1%40example.com&session%5Bpassword%5D=examplepass&scribe_log=&redirect_after_login=%2F&authenticity_token=4d27cf47496ec391e055eb78a6f4fa50a5a87e5b

So, what I'd like to pull out of that information is this:
examplesite.com (This is a changing variable depending on the output of it's parent script.)
username_or_email (This is static.)
user1%40example.com (Changing.)
password (Static)
examplepass (Changing.)

The desired output is as such:
Code:
Examplesite.com
Username = user1%40example.com
Password = examplepass

If you could at least point me in a direction to figure this out on my own, I'd be happy with that. If you would like to provide the answer to my problem for me, if you could break it out so I can learn from it, I'd appreciate that even more.

As I said before, I'm not just looking for the simple answer of how to make it work. I'd like to learn why it works as well Smilie.

Cheers,
-Shadow

Last edited by Scrutinizer; 01-13-2013 at 02:42 PM.. Reason: code tags
# 2  
Old 01-11-2013
Code:
awk -F'[ |=]' ' \
 {
   for(i=1;i<=NF;i++)
   {
     if($i ~ /^\(/)
     {
       site=$i; gsub(/\(/,"",site); gsub(/\)/,"",site); gsub(":","",site);
     }
     if($i ~ /username/)
     {
       user=$(i+1); gsub(/\&.*/,"",user);
     }
     if($i ~ /password/)
     {
       pass=$(i+1); gsub(/\&.*/,"",pass);
     }
   }
 }
END {
   printf "%s\nUsername = %s\nPassword = %s\n", site, user, pass;
} ' filename

This User Gave Thanks to Yoda For This Post:
# 3  
Old 01-11-2013
Any chance you could break down what each part of this does, or should I walk my happy ass on over to Google and research Awk and Gsub?

Thank you for the help bipinajith! I really do appreciate it Smilie.
# 4  
Old 01-11-2013
Condensing bipinajith's proposal slightly, here's some explanatory hints:
Code:
awk -F'[ =&]' '                               # set field separator to " ", "=", or "&" single char at which the line is broken into fields
     {for(i=1;i<=NF;i++)                      # check field one to last (NF is no. of fields)
       {if ($i ~ /^\(/)     {site=$i;         # parentheses enclose the site name ( in this case, but this is not necessarily an unambiguous identifier...)
                             gsub(/\(|\)|:/,"",site)}    # gsub removes them
        if ($i ~ /username/) user=$(++i)      # if field contains "username" string, the next field will hold the actual username
        if ($i ~ /password/) pass=$(++i)      # same - ++i is a bit safer than i+1 as it will increment i and thus skip the next field and not evaluate
       }
     }
     END {printf "%s\nUsername = %s\nPassword = %s\n", site, user, pass;}
    ' file

These 2 Users Gave Thanks to RudiC For This Post:
# 5  
Old 01-11-2013
Thank you Rudi! I really appreciate the breakdown. I'd much rather know how it works so I can try to do it myself next time Smilie!

---------- Post updated at 03:46 PM ---------- Previous update was at 01:13 PM ----------

Question about the code. Where can I find more information on the syntax used on these lines:

Code:
{if ($i ~ /^\(/)     {site=$i;
gsub(/\(|\)|:/,"",site)}

Specifically "/^\(/" and the "/\(|\)|:/,"",site"

Where can I find out what those symbols represent and how to use them? Also, do you have any suggetions for some good books to learn awk?

Last edited by Scrutinizer; 01-11-2013 at 04:50 PM.. Reason: code tags
# 6  
Old 01-11-2013
In this pattern we are escaping open round bracket ( and close round bracket ) because these meta-characters have special meaning.
  • / Opening pattern
  • \( Escaping (
  • | Represents OR
  • \( Escaping )
  • | Represents OR
  • : No need to escape this sign.
  • / Closing pattern
I hope you understood.
This User Gave Thanks to Yoda For This Post:
# 7  
Old 01-11-2013
Quote:
Originally Posted by bipinajith
In this pattern we are escaping open round bracket ( and close round bracket ) because these meta-characters have special meaning.
  • / Opening pattern
  • \( Escaping (
  • | Represents OR
  • \( Escaping )
  • | Represents OR
  • : No need to escape this sign.
  • / Closing pattern
I hope you understood.
Thank you! That's exactly what I was looking for! I'll go read up on awk's metacharacters.
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Need HELP with AWK split. Need to check for "special characters" in string before splitting the file

Hi Experts. I'm stuck with the below AWK code where i'm trying to move the records containing any special characters in the last field to a bad file. awk -F, '{if ($NF ~ /^|^/) print >"goodfile";else print >"badfile"}' filename sample data 1,abc,def,1234,A * 2,bed,dec,342,* A ... (6 Replies)
Discussion started by: shell_boy23
6 Replies

2. Shell Programming and Scripting

sed or awk command to replace a string pattern with another string based on position of this string

here is what i want to achieve... consider a file contains below contents. the file size is large about 60mb cat dump.sql INSERT INTO `table1` (`id`, `action`, `date`, `descrip`, `lastModified`) VALUES (1,'Change','2011-05-05 00:00:00','Account Updated','2012-02-10... (10 Replies)
Discussion started by: vivek d r
10 Replies

3. Shell Programming and Scripting

awk to split one field and print the last two fields within the split part.

Hello; I have a file consists of 4 columns separated by tab. The problem is the third fields. Some of the them are very long but can be split by the vertical bar "|". Also some of them do not contain the string "UniProt", but I could ignore it at this moment, and sort the file afterwards. Here is... (5 Replies)
Discussion started by: yifangt
5 Replies

4. Shell Programming and Scripting

split string

Hi I am facing a problem in spitting a string. Here is the string -------------------- subject1=10;subject2=30;subject3=40;subjectcode=10001;... Now, I want only marks not the subject code. (there can be 'n' subjects) ie. 10 30 40 My doubt ---------- How do I... (5 Replies)
Discussion started by: jionnet
5 Replies

5. Shell Programming and Scripting

awk to split string

Hello Friends, Im trying to split a string. When i use first method of awk like below i have an error: method1 (I specified the FS as ":" so is this wrong?) servert1{root}>awk -f split.txt awk: syntax error near line 2 awk: bailing out near line 2 split.txt:... (5 Replies)
Discussion started by: EAGL€
5 Replies

6. Shell Programming and Scripting

awk: split a file using a string problems

Hi, if i use this code awk '/String/{n++}{print > f n}' f=file input I get "input" splited this way file1 String 1515 1354 2356 file 2 String 4531 0345 5345 (3 Replies)
Discussion started by: aloctavodia
3 Replies

7. Shell Programming and Scripting

split the string

I need to split the string msu1_2 It should be generic for any string of the form msu<digits>_<digits> so that i get $X =1 and $Y = 2 Please help Thanks (5 Replies)
Discussion started by: asth
5 Replies

8. Shell Programming and Scripting

Split A String

Hi, I am new to scripting and need help splitting a string using space as the delimiter. How can I do that? I want the result to be stored in an Array. I tried using set -A arr $(echo $FILE) echo $arr The result of the above was ''. Thanks. (2 Replies)
Discussion started by: newbie187
2 Replies

9. UNIX for Dummies Questions & Answers

Split a file with no pattern -- Split, Csplit, Awk

I have gone through all the threads in the forum and tested out different things. I am trying to split a 3GB file into multiple files. Some files are even larger than this. For example: split -l 3000000 filename.txt This is very slow and it splits the file with 3 million records in each... (10 Replies)
Discussion started by: madhunk
10 Replies

10. Shell Programming and Scripting

split a string

Hi I have a script that loops though lines of a file and reads each line in to a variable ($LINE). I want to look at the line and split it into it's constituent parts. e.g. a line might be "This is a string" I want to then have variables set to each element thus: A=This B=is C=a... (3 Replies)
Discussion started by: gazingdown
3 Replies
Login or Register to Ask a Question