Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Extracting key/value pairs in awk

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Extracting key/value pairs in awk
# 1  
Old 08-06-2013
Extracting key/value pairs in awk
I am extracting a number of key/value pairs in awk using following:
Code:
 
awk '
/xyz_session_id/ {
        n=index($0,"xyz_session_id");
        id=substr($0,n+15,25);
        a[id]=$4;
}
END{
        for (ix in a) { print a[ix] }
}'

I don't like this Index + substr with manually calculated length construct - is there a good techinque to do it?

Sample of my input:
Code:
69.25.74.121 - - [30/Jul/2013:12:09:44 -0700] "GET /B/?client_id=628637&confirmationNumber=7302588&fare=%242%2e50&ivr_phone=6262581325&xyz_session_id/=200032%3b005%3b1375211171&req=1248%3f&sch=0105PM HTTP/1.1" 200 182
69.25.74.121 - - [30/Jul/2013:12:09:44 -0700] "GET /B/?client_id=628637&xyz_session_id/=200032%3b005%3b1375211171&keyword=XYZ&message=Confirmation%20number%3a%207302589 HTTP/1.1" 200 62
69.25.74.121 - - [30/Jul/2013:12:09:53 -0700] "GET /B/?client_id=628637&xyz_session_id/=200032%3b005%3b1375211171&keyword=XYZ&message=Ask%20if%20client%20wants%20to%20hear%20confirmation%20number%20one%20more%20time HTTP/1.1" 200 62
69.25.74.121 - - [30/Jul/2013:12:10:00 -0700] "GET /B/?client_id=628637&xyz_session_id/=200032%3b005%3b1375211171&keyword=Client&message=Client%20selected%20to%20disconnect HTTP/1.1" 200 62
69.25.74.121 - - [30/Jul/2013:12:10:00 -0700] "GET /B/?xyz_session_id/=200032%3b005%3b1375211171 HTTP/1.1" 200 62
69.25.74.121 - - [30/Jul/2013:12:10:01 -0700] "GET /B/?client_id=628637&xyz_session_id/=200032%3b005%3b1375211171&keyword=XYZ&message=Disconnect%20Event%2c%20Time%20Elapsed%20%3d%20210%2e097%20seconds HTTP/1.1" 200 62

I run sed 's/\[//;s/\]//;s/\?/ /' $1 | awk -f my-code-above
my output:
Code:
 
30/Jul/2013:12:10:01
30/Jul/2013:12:10:04

I do need to show a couple more values, such as phone etc., I'd hate to code another index + substr.

Ideal for me would be to have array of keys defined, such as :
Code:
 
BEGIN {
 arr[1] = "key1";
 arr[2] = "key2";
... etc ...
}

Then in processing each line I'd cycle through my arr of keys and get each key's length, find its position, position + lenght gives me beginning of the value for the key and so on. Is it doable?

Would anyone help in putting pieces together? Or some other approach?
# 2  
Old 08-06-2013
Can you please post what is the expected output from that 6 line input file?
# 3  
Old 08-06-2013
I have shown it there, see "my output", the sample outputs just date/time, but as I said I will need to output several other key values
# 4  
Old 08-06-2013
Your sample output is not clear:
Code:
30/Jul/2013:12:10:01
30/Jul/2013:12:10:04

Because I don't see the highlighted value in your input file.

By the way if you are interested in just another approach to extract required fields. I would suggest using match function. Here is an example:
Code:
awk '
        /xyz_session_id/ {
                match ( $0, /[0-9]+\/[a-zA-Z]+\/[0-9]+:[0-9]+:[0-9]+/ )
                dt = substr ( $0, RSTART, RLENGTH )
                match ( $0, /xyz_session_id\/[^& ]*/ )
                id =  substr ( $0, RSTART, RLENGTH )
                sub ( /xyz_session_id.*=/, X, id )
                print dt, id
        }
' file

# 5  
Old 08-06-2013
Thanks, this match looks very interesting.
I just found that gawk has a great extension to allow array as a parameter to match, very nice feature.
Thanks again.
Last edited by migurus; 08-06-2013 at 05:33 PM.. Reason: found more info
# 6  
Old 08-07-2013
Speaking of gawk's extended match this is one way for that:
Code:
#!/usr/bin/gawk -f

{
    if (match($0, /^[^\[]+\[([^ ]+).*xyz_session_id\/=([^ &]+)/, t)) {
        k = t[1]; v = t[2]
        a[k] = v
    }
}

END {
    for (i in a) {
        print i " : " a[i]
    }
}

And you can add other parameters like a[k, "keywords"] = keywords etc.
This User Gave Thanks to konsolebox For This Post:
migurus
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Extracting data between two tag pairs

In a huge log file (43MB, 43k lines) I am trying to extract data between two tag pairs on same line and export it to a file so I can pull it into Excel for a report. One Pair is <Text>data I need</Text> Other pair follows on same line and is <TimeStamp>more data I need</TimeStamp> I would need... (2 Replies)
Discussion started by: NanookArctic
2 Replies

2. Shell Programming and Scripting

Query related to the SSH key pairs

Hi All, I have question , How to generate the SSH keys for the two servers in the pair. So if the server will be replicated , same SSH key pairs will work on each server. No need to generate the SSH keys for the second server. I have created the SSH keys for the single server. I was... (1 Reply)
Discussion started by: sharsour
1 Replies

3. Shell Programming and Scripting

Looping through pairs of files with awk

Hi all, please help me construct the command. i want to loop through all files named bam* and bed*. My awk works for a particular pair but there are too many pairs to do manually. I have generated multiple files in a folder in a given pattern. The files are named like bam_fixed1.bam... (2 Replies)
Discussion started by: newbie83
2 Replies

4. UNIX for Advanced & Expert Users

Parse (delimited string) key-value pairs in a column into separate lines

Hi experts, e.g. i/p data looks like 0000xm7zcNDIkP888vRqGv93xA7:176n00qql||9700005405552747,9700005405717924,9700005405733788|unidentified,unidentified,unidentified|| o/p data should like - row1: 0000xm7zcNDIkP888vRqGv93xA7:176n00qql||9700005405552747|unidentified ... (1 Reply)
Discussion started by: sumoka
1 Replies

5. Shell Programming and Scripting

extracting non-zero pairs of numbers from each row

Hi all, I do have a tab delimited file a1 a2 b1 b2 c1 c2 d1 d2 e1 e2 f1 f2 0 0 123 546 0 0 0 0 0 0 0 0 0 0 345 456 765 890 902 1003 0 0 0 0 534 768 0 0 0 0 0 0 0 0 0 0 0 0 0 0 456 765 0 0 0 0 0 0 0 0 0 0 0 0 12 102 0 0 0 0 456 578 789 1003 678 765 345 400 801 1003 134 765... (5 Replies)
Discussion started by: Lucky Ali
5 Replies

6. Shell Programming and Scripting

extracting KEY from a line

consider a line from a file is stored in a variable called $var i.e. var=" KEY `controllingZoneId` (`controllingZoneId`)," or it can be (i.e without comma at the end) var="KEY `requestingUserId` (`requestingUserId`)" here i need to extract KEY and 1st and 2nd word i want to know how... (7 Replies)
Discussion started by: vivek d r
7 Replies

7. UNIX for Dummies Questions & Answers

Extracting a Private key from a keystore?

Hi everyone! I know you can extract public keys from a keystore using the keytool command. But what is the process to extract a private key from a jks keystore and import into another jks keystore using keytool? Any guidance would be greatly appreciated! I can't seem to find anything, I do... (0 Replies)
Discussion started by: Keepcase
0 Replies

8. Shell Programming and Scripting

Searching a delimited Key value pairs in shell script

Hello, I have property file with key value pairs separated by pipe , I am trying to write a script which reads the property file and search and print value of specific key. I tried with Sed, I am successfull. The file is as follows ... (4 Replies)
Discussion started by: ANK
4 Replies

9. Shell Programming and Scripting

Matching Pairs with AWK or Shell

Hello, I have a file Pairs.txt composed of 3 columns. For simplicity, this is the file: A B C B D W X Y Z Z W K B A C Y J Q F M P Y X Z I am trying to reduce the file such that I only keep the first occurrence of any "pair". In the example above, I would only want to keep one... (6 Replies)
Discussion started by: InfoSeeker
6 Replies

10. Shell Programming and Scripting

Parsing /proc/net/dev into key:value pairs (self-answered)

Hi all, I need some help with using sed/awk/other linux tools to meet the following goal: I'm trying to take the output of /proc/net/dev: Inter-| Receive | Transmit face |bytes packets errs drop fifo frame compressed multicast|bytes ... (0 Replies)
Discussion started by: felbane
0 Replies
Login or Register to Ask a Question