Login or Register to Ask a Question and Join Our Community

Sponsored Content
Full Discussion: Extracting key/value pairs in awk
Top Forums Shell Programming and Scripting Extracting key/value pairs in awk Post 302840845 by migurus on Tuesday 6th of August 2013 03:30:18 PM
Old 08-06-2013
Extracting key/value pairs in awk
I am extracting a number of key/value pairs in awk using following:
Code:
 
awk '
/xyz_session_id/ {
        n=index($0,"xyz_session_id");
        id=substr($0,n+15,25);
        a[id]=$4;
}
END{
        for (ix in a) { print a[ix] }
}'

I don't like this Index + substr with manually calculated length construct - is there a good techinque to do it?

Sample of my input:
Code:
69.25.74.121 - - [30/Jul/2013:12:09:44 -0700] "GET /B/?client_id=628637&confirmationNumber=7302588&fare=%242%2e50&ivr_phone=6262581325&xyz_session_id/=200032%3b005%3b1375211171&req=1248%3f&sch=0105PM HTTP/1.1" 200 182
69.25.74.121 - - [30/Jul/2013:12:09:44 -0700] "GET /B/?client_id=628637&xyz_session_id/=200032%3b005%3b1375211171&keyword=XYZ&message=Confirmation%20number%3a%207302589 HTTP/1.1" 200 62
69.25.74.121 - - [30/Jul/2013:12:09:53 -0700] "GET /B/?client_id=628637&xyz_session_id/=200032%3b005%3b1375211171&keyword=XYZ&message=Ask%20if%20client%20wants%20to%20hear%20confirmation%20number%20one%20more%20time HTTP/1.1" 200 62
69.25.74.121 - - [30/Jul/2013:12:10:00 -0700] "GET /B/?client_id=628637&xyz_session_id/=200032%3b005%3b1375211171&keyword=Client&message=Client%20selected%20to%20disconnect HTTP/1.1" 200 62
69.25.74.121 - - [30/Jul/2013:12:10:00 -0700] "GET /B/?xyz_session_id/=200032%3b005%3b1375211171 HTTP/1.1" 200 62
69.25.74.121 - - [30/Jul/2013:12:10:01 -0700] "GET /B/?client_id=628637&xyz_session_id/=200032%3b005%3b1375211171&keyword=XYZ&message=Disconnect%20Event%2c%20Time%20Elapsed%20%3d%20210%2e097%20seconds HTTP/1.1" 200 62

I run sed 's/\[//;s/\]//;s/\?/ /' $1 | awk -f my-code-above
my output:
Code:
 
30/Jul/2013:12:10:01
30/Jul/2013:12:10:04

I do need to show a couple more values, such as phone etc., I'd hate to code another index + substr.

Ideal for me would be to have array of keys defined, such as :
Code:
 
BEGIN {
 arr[1] = "key1";
 arr[2] = "key2";
... etc ...
}

Then in processing each line I'd cycle through my arr of keys and get each key's length, find its position, position + lenght gives me beginning of the value for the key and so on. Is it doable?

Would anyone help in putting pieces together? Or some other approach?
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Parsing /proc/net/dev into key:value pairs (self-answered)

Hi all, I need some help with using sed/awk/other linux tools to meet the following goal: I'm trying to take the output of /proc/net/dev: Inter-| Receive | Transmit face |bytes packets errs drop fifo frame compressed multicast|bytes ... (0 Replies)
Discussion started by: felbane
0 Replies

2. Shell Programming and Scripting

Matching Pairs with AWK or Shell

Hello, I have a file Pairs.txt composed of 3 columns. For simplicity, this is the file: A B C B D W X Y Z Z W K B A C Y J Q F M P Y X Z I am trying to reduce the file such that I only keep the first occurrence of any "pair". In the example above, I would only want to keep one... (6 Replies)
Discussion started by: InfoSeeker
6 Replies

3. Shell Programming and Scripting

Searching a delimited Key value pairs in shell script

Hello, I have property file with key value pairs separated by pipe , I am trying to write a script which reads the property file and search and print value of specific key. I tried with Sed, I am successfull. The file is as follows ... (4 Replies)
Discussion started by: ANK
4 Replies

4. UNIX for Dummies Questions & Answers

Extracting a Private key from a keystore?

Hi everyone! I know you can extract public keys from a keystore using the keytool command. But what is the process to extract a private key from a jks keystore and import into another jks keystore using keytool? Any guidance would be greatly appreciated! I can't seem to find anything, I do... (0 Replies)
Discussion started by: Keepcase
0 Replies

5. Shell Programming and Scripting

extracting KEY from a line

consider a line from a file is stored in a variable called $var i.e. var=" KEY `controllingZoneId` (`controllingZoneId`)," or it can be (i.e without comma at the end) var="KEY `requestingUserId` (`requestingUserId`)" here i need to extract KEY and 1st and 2nd word i want to know how... (7 Replies)
Discussion started by: vivek d r
7 Replies

6. Shell Programming and Scripting

extracting non-zero pairs of numbers from each row

Hi all, I do have a tab delimited file a1 a2 b1 b2 c1 c2 d1 d2 e1 e2 f1 f2 0 0 123 546 0 0 0 0 0 0 0 0 0 0 345 456 765 890 902 1003 0 0 0 0 534 768 0 0 0 0 0 0 0 0 0 0 0 0 0 0 456 765 0 0 0 0 0 0 0 0 0 0 0 0 12 102 0 0 0 0 456 578 789 1003 678 765 345 400 801 1003 134 765... (5 Replies)
Discussion started by: Lucky Ali
5 Replies

7. UNIX for Advanced & Expert Users

Parse (delimited string) key-value pairs in a column into separate lines

Hi experts, e.g. i/p data looks like 0000xm7zcNDIkP888vRqGv93xA7:176n00qql||9700005405552747,9700005405717924,9700005405733788|unidentified,unidentified,unidentified|| o/p data should like - row1: 0000xm7zcNDIkP888vRqGv93xA7:176n00qql||9700005405552747|unidentified ... (1 Reply)
Discussion started by: sumoka
1 Replies

8. Shell Programming and Scripting

Looping through pairs of files with awk

Hi all, please help me construct the command. i want to loop through all files named bam* and bed*. My awk works for a particular pair but there are too many pairs to do manually. I have generated multiple files in a folder in a given pattern. The files are named like bam_fixed1.bam... (2 Replies)
Discussion started by: newbie83
2 Replies

9. Shell Programming and Scripting

Query related to the SSH key pairs

Hi All, I have question , How to generate the SSH keys for the two servers in the pair. So if the server will be replicated , same SSH key pairs will work on each server. No need to generate the SSH keys for the second server. I have created the SSH keys for the single server. I was... (1 Reply)
Discussion started by: sharsour
1 Replies

10. Shell Programming and Scripting

Extracting data between two tag pairs

In a huge log file (43MB, 43k lines) I am trying to extract data between two tag pairs on same line and export it to a file so I can pull it into Excel for a report. One Pair is <Text>data I need</Text> Other pair follows on same line and is <TimeStamp>more data I need</TimeStamp> I would need... (2 Replies)
Discussion started by: NanookArctic
2 Replies

LEARN ABOUT REDHAT

igawk

IGAWK(1)							 Utility Commands							  IGAWK(1)

NAME

       igawk - gawk with include files

SYNOPSIS

       igawk [ all gawk options ] -f program-file [ -- ] file ...
       igawk [ all gawk options ] [ -- ] program-text file ...

DESCRIPTION

       Igawk is a simple shell script that adds the ability to have ``include files'' to gawk(1).

       AWK programs for igawk are the same as for gawk, except that, in addition, you may have lines like

	      @include getopt.awk

       in your program to include the file getopt.awk from either the current directory or one of the other directories in the search path.

OPTIONS

       See gawk(1) for a full description of the AWK language and the options that gawk supports.

EXAMPLES

       cat << EOF > test.awk
       @include getopt.awk

       BEGIN {
	    while (getopt(ARGC, ARGV, "am:q") != -1)
		 ...
       }
       EOF

       igawk -f test.awk

SEE ALSO

       gawk(1)

       Effective AWK Programming, Edition 1.0, published by the Free Software Foundation, 1995.

AUTHOR

       Arnold Robbins (arnold@skeeve.com).

Free Software Foundation					    Nov 3 1999								  IGAWK(1)
All times are GMT -4. The time now is 10:49 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy