Basic awk help


 
Thread Tools Search this Thread
Top Forums UNIX for Dummies Questions & Answers Basic awk help
# 1  
Old 03-03-2013
Basic awk help

Im sure this is an easy question, but Ive tried and tried to get this to print all on one line and cant figure out why its not, so maybe someone can help
Code:
awk '/AP/{sub(/:80/, "", $4);printf $4"\t"} /User-Agent/{sub(/^[^:][^:]*:/,"");print};sub(/\.80/,"", $4);/Host/{sub(/^[^:][^:]*:/,""); print}'

What this prints is "AP" then "User-Agent" on one line(good) but then prints "HOST" on a separate line? How can I get all three to print on a single line?

Thanks!Smilie

Last edited by sectech; 03-03-2013 at 01:58 AM..
# 2  
Old 03-03-2013
Without knowing your input:
Code:
awk '/AP/{sub(/:80/, "", $4);printf $4"\t"} /User-Agent/{sub(/^[^:][^:]*:/,"");printf $0}/Host/{sub(/^[^:][^:]*:/,"");print}'

This User Gave Thanks to elixir_sinari For This Post:
# 3  
Old 03-03-2013
Thank you!!

Quote:
Originally Posted by elixir_sinari
Without knowing your input:
Code:
awk '/AP/{sub(/:80/, "", $4);printf $4"\t"} /User-Agent/{sub(/^[^:][^:]*:/,"");printf $0}/Host/{sub(/^[^:][^:]*:/,"");print}'

That did it!!! I dont know why I never tried that! Ugh banging my head!!
# 4  
Old 03-03-2013
Each call to print produces a <newline> character in the output. You could try something like:
Code:
awk '/AP/{sub(/:80/, "", $4);printf "%s\t", $4} /User-Agent/{sub(/^[^:][^:]*:/,"");printf "%s\t", $0}/Host/{sub(/^[^:][^:]*:/,"");print}'

Note that I changed the format string for your printf call. If $4 happens to contain any % or \ characters, you could end up with output significantly different from what you might expect; using the format %s avoids that possibility.
# 5  
Old 03-03-2013
Strange format now

Quote:
Originally Posted by Don Cragun
Each call to print produces a <newline> character in the output. You could try something like:
Code:
awk '/AP/{sub(/:80/, "", $4);printf "%s\t", $4} /User-Agent/{sub(/^[^:][^:]*:/,"");printf "%s\t", $0}/Host/{sub(/^[^:][^:]*:/,"");print}'

Note that I changed the format string for your printf call. If $4 happens to contain any % or \ characters, you could end up with output significantly different from what you might expect; using the format %s avoids that possibility.
Thanks! So I tried the last one and man the output format is kind of jacked up Smilie
Code:
8.254.53.254     Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.1; WOW64; Trident/5.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0).      mi.adinterax.com.
98.139.200.238   Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.1; WOW64; Trident/5    68.142.250.161   Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.1; WOW64; Trident/5.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0).      ads.yimg.com.

Sometimes it appends other lines and sometimes it throws the the first AP in at the end or middle?

This is what it looks like if I go back to my original awk commands.
Code:
63.241.108.124   Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.1; WOW64; Trident/5.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0).
 bs.serving-sys.com.
8.254.53.254     Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.1; WOW64; Trident/5.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0).
 mi.adinterax.com.

I will play with this and see what I can figure out...it seems like its combining or adding the User-Agent field. strange output. Thanks again for your help!

---------- Post updated at 03:50 PM ---------- Previous update was at 01:25 AM ----------

Is there anyway to force this to always print the "/AP/" field first?
Code:
 awk '/AP/{sub(/:80/, "", $4);printf "%s\t", $4} /User-Agent/{sub(/^[^:][^:]*:/,"");printf "%s\t", $0}/Host/{sub(/^[^:][^:]*:/,"");sub(/\.80/,"", $4);print}'

Right now it does maybe 50% sometimes it ends up after the User-Agent. There must be something causing this but I cannot figure it out?

Thanks!!

Last edited by sectech; 03-03-2013 at 02:40 AM..
# 6  
Old 03-03-2013
Quote:
Originally Posted by sectech
Thanks! So I tried the last one and man the output format is kind of jacked up Smilie
Code:
8.254.53.254     Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.1; WOW64; Trident/5.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0).      mi.adinterax.com.
98.139.200.238   Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.1; WOW64; Trident/5    68.142.250.161   Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.1; WOW64; Trident/5.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0).      ads.yimg.com.

Sometimes it appends other lines and sometimes it throws the the first AP in at the end or middle?

This is what it looks like if I go back to my original awk commands.
Code:
63.241.108.124   Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.1; WOW64; Trident/5.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0).
 bs.serving-sys.com.
8.254.53.254     Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.1; WOW64; Trident/5.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0).
 mi.adinterax.com.

I will play with this and see what I can figure out...it seems like its combining or adding the User-Agent field. strange output. Thanks again for your help!

---------- Post updated at 03:50 PM ---------- Previous update was at 01:25 AM ----------

Is there anyway to force this to always print the "/AP/" field first?
Code:
 awk '/AP/{sub(/:80/, "", $4);printf "%s\t", $4} /User-Agent/{sub(/^[^:][^:]*:/,"");printf "%s\t", $0}/Host/{sub(/^[^:][^:]*:/,"");sub(/\.80/,"", $4);print}'

Right now it does maybe 50% sometimes it ends up after the User-Agent. There must be something causing this but I cannot figure it out?

Thanks!!
What do you mean you cannot figure it out? It is obvious! With the code suggested for handling your problem, it is simple: when the string AP appears first in your input file, it will appear first in your output file.

Is there something in your input file that can be used to indicate that it is the first line of a set or the last line of a set?

Will it always be true that there is exactly one line containing the string AP, one line containing the string User-Agent, and one line containing the string Host in a group of lines to be treated as a set?

You know what your input looks like, you have left the rest of us guessing.
# 7  
Old 03-03-2013
input code

Found that User-Agent and Host are switching spots when using different browsers!?!!?!?!?!?!! See below and let me know if there is a solution to these variations.
When IE does the request:
Code:
T 12.237.222.221:57578 -> 70.37.131.11:80 [AP]
GET /c.gif?clid=3A8ACC9C01566FAB3DA8C8E105566FE0%26TUID%3D1&rid=CDA92F497C2347BEBCAD28088CA12CB7&cts=1362349841890&evt=unload HTTP/1.1.
Accept: image/png, image/svg+xml, image/*;q=0.8, */*;q=0.5.
Referer: http://photos.msn.com/browse/places.
Accept-Language: en-US.
User-Agent: Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; WOW64; Trident/5.0).
Accept-Encoding: gzip, deflate.
Host: udc.msn.com.
DNT: 1.
Connection: Keep-Alive.
Cookie:

Chrome and Safari/Iphone/Android
Code:
T 12.237.222.221:57991 -> 4.59.125.171:80 [AP]
GET /unix-dummies-questions-answers/217117-basic-awk-help.html HTTP/1.1.
Host: www.unix.com.
Connection: keep-alive.
Cache-Control: max-age=0.
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8.
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.22 (KHTML, like Gecko) Chrome/25.0.1364.97 Safari/537.22.
Referer: https://www.unix.com/unix-dummies-questions-answers/217117-basic-awk-help.html
Accept-Encoding: gzip,deflate,sdch.
Accept-Language: en-GB,en-US;q=0.8,en;q=0.6.
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.3.
Cookie:

Is there any way to be strict about always printing HOST first even if its after User-Agent? or vice-versa

Last edited by sectech; 03-03-2013 at 06:39 PM.. Reason: updated with more info.
 
Login or Register to Ask a Question

Previous Thread | Next Thread

8 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Confusing of some basic awk

1. increase file space first, double space a file: awk '1;{print ""}' I probably can understand it:print a blank line every time.But when I read triple space a file I am confused: awk '1;{print "\n"}' doesn't it meaning print a blank line every time too? 2. number each line of file, but... (6 Replies)
Discussion started by: hhdzhu
6 Replies

2. UNIX for Dummies Questions & Answers

Basic arithmetic operation with awk?

input: Name|Operation rec_10|1+2+2- Output: rec_10|1 Basically I am trying to calculate the result of "the path" in $3 where the operators follow the number and not preceding them like we normally do: rec_10: +1+2-2=1 But I realise (I am sure there is a good reason for that) that awk... (7 Replies)
Discussion started by: beca123456
7 Replies

3. UNIX for Dummies Questions & Answers

Basic awk...newbie quetion

Hi, I was trying to change the value of the 4th column (put '1' in the 4th column of each row). My awk command is: awk -F, '{$3=1;}1' OFS= input.txt > ./test_out.txt My input file is: a 1 2 31 b 4 5 61 c 7 8 91 My output file (test_out.txt)is: a 1 2 31 b 4 5 61 c 7 8 91 What... (4 Replies)
Discussion started by: pc2001
4 Replies

4. UNIX for Dummies Questions & Answers

Basic loop awk/shell script question..

Hi, Sorry if this is a newbie question. I guess you can use either awk or shell script for this sequence of operations, but knowing very little about either of them I'm not sure how I should try to write this. The basic objective is to copy certain files that are scattered all over my... (10 Replies)
Discussion started by: pc2001
10 Replies

5. Shell Programming and Scripting

Issue with basic Awk script

Here's a basic awk program I am trying to run. It shows no error but shows no result either too. If someone can look up and tell me what's wrong I will be obliged. Thanks. :) Code Snippet. #!/bin/bash awk '{ for (i = 1 ; i <= 3 ; i++) for ( j = 1 ; j <= 3 ; j++ ) { ... (2 Replies)
Discussion started by: mr.amitkc
2 Replies

6. Shell Programming and Scripting

basic awk questions

I find an script with awk sitting around. I went through some online manuals, but I can't figure out exactly how it works. I can't post the whole program. Not allowed. This is the line that is confusing me. I get when else is in the script grep -v "^REM " $1| grep -v "JUNK;" | awk -F" "... (2 Replies)
Discussion started by: guessingo
2 Replies

7. UNIX for Dummies Questions & Answers

Basic awk question...getting awk to act on $1 of the command itself

I have a script problem that I am not able to solve due my very limited understanding of unix/awk. This is the contents of test.sh awk '{print $1}' From the prompt if I enter: ./test.sh Hello World I would expect to see "Hello" but all I get is a blank line. Only then if I enter "Hello... (2 Replies)
Discussion started by: JasonHamm
2 Replies

8. Shell Programming and Scripting

awk basic issue

Hi all, I have an awk basic question. file.text Our Location: Our home has light yellow siding, and is a duplex on Main Street, across from the High School, and across the lane from the Health Center If I run: cat file.txt | awk '{print $2}' | grep... (7 Replies)
Discussion started by: research3
7 Replies
Login or Register to Ask a Question