Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

A script needs to be created to collect all HTTP GET requests containing a particular string

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting A script needs to be created to collect all HTTP GET requests containing a particular string
# 1  
Old 07-23-2012
A script needs to be created to collect all HTTP GET requests containing a particular string
Hi friends,

A script needs to be created to collect all HTTP GET requests containing a particular string say abcd.gif in the url path along with the IP address of the client that issued the request. The source of this data is the web server logs. Also Each script execution should extract client IP address and timestamp and record this either in a file or in a DB.
Logs are like
Code:
10.252.33.251 - - [13/Jul/2012:05:17:46 -0400] "GET /keepalive.html HTTP/1.1" 200 299 
10.254.17.140 - - [13/Jul/2012:05:17:48 -0400] "GET /webapp/wcs/stores/servlet/KioskGiftRegistryMainView?catalogId=10051&langId=-1&storeId=10151 HTTP/1.1" 200 7881 
................................... 
10.252.33.251 - - [13/Jul/2012:05:20:46 -0400] "GET /keepalive.html HTTP/1.1" 200 299 
10.254.17.140 - - [13/Jul/2012:05:20:49 -0400] "GET /wcsstore/GiftRegistryStorefrontAssetStore/KioskArea/images/abcd.gif?1342171249161 HTTP/1.1" 200 799 
10.252.33.252 - - [13/Jul/2012:05:20:50 -0400] "GET /keepalive.html HTTP/1.1" 200 299 
Here we need to capture 
10.254.17.140 - - [13/Jul/2012:05:20:49 -0400] "GET /wcsstore/GiftRegistryStorefrontAssetStore/KioskArea/images/abcd.gif?1342171249161 HTTP/1.1" 200 799

and capture IP address 10.254.17.140 and timestamp 13/Jul/2012:05:20:49 in a file

Any help will be greatly appreciated
Regards,
Surendra


Moderator's Comments:
Mod Comment Please use code tags next time for your code and data.
Last edited by zaxxon; 07-23-2012 at 10:54 AM.. Reason: code tags, see PM
# 2  
Old 07-23-2012
Code:
 awk '/abcd.gif/{print $1$4}' access_log | tr "[" " " >> yourlogfile

# 3  
Old 07-23-2012
That is nothing AIX related, moving.
# 4  
Old 07-23-2012
Have to include with HTTP/1.1" 200
Hi,

Thanks for your response,

Here I want to add 2 more conditions in my extract, I need to extract the lines with abcd.gif which contains HTTP GET requests, so I need to add strings HTTP and GET in conditions.
Apart from this it should be 200 request, sometimes in access logs it also comes as HTTP/1.1" 404, so effectively we have conditions to include abcd.gif, HTTP, GET, 200

Code:
10.254.17.139 - - [09/Jul/2012:09:53:03 -0400] "GET /wcsstore/GiftRegistryStorefrontAssetStore/KioskArea/images/abcd.gif?1341841982900 HTTP/1.1" 200 799
10.252.33.252 - - [09/Jul/2012:09:53:06 -0400] "GET /keepalive.html HTTP/1.1" 200 299
10.254.17.139 - - [09/Jul/2012:09:53:09 -0400] "GET /wcsstore/GiftRegistryStorefrontAssetStore/KioskArea/images/abcd.gif?1341841988900 HTTP/1.1" 200 799
10.252.33.252 - - [09/Jul/2012:09:53:11 -0400] "GET /keepalive.html HTTP/1.1" 200 299
10.254.17.139 - - [09/Jul/2012:09:53:15 -0400] "GET /wcsstore/GiftRegistryStorefrontAssetStore/KioskArea/images/abcd.gif?
1341841994901 HTTP/1.1" 200 799

Thanks again.

Regards,
Surendra
Last edited by Scott; 07-23-2012 at 12:20 PM.. Reason: Code tags
# 5  
Old 07-23-2012
That's only a trivial modification of what you've been given, you know. Have you tried anything yourself?

Code:
awk '/HTTP/ && /GET/ && /abcd[.]gif/ { print $1$4 }' access.log | tr "[" " "

Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Expect script to collect logs in expert mode

I am trying to collect logs from a system in expert mode. Where I login to system and then change mode to expert mode and issue command to collect the log. I have a script which works but It waits for 30 mins even though log collection is complete before that. I have two issues with existing... (4 Replies)
Discussion started by: shunya
4 Replies

2. Shell Programming and Scripting

awk script to find time difference between HTTP PUT and HTTP DELETE requests in access.log

Hi, I'm trying to write a script to determine the time gap between HTTP PUT and HTTP DELETE requests in the HTTP Servers access log. Normally client will do HTTP PUT to push content e.g. file_1.txt and 21 seconds later it will do HTTP DELETE, but sometimes the time varies causing some issues... (3 Replies)
Discussion started by: Juha
3 Replies

3. UNIX for Dummies Questions & Answers

Script to collect log files in case of server crash

Environmnet: HP-UX B.11.31 U ia64 RDBMS: Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production Question: When server hangs or node evicts, we open up tickets with Oracle Support and Oracle Support ask for some list of log files. Currently we can not use the... (2 Replies)
Discussion started by: Siva SQL
2 Replies

4. UNIX for Dummies Questions & Answers

Shell script to collect inbound connections

Hi Kindly provide me shell script on server to collect inbound connections, using port number on Linux. (4 Replies)
Discussion started by: adminhelp
4 Replies

5. Shell Programming and Scripting

Korn script to access multiple host and collect data

Hi, I'm newbie in Korn scripting as well as in Unix. I need to have a script to do the following. 1) ssh multiple hosts -In my case, login credentials are not an issue as all keys authorized by default. Example of host IP (24.35.14.6, 24.35.14.7, 24.35.14.8, 24.35.14.9, ... (2 Replies)
Discussion started by: blinkingdan
2 Replies

6. Shell Programming and Scripting

Looking for script to collect SAN usage

My management is looking for me to sweep my Unix servers and report on (1) how much SAN space has been allocated to each server and (2) how much is used (defined as having a file on those bits). Basically stuff one can get from "df", but they just want SAN totals (not local disks). NAS is exempt. ... (1 Reply)
Discussion started by: MikJik
1 Replies

7. Shell Programming and Scripting

script to collect all db information

hi all I am looking for shell script ,by which i need to gather all the DB information in Sybase server . Thanks in advance :) (3 Replies)
Discussion started by: mvsramarao
3 Replies

8. Shell Programming and Scripting

How do i collect Date & Time from Different Directories in a script

How do i collect Date & Time from Different Directories in a script The script iam using for a single directory is : ls -l | grep awk '{print $8}' (2 Replies)
Discussion started by: laknar
2 Replies

9. UNIX for Dummies Questions & Answers

Need to log http requests

Hi folks, I am trying to build this GUI application that will perform some http requests to a specific server. Basically I will use curl to "pretend" that it is a browser a sending an http request from a form. My http knowledge is very low, and the site is tricky, but I think if I could... (3 Replies)
Discussion started by: fundidor
3 Replies

10. UNIX for Dummies Questions & Answers

turning off certain http requests

On a sparc solaris 8 host running sunone webserver 6 I would like to limit the http requests that can be used when port 80 is accessed. We currently have http/1.0 enabled. For example I would like to remove the http request DELETE. Regards, BLP (1 Reply)
Discussion started by: blp001
1 Replies
Login or Register to Ask a Question