Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Extract various information from a log file

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Extract various information from a log file
# 1  
Old 09-26-2011
Extract various information from a log file
Hye ShamRock

If you can help me with this difficult task for me then it will save my day


Logs :

==================================================================================================== ==============

Quote:
--f42e2544-A--
[26/Sep/2011:16:03:13 +0100] ToCUMdXlTpYAACTqNMsAAAAO 80.33.86.223 53424 91.186.30.249 80
--f42e2544-B--
GET /im/qs_menu.php?text=Contact%20Us&bt_img=bt_contact HTTP/1.1
Accept: */*
Referer: http://www.domainname.com/
Accept-Language: en-GB
User-Agent: Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; Trident/4.0; GTB7.1; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; .NET4.0C; InfoPath.1)
Accept-Encoding: gzip, deflate
Host: www.domainname.com
Connection: Keep-Alive
Cookie: PHPSESSID=f933fb642e1c3e258b7c9787b49d2408; lang=en

--f42e2544-F--
HTTP/1.1 406 Not Acceptable
Content-Length: 384
Keep-Alive: timeout=5, max=96
Connection: Keep-Alive
Content-Type: text/html; charset=iso-8859-1

--f42e2544-H--
Message: Access denied with code 406 (phase 2). Pattern match "_img|amature-big-titties|amature-big-titties|avril-laveign-porn|breast-touch-video|gingers-having-sex|naked-indian-models" at REQUEST_URI. [file "/usr/local/apache/conf/modsec2.user.conf"] [line "109"] [id "950013"] [msg "PHP/FTP Injection Attack. Matched signature <_img>"] [severity "CRITICAL"]
Apache-Error: [file "core.c"] [line 3650] [level 3] File does not exist: /home/costadel/domains/domainname.com/public_html/406.shtml, referer: http://www.domainname.com/
Action: Intercepted (phase 2)
Stopwatch: 1317049393646593 1950 (402 1648 -)
Producer: ModSecurity for Apache/2.5.13 (http://www.modsecurity.org/).


--f42e2544-Z--

--2ed66772-A--
[26/Sep/2011:16:03:14 +0100] ToCUMtXlTpYAACTqNMwAAAAO 80.33.86.223 53424 91.186.30.249 80
--2ed66772-B--
GET /im/qs_menu.php?text=Map&bt_img=bt_map HTTP/1.1
Accept: */*
Referer: http://www.domainname.com/
Accept-Language: en-GB
User-Agent: Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; Trident/4.0; GTB7.1; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; .NET4.0C; InfoPath.1)
Accept-Encoding: gzip, deflate
Host: www.domainname.com
Connection: Keep-Alive
Cookie: PHPSESSID=f933fb642e1c3e258b7c9787b49d2408; lang=en

--2ed66772-F--
HTTP/1.1 406 Not Acceptable
Content-Length: 384
Keep-Alive: timeout=5, max=95
Connection: Keep-Alive
Content-Type: text/html; charset=iso-8859-1

--2ed66772-H--
Message: Access denied with code 406 (phase 2). Pattern match "_img|amature-big-titties|amature-big-titties|avril-laveign-porn|breast-touch-video|gingers-having-sex|naked-indian-models" at REQUEST_URI. [file "/usr/local/apache/conf/modsec2.user.conf"] [line "109"] [id "950013"] [msg "PHP/FTP Injection Attack. Matched signature <_img>"] [severity "CRITICAL"]
Apache-Error: [file "core.c"] [line 3650] [level 3] File does not exist: /home/costadel/domains/domainname.com/public_html/406.shtml, referer: http://www.domainname.com/
Action: Intercepted (phase 2)
Stopwatch: 1317049394307033 2032 (448 1733 -)
Producer: ModSecurity for Apache/2.5.13 (http://www.modsecurity.org/).
Server: Apache
============================================================================



This is modsecurity rules i need to add the rules ID mention in the logs for the particular domain and URL, but i am not able to write the script i am sure awk will help me here too which will give me domain name and the ID for which its got block with the URL

THANK in advance

Moderator's Comments:
Mod Comment This thread was split from another thread. SilvesterJ: Do not "hijack" others threads!
# 2  
Old 09-26-2011
So what is the output you expect.
# 3  
Old 09-26-2011
I need
Domain : URL : and ID

Like
DOMAIN :domainname.com
URL : /im/qs_menu.php
ID : 950013

Which are shown in read [id "950013"]

Smilie thanks for looking into it
# 4  
Old 09-26-2011
How should it know to tie the URL from an error three things up with the URL-less error three things down?

---------- Post updated at 01:08 PM ---------- Previous update was at 12:52 PM ----------

Here's something that sort of does it:

Code:
$ cat get.awk
BEGIN { RS="";  FS="\n" }

{
        split($1, L, "[-]*");

        if(L[2] != LAST)
        {
                if(ID)
                {
                        print "dom:", DOM;
                        print "url:", URL;
                        print "id:", ID;
                        printf("\n");
                }
                DOM=""; URL=""; ID=""

                LAST=L[2];
        }


        for(N=1; N<=NF; N++)
        {
                if($N ~ /referer: http:/)       NEWDOM=$N
                if($N ~ /\[id /)                NEWID=$N
                if($N ~ /^GET/)                 NEWURL=$N
        }

        if(NEWURL)
        {
                split(NEWURL, a, "[ ?]");
                NEWURL=a[2];
                URL=NEWURL
                NEWURL=""
        }

        if(NEWID)
        {
                # Id string will be in a[2]
                split(NEWID, a, "\\[id ");
                # Split on ], ", ' ' chars.
                split(a[2], a, "[\"\\] ]");
                NEWID=a[2];
                ID=NEWID;
                NEWID=""
        }

        if(NEWDOM)
        {
                # Extract everything after 'referer:'
                split(NEWDOM, a, "referer: ");  NEWDOM=a[2];
                # Reduce http://whatever/ to whatever
                sub(/http:\/\//, "", NEWDOM);
                sub(/\/$/, "", NEWDOM);
                # Turn www.whatever.com into www, whatever, com.
                N=split(NEWDOM, a, ".");
                # Paste the last two together.
                NEWDOM=a[N-1];  NEWDOM=NEWDOM "." a[N];
                DOM=NEWDOM
                NEWDOM=""
        }
}

END {
        if(ID)
        {
                print "dom:", DOM;
                print "url:", URL;
                print "id:", ID;
        }
}
$ awk -f get.awk < data
dom: domainname.com
url: /im/qs_menu.php
id: 950013

dom: domainname.com
url: /im/qs_menu.php
id: 950013

$

I'm not sure how to remove the doubles, since I don't know what criteria they should or shouldn't be duplicated on.
Last edited by Corona688; 09-26-2011 at 04:18 PM..
# 5  
Old 09-27-2011
CORONA!!!! I love you :)
This has fabulous work Smilie

THANK YOU Smilie

just please my log getting fill up with POST and GET methods too


the perfect example to try the script is



Quote:
--fecb387d-A--
[27/Sep/2011:01:04:14 +0100] ToES-dXlfQYAAGD-UgsAAAAn 209.172.61.41 58098 109.75.170.170 80
--fecb387d-B--
POST /xmlrpc.php HTTP/1.0
User-Agent: The Incutio XML-RPC PHP Library -- WordPress/3.2.1
Host: www.domainname.co.uk
Accept: */*
Content-Type: text/xml
Accept-Encoding: deflate;q=1.0, compress;q=0.5
Content-Length: 359

--fecb387d-C--
<?xml version="1.0"?>
<methodCall>
<methodName>pingback.ping</methodName>
<params>
<param><value><string>http://www.domain.com/relationships/relationships-weddings/trinkets-perfect-presents-for-a-wedding/</string></value></param>
<param><value><string>http://www.domainname.co.uk/wedding-shoes-sale</string></value></param>
</params></methodCall>
--fecb387d-F--
HTTP/1.1 404 Not Found
X-Powered-By: PHP/5.2.17
X-Pingback: http://www.domainname.co.uk/xmlrpc.php
Expires: Wed, 11 Jan 1984 05:00:00 GMT
Cache-Control: no-cache, must-revalidate, max-age=0
Pragma: no-cache
Last-Modified: Tue, 27 Sep 2011 00:04:14 GMT
Connection: close
Content-Type: text/html; charset=UTF-8

--fecb387d-H--
Message: Access denied with code 406 (phase 2). Match of "rx (^application/x-www-form-urlencoded|^multipart/form-dataSmilie.*$" against "REQUEST_HEADERS:Content-Type" required. [file "/usr/local/apache/conf/modsec2.user.conf"] [line "14"] [id "90111"]
Action: Intercepted (phase 2)
Stopwatch: 1317081853500200 736240 (2276* 2408 -)
Producer: ModSecurity for Apache/2.5.13 (ModSecurity: Open Source Web Application Firewall).


--f42e2544-A--
[26/Sep/2011:16:03:13 +0100] ToCUMdXlTpYAACTqNMsAAAAO 80.33.86.223 53424 91.186.30.249 80
--f42e2544-B--
GET /im/qs_menu.php?text=Contact%20Us&bt_img=bt_contact HTTP/1.1
Accept: */*
Referer: http://www.domainname.com/
Accept-Language: en-GB
User-Agent: Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; Trident/4.0; GTB7.1; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; .NET4.0C; InfoPath.1)
Accept-Encoding: gzip, deflate
Host: www.domainname.com
Connection: Keep-Alive
Cookie: PHPSESSID=f933fb642e1c3e258b7c9787b49d2408; lang=en

--f42e2544-F--
HTTP/1.1 406 Not Acceptable
Content-Length: 384
Keep-Alive: timeout=5, max=96
Connection: Keep-Alive
Content-Type: text/html; charset=iso-8859-1

--f42e2544-H--
Message: Access denied with code 406 (phase 2). Pattern match "_img|amature-big-titties|amature-big-titties|avril-laveign-porn|breast-touch-video|gingers-having-sex|naked-indian-models" at REQUEST_URI. [file "/usr/local/apache/conf/modsec2.user.conf"] [line "109"] [id "950013"] [msg "PHP/FTP Injection Attack. Matched signature <_img>"] [severity "CRITICAL"]
Apache-Error: [file "core.c"] [line 3650] [level 3] File does not exist: /home/costadel/domains/domainname.com/public_html/406.shtml, referer: http://www.domainname.com/
Action: Intercepted (phase 2)
Stopwatch: 1317049393646593 1950 (402 1648 -)
Producer: ModSecurity for Apache/2.5.13 (http://www.modsecurity.org/).
This need to be done and i am yours Smilie

I know its will take mins for you to do it you are champ


Thanks

---------- Post updated at 08:49 PM ---------- Previous update was at 07:59 PM ----------

I managed to get it with "|" pipe

Quote:
for(N=1; N<=NF; N++)
{
if($N ~ /Host: /) NEWDOM=$N
if($N ~ /\[id /) NEWID=$N
if($N ~ /^GET|POST/) NEWURL=$N
}
Now i need to know if can i just get the URL and ID(multiple IDs if it has multiple block id for same doamin) for specific domain name which i will provide from command line

may be like
[#]awk -f get.awk /usr/local/apache/logs/modsec_audit.log domain1.co.uk
URL : /wonder/all.php
ID : 910011 , 910023


is this possible ? if yes how ??

---------- Post updated 09-27-11 at 11:00 AM ---------- Previous update was 09-26-11 at 08:49 PM ----------

Please, Can any one from Unix team can help me with this ?
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

sed / awk / grep to extract information from log

Hi all, I have a query that runs that outputs data in the following format - 01/09/12 11:43:40,ADMIN,4,77,Application Group Load: Name(TESTED) LoadId(5137-1-0-1XX-15343-15343) File(/dir/dir/File.T03.CI2.RYR.2012009.11433350806.ARD) InputSize(5344) OutputSize(1359) Rows(2) Time(1.9960)... (8 Replies)
Discussion started by: jeffs42885
8 Replies

2. Shell Programming and Scripting

Extract information from file

In a particular directory, there can be 1000 files like below. filename is job901.ksh #!/bin/ksh cront -x << EOJ submit file=$PRODPATH/scripts/genReport.sh maxdelay=30 &node=xnode01 tname=job901 &pfile1=/prod/mldata/data/test1.dat ... (17 Replies)
Discussion started by: vedanta
17 Replies

3. Shell Programming and Scripting

Extract information from file

Gents, If is possible please help. I have a big file (example attached) which contends exactly same value in column, but from column 2 to 6 these values are diff. I will like to compile for all records all columns like the example attached in .csv format (output.rar ).. The last column in the... (11 Replies)
Discussion started by: jiam912
11 Replies

4. Shell Programming and Scripting

Extract information from txt file

Hello! I need help :) I have a file like this: AA BC FG RF TT GH DD FF HH (a few number of rows and three columns) and I want to put the letters of each column in a variable step by step in order to give them as input in another script. So I would like to obtain: for the 1° loop:... (11 Replies)
Discussion started by: edekP
11 Replies

5. Shell Programming and Scripting

How to extract information from a file?

Hi, i have a file like this: <Iteration> <Iteration_iter-num>3</Iteration_iter-num> <Iteration_query-ID>lcl|3_0</Iteration_query-ID> <Iteration_query-def>G383C4U01EQA0A length=197</Iteration_query-def> <Iteration_query-len>197</Iteration_query-len> ... (9 Replies)
Discussion started by: the_simpsons
9 Replies

6. Shell Programming and Scripting

extract information from a log file (last days)

I'm still new to bash script , I have a log file and I want to extract the items within the last 5 days . and also within the last 10 hours the log file is like this : it has 14000 items started from march 2002 to january 2003 awk '{print $4}' < *.log |uniq -c|sort -g|tail -10 but... (14 Replies)
Discussion started by: matarsak
14 Replies

7. Shell Programming and Scripting

Create shell script to extract unique information from one file to a new file.

Hi to all, I got this content/pattern from file http.log.20110808.gz mail1 httpd: Account Notice: close igchung@abc.com 2011/8/7 7:37:36 0:00:03 0 0 1 mail1 httpd: Account Information: login sastria9@abc.com proxy sid=gFp4DLm5HnU mail1 httpd: Account Notice: close sastria9@abc.com... (16 Replies)
Discussion started by: Mr_47
16 Replies

8. Shell Programming and Scripting

Extract information from Log file formatted

Good evening! Trying to make a shell script to parse log file and show only required information. log file has 44 fields and alot of lines, each columns separated by ":". log file is like: first_1:3:4:5:6:1:3:4:5:something:notinterested second_2:3:4:3:4:2 first_1:3:4:6:6:7:8 I am interested... (3 Replies)
Discussion started by: dummie55
3 Replies

9. Shell Programming and Scripting

extract and format information from a file

Hi, Following is sample portion of the file; <JDBCConnectionPool DriverName="oracle.jdbc.OracleDriver" MaxCapacity="10" Name="MyApp_DevPool" PasswordEncrypted="{3DES}7tXFH69Xg1c=" Properties="user=MYAPP_ADMIN" ShrinkingEnabled="false" ... (12 Replies)
Discussion started by: sujoy101
12 Replies

10. Shell Programming and Scripting

How to extract a piece of information from a huge file

Hello All, I need some assistance to extract a piece of information from a huge file. The file is like this one : database information ccccccccccccccccc ccccccccccccccccc ccccccccccccccccc ccccccccccccccccc os information cccccccccccccccccc cccccccccccccccccc... (2 Replies)
Discussion started by: Marcor
2 Replies
Login or Register to Ask a Question