Login or Register to Ask a Question and Join Our Community

Sponsored Content
Full Discussion: Extract various information from a log file
Top Forums Shell Programming and Scripting Extract various information from a log file Post 302559265 by Corona688 on Monday 26th of September 2011 03:08:56 PM
Old 09-26-2011
How should it know to tie the URL from an error three things up with the URL-less error three things down?

---------- Post updated at 01:08 PM ---------- Previous update was at 12:52 PM ----------

Here's something that sort of does it:

Code:
$ cat get.awk
BEGIN { RS="";  FS="\n" }

{
        split($1, L, "[-]*");

        if(L[2] != LAST)
        {
                if(ID)
                {
                        print "dom:", DOM;
                        print "url:", URL;
                        print "id:", ID;
                        printf("\n");
                }
                DOM=""; URL=""; ID=""

                LAST=L[2];
        }


        for(N=1; N<=NF; N++)
        {
                if($N ~ /referer: http:/)       NEWDOM=$N
                if($N ~ /\[id /)                NEWID=$N
                if($N ~ /^GET/)                 NEWURL=$N
        }

        if(NEWURL)
        {
                split(NEWURL, a, "[ ?]");
                NEWURL=a[2];
                URL=NEWURL
                NEWURL=""
        }

        if(NEWID)
        {
                # Id string will be in a[2]
                split(NEWID, a, "\\[id ");
                # Split on ], ", ' ' chars.
                split(a[2], a, "[\"\\] ]");
                NEWID=a[2];
                ID=NEWID;
                NEWID=""
        }

        if(NEWDOM)
        {
                # Extract everything after 'referer:'
                split(NEWDOM, a, "referer: ");  NEWDOM=a[2];
                # Reduce http://whatever/ to whatever
                sub(/http:\/\//, "", NEWDOM);
                sub(/\/$/, "", NEWDOM);
                # Turn www.whatever.com into www, whatever, com.
                N=split(NEWDOM, a, ".");
                # Paste the last two together.
                NEWDOM=a[N-1];  NEWDOM=NEWDOM "." a[N];
                DOM=NEWDOM
                NEWDOM=""
        }
}

END {
        if(ID)
        {
                print "dom:", DOM;
                print "url:", URL;
                print "id:", ID;
        }
}
$ awk -f get.awk < data
dom: domainname.com
url: /im/qs_menu.php
id: 950013

dom: domainname.com
url: /im/qs_menu.php
id: 950013

$

I'm not sure how to remove the doubles, since I don't know what criteria they should or shouldn't be duplicated on.
Last edited by Corona688; 09-26-2011 at 04:18 PM..
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

How to extract a piece of information from a huge file

Hello All, I need some assistance to extract a piece of information from a huge file. The file is like this one : database information ccccccccccccccccc ccccccccccccccccc ccccccccccccccccc ccccccccccccccccc os information cccccccccccccccccc cccccccccccccccccc... (2 Replies)
Discussion started by: Marcor
2 Replies

2. Shell Programming and Scripting

extract and format information from a file

Hi, Following is sample portion of the file; <JDBCConnectionPool DriverName="oracle.jdbc.OracleDriver" MaxCapacity="10" Name="MyApp_DevPool" PasswordEncrypted="{3DES}7tXFH69Xg1c=" Properties="user=MYAPP_ADMIN" ShrinkingEnabled="false" ... (12 Replies)
Discussion started by: sujoy101
12 Replies

3. Shell Programming and Scripting

Extract information from Log file formatted

Good evening! Trying to make a shell script to parse log file and show only required information. log file has 44 fields and alot of lines, each columns separated by ":". log file is like: first_1:3:4:5:6:1:3:4:5:something:notinterested second_2:3:4:3:4:2 first_1:3:4:6:6:7:8 I am interested... (3 Replies)
Discussion started by: dummie55
3 Replies

4. Shell Programming and Scripting

Create shell script to extract unique information from one file to a new file.

Hi to all, I got this content/pattern from file http.log.20110808.gz mail1 httpd: Account Notice: close igchung@abc.com 2011/8/7 7:37:36 0:00:03 0 0 1 mail1 httpd: Account Information: login sastria9@abc.com proxy sid=gFp4DLm5HnU mail1 httpd: Account Notice: close sastria9@abc.com... (16 Replies)
Discussion started by: Mr_47
16 Replies

5. Shell Programming and Scripting

extract information from a log file (last days)

I'm still new to bash script , I have a log file and I want to extract the items within the last 5 days . and also within the last 10 hours the log file is like this : it has 14000 items started from march 2002 to january 2003 awk '{print $4}' < *.log |uniq -c|sort -g|tail -10 but... (14 Replies)
Discussion started by: matarsak
14 Replies

6. Shell Programming and Scripting

How to extract information from a file?

Hi, i have a file like this: <Iteration> <Iteration_iter-num>3</Iteration_iter-num> <Iteration_query-ID>lcl|3_0</Iteration_query-ID> <Iteration_query-def>G383C4U01EQA0A length=197</Iteration_query-def> <Iteration_query-len>197</Iteration_query-len> ... (9 Replies)
Discussion started by: the_simpsons
9 Replies

7. Shell Programming and Scripting

Extract information from txt file

Hello! I need help :) I have a file like this: AA BC FG RF TT GH DD FF HH (a few number of rows and three columns) and I want to put the letters of each column in a variable step by step in order to give them as input in another script. So I would like to obtain: for the 1° loop:... (11 Replies)
Discussion started by: edekP
11 Replies

8. Shell Programming and Scripting

Extract information from file

Gents, If is possible please help. I have a big file (example attached) which contends exactly same value in column, but from column 2 to 6 these values are diff. I will like to compile for all records all columns like the example attached in .csv format (output.rar ).. The last column in the... (11 Replies)
Discussion started by: jiam912
11 Replies

9. Shell Programming and Scripting

Extract information from file

In a particular directory, there can be 1000 files like below. filename is job901.ksh #!/bin/ksh cront -x << EOJ submit file=$PRODPATH/scripts/genReport.sh maxdelay=30 &node=xnode01 tname=job901 &pfile1=/prod/mldata/data/test1.dat ... (17 Replies)
Discussion started by: vedanta
17 Replies

10. Shell Programming and Scripting

sed / awk / grep to extract information from log

Hi all, I have a query that runs that outputs data in the following format - 01/09/12 11:43:40,ADMIN,4,77,Application Group Load: Name(TESTED) LoadId(5137-1-0-1XX-15343-15343) File(/dir/dir/File.T03.CI2.RYR.2012009.11433350806.ARD) InputSize(5344) OutputSize(1359) Rows(2) Time(1.9960)... (8 Replies)
Discussion started by: jeffs42885
8 Replies

LEARN ABOUT CENTOS

abrt-retrace-client

ABRT-RETRACE-CLIEN(1)						    ABRT Manual 					     ABRT-RETRACE-CLIEN(1)

NAME

       abrt-retrace-client - Handles the communication with Retrace server.

SYNOPSIS

       abrt-retrace-client <operation> [options]

DESCRIPTION

       This tool is able to communicate with Retrace server: create a new task, ask about task's status, download log or backtrace of a finished
       task.

   Integration with libreport events
       abrt-retrace-client can be used as an analyzer for application crashes which dump core.

       Example usage in report_event.conf:

	   EVENT=analyze analyzer=CCpp
		   abrt-retrace-client

OPERATIONS

       create
	   Creates a new task. Prints task ID and password to stdout. Either -d or -c is required.

       status
	   Prints task's status to stdout. Both -t and -p are required.

       log
	   Prints finished task's log to stdout. Both -t and -p are required.

       backtrace
	   Print finished task's backtrace to stdout. Both -t and -p are required.

       batch
	   Runs all operations in one step: creates a new task, periodically asks for status (the period is specified by --status-delay option)
	   and downloads the result when finished. If the task was successful backtrace file is saved, otherwise log is printed to stdout. Either
	   -c or -d is required.

OPTIONS

       -v, --verbose
	   be verbose

       -s, --syslog
	   log to syslog

       -k, --insecure
	   allow insecure connection to retrace server

       --url URL
	   retrace server URL

       --headers
	   (debug) show received HTTP headers

       -d, --dir DIR
	   read data from ABRT problem directory

       -c, --core COREDUMP
	   read data from coredump

       -l, --status-delay
	   delay for polling operations (seconds)

       --no-unlink
	   (debug) do not delete temporary archive created in /tmp

       -t, --task ID
	   ID of the task on server

       -p, --password PWD
	   password of the task on server

AUTHORS

       o   ABRT team

abrt 2.1.11							    06/18/2014						     ABRT-RETRACE-CLIEN(1)
All times are GMT -4. The time now is 02:00 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy