Login or Register to Ask a Question and Join Our Community

Sponsored Content
Full Discussion: Extract various information from a log file
Top Forums Shell Programming and Scripting Extract various information from a log file Post 302559265 by Corona688 on Monday 26th of September 2011 03:08:56 PM
Old 09-26-2011
How should it know to tie the URL from an error three things up with the URL-less error three things down?

---------- Post updated at 01:08 PM ---------- Previous update was at 12:52 PM ----------

Here's something that sort of does it:

Code:
$ cat get.awk
BEGIN { RS="";  FS="\n" }

{
        split($1, L, "[-]*");

        if(L[2] != LAST)
        {
                if(ID)
                {
                        print "dom:", DOM;
                        print "url:", URL;
                        print "id:", ID;
                        printf("\n");
                }
                DOM=""; URL=""; ID=""

                LAST=L[2];
        }


        for(N=1; N<=NF; N++)
        {
                if($N ~ /referer: http:/)       NEWDOM=$N
                if($N ~ /\[id /)                NEWID=$N
                if($N ~ /^GET/)                 NEWURL=$N
        }

        if(NEWURL)
        {
                split(NEWURL, a, "[ ?]");
                NEWURL=a[2];
                URL=NEWURL
                NEWURL=""
        }

        if(NEWID)
        {
                # Id string will be in a[2]
                split(NEWID, a, "\\[id ");
                # Split on ], ", ' ' chars.
                split(a[2], a, "[\"\\] ]");
                NEWID=a[2];
                ID=NEWID;
                NEWID=""
        }

        if(NEWDOM)
        {
                # Extract everything after 'referer:'
                split(NEWDOM, a, "referer: ");  NEWDOM=a[2];
                # Reduce http://whatever/ to whatever
                sub(/http:\/\//, "", NEWDOM);
                sub(/\/$/, "", NEWDOM);
                # Turn www.whatever.com into www, whatever, com.
                N=split(NEWDOM, a, ".");
                # Paste the last two together.
                NEWDOM=a[N-1];  NEWDOM=NEWDOM "." a[N];
                DOM=NEWDOM
                NEWDOM=""
        }
}

END {
        if(ID)
        {
                print "dom:", DOM;
                print "url:", URL;
                print "id:", ID;
        }
}
$ awk -f get.awk < data
dom: domainname.com
url: /im/qs_menu.php
id: 950013

dom: domainname.com
url: /im/qs_menu.php
id: 950013

$

I'm not sure how to remove the doubles, since I don't know what criteria they should or shouldn't be duplicated on.
Last edited by Corona688; 09-26-2011 at 04:18 PM..
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

How to extract a piece of information from a huge file

Hello All, I need some assistance to extract a piece of information from a huge file. The file is like this one : database information ccccccccccccccccc ccccccccccccccccc ccccccccccccccccc ccccccccccccccccc os information cccccccccccccccccc cccccccccccccccccc... (2 Replies)
Discussion started by: Marcor
2 Replies

2. Shell Programming and Scripting

extract and format information from a file

Hi, Following is sample portion of the file; <JDBCConnectionPool DriverName="oracle.jdbc.OracleDriver" MaxCapacity="10" Name="MyApp_DevPool" PasswordEncrypted="{3DES}7tXFH69Xg1c=" Properties="user=MYAPP_ADMIN" ShrinkingEnabled="false" ... (12 Replies)
Discussion started by: sujoy101
12 Replies

3. Shell Programming and Scripting

Extract information from Log file formatted

Good evening! Trying to make a shell script to parse log file and show only required information. log file has 44 fields and alot of lines, each columns separated by ":". log file is like: first_1:3:4:5:6:1:3:4:5:something:notinterested second_2:3:4:3:4:2 first_1:3:4:6:6:7:8 I am interested... (3 Replies)
Discussion started by: dummie55
3 Replies

4. Shell Programming and Scripting

Create shell script to extract unique information from one file to a new file.

Hi to all, I got this content/pattern from file http.log.20110808.gz mail1 httpd: Account Notice: close igchung@abc.com 2011/8/7 7:37:36 0:00:03 0 0 1 mail1 httpd: Account Information: login sastria9@abc.com proxy sid=gFp4DLm5HnU mail1 httpd: Account Notice: close sastria9@abc.com... (16 Replies)
Discussion started by: Mr_47
16 Replies

5. Shell Programming and Scripting

extract information from a log file (last days)

I'm still new to bash script , I have a log file and I want to extract the items within the last 5 days . and also within the last 10 hours the log file is like this : it has 14000 items started from march 2002 to january 2003 awk '{print $4}' < *.log |uniq -c|sort -g|tail -10 but... (14 Replies)
Discussion started by: matarsak
14 Replies

6. Shell Programming and Scripting

How to extract information from a file?

Hi, i have a file like this: <Iteration> <Iteration_iter-num>3</Iteration_iter-num> <Iteration_query-ID>lcl|3_0</Iteration_query-ID> <Iteration_query-def>G383C4U01EQA0A length=197</Iteration_query-def> <Iteration_query-len>197</Iteration_query-len> ... (9 Replies)
Discussion started by: the_simpsons
9 Replies

7. Shell Programming and Scripting

Extract information from txt file

Hello! I need help :) I have a file like this: AA BC FG RF TT GH DD FF HH (a few number of rows and three columns) and I want to put the letters of each column in a variable step by step in order to give them as input in another script. So I would like to obtain: for the 1° loop:... (11 Replies)
Discussion started by: edekP
11 Replies

8. Shell Programming and Scripting

Extract information from file

Gents, If is possible please help. I have a big file (example attached) which contends exactly same value in column, but from column 2 to 6 these values are diff. I will like to compile for all records all columns like the example attached in .csv format (output.rar ).. The last column in the... (11 Replies)
Discussion started by: jiam912
11 Replies

9. Shell Programming and Scripting

Extract information from file

In a particular directory, there can be 1000 files like below. filename is job901.ksh #!/bin/ksh cront -x << EOJ submit file=$PRODPATH/scripts/genReport.sh maxdelay=30 &node=xnode01 tname=job901 &pfile1=/prod/mldata/data/test1.dat ... (17 Replies)
Discussion started by: vedanta
17 Replies

10. Shell Programming and Scripting

sed / awk / grep to extract information from log

Hi all, I have a query that runs that outputs data in the following format - 01/09/12 11:43:40,ADMIN,4,77,Application Group Load: Name(TESTED) LoadId(5137-1-0-1XX-15343-15343) File(/dir/dir/File.T03.CI2.RYR.2012009.11433350806.ARD) InputSize(5344) OutputSize(1359) Rows(2) Time(1.9960)... (8 Replies)
Discussion started by: jeffs42885
8 Replies

LEARN ABOUT HPUX

pppoec

pppoec(1)						      General Commands Manual							 pppoec(1)

NAME

       pppoec - PPPoE (Point to Point Protocol over Ethernet) client

SYNOPSIS

       interface-name

       interface-name config-file] debug-level] log-file] retry-number]

DESCRIPTION

       discovers  an access concentrator that is willing to offer services to the client.  It obtains a unique session-id from the selected access
       concentrator.  then forks a daemon which performs the session phase functionalities of PPPoE as mentioned in RFC 2516.

   Options
       supports the following options:

	      Specify the configuration file with absolute path
				to be used by The default configuration file is

	      Specify the debug level at which logging has to be
				enabled.  The three debug levels are:

				Logs all error messages.
					      To log these messages, type on the command line.

				Logs all warning messages.
					      To log these messages, type on the command line.

				Logs function level debug messages.
					      To log these messages, type on the command line.

	      Specify the name of the network interface (e.g.,
				to be used for the PPPoE session.

	      Specify the log file with absolute path, to which pppoec must
				log all messages.  The default log file is

	      Specify the frequency of PADI (PPPoE Active
				Discovery Initiation) or PADR (PPPoE Active Discovery Request) packet retransmission when an  access  concentrator
				does not respond to the client's request.  The default value is 3.

EXAMPLES

       An example usage of the command is as follows:

WARNINGS

       Note that, you cannot run the client daemon without file.  See pppoec.conf(4) for more information.  Also, do not use to terminate Instead,
       use to kill the client daemon.

AUTHOR

       was developed by Hewlett-Packard.

FILES

       PPPoE server daemon
       PPPoE relay
       PPPoE client configuration file
       PPP daemon

SEE ALSO

       pppd(1), pppoerd(1M), pppoesd(1M), pppoec.conf(4).

																	 pppoec(1)
All times are GMT -4. The time now is 06:54 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy