extract information from a log file (last days)


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting extract information from a log file (last days)
# 8  
Old 09-19-2011
gawk does not work !

line 1: gawk: command not found
matarsak
# 9  
Old 09-19-2011
I did wonder what your system was, but you never said. What is it?
# 10  
Old 09-19-2011
Do you have perl on your system.
# 11  
Old 09-19-2011
It's Mac OSX ...
I don't want to use perl ! simple shell script not perl or python or anything else !

I searched a lot but i could not find anything on the web related to what i want !
it's actually part of my assignments ! ( a little part )


So I should extract the last X hours/days from the log file

Last edited by matarsak; 09-19-2011 at 04:24 PM..
matarsak
# 12  
Old 09-19-2011
I repeat: What you're asking for isn't easy.

1) Date math is not easy. The only time it's easy is when someone else has done all the work for you. How does one subtract dates if they're not numbers? If the language can't do it for you, convert them the hard way into something the language can subtract. That means worrying about things like calendars and leap-years.

2) This isn't a database. There's no "query". There's no "SELECT X from Y WHERE ..." to select data you want from known datatypes, this is a text file with no datatypes except columns, maybe, if you're lucky. To get text out of it, you match text against text. The closest thing there is to 'select' for text is awk, the flatfile powertool, which organizes text into records and columns for you if you tell it how the textfile's laid out, and understands numbers.

If you have no nice, clean tools which do date math for you, you have to do it the hard way. Fortunately, you might not have been the first person in the world to do so. OSX's awk is not GNU awk, but it does at least support functions, so there's an alternative mktime() you can try:

Code:
awk -v EDATE="26/Oct/2002:21:02:19" '
function _tm_isleap(year,    ret)
{
    ret = (year % 4 == 0 && year % 100 != 0) ||
            (year % 400 == 0)

    return ret
}

function _tm_addup(a,    total, yearsecs, daysecs,
                         hoursecs, i, j)
{
    hoursecs = 60 * 60
    daysecs = 24 * hoursecs
    yearsecs = 365 * daysecs

    total = (a[1] - 1970) * yearsecs

    # extra day for leap years
    for (i = 1970; i < a[1]; i++)
        if (_tm_isleap(i))
            total += daysecs

    j = _tm_isleap(a[1])
    for (i = 1; i < a[2]; i++)
        total += _tm_months[j, i] * daysecs

    total += (a[3] - 1) * daysecs
    total += a[4] * hoursecs
    total += a[5] * 60
    total += a[6]

    return total
}

function mktime(str,    res1, res2, a, b, i, j, t, diff)
{
    i = split(str, a, " ")    # don't rely on FS

    if (i != 6)
        return -1

    # force numeric
    for (j in a)
        a[j] += 0

    # validate
    if (a[1] < 1970 ||
        a[2] < 1 || a[2] > 12 ||
        a[3] < 1 || a[3] > 31 ||
        a[4] < 0 || a[4] > 23 ||
        a[5] < 0 || a[5] > 59 ||
        a[6] < 0 || a[6] > 61 )
            return -1

    res1 = _tm_addup(a)
    t = strftime("%Y %m %d %H %M %S", res1)

    if (_tm_debug)
        printf("(%s) -> (%s)\n", str, t) > "/dev/stderr"

    split(t, b, " ")
    res2 = _tm_addup(b)

    diff = res1 - res2

    if (_tm_debug)
        printf("diff = %d seconds\n", diff) > "/dev/stderr"

    res1 += diff

    return res1
}

BEGIN {
        # Initialize data for mktime()
    # Initialize table of month lengths
    _tm_months[0,1] = _tm_months[1,1] = 31
    _tm_months[0,2] = 28; _tm_months[1,2] = 29
    _tm_months[0,3] = _tm_months[1,3] = 31
    _tm_months[0,4] = _tm_months[1,4] = 30
    _tm_months[0,5] = _tm_months[1,5] = 31
    _tm_months[0,6] = _tm_months[1,6] = 30
    _tm_months[0,7] = _tm_months[1,7] = 31
    _tm_months[0,8] = _tm_months[1,8] = 31
    _tm_months[0,9] = _tm_months[1,9] = 30
    _tm_months[0,10] = _tm_months[1,10] = 31
    _tm_months[0,11] = _tm_months[1,11] = 30
    _tm_months[0,12] = _tm_months[1,12] = 31

        # Set up arrays for name-to-monthnumber
        split("Jan|Feb|Mar|Apr|May|Jun|Jul|Aug|Sep|Oct|Nov|Dec", MON, "|");
        for(N=1; N<=12; N++) MNUM[MON[N]]=sprintf("%02d", N);

        # Split [DD/MON/YYYY:HH:MM:SS into DD MM YYYY HH MM SS stored in D[1]-D[6].
        split(EDATE, D, "[:/]");
        # Convert "YYYY MM DD HH MM SS" into epoch time, i.e. seconds since 1970
        EDATE=mktime(D[3] " " MNUM[D[2]] " " D[1] " " D[4] " " D[5] " " D[6]);
        # Starting date is 5 days earlier
        SDATE=EDATE-(60*60*24*5);
}

{
                # Need the \\[ in there to ignore the [ at the beginning of the field
                split($(NF-1), D, "[\\[:/]");
                DATE=mktime(D[4] " " MNUM[D[3]] " " D[2] " " D[5] " " D[6] " " D[7]);
                # Print the line if it falls in the correct range
                if((DATE >= SDATE) && (DATE <= EDATE)) print;
        }' < datafile

---------- Post updated at 01:04 PM ---------- Previous update was at 12:58 PM ----------

Quote:
Originally Posted by matarsak
it's actually part of my assignments ! ( a little part )
Wait, what? This is homework?
# 13  
Old 09-19-2011
actually I wrote the assignment with python without any problem ! but I should write the whole thing with shell script too ! and i've done that except this part ! all other parts was not this much complicated ! So, I suppose there should be another way
matarsak
# 14  
Old 09-19-2011
I repeat for the third time: Date arithmetic is only easy when the language does it for you.

Date math is one of shell programming's blind spots. The enhancements GNU/Linux has for them mostly filled them in, but you only get them with the GNU utilities.

I also ask again: Is this homework?
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

sed / awk / grep to extract information from log

Hi all, I have a query that runs that outputs data in the following format - 01/09/12 11:43:40,ADMIN,4,77,Application Group Load: Name(TESTED) LoadId(5137-1-0-1XX-15343-15343) File(/dir/dir/File.T03.CI2.RYR.2012009.11433350806.ARD) InputSize(5344) OutputSize(1359) Rows(2) Time(1.9960)... (8 Replies)
Discussion started by: jeffs42885
8 Replies

2. Shell Programming and Scripting

Extract information from file

In a particular directory, there can be 1000 files like below. filename is job901.ksh #!/bin/ksh cront -x << EOJ submit file=$PRODPATH/scripts/genReport.sh maxdelay=30 &node=xnode01 tname=job901 &pfile1=/prod/mldata/data/test1.dat ... (17 Replies)
Discussion started by: vedanta
17 Replies

3. Shell Programming and Scripting

Extract information from file

Gents, If is possible please help. I have a big file (example attached) which contends exactly same value in column, but from column 2 to 6 these values are diff. I will like to compile for all records all columns like the example attached in .csv format (output.rar ).. The last column in the... (11 Replies)
Discussion started by: jiam912
11 Replies

4. Shell Programming and Scripting

Extract information from txt file

Hello! I need help :) I have a file like this: AA BC FG RF TT GH DD FF HH (a few number of rows and three columns) and I want to put the letters of each column in a variable step by step in order to give them as input in another script. So I would like to obtain: for the 1 loop:... (11 Replies)
Discussion started by: edekP
11 Replies

5. Shell Programming and Scripting

How to extract information from a file?

Hi, i have a file like this: <Iteration> <Iteration_iter-num>3</Iteration_iter-num> <Iteration_query-ID>lcl|3_0</Iteration_query-ID> <Iteration_query-def>G383C4U01EQA0A length=197</Iteration_query-def> <Iteration_query-len>197</Iteration_query-len> ... (9 Replies)
Discussion started by: the_simpsons
9 Replies

6. Shell Programming and Scripting

Extract various information from a log file

Hye ShamRock If you can help me with this difficult task for me then it will save my day Logs : ================================================================================================================== ... (4 Replies)
Discussion started by: SilvesterJ
4 Replies

7. Shell Programming and Scripting

Create shell script to extract unique information from one file to a new file.

Hi to all, I got this content/pattern from file http.log.20110808.gz mail1 httpd: Account Notice: close igchung@abc.com 2011/8/7 7:37:36 0:00:03 0 0 1 mail1 httpd: Account Information: login sastria9@abc.com proxy sid=gFp4DLm5HnU mail1 httpd: Account Notice: close sastria9@abc.com... (16 Replies)
Discussion started by: Mr_47
16 Replies

8. Shell Programming and Scripting

Extract information from Log file formatted

Good evening! Trying to make a shell script to parse log file and show only required information. log file has 44 fields and alot of lines, each columns separated by ":". log file is like: first_1:3:4:5:6:1:3:4:5:something:notinterested second_2:3:4:3:4:2 first_1:3:4:6:6:7:8 I am interested... (3 Replies)
Discussion started by: dummie55
3 Replies

9. Shell Programming and Scripting

extract and format information from a file

Hi, Following is sample portion of the file; <JDBCConnectionPool DriverName="oracle.jdbc.OracleDriver" MaxCapacity="10" Name="MyApp_DevPool" PasswordEncrypted="{3DES}7tXFH69Xg1c=" Properties="user=MYAPP_ADMIN" ShrinkingEnabled="false" ... (12 Replies)
Discussion started by: sujoy101
12 Replies

10. Shell Programming and Scripting

How to extract a piece of information from a huge file

Hello All, I need some assistance to extract a piece of information from a huge file. The file is like this one : database information ccccccccccccccccc ccccccccccccccccc ccccccccccccccccc ccccccccccccccccc os information cccccccccccccccccc cccccccccccccccccc... (2 Replies)
Discussion started by: Marcor
2 Replies
Login or Register to Ask a Question