Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

More complicated log parsing

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting More complicated log parsing
# 15  
Old 06-07-2007
Sorry, it seemed clearer to me.


Since this script will be run frequently on the same log file(s), I would like to:

-Add a check to make sure the script does not overwrite existing xml files (a check to see if the file exists, before it is written). I know how to do this as a shell script, but I am not sure how to do it in the awk script.

-While writing the new xml file (whose filename follow the format of 01-YYYYMMDD-XXXXXX - as read from the original log), move the xml into an appropriate directory (that can share the same YYYYMMDD format from the xml filename). If the filename does not follow the format (as above), then put it in an arbitrary saftey directory (any name).
# 16  
Old 06-07-2007
Quote:
Originally Posted by sjug
-Add a check to make sure the script does not overwrite existing xml files (a check to see if the file exists, before it is written). I know how to do this as a shell script, but I am not sure how to do it in the awk script.
its not clear why you want to do it in an awk script, but anyway, you can use getline. something like this snippet...
Code:
awk 'BEGIN { 
       if ((getline < "ljalsdfls") == -1) {
            print "not exists"
        } 
 } '

Quote:
-While writing the new xml file (whose filename follow the format of 01-YYYYMMDD-XXXXXX - as read from the original log), move the xml into an appropriate directory (that can share the same YYYYMMDD format from the xml filename). If the filename does not follow the format (as above), then put it in an arbitrary saftey directory (any name).
just an example only. apply the concepts to your code as needed.
Code:
awk 'BEGIN { 
      string="01-YYYYMMDD-XXXXXX"
      n=split(string,array,"-")
      print array[2] #this contains your format.
      cmd="mkdir -p " array[2] 
      print cmd
      cmd | getline result #execute mkdir
      close(cmd)
      print "your xml lines " > array[2]"/"string
 } '

# 17  
Old 06-07-2007
Thanks alot for all your help ghostdog! Smilie

My combined script looks like:

Code:
#!/usr/bin/awk -f
# Awk script: extract3.awk

/HandleRequest.*OrderControlService/{  
    if ( b= match($0,"xml") ) {
      xml = substr($0,b-2)
      print $0
    }
    else {
      next
    }
         
    n=split($0,line," ")
    m=split(line[5],file,"|")     
      
    if ( file[2] ~ /^[0-9]/) {
        filename=file[2]

       if ((getline < filename) == -1) {
          n=split(filename,array,"-")
          print array[2]
          cmd="mkdir -p " array[2] 
          print cmd
          cmd | getline result
          close(cmd)
          print xml > array[2]"/"filename".xml"
       }
       else {
       print filename " - File Exists"
       }
        
    }     
}

Without the directory creation it works fine, however with this full version, when I run it I get:

Quote:
Syntax Error The source line is 3.
The error context is
<<< >>>
awk: 0602-500 Quitting The source line is 3.

It works on some shells, and gives me a mkdir missing operand error, but on the log server it just gives me the above error and does nothing.
What is wrong?
# 18  
Old 06-07-2007
what is your OS platform, and awk version? what shell are you using? any such information will be good
# 19  
Old 06-08-2007
running
AIX 5.3, bash,
and I don't know what version of awk (there is no version flag?)

I fixed the original error I posted above, as my version of awk seems to have a problem with spaces in the script, so that is resolved for now.

But I have one more issue, which will hopefully be the last.

The way in which directories are created is by splitting the 01-YYYYMMDD-XXXXX by '-' then putting each piece in an array then creating a directory based on the second piece.

However there are some entries between the pipes that do not follow the format and have no '-' in them to give them a second piece.

For example there are some lines that look like:
Code:
2007-06-06 16:52:38,805 INFO  External- |3676482||>> [HandleRequest] AService<?xml version="1.0" encoding="utf-8" standalone="yes"?>

I am not sure what would be easier, simply to put them in the same folder as the previous one that was created (rolling over the folder filename to the next loop), or to just put all all these seven digit standalone numbers that do not follow the array format into a fixed existing directory (by checking the size of array[2] before mkdir?).
Last edited by sjug; 06-08-2007 at 10:01 AM..
# 20  
Old 06-08-2007
Quote:
Originally Posted by sjug
I am not sure what would be easier, simply to put them in the same folder as the previous one that was created (rolling over the folder filename to the next loop), or to just put all all these seven digit standalone numbers that do not follow the array format into a fixed existing directory (by checking the size of array[2] before mkdir?).
it depends on what you are comfortable with. and you are on the right track. one way is to check the array[2], if empty then put to another defined directory..
# 21  
Old 06-08-2007
I think a predefined directory would be simpler in the long run, what would the comparison be for array[2] being null?
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Parsing Log File help

Hi, I am a newbie to scripting. I have multiple log files (saved as .gz) in a directory that looks like this 01-01-2013 10:00 pn: 123 01-01-2013 10:00 sn: 987 01-01-2013 10:00 Test1 01-01-2013 10:00 Result: Pass 01-01-2013 10:00 Time: 5:00 01-01-2013 10:00 Test2 01-01-2013 10:00... (3 Replies)
Discussion started by: linuxnew
3 Replies

2. Shell Programming and Scripting

Log parsing

I have a directory with daily logs that have records like this: Date: 04/17/13 Time: 09:29:15 IP: 123.123.123.123 URL: usr/local/file1 and I want to only count how many times each file was accessed (e.g. file1 in that example above), and I want to also look in all the logs in the current... (3 Replies)
Discussion started by: Jaymz
3 Replies

3. Shell Programming and Scripting

Help Parsing a Log File

Hello all, I am new to scripting and I have written a script that performs an Rsync on my NAS and then moves on to send me an email with the status etc. The problem is that I think Rsync is taking to long to complete and the IF statement is timing out, as it doesn't appear to move on. Here... (1 Reply)
Discussion started by: Mongrel
1 Replies

4. Shell Programming and Scripting

Log parsing script

Hello, I have a script that parses logs and sends the output via digitally signed and encrypted email. This script uses grep -v to exclude patterns in a file. The problem I have is if this is run via cron none of the pattern matching seems to occur. If I run it by hand it runs exactly as it is... (2 Replies)
Discussion started by: wpfontenot
2 Replies

5. Shell Programming and Scripting

Parsing complicated CSV file with sed

Yes, there is a great doc out there that discusses parsing csv files with sed, and this topic has been covered before but not enough to answer my question (unix.com forums). I'm trying to parse a CSV file that has optional quotes like the following: "Apple","Apples, are fun",3.60,4.4,"I... (3 Replies)
Discussion started by: analog999
3 Replies

6. Shell Programming and Scripting

Perl log parsing help

Hello, I'm sure this is a very simple problem, but I'm having trouble thinking of an efficient way to do the following: given a large centralized ssh-log, one file on a syslog server, not separated by machines (I wish it were), that looks something like this: Sep 27 16:20:56 machine-name... (1 Reply)
Discussion started by: droog72
1 Replies

7. Shell Programming and Scripting

Parsing a Complicated properties file

Hi All, I have a requirement to parse a file. Let me clear you all on the req. I have a job which contains multiple tasks and each task will have multiple attributes that will be in the below format. Each task will have some sequence number according to that sequence number tasks shld... (1 Reply)
Discussion started by: rajeshorpu
1 Replies

8. Shell Programming and Scripting

XML Log Parsing

I have a log file that is around 300 MB of data having continours soap responses as shown below( I have attached only one sample SOAP). I would require to have the following extracted and written onto a new file. timestamp WebPartId bus:block bus:unblock endpt:operation Please help me. ... (3 Replies)
Discussion started by: pk_eee
3 Replies

9. Shell Programming and Scripting

Parsing a large log

I need to parse a large log say 300-400 mb The commands like awk and cat etc are taking time. Please help how to process. I need to process the log for certain values of current date. But I am unbale to do so. (17 Replies)
Discussion started by: asth
17 Replies

10. Shell Programming and Scripting

parsing email log

Can anyone give me some examples of how I can parse the following lines of text so that all characters up to and including the @ symbol are deleted? Also, any duplicates would need to be deleted in order to produce the desired output. Any help is much appreciated and explanations of any... (5 Replies)
Discussion started by: jjamd64
5 Replies
Login or Register to Ask a Question