Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Sed to parse log file

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Sed to parse log file
# 1  
Old 05-01-2012
Sed to parse log file
Hi all, thanks for reading the post.

I'm trying to parse hundreds of log files in a directory. One log file looks similar to below:

Code:
Investigator  : Jim_Foo
Custodian     : Jim_Foo-HDD1-FOO-1234
Export Path   : N:\FOO-1234\Foo_Foo
Compute MD5   : No
File List Only: No
Extensions Selected:
     DOC
     DOCX
     EML
     MSG
     OST
     PDF
     PPT
     PPTX
     PST
     XLS
     XLSX
     XLSM
     ZIP
     REP

Matched File : foo_file1.pdf
Match Type   : File Extension Match
File Number  : 0
File Size    : 640
MD5 Hash     : 'Compute MD5 Hash' not selected
MAC Times    : M=09/27/10 02:05:26PM  A=02/01/12 09:59:49AM  C=09/27/10 02:05:26PM  
Original Path: foopath\foopath\foopath\foobar\foo_file.pdf

Matched File : foo_file2.pdf
Match Type   : File Extension Match
File Number  : 0
File Size    : 123
MD5 Hash     : 'Compute MD5 Hash' not selected
MAC Times    : M=09/27/10 02:05:26PM  A=02/01/12 09:59:49AM  C=09/27/10 02:05:26PM  
Original Path: foopath\foopath\foopath\foobar\foo_file.pdf

I would like to search for strings with sed and put the results into a csv. I can search for the right patterns, but I'm having trouble printing it correctly.

Code:
$ sed -e '/Custodian/b' -e '/Matched File/b' -e '/File Size/b' -e '/Original Path/b' -e d *

Custodian : Jim_Foo-HDD1-FOO-1234
Export Path : N:\FOO-1234\Foo_Foo
Matched File : foo_file1.pdf
File Size : 640
Original Path: foopath\foopath\foopath\foobar\foo_file1.pdf
Matched File : foo_file2.pdf
File Size : 123
Original Path: foopath\foopath\foopath\foobar\foo_file.pdf[/CODE]

CSV needs to look something like this:
Code:
Custodian,Export Path,Matched File,File Size,Original Path
Custodian,Export Path,Matched File,File Size,Original Path

Code:
"Jim_Foo-HDD1-FOO-1234","N:\FOO-1234\Foo_Foo","foo_file1.pdf","640","foopath\foopath\foopath\foobar\foo_file1.pdf"
"Jim_Foo-HDD1-FOO-1234","N:\FOO-1234\Foo_Foo","foo_file2.pdf","123","foopath\foopath\foopath\foobar\foo_file.pdf"

Last edited by chipperuga; 05-01-2012 at 12:52 PM..
# 2  
Old 05-01-2012
I'd use awk to do this, since it has much easier recall with variables and sparse arrays. Workign on it.
# 3  
Old 05-01-2012
Code:
$ awk -F": " '/Custodian/{c=$2}/Export Path/{p=$2}/Matched File/{m=$2}/File Size/{s=$2}/Original Path/{o=$2} {if(c && p && m && o && s){printf("\"%s\",\"%s\",\"%s\",\"%s\",\"%s\"\n",c,p,m,s,o);m=s=o=0;}}' test.txt
"Jim_Foo-HDD1-FOO-1234","N:\FOO-1234\Foo_Foo","foo_file1.pdf","640","foopath\foopath\foopath\foobar\foo_file.pdf"
"Jim_Foo-HDD1-FOO-1234","N:\FOO-1234\Foo_Foo","foo_file2.pdf","123","foopath\foopath\foopath\foobar\foo_file.pdf"

This User Gave Thanks to itkamaraj For This Post:
chipperuga
# 4  
Old 05-01-2012
perhaps more easily configurable:

Code:
$ cat myscript.awk

BEGIN { OFS="\",\"" }

FNR==1 {        for(X in A) delete A[X];        }

match($0, /:/) {
        N=substr($0, 1, RSTART-1);
        V=substr($0, RSTART+2);
        sub(/ *$/, "", N);
        A[N]=V;
}

/^Original Path/ {
        print "\"" A["Custodian"], A["Export Path"], A["Matched File"], A["File Size"], A["Original Path"] "\""
}

$ awk -f myscript.awk datafile datafile datafile

"Jim_Foo-HDD1-FOO-1234","N:\FOO-1234\Foo_Foo","foo_file1.pdf","640","foopath\foopath\foopath\foobar\foo_file.pdf"
"Jim_Foo-HDD1-FOO-1234","N:\FOO-1234\Foo_Foo","foo_file2.pdf","123","foopath\foopath\foopath\foobar\foo_file.pdf"
"Jim_Foo-HDD1-FOO-1234","N:\FOO-1234\Foo_Foo","foo_file1.pdf","640","foopath\foopath\foopath\foobar\foo_file.pdf"
"Jim_Foo-HDD1-FOO-1234","N:\FOO-1234\Foo_Foo","foo_file2.pdf","123","foopath\foopath\foopath\foobar\foo_file.pdf"
"Jim_Foo-HDD1-FOO-1234","N:\FOO-1234\Foo_Foo","foo_file1.pdf","640","foopath\foopath\foopath\foobar\foo_file.pdf"
"Jim_Foo-HDD1-FOO-1234","N:\FOO-1234\Foo_Foo","foo_file2.pdf","123","foopath\foopath\foopath\foobar\foo_file.pdf"

$

Last edited by Corona688; 05-01-2012 at 01:14 PM..
This User Gave Thanks to Corona688 For This Post:
chipperuga
# 5  
Old 05-01-2012
Made some progress, but still need help formatting the output:
Code:
$sed -e '/Custodian/b' -e '/Matched File/b' -e '/File Size/b' -e '/Original Path/b' -e d * | sed -e 's/.*= \(.*\) \(.*\)/"\2, \1",/' -e 's/.*= \(.*\)/"\1",/' -e 's/.*: \(.*\)/"\1",/' -e 's/"\(.*\) ..:..:.*/"\1"/'

"Jim_Foo-HDD1-FOO-1234",
"foo_file1.pdf",
"640",
"foopath\foopath\foopath\foobar\foo_file.pdf",
"foo_file2.pdf",
"123",
"foopath\foopath\foopath\foobar\foo_file.pdf",

It should print as:
Code:
"Jim_Foo-HDD1-FOO-1234","foo_file1.pdf","640","foopath\foopath\foopath\foobar\foo_file.pdf"
"Jim_Foo-HDD1-FOO-1234","foo_file2.pdf","123","foopath\foopath\foopath\foobar\foo_file.pdf"

Need to:
  1. Repeat "Custodian" before each "Matched File"
  2. Remove some of the extra commas

---------- Post updated at 11:17 AM ---------- Previous update was at 11:04 AM ----------

This is perfect, thank you sir.

Quote:
Originally Posted by Corona688
perhaps more easily configurable:

Code:
$ cat myscript.awk

BEGIN { OFS="\",\"" }

FNR==1 {        for(X in A) delete A[X];        }

match($0, /:/) {
        N=substr($0, 1, RSTART-1);
        V=substr($0, RSTART+2);
        sub(/ *$/, "", N);
        A[N]=V;
}

/^Original Path/ {
        print "\"" A["Custodian"], A["Export Path"], A["Matched File"], A["File Size"], A["Original Path"] "\""
}

$ awk -f myscript.awk datafile datafile datafile

"Jim_Foo-HDD1-FOO-1234","N:\FOO-1234\Foo_Foo","foo_file1.pdf","640","foopath\foopath\foopath\foobar\foo_file.pdf"
"Jim_Foo-HDD1-FOO-1234","N:\FOO-1234\Foo_Foo","foo_file2.pdf","123","foopath\foopath\foopath\foobar\foo_file.pdf"
"Jim_Foo-HDD1-FOO-1234","N:\FOO-1234\Foo_Foo","foo_file1.pdf","640","foopath\foopath\foopath\foobar\foo_file.pdf"
"Jim_Foo-HDD1-FOO-1234","N:\FOO-1234\Foo_Foo","foo_file2.pdf","123","foopath\foopath\foopath\foobar\foo_file.pdf"
"Jim_Foo-HDD1-FOO-1234","N:\FOO-1234\Foo_Foo","foo_file1.pdf","640","foopath\foopath\foopath\foobar\foo_file.pdf"
"Jim_Foo-HDD1-FOO-1234","N:\FOO-1234\Foo_Foo","foo_file2.pdf","123","foopath\foopath\foopath\foobar\foo_file.pdf"

$

Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Parse log file to insert into database

I have a log file that's created daily by this command: sar -u 300 288 >> /var/log/usage/$(date "+%Y-%m-%d")_$(hostname)_cpu.log It that contains data like this: Linux 3.16.0-4-amd64 (myhostname) 08/15/2015 _x86_64_ (1 CPU) 11:34:17 PM CPU %user %nice ... (12 Replies)
Discussion started by: unplugme71
12 Replies

2. Shell Programming and Scripting

Parse A Log File

Hello All, Below is the excerpt from my Informatica log file which has 4 blocks of lines (starting with WRITER_1_*_1). Like these my log file will have multiple blocks of same pattern. WRITER_1_*_1> WRT_8161 TARGET BASED COMMIT POINT Thu May 08 09:33:21 2014... (13 Replies)
Discussion started by: Ariean
13 Replies

3. Shell Programming and Scripting

sed parse a lign into a new sql file

Hi everyone, another question while using sed. my sed statement should parse every line in a file and store all "i" variable item a a new file. any wrong arguments here? Thanks a million. task_name => name, object_type => 'TABLE', attr1 => 'TestR3', attr2 => '$i', for i... (4 Replies)
Discussion started by: sundaygeek
4 Replies

4. Shell Programming and Scripting

parse log with sed

I've been searching for an hour on how to parse a file like this: 10.200.5.83 - - "GET /portal/edits.js HTTP/1.1" 200 24324 10.200.5.83 - - "GET /portal/objects/PortalConfig.js HTTP/1.1" 200 12187 10.200.5.84 - - "GET /portal/objects/CommonDialog.js HTTP/1.1" 200 8283 10.200.5.84 - - "GET... (4 Replies)
Discussion started by: dba_frog
4 Replies

5. Shell Programming and Scripting

Parse the log file

./abc.sh started at Sun Oct 24 06:42:04 PDT 2010 Message: ======= Summary Report of NAME count ----------------------------------------------------------------- Below is the output of the SQL query :- NAME COUNT... (2 Replies)
Discussion started by: sandy1028
2 Replies

6. Shell Programming and Scripting

parse a log file and remember last line

Hi all: I'm working on a HPUX 11.23 system and I am needing to parse a tomcat-jakarta log file for memory use. Getting the desired data is easy, assuming the log file does not grow. This file grows constantly and I want to check it q 5 min. The next check will pick up from where it left off 5... (4 Replies)
Discussion started by: raggmopp
4 Replies

7. Shell Programming and Scripting

sed parse small xml file

I have a tmp.xml file like: <?xml version="1.0" encoding="UTF-8"?> <Response> <Ip>193.143.121.198</Ip> <Status>OK</Status> <CountryCode>PL</CountryCode> <CountryName>Poland</CountryName> <RegionCode>82</RegionCode> <RegionName>Pomorskie</RegionName> <City>Gdansk</City> ... (9 Replies)
Discussion started by: unclecameron
9 Replies

8. Shell Programming and Scripting

sed command to parse Apache config file

Hi there, am trying to parse an Apache 'server' config file. A snippet of the config file is shown below: ..... ProxyPassReverse /foo http://foo.example.com/bar ..... ..... RewriteRule ^/(.*) http://www.example.com/$1 RewriteRule /redirect https://www.example1.com/$1 ........ (7 Replies)
Discussion started by: jy2k7ca
7 Replies

9. Shell Programming and Scripting

Parse out known messages from a log file

I am looking for a script to do the following. I have a large log file that contains hundreds of warnings, a lot of which can be ignored. The tool doesn't allow me to suppress it, so I like to parse it out from the log file and isolate just the new messages/warnings, based on an exception file. ... (12 Replies)
Discussion started by: cdn2008
12 Replies

10. Shell Programming and Scripting

To parse through the file and print output using awk or sed script

suppose if u have a file like that Hen ABCCSGSGSGJJJJK 15 Cock ABCCSGGGSGIJJJL 15 * * * * * * : * * * . * * * : Hen CFCDFCSDFCDERTF 30 Cock CHCDFCSDHCDEGFI 30 * . * * * * * * * : * * :* : : . The output shud be where there is : and . It shud... (4 Replies)
Discussion started by: cdfd123
4 Replies
Login or Register to Ask a Question