Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

parse log with sed

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting parse log with sed
# 1  
Old 02-02-2011
parse log with sed
I've been searching for an hour on how to parse a file like this:
Code:
10.200.5.83 - - [02/Feb/2011:08:50:31 -0500] "GET /portal/edits.js HTTP/1.1" 200 24324
10.200.5.83 - - [02/Feb/2011:08:50:31 -0500] "GET /portal/objects/PortalConfig.js HTTP/1.1" 200 12187
10.200.5.84 - - [02/Feb/2011:08:50:31 -0500] "GET /portal/objects/CommonDialog.js HTTP/1.1" 200 8283
10.200.5.84 - - [02/Feb/2011:08:50:31 -0500] "GET /portal/objects/ErrorObject.js HTTP/1.1" 200 8248
10.200.5.83 - - [02/Feb/2011:08:50:31 -0500] "GET /portal/objects/FilterObject.js HTTP/1.1" 200 15520
10.200.5.83 - - [02/Feb/2011:08:50:31 -0500] "GET /portal/objects/DME.js HTTP/1.1" 200 13425
10.200.5.83 - - [02/Feb/2011:08:50:31 -0500] "GET /portal/objects/HotkeysManager.js HTTP/1.1" 200 16475
10.200.5.83 - - [02/Feb/2011:08:50:31 -0500] "GET /portal/objects/ValueStorage.js HTTP/1.1" 200 1922

When complete I need:
Code:
10.200.5.83  02/Feb/2011
10.200.5.84  02/Feb/2011

So in the end I really need a distinct list of each IP rolled up by Day so that we can get an idea of traffic.

I've been getting hung up on this:
Code:
sed '/\:/s/[0-9]*//' $file > tmpfile ; mv tmpfile test2.log

it keeps stripping out the first octet of the ip, not the time portion.
This User Gave Thanks to dba_frog For This Post:
rocking77
# 2  
Old 02-02-2011
Try
Code:
sed 's/ .*\[/ /;s/:.*//' $file | sort | uniq

# 3  
Old 02-02-2011
Awesome!
Thank you very much.
# 4  
Old 02-02-2011
I was not sure if you want to cumulate the information up, which IP shows up how often per day:
Code:
awk -F"[ \t]*|:" '{gsub(/\[/,""); a[$1" "$4]++} END{for(e in a){print e,"-- Times shown up:",a[e]}}' infile
10.200.5.84 02/Feb/2011 -- Times shown up: 2
10.200.5.83 02/Feb/2011 -- Times shown up: 6

# 5  
Old 02-02-2011
Zaxxon;
That was my next step. Thanks for this ! Now if I can count unique IP's for each day, I'd have all my 'wish list' items.
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

sed script to parse logs issue

I have this script to parse some logs: #!/bin/bash id=$1 shift sed "/(id=$id)/,/^$/!d" "$@" Usage: ./script.sh 1234 logfile The logs have an empty line before the logged events/timestamps -- most of the time. And this is my issue, since when there is no empty line, it will catch things... (4 Replies)
Discussion started by: KidCactus
4 Replies

2. Shell Programming and Scripting

Sed to parse log file

Hi all, thanks for reading the post. I'm trying to parse hundreds of log files in a directory. One log file looks similar to below: Investigator : Jim_Foo Custodian : Jim_Foo-HDD1-FOO-1234 Export Path : N:\FOO-1234\Foo_Foo Compute MD5 : No File List Only: No Extensions Selected:... (4 Replies)
Discussion started by: chipperuga
4 Replies

3. Shell Programming and Scripting

sed parse a lign into a new sql file

Hi everyone, another question while using sed. my sed statement should parse every line in a file and store all "i" variable item a a new file. any wrong arguments here? Thanks a million. task_name => name, object_type => 'TABLE', attr1 => 'TestR3', attr2 => '$i', for i... (4 Replies)
Discussion started by: sundaygeek
4 Replies

4. Shell Programming and Scripting

parse xm entry with awk/sed

Hi folks, I have XML files with the following sections (section occurs once per file) in them: <AuthorList CompleteYN="Y"> <Author ValidYN="Y"> <LastName>Bernal</LastName> <ForeName>Federico</ForeName> ... (3 Replies)
Discussion started by: euval
3 Replies

5. Shell Programming and Scripting

sed to parse html

Hello, I have a html file like this : <html> ... ... ... <table> ....... ...... </table> <table name = "hi"> ...... ..... ... </table> <h1> Welcome </h1> ....... ...... </html> (11 Replies)
Discussion started by: prasanna1157
11 Replies

6. Shell Programming and Scripting

sed or awk to parse this text

I am just beginning with sed and awk and understand that they are "per" line input. That is, they operate on each line individually, and output based on rules, etc. But I have multi-line text blocks that looks as follows, and wish to ONLY extract the text between the first hyphen (-) and the... (13 Replies)
Discussion started by: bulgin
13 Replies

7. Shell Programming and Scripting

sed parse small xml file

I have a tmp.xml file like: <?xml version="1.0" encoding="UTF-8"?> <Response> <Ip>193.143.121.198</Ip> <Status>OK</Status> <CountryCode>PL</CountryCode> <CountryName>Poland</CountryName> <RegionCode>82</RegionCode> <RegionName>Pomorskie</RegionName> <City>Gdansk</City> ... (9 Replies)
Discussion started by: unclecameron
9 Replies

8. Shell Programming and Scripting

How to parse slash / in sed command..

Hi All, I am trying to replace paths with \ (as in NT) with / (as in Unix) in a script using sed. Following is the command I use: sed "s/e:\Kenny_test\csv_files/var/opt/ciw/data/outbound/g" EMEA4710.SQL > test the string to replace is e:\Kenny_test\csv_files the target string needs to... (3 Replies)
Discussion started by: Abhidey
3 Replies

9. Shell Programming and Scripting

Parse String Using Sed

Hi, I am wondering if there's a simpler way to extract the second occurrence of a word enclosed in that matches my search criteria. Sample Input is as follows: Error installing feature - com.er.nms.cif.ist.NoMatchingUpgra Error installing feature -... (4 Replies)
Discussion started by: racbern
4 Replies

10. Shell Programming and Scripting

Parse Line Using Sed

Hello All, I am new to using sed, and I need to extract from the string data after : delimeter. Can you help me please with the sed command? Here's the input: ipAddress: 10.20.10.11 ioIpAddressNodeB: 10.20.10.10 ioIpAddressNodeA: 10.20.10.9 ipAddress: 0.0.0.0 Expected Output:... (7 Replies)
Discussion started by: racbern
7 Replies
Login or Register to Ask a Question