Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Extract IP from logs and

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Extract IP from logs and
# 15  
Old 03-19-2013
It's pretty straightforward to add the file writting:
Code:
#!/usr/bin/python

import re
import sys
from datetime import datetime 
import time
strptime = lambda date_string, format: datetime(*(time.strptime(date_string, format)[0:6]))

try:
   f = open('./a.txt', 'r')
   out_fic=f.read()
   f.close()
except IOError:
   print "File error"
   sys.exit(5)

FMT = '%b %d, %Y %H:%M:%S %p'
pat='^(.+\s+(AM|PM)).+login:(\d+\.\d+\.\d+\.\d+):'
seek_p=re.compile(pat,re.MULTILINE)

res_h={}
res_f={}
for match_p in seek_p.findall(out_fic):
   time_p=match_p[0]
   ip_p=match_p[2]
   try:
      last=res_h[ip_p]
      res_h[ip_p]=time_p
      delta=strptime(time_p,FMT) - strptime(last,FMT)
      delta_segs=delta.seconds 
      if delta_segs <= 4 : 
          res_f[ip_p]=True
   except KeyError:
      res_h[ip_p]=time_p
      res_f[ip_p]=False

human = open('./b.out','w')
not_human = open('./c.out','w')

for ip in res_f: 
    if res_f[ip]:
        not_human.write('%s\n' % ip)
    else:
        human.write('%s\n' % ip)

human.close()
not_human.close()
sys.exit(0)

This User Gave Thanks to Klashxx For This Post:
Mr_47
# 16  
Old 03-20-2013
yup thats works great, anyway anyone can do it in shell script as my unix server did not have any python inside.
# 17  
Old 03-20-2013
Here is a shell script (bash/ksh) version

Code:
#!/bin/bash

rm -f b.out c.out
human=0
machine=0

while read line
do
   TM=${line%%AM*}AM
   [ ${#TM} -ge ${#line} ] && TM=${line%%PM*}PM
   IP=${line##*login:}
   IP=${IP%%:*}
   printf "%s %d\n" "$IP" $(date -d"$TM" +%s)
done < a.out | sort -t' ' -k1,1 -k2,2n | while read IP time
do
   if [ "$IP" = "$prev_ip" ]
   then
      if [ $human -eq 1 -o $(( time - prev_time )) -gt 4 ]
      then
         human=1
      else
         machine=1
      fi
   else
      [ $human -eq 1 ] && echo $prev_ip >> b.out
      [ $machine -eq 1 ] && echo $prev_ip >> c.out
      prev_ip=$IP
      prev_time=$time
      human=0
      machine=0
   fi
done
[ $human -eq 1 ] && echo $prev_ip >> b.out
[ $machine -eq 1 ] && echo $prev_ip >> c.out

Last edited by Chubler_XL; 03-20-2013 at 03:07 AM.. Reason: Correct sort params
# 18  
Old 03-20-2013
got error when run this,

bash-3.00# more code1.out
Code:
rm -f b.out c.out
human=0
machine=0

while read line
do
   TM=${line%%AM*}AM
   [ ${#TM} -ge ${#line} ] && TM=${line%%PM*}PM
   IP=${line##*login:}
   IP=${IP%%:*}
   printf "%s %d\n" "$IP" $(date -d"$TM" +%s)
done < a.out | sort -t' ' -k1,1 -k2,2n | while read IP time
do
   if [ "$IP" = "$prev_ip" ]
   then
      if [ $human -eq 1 -o $(( time - prev_time )) -gt 4 ]
      then
         human=1
      else
         machine=1
      fi
   else
      [ $human -eq 1 ] && echo $prev_ip >> b.out
      [ $machine -eq 1 ] && echo $prev_ip >> c.out
      prev_ip=$IP
      prev_time=$time
      human=0
      machine=0
   fi
done
[ $human -eq 1 ] && echo $prev_ip >> b.out
[ $machine -eq 1 ] && echo $prev_ip >> c.out


bash-3.00# ./code1.out
Code:
date: illegal option -- d
date: illegal option -- J
date: invalid argument -- n 9, 2013 2:32:29 AM
usage:  date [-u] mmddHHMM[[cc]yy][.SS]
        date [-u] [+format]
        date -a [-]sss[.fff]
date: illegal option -- d
date: illegal option -- J
date: invalid argument -- n 9, 2013 2:34:31 AM
usage:  date [-u] mmddHHMM[[cc]yy][.SS]
        date [-u] [+format]
        date -a [-]sss[.fff]
date: illegal option -- d
date: illegal option -- J
date: invalid argument -- n 9, 2013 2:36:51 AM
usage:  date [-u] mmddHHMM[[cc]yy][.SS]
        date [-u] [+format]
        date -a [-]sss[.fff]
date: illegal option -- d
date: illegal option -- J
date: invalid argument -- n 9, 2013 2:37:00 AM
usage:  date [-u] mmddHHMM[[cc]yy][.SS]
        date [-u] [+format]
        date -a [-]sss[.fff]
date: illegal option -- d
date: illegal option -- J
date: invalid argument -- n 9, 2013 2:37:12 AM
usage:  date [-u] mmddHHMM[[cc]yy][.SS]
        date [-u] [+format]
        date -a [-]sss[.fff]
date: illegal option -- d
date: illegal option -- J
date: invalid argument -- n 9, 2013 2:38:11 AM
usage:  date [-u] mmddHHMM[[cc]yy][.SS]
        date [-u] [+format]
        date -a [-]sss[.fff]
date: illegal option -- d
date: illegal option -- J
date: invalid argument -- n 9, 2013 2:41:06 AM
usage:  date [-u] mmddHHMM[[cc]yy][.SS]
        date [-u] [+format]
        date -a [-]sss[.fff]
date: illegal option -- d
date: illegal option -- J
date: invalid argument -- n 9, 2013 2:42:34 AM
usage:  date [-u] mmddHHMM[[cc]yy][.SS]
        date [-u] [+format]
        date -a [-]sss[.fff]
date: illegal option -- d
date: illegal option -- J
date: invalid argument -- n 9, 2013 2:45:28 AM
usage:  date [-u] mmddHHMM[[cc]yy][.SS]
        date [-u] [+format]
        date -a [-]sss[.fff]
date: illegal option -- d
date: illegal option -- J
date: invalid argument -- n 9, 2013 2:48:08 AM
usage:  date [-u] mmddHHMM[[cc]yy][.SS]
        date [-u] [+format]
        date -a [-]sss[.fff]
date: illegal option -- d
date: illegal option -- J
date: invalid argument -- n 9, 2013 2:49:22 AM
usage:  date [-u] mmddHHMM[[cc]yy][.SS]
        date [-u] [+format]
        date -a [-]sss[.fff]
date: illegal option -- d
date: illegal option -- J
date: invalid argument -- n 9, 2013 2:52:39 AM
usage:  date [-u] mmddHHMM[[cc]yy][.SS]
        date [-u] [+format]
        date -a [-]sss[.fff]
date: illegal option -- d
date: illegal option -- J
date: invalid argument -- n 9, 2013 2:55:36 AM
usage:  date [-u] mmddHHMM[[cc]yy][.SS]
        date [-u] [+format]
        date -a [-]sss[.fff]
date: illegal option -- d
date: illegal option -- J
date: invalid argument -- n 9, 2013 2:56:04 AM
usage:  date [-u] mmddHHMM[[cc]yy][.SS]
        date [-u] [+format]
        date -a [-]sss[.fff]
date: illegal option -- d
date: illegal option -- J
date: invalid argument -- n 9, 2013 2:58:17 AM
usage:  date [-u] mmddHHMM[[cc]yy][.SS]
        date [-u] [+format]
        date -a [-]sss[.fff]
date: illegal option -- d
date: illegal option -- J
date: invalid argument -- n 9, 2013 3:01:36 AM
usage:  date [-u] mmddHHMM[[cc]yy][.SS]
        date [-u] [+format]
        date -a [-]sss[.fff]
date: illegal option -- d
date: illegal option -- J
date: invalid argument -- n 9, 2013 3:08:25 AM
usage:  date [-u] mmddHHMM[[cc]yy][.SS]
        date [-u] [+format]
        date -a [-]sss[.fff]
date: illegal option -- d
date: illegal option -- J
date: invalid argument -- n 9, 2013 3:17:03 AM
usage:  date [-u] mmddHHMM[[cc]yy][.SS]
        date [-u] [+format]
        date -a [-]sss[.fff]
date: illegal option -- d
date: illegal option -- J
date: invalid argument -- n 9, 2013 3:19:10 AM
usage:  date [-u] mmddHHMM[[cc]yy][.SS]
        date [-u] [+format]
        date -a [-]sss[.fff]
date: illegal option -- d
date: illegal option -- J
date: invalid argument -- n 9, 2013 3:26:57 AM
usage:  date [-u] mmddHHMM[[cc]yy][.SS]
        date [-u] [+format]
        date -a [-]sss[.fff]
date: illegal option -- d
date: illegal option -- J
date: invalid argument -- n 9, 2013 3:28:53 AM
usage:  date [-u] mmddHHMM[[cc]yy][.SS]
        date [-u] [+format]
        date -a [-]sss[.fff]
date: illegal option -- d
date: illegal option -- J
date: invalid argument -- n 9, 2013 3:30:30 AM
usage:  date [-u] mmddHHMM[[cc]yy][.SS]
        date [-u] [+format]
        date -a [-]sss[.fff]
date: illegal option -- d
date: illegal option -- J
date: invalid argument -- n 9, 2013 3:43:08 AM
usage:  date [-u] mmddHHMM[[cc]yy][.SS]
        date [-u] [+format]
        date -a [-]sss[.fff]
date: illegal option -- d
date: illegal option -- J
date: invalid argument -- n 9, 2013 3:47:46 AM
usage:  date [-u] mmddHHMM[[cc]yy][.SS]
        date [-u] [+format]
        date -a [-]sss[.fff]
date: illegal option -- d
date: illegal option -- J
date: invalid argument -- n 9, 2013 3:47:47 AM
usage:  date [-u] mmddHHMM[[cc]yy][.SS]
        date [-u] [+format]
        date -a [-]sss[.fff]
date: illegal option -- d
date: illegal option -- J
date: invalid argument -- n 9, 2013 3:49:22 AM
usage:  date [-u] mmddHHMM[[cc]yy][.SS]
        date [-u] [+format]
        date -a [-]sss[.fff]
date: illegal option -- d
date: illegal option -- J
date: invalid argument -- n 9, 2013 3:49:35 AM
usage:  date [-u] mmddHHMM[[cc]yy][.SS]
        date [-u] [+format]
        date -a [-]sss[.fff]
date: illegal option -- d
date: illegal option -- J
date: invalid argument -- n 9, 2013 3:49:54 AM
usage:  date [-u] mmddHHMM[[cc]yy][.SS]
        date [-u] [+format]
        date -a [-]sss[.fff]
date: illegal option -- d
date: illegal option -- J
date: invalid argument -- n 9, 2013 3:51:45 AM
usage:  date [-u] mmddHHMM[[cc]yy][.SS]
        date [-u] [+format]
        date -a [-]sss[.fff]
date: illegal option -- d
date: illegal option -- J
date: invalid argument -- n 9, 2013 3:52:32 AM
usage:  date [-u] mmddHHMM[[cc]yy][.SS]
        date [-u] [+format]
        date -a [-]sss[.fff]
date: illegal option -- d
date: illegal option -- J
date: invalid argument -- n 9, 2013 3:54:25 AM
usage:  date [-u] mmddHHMM[[cc]yy][.SS]
        date [-u] [+format]
        date -a [-]sss[.fff]
date: illegal option -- d
date: illegal option -- J
date: invalid argument -- n 9, 2013 3:56:09 AM
usage:  date [-u] mmddHHMM[[cc]yy][.SS]
        date [-u] [+format]
        date -a [-]sss[.fff]
date: illegal option -- d
date: illegal option -- J
date: invalid argument -- n 9, 2013 3:59:38 AM
usage:  date [-u] mmddHHMM[[cc]yy][.SS]
        date [-u] [+format]
        date -a [-]sss[.fff]

bash-3.00# ls
Code:
a.out      c.out      code1.out

Last edited by Mr_47; 03-20-2013 at 04:10 AM.. Reason: missing separated code
# 19  
Old 03-20-2013
Yes, your date command doesn't support -d option (you need gnudate for the posted script). What OS are you using?
# 20  
Old 03-20-2013
bash-3.00# uname -a
SunOS mldap1 5.10 Generic_137138-09 i86pc i386 i86pc


its Solaris 10
# 21  
Old 03-20-2013
You will probably need to write a perl script to convert the date to epoch time
like this (name it strepoch.pl):
Code:
#!/usr/bin/perl
my @months = ("JAN", "FEB", "MAR", "APR", "MAY",
"JUN","JUL", "AUG", "SEP", "OCT", "NOV", "DEC");
use Time::Local 'timelocal';

my $mname, $month, $year, $month, $day, $hour, $min, $sec, $pm;

my $ln = $ARGV[0];
if($ln =~ /([A-z]{3}) (\d+), (\d{4}) (\d+):(\d\d):(\d\d) ([PA]M)/) {
  $ln =~ s/([A-z]{3}) (\d+), (\d{4}) (\d+):(\d\d):(\d\d) ([PA]M)/\1 \2 \3 \4 \5 \6 \7/;
  ($mname, $day, $year, $hour, $min, $sec, $pm) = split(/\s/, $ln);
  $hour += 12 if ($pm eq "PM" && $hour lt 12);
  ($month)=grep { $months[$_] eq uc($mname) } 0..$#months;
  printf "%d\n", timelocal($sec,$min,$hour,$day,$month,$year);
  exit(0);
} else {
  print { *STDERR } "Invalid date must be \"mmm dd, YYYY HH:MM:SS PM\"\n";
  exit(1);
}

Then in the script I posted before replace $(date -d"$TM" +%s) with $(./strepoch.pl "$TM")

---------- Post updated at 06:40 AM ---------- Previous update was at 06:38 AM ----------

Don't forget to chmod 755 strepoch.pl, to test it try:

Code:
$ ./strepoch.pl "Mar 9, 2013 11:59:59 AM"
1362794399

Last edited by Chubler_XL; 03-20-2013 at 05:53 PM..
This User Gave Thanks to Chubler_XL For This Post:
Mr_47
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

If I ran perl script again,old logs should move with today date and new logs should generate.

Appreciate help for the below issue. Im using below code.....I dont want to attach the logs when I ran the perl twice...I just want to take backup with today date and generate new logs...What I need to do for the below scirpt.............. 1)if logs exist it should move the logs with extention... (1 Reply)
Discussion started by: Sanjeev G
1 Replies

2. Shell Programming and Scripting

How to extract logs between the current time and the last 15 minutes ?

I want to extract the logs between the current time stamp and 15 minutes before and sent an email to the people configured. I developed the below script but it's not working properly; can someone help me?? I have a log file containing this pattern: Constructor QuartzJob ... (3 Replies)
Discussion started by: puneetkhullar
3 Replies

3. Shell Programming and Scripting

Perl script to extract last 24 hrs logs from cronlog

Hi Friends, Can anybody help me to create a perl script to generate log file for last 24 hrs from cron log file ?? Thank You (2 Replies)
Discussion started by: syamji.vm
2 Replies

4. Shell Programming and Scripting

Help with extract application logs through shell script in performance testing

Hi Experts, I am new to shell.How to extract logs (Web,APP,Database) using shell in performance testing? Need for webserver logs,app server logs and d/b logs code. Thanks in advance Sree (3 Replies)
Discussion started by: sree vasu
3 Replies

5. Shell Programming and Scripting

Search for a pattern,extract value(s) from next line, extract lines having those extracted value(s)

I have hundreds of files to process. In each file I need to look for a pattern then extract value(s) from next line and then search for value(s) selected from point (2) in the same file at a specific position. HEADER ELECTRON TRANSPORT 18-MAR-98 1A7V TITLE CYTOCHROME... (7 Replies)
Discussion started by: AshwaniSharma09
7 Replies

6. Shell Programming and Scripting

Help needed to extract distinct logs from a Log File

Hi, I urgently need some help how to extract distinct entries from a Log file. The Log File may have same error occuring many times so how do i count the occurance of an error in file and also extract out distinct errors in a file. Eg:- I have a file name A.log it contains entries as below:-... (5 Replies)
Discussion started by: roro
5 Replies

7. Shell Programming and Scripting

Grep yesterday logs from weblogic logs

Hi, I am trying to write a script which would go search and get the info from the logs based on yesterday timestamp and write yesterday logs in new file. The log file format is as follows: """"""""""""""""""""""""""... (3 Replies)
Discussion started by: harish.parker
3 Replies

8. UNIX for Advanced & Expert Users

logs

Hy, I have a question I have a directory in a unix server, Some of my files have a diffrent access time, from the time i accessed them last, I think some one has copied it,it's not an important file,but none the less,it is my file,It mistakenly had a 777 permission( yes ,I know it is a noob's... (1 Reply)
Discussion started by: lordmod
1 Replies

9. Shell Programming and Scripting

Logs

Hey Guys, i am new into shell programming and i have to do one script which have to record all the commands entered by a specific user. Example of that, i have a system running on unix, several users are using this system, i have to create like a databse which will record every user entered that... (5 Replies)
Discussion started by: charbel
5 Replies

10. UNIX for Dummies Questions & Answers

logs

can i include this command into my crontab file > /var/adm/wtmp to clear the contents on a regular basis ? what about file permissions ? (6 Replies)
Discussion started by: cubicle^dweller
6 Replies
Login or Register to Ask a Question