Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Extract IP from logs and

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Extract IP from logs and
# 1  
Old 03-11-2013
Extract IP from logs and
story
------------------------
The file is a.out, it consist a set of logs from user access to my system ( email system)


question
--------------------------
using shell script, how can we extract 2 sets of IP output from the a.out log by separate the IP, determine human and non-human access base on rules of multiple same IP access within maximum 4 sec time.

the rules,
1) multiple same IP access that maximum time more than 4 second time = human

2) multiple same IP access within maximum 4 second time = non-human

output
-------------------
output should be in b.out and c.out

meaning
1) b.out = IP list with rule 1
2) c.out = IP list with rule 2



more a.out log details,
Code:
Jan 9, 2013 2:32:29 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:220.255.2.108:null:login successful
Jan 9, 2013 2:34:31 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:116.197.5.39:null:login successful
Jan 9, 2013 2:36:51 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:210.186.188.172:null:login successful
Jan 9, 2013 2:37:00 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:202.162.27.126:null:login successful
Jan 9, 2013 2:37:12 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:220.255.2.128:null:login successful
Jan 9, 2013 2:38:11 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:210.186.241.248:null:login successful
Jan 9, 2013 2:41:06 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:220.255.2.159:null:login successful
Jan 9, 2013 2:42:34 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:124.13.171.171:null:login successful
Jan 9, 2013 2:45:28 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:202.188.41.226:null:login successful
Jan 9, 2013 2:48:08 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:210.195.222.207:null:login successful
Jan 9, 2013 2:49:22 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:60.52.28.59:null:login successful
Jan 9, 2013 2:52:39 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:41.138.175.227:null:login successful
Jan 9, 2013 2:55:36 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:116.197.5.39:null:login successful
Jan 9, 2013 2:56:04 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:175.144.156.85:null:login successful
Jan 9, 2013 2:58:17 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:58.71.157.170:null:login successful
Jan 9, 2013 3:01:36 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:41.203.67.54:null:login successful
Jan 9, 2013 3:08:25 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:175.136.58.228:null:login successful
Jan 9, 2013 3:17:03 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:41.138.180.156:null:login successful
Jan 9, 2013 3:19:10 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:41.138.180.156:null:login successful
Jan 9, 2013 3:26:57 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:41.138.180.156:null:login successful
Jan 9, 2013 3:28:53 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:49.249.114.228:null:login successful
Jan 9, 2013 3:30:30 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:41.138.180.156:null:login successful
Jan 9, 2013 3:43:08 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:188.52.46.85:null:login successful
Jan 9, 2013 3:47:46 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:58.27.85.20:null:login successful
Jan 9, 2013 3:47:47 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:202.162.27.126:null:login successful
Jan 9, 2013 3:49:22 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:175.140.126.74:null:login successful
Jan 9, 2013 3:49:35 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:58.27.85.20:null:login successful
Jan 9, 2013 3:49:54 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:58.27.85.20:null:login successful
Jan 9, 2013 3:51:45 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:60.50.3.143:null:login successful
Jan 9, 2013 3:52:32 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:175.141.95.105:null:login successful
Jan 9, 2013 3:54:25 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:175.143.242.152:null:login successful
Jan 9, 2013 3:56:09 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:175.143.242.152:null:login successful
Jan 9, 2013 3:59:38 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:124.13.188.59:null:login successful

# 2  
Old 03-11-2013
So, just to be clear, what you want given the a.out in your example is for b.out to be an exact copy of a.out, and either there will be no c.out or c.out will be an empty file. Is that what you want? If that isn't what you want please explain in more detail what the output is supposed to be.
# 3  
Old 03-11-2013
Although not entirely clear about what you want, this
Code:
awk     '{gsub (/:*login:*|:*null:*/,"")
          if (!max[$9] || $4>max[$9]) max[$9]=$4
          if (!min[$9] || $4<min[$9]) min[$9]=$4
         }
         END    {for (i in max) {split (max[i], MX, ":")
                                 split (min[i], MN, ":")
                                 Delta = (MX[1]-MN[1]) * 3600 + (MX[2]-MN[2]) * 60 + MX[3]-MN[3]
                                 print i, min[i], max[i] > ((Delta > 4)?"b":"c")".out"
                                }
                }
        ' OFS="\t" a.out

will list logins with a span larger than 4 sec between earliest and latest access to b.out, and the rest to c.out. It doesn't care if there's four accesses within four sec, indicating a non-human attempt. And, it does NOT take care of periods spanning across midnight - this would open a wide field, then.
# 4  
Old 03-11-2013
Quote:
Originally Posted by Don Cragun
So, just to be clear, what you want given the a.out in your example is for b.out to be an exact copy of a.out, and either there will be no c.out or c.out will be an empty file. Is that what you want? If that isn't what you want please explain in more detail what the output is supposed to be.
nope, a.out is the existing file. after we ran the shell script the command will generate 2 set of files, one is b.out which contain only IP from the a.out file and the other one is c.out which also contain only IP list from a.out file.
# 5  
Old 03-11-2013
Quote:
Originally Posted by Mr_47
nope, a.out is the existing file. after we ran the shell script the command will generate 2 set of files, one is b.out which contain only IP from the a.out file and the other one is c.out which also contain only IP list from a.out file.
And, (since there are only two records in your sample a.out with timestamps within 4 seconds of each other and those two records have different IPs), nothing will be written to c.out.
# 6  
Old 03-11-2013
I modified your file to make the test posible:
Code:
# cat a.txt
Jan 9, 2013 23:59:59 PM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:220.255.2.108:null:login successful
Jan 9, 2013 0:00:01 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:220.255.2.108:null:login successful
Jan 9, 2013 2:32:29 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:220.255.2.108:null:login successful
Jan 9, 2013 2:34:31 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:116.197.5.39:null:login successful
Jan 9, 2013 2:36:51 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:210.186.188.172:null:login successful
Jan 9, 2013 2:37:00 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:202.162.27.126:null:login successful
Jan 9, 2013 2:37:12 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:220.255.2.128:null:login successful
Jan 9, 2013 2:38:11 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:210.186.241.248:null:login successful
Jan 9, 2013 2:41:06 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:220.255.2.159:null:login successful
Jan 9, 2013 2:42:34 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:124.13.171.171:null:login successful
Jan 9, 2013 2:45:28 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:202.188.41.226:null:login successful
Jan 9, 2013 2:48:08 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:210.195.222.207:null:login successful
Jan 9, 2013 2:49:22 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:60.52.28.59:null:login successful
Jan 9, 2013 2:52:39 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:41.138.175.227:null:login successful
Jan 9, 2013 2:55:36 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:116.197.5.39:null:login successful
Jan 9, 2013 2:56:04 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:175.144.156.85:null:login successful
Jan 9, 2013 2:58:17 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:58.71.157.170:null:login successful
Jan 9, 2013 2:58:18 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:58.71.157.170:null:login successful
Jan 9, 2013 2:58:19 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:58.71.157.170:null:login successful
Jan 9, 2013 3:01:36 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:41.203.67.54:null:login successful
Jan 9, 2013 3:08:25 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:175.136.58.228:null:login successful
Jan 9, 2013 3:17:03 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:41.138.180.156:null:login successful
Jan 9, 2013 3:19:10 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:41.138.180.156:null:login successful
Jan 9, 2013 3:26:57 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:41.138.180.156:null:login successful
Jan 9, 2013 3:28:53 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:49.249.114.228:null:login successful
Jan 9, 2013 3:30:30 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:41.138.180.156:null:login successful
Jan 9, 2013 3:43:08 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:188.52.46.85:null:login successful
Jan 9, 2013 3:47:46 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:58.27.85.20:null:login successful
Jan 9, 2013 3:47:47 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:202.162.27.126:null:login successful
Jan 9, 2013 3:49:22 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:175.140.126.74:null:login successful
Jan 9, 2013 3:49:35 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:58.27.85.20:null:login successful
Jan 9, 2013 3:49:54 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:58.27.85.20:null:login successful
Jan 9, 2013 3:51:45 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:60.50.3.143:null:login successful
Jan 9, 2013 3:52:32 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:175.141.95.105:null:login successful
Jan 9, 2013 3:52:34 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:175.141.95.105:null:login successful
Jan 9, 2013 3:52:36 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:175.141.95.105:null:login successful
Jan 9, 2013 3:54:25 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:175.143.242.152:null:login successful
Jan 9, 2013 3:56:09 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:175.143.242.152:null:login successful
Jan 9, 2013 3:59:38 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:124.13.188.59:null:login successful

Then (if python is ok):
Code:
# python -V
Python 2.4.2

Code:
cat ips.py 
#!/usr/bin/python

import re
import sys
from datetime import datetime 
import time
strptime = lambda date_string, format: datetime(*(time.strptime(date_string, format)[0:6]))

try:
   f = open('./a.txt', 'r')
   out_fic=f.read()
   f.close()
except IOError:
   print "File error"
   sys.exit(5)

FMT = '%b %d, %Y %H:%M:%S %p'
pat='^(.+\s+(AM|PM)).+login:(\d+\.\d+\.\d+\.\d+):'
seek_p=re.compile(pat,re.MULTILINE)

res_h={}
res_f={}
for match_p in seek_p.findall(out_fic):
   time_p=match_p[0]
   ip_p=match_p[2]
   try:
      last=res_h[ip_p]
      res_h[ip_p]=time_p
      delta=strptime(time_p,FMT) - strptime(last,FMT)
      delta_segs=delta.seconds 
      if delta_segs <= 4 : 
          res_f[ip_p]=True
   except KeyError:
      res_h[ip_p]=time_p
      res_f[ip_p]=False

for ip in res_f: 
    if res_f[ip]:print 'Not human ip connect : %s' % ip

Code:
./ips.py
Not human ip connect : 58.71.157.170
Not human ip connect : 220.255.2.108
Not human ip connect : 175.141.95.105

This User Gave Thanks to Klashxx For This Post:
Mr_47
# 7  
Old 03-13-2013
thanks, need to test it out first. will let you know the result soon.

---------- Post updated at 03:40 PM ---------- Previous update was at 03:01 PM ----------

seems there is no python in my unix server
Code:
[root|webmail5.tm.net.my:/tmp] python -V
ksh: python:  not found

---------- Post updated at 03:46 PM ---------- Previous update was at 03:40 PM ----------

Quote:
Originally Posted by RudiC
Although not entirely clear about what you want, this
Code:
awk     '{gsub (/:*login:*|:*null:*/,"")
          if (!max[$9] || $4>max[$9]) max[$9]=$4
          if (!min[$9] || $4<min[$9]) min[$9]=$4
         }
         END    {for (i in max) {split (max[i], MX, ":")
                                 split (min[i], MN, ":")
                                 Delta = (MX[1]-MN[1]) * 3600 + (MX[2]-MN[2]) * 60 + MX[3]-MN[3]
                                 print i, min[i], max[i] > ((Delta > 4)?"b":"c")".out"
                                }
                }
        ' OFS="\t" a.out

will list logins with a span larger than 4 sec between earliest and latest access to b.out, and the rest to c.out. It doesn't care if there's four accesses within four sec, indicating a non-human attempt. And, it does NOT take care of periods spanning across midnight - this would open a wide field, then.

how to test this,

already put the code in code1.sh with the same directory to a.out file,

then enable all access to the code file
Code:
chmod 777 code1.sh

finally execute this with
Code:
./code1.sh

result,
Code:
[root|webmail5.tm.net.my:/tmp] ./code1.sh
awk: syntax error near line 1
awk: illegal statement near line 1
awk: syntax error near line 8
awk: illegal statement near line 8
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

If I ran perl script again,old logs should move with today date and new logs should generate.

Appreciate help for the below issue. Im using below code.....I dont want to attach the logs when I ran the perl twice...I just want to take backup with today date and generate new logs...What I need to do for the below scirpt.............. 1)if logs exist it should move the logs with extention... (1 Reply)
Discussion started by: Sanjeev G
1 Replies

2. Shell Programming and Scripting

How to extract logs between the current time and the last 15 minutes ?

I want to extract the logs between the current time stamp and 15 minutes before and sent an email to the people configured. I developed the below script but it's not working properly; can someone help me?? I have a log file containing this pattern: Constructor QuartzJob ... (3 Replies)
Discussion started by: puneetkhullar
3 Replies

3. Shell Programming and Scripting

Perl script to extract last 24 hrs logs from cronlog

Hi Friends, Can anybody help me to create a perl script to generate log file for last 24 hrs from cron log file ?? Thank You (2 Replies)
Discussion started by: syamji.vm
2 Replies

4. Shell Programming and Scripting

Help with extract application logs through shell script in performance testing

Hi Experts, I am new to shell.How to extract logs (Web,APP,Database) using shell in performance testing? Need for webserver logs,app server logs and d/b logs code. Thanks in advance Sree (3 Replies)
Discussion started by: sree vasu
3 Replies

5. Shell Programming and Scripting

Search for a pattern,extract value(s) from next line, extract lines having those extracted value(s)

I have hundreds of files to process. In each file I need to look for a pattern then extract value(s) from next line and then search for value(s) selected from point (2) in the same file at a specific position. HEADER ELECTRON TRANSPORT 18-MAR-98 1A7V TITLE CYTOCHROME... (7 Replies)
Discussion started by: AshwaniSharma09
7 Replies

6. Shell Programming and Scripting

Help needed to extract distinct logs from a Log File

Hi, I urgently need some help how to extract distinct entries from a Log file. The Log File may have same error occuring many times so how do i count the occurance of an error in file and also extract out distinct errors in a file. Eg:- I have a file name A.log it contains entries as below:-... (5 Replies)
Discussion started by: roro
5 Replies

7. Shell Programming and Scripting

Grep yesterday logs from weblogic logs

Hi, I am trying to write a script which would go search and get the info from the logs based on yesterday timestamp and write yesterday logs in new file. The log file format is as follows: """"""""""""""""""""""""""... (3 Replies)
Discussion started by: harish.parker
3 Replies

8. UNIX for Advanced & Expert Users

logs

Hy, I have a question I have a directory in a unix server, Some of my files have a diffrent access time, from the time i accessed them last, I think some one has copied it,it's not an important file,but none the less,it is my file,It mistakenly had a 777 permission( yes ,I know it is a noob's... (1 Reply)
Discussion started by: lordmod
1 Replies

9. Shell Programming and Scripting

Logs

Hey Guys, i am new into shell programming and i have to do one script which have to record all the commands entered by a specific user. Example of that, i have a system running on unix, several users are using this system, i have to create like a databse which will record every user entered that... (5 Replies)
Discussion started by: charbel
5 Replies

10. UNIX for Dummies Questions & Answers

logs

can i include this command into my crontab file > /var/adm/wtmp to clear the contents on a regular basis ? what about file permissions ? (6 Replies)
Discussion started by: cubicle^dweller
6 Replies
Login or Register to Ask a Question