Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Extract IP from logs and

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Extract IP from logs and
# 8  
Old 03-13-2013
I've tested this on my linux platform; it works fine. What system /awk version do you have? You may need to explicitly invoke e.g. nawk...
# 9  
Old 03-13-2013
Quote:
Originally Posted by RudiC
I've tested this on my linux platform; it works fine. What system /awk version do you have? You may need to explicitly invoke e.g. nawk...
Hi RudiC,
While trying to run your code on OS/X, I got the error:
Code:
awk: syntax error at source line 8
 context is
	                                 print i, min[i], max[i] > ((Delta > >>>  4)?"b":"c")".out" <<< 
awk: illegal statement at source line 9

I reformatted your script and changed line 8 to the following:
Code:
awk '
{       gsub (/:*login:*|:*null:*/,"")
        if (!max[$9] || $4>max[$9]) max[$9]=$4
        if (!min[$9] || $4<min[$9]) min[$9]=$4
}
END {   for (i in max) {
                split (max[i], MX, ":")
                split (min[i], MN, ":")
                Delta = (MX[1]-MN[1]) * 3600 + (MX[2]-MN[2]) * 60 + MX[3]-MN[3]
                if(Delta > 4)   print i, min[i], max[i] > "b.out"
                else            print i, min[i], max[i] > "c.out"
        }
}' OFS="\t" a.out

That produces the output files:
b.out:
Code:
116.197.5.39    2:34:31 2:55:36
175.143.242.152 3:54:25 3:56:09
41.138.180.156  3:17:03 3:30:30
202.162.27.126  2:37:00 3:47:47
58.27.85.20     3:47:46 3:49:54

and c.out:
Code:
202.188.41.226  2:45:28 2:45:28
175.141.95.105  3:52:32 3:52:32
188.52.46.85    3:43:08 3:43:08
210.186.188.172 2:36:51 2:36:51
220.255.2.159   2:41:06 2:41:06
175.140.126.74  3:49:22 3:49:22
210.186.241.248 2:38:11 2:38:11
210.195.222.207 2:48:08 2:48:08
220.255.2.128   2:37:12 2:37:12
41.203.67.54    3:01:36 3:01:36
58.71.157.170   2:58:17 2:58:17
60.50.3.143     3:51:45 3:51:45
49.249.114.228  3:28:53 3:28:53
60.52.28.59     2:49:22 2:49:22
175.136.58.228  3:08:25 3:08:25
41.138.175.227  2:52:39 2:52:39
175.144.156.85  2:56:04 2:56:04
124.13.171.171  2:42:34 2:42:34
124.13.188.59   3:59:38 3:59:38
220.255.2.108   2:32:26 2:32:29

I would have thought that the IP addresses that appeared in the input file only once and the IP addresses where all of the timestamps of consecutive entries for that IP address were each more than 4 seconds apart should appear in b.out rather than c.out. But, I agree that the requirements are not clear.

Note also that your calculation of Delta assumes 24 hour time notation in field 4. But with field 5 being "AM" in all of the sample input, I believe the timestamps are 12 hour time notation using fields 4 and 5.

Hi Mr_47,
I am still not clear as to what output you want. Exactly what output do you want in b.out and c.out if a.out contains the following:
Code:
Jan 9, 2013 3:01:36 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:41.203.67.54:null:login successful
Jan 9, 2013 3:01:41 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:41.203.67.54:null:login successful
Jan 9, 2013 3:01:42 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:175.136.58.228:null:login successful
Jan 9, 2013 3:01:43 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:41.203.67.54:null:login successful
Jan 9, 2013 3:02:43 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:175.136.58.228:null:login successful
Jan 9, 2013 3:03:43 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:175.136.58.228:null:login successful

This User Gave Thanks to Don Cragun For This Post:
RudiC
# 10  
Old 03-13-2013
@Don Cragun: LOL - that was my first attempt
Quote:
if(Delta > 4) print i, min[i], max[i] > "b.out"
else print i, min[i], max[i] > "c.out"
and I thought: "I can do this smarter!"

And, you're right - that "AM" time postfix slipped through. Sorry and thanks for that, too. But there's even more to correct date/time arithmetics, as I stated in my post.

After all I share the point of view that the specification should be way more precise...
# 11  
Old 03-14-2013
if a.out contains,

Code:
Jan 9, 2013 3:01:36 AM   com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:41.203.67.54:null:login successful
Jan 9, 2013 3:01:41 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:41.203.67.54:null:login successful
Jan 9, 2013 3:01:42 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:175.136.58.228:null:login successful
Jan 9, 2013 3:01:43 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:41.203.67.54:null:login successful
Jan 9, 2013 3:02:43 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:175.136.58.228:null:login successful
Jan 9, 2013 3:03:43 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:175.136.58.228:null:login successful

then the output b.out should be

b.out
Code:
175.136.58.228


c.out
Code:
41.203.67.54

# 12  
Old 03-14-2013
Give this a shot:
Code:
awk     '       {gsub (/:*login:*|:*null:*/,"")
                 split ($4, TMP, ":")
                 TIME = TMP[1] * 3600 + TMP[2] * 60 + TMP[3]
                 if ($5 = "PM") TIME = TIME + 43200
                 if (LAST[$9])  {DELTA = TIME - LAST[$9]
                                 if (!DT[$9] || (DELTA < DT[$9])) DT[$9] = DELTA
                                }
                 LAST[$9] = TIME
                }
         END    {for (i in DT)  if (DT[i] > 4)  print i > "b.out"
                                  else          print i > "c.out"
                }
        ' OFS="\t" file

# 13  
Old 03-14-2013
Quote:
Originally Posted by Mr_47
if a.out contains,

Code:
Jan 9, 2013 3:01:36 AM   com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:41.203.67.54:null:login successful
Jan 9, 2013 3:01:41 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:41.203.67.54:null:login successful
Jan 9, 2013 3:01:42 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:175.136.58.228:null:login successful
Jan 9, 2013 3:01:43 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:41.203.67.54:null:login successful
Jan 9, 2013 3:02:43 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:175.136.58.228:null:login successful
Jan 9, 2013 3:03:43 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:175.136.58.228:null:login successful

then the output b.out should be

b.out
Code:
175.136.58.228


c.out
Code:
41.203.67.54

Given that there is more than 4 seconds between:
Code:
Jan 9, 2013 3:01:36 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:41.203.67.54:null:login successful
Jan 9, 2013 3:01:41 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:41.203.67.54:null:login successful

and less than 4 seconds between:
Code:
Jan 9, 2013 3:01:41 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:41.203.67.54:null:login successful
Jan 9, 2013 3:01:43 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:41.203.67.54:null:login successful

why isn't IP address 41.203.67.54 in both b.out and c.out?
# 14  
Old 03-19-2013
Quote:
Originally Posted by Don Cragun
Given that there is more than 4 seconds between:
Code:
Jan 9, 2013 3:01:36 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:41.203.67.54:null:login successful
Jan 9, 2013 3:01:41 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:41.203.67.54:null:login successful

and less than 4 seconds between:
Code:
Jan 9, 2013 3:01:41 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:41.203.67.54:null:login successful
Jan 9, 2013 3:01:43 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:41.203.67.54:null:login successful

why isn't IP address 41.203.67.54 in both b.out and c.out?

Hi,

from
Code:
Jan 9, 2013 3:01:41 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:41.203.67.54:null:login successful
Jan 9, 2013 3:01:43 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:41.203.67.54:null:login successful


the IP was confirmed a non-human access because it authenticate more than 1 time within less than 4 second. so it will straight go to non human list in c.out.

---------- Post updated at 02:16 PM ---------- Previous update was at 02:14 PM ----------

Quote:
Originally Posted by RudiC
Give this a shot:
Code:
awk     '       {gsub (/:*login:*|:*null:*/,"")
                 split ($4, TMP, ":")
                 TIME = TMP[1] * 3600 + TMP[2] * 60 + TMP[3]
                 if ($5 = "PM") TIME = TIME + 43200
                 if (LAST[$9])  {DELTA = TIME - LAST[$9]
                                 if (!DT[$9] || (DELTA < DT[$9])) DT[$9] = DELTA
                                }
                 LAST[$9] = TIME
                }
         END    {for (i in DT)  if (DT[i] > 4)  print i > "b.out"
                                  else          print i > "c.out"
                }
        ' OFS="\t" file

this wont work as using,

more code1.sh
Code:
awk     '       {gsub (/:*login:*|:*null:*/,"")
                 split ($4, TMP, ":")
                 TIME = TMP[1] * 3600 + TMP[2] * 60 + TMP[3]
                 if ($5 = "PM") TIME = TIME + 43200
                 if (LAST[$9])  {DELTA = TIME - LAST[$9]
                                 if (!DT[$9] || (DELTA < DT[$9])) DT[$9] = DELTA
                                }
                 LAST[$9] = TIME
                }
         END    {for (i in DT)  if (DT[i] > 4)  print i > "b.out"
                                  else          print i > "c.out"
                }
        ' OFS="\t" file

then
Code:
./code1.sh
 ./code1.sh
awk: syntax error near line 1
awk: illegal statement near line 1

---------- Post updated at 05:08 PM ---------- Previous update was at 02:16 PM ----------

Quote:
Originally Posted by Klashxx
I modified your file to make the test posible:
Code:
# cat a.txt
Jan 9, 2013 23:59:59 PM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:220.255.2.108:null:login successful
Jan 9, 2013 0:00:01 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:220.255.2.108:null:login successful
Jan 9, 2013 2:32:29 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:220.255.2.108:null:login successful
Jan 9, 2013 2:34:31 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:116.197.5.39:null:login successful
Jan 9, 2013 2:36:51 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:210.186.188.172:null:login successful
Jan 9, 2013 2:37:00 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:202.162.27.126:null:login successful
Jan 9, 2013 2:37:12 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:220.255.2.128:null:login successful
Jan 9, 2013 2:38:11 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:210.186.241.248:null:login successful
Jan 9, 2013 2:41:06 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:220.255.2.159:null:login successful
Jan 9, 2013 2:42:34 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:124.13.171.171:null:login successful
Jan 9, 2013 2:45:28 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:202.188.41.226:null:login successful
Jan 9, 2013 2:48:08 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:210.195.222.207:null:login successful
Jan 9, 2013 2:49:22 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:60.52.28.59:null:login successful
Jan 9, 2013 2:52:39 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:41.138.175.227:null:login successful
Jan 9, 2013 2:55:36 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:116.197.5.39:null:login successful
Jan 9, 2013 2:56:04 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:175.144.156.85:null:login successful
Jan 9, 2013 2:58:17 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:58.71.157.170:null:login successful
Jan 9, 2013 2:58:18 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:58.71.157.170:null:login successful
Jan 9, 2013 2:58:19 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:58.71.157.170:null:login successful
Jan 9, 2013 3:01:36 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:41.203.67.54:null:login successful
Jan 9, 2013 3:08:25 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:175.136.58.228:null:login successful
Jan 9, 2013 3:17:03 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:41.138.180.156:null:login successful
Jan 9, 2013 3:19:10 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:41.138.180.156:null:login successful
Jan 9, 2013 3:26:57 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:41.138.180.156:null:login successful
Jan 9, 2013 3:28:53 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:49.249.114.228:null:login successful
Jan 9, 2013 3:30:30 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:41.138.180.156:null:login successful
Jan 9, 2013 3:43:08 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:188.52.46.85:null:login successful
Jan 9, 2013 3:47:46 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:58.27.85.20:null:login successful
Jan 9, 2013 3:47:47 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:202.162.27.126:null:login successful
Jan 9, 2013 3:49:22 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:175.140.126.74:null:login successful
Jan 9, 2013 3:49:35 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:58.27.85.20:null:login successful
Jan 9, 2013 3:49:54 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:58.27.85.20:null:login successful
Jan 9, 2013 3:51:45 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:60.50.3.143:null:login successful
Jan 9, 2013 3:52:32 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:175.141.95.105:null:login successful
Jan 9, 2013 3:52:34 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:175.141.95.105:null:login successful
Jan 9, 2013 3:52:36 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:175.141.95.105:null:login successful
Jan 9, 2013 3:54:25 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:175.143.242.152:null:login successful
Jan 9, 2013 3:56:09 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:175.143.242.152:null:login successful
Jan 9, 2013 3:59:38 AM com.sun.uwc.common.auth.IdentitySSOAuthFilter doFilter INFO login:124.13.188.59:null:login successful

Then (if python is ok):
Code:
# python -V
Python 2.4.2

Code:
cat ips.py 
#!/usr/bin/python

import re
import sys
from datetime import datetime 
import time
strptime = lambda date_string, format: datetime(*(time.strptime(date_string, format)[0:6]))

try:
   f = open('./a.txt', 'r')
   out_fic=f.read()
   f.close()
except IOError:
   print "File error"
   sys.exit(5)

FMT = '%b %d, %Y %H:%M:%S %p'
pat='^(.+\s+(AM|PM)).+login:(\d+\.\d+\.\d+\.\d+):'
seek_p=re.compile(pat,re.MULTILINE)

res_h={}
res_f={}
for match_p in seek_p.findall(out_fic):
   time_p=match_p[0]
   ip_p=match_p[2]
   try:
      last=res_h[ip_p]
      res_h[ip_p]=time_p
      delta=strptime(time_p,FMT) - strptime(last,FMT)
      delta_segs=delta.seconds 
      if delta_segs <= 4 : 
          res_f[ip_p]=True
   except KeyError:
      res_h[ip_p]=time_p
      res_f[ip_p]=False

for ip in res_f: 
    if res_f[ip]:print 'Not human ip connect : %s' % ip

Code:
./ips.py
Not human ip connect : 58.71.157.170
Not human ip connect : 220.255.2.108
Not human ip connect : 175.141.95.105


well this simply works, but without a list in b.out ( human access) and c.out ( non human access)
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

If I ran perl script again,old logs should move with today date and new logs should generate.

Appreciate help for the below issue. Im using below code.....I dont want to attach the logs when I ran the perl twice...I just want to take backup with today date and generate new logs...What I need to do for the below scirpt.............. 1)if logs exist it should move the logs with extention... (1 Reply)
Discussion started by: Sanjeev G
1 Replies

2. Shell Programming and Scripting

How to extract logs between the current time and the last 15 minutes ?

I want to extract the logs between the current time stamp and 15 minutes before and sent an email to the people configured. I developed the below script but it's not working properly; can someone help me?? I have a log file containing this pattern: Constructor QuartzJob ... (3 Replies)
Discussion started by: puneetkhullar
3 Replies

3. Shell Programming and Scripting

Perl script to extract last 24 hrs logs from cronlog

Hi Friends, Can anybody help me to create a perl script to generate log file for last 24 hrs from cron log file ?? Thank You (2 Replies)
Discussion started by: syamji.vm
2 Replies

4. Shell Programming and Scripting

Help with extract application logs through shell script in performance testing

Hi Experts, I am new to shell.How to extract logs (Web,APP,Database) using shell in performance testing? Need for webserver logs,app server logs and d/b logs code. Thanks in advance Sree (3 Replies)
Discussion started by: sree vasu
3 Replies

5. Shell Programming and Scripting

Search for a pattern,extract value(s) from next line, extract lines having those extracted value(s)

I have hundreds of files to process. In each file I need to look for a pattern then extract value(s) from next line and then search for value(s) selected from point (2) in the same file at a specific position. HEADER ELECTRON TRANSPORT 18-MAR-98 1A7V TITLE CYTOCHROME... (7 Replies)
Discussion started by: AshwaniSharma09
7 Replies

6. Shell Programming and Scripting

Help needed to extract distinct logs from a Log File

Hi, I urgently need some help how to extract distinct entries from a Log file. The Log File may have same error occuring many times so how do i count the occurance of an error in file and also extract out distinct errors in a file. Eg:- I have a file name A.log it contains entries as below:-... (5 Replies)
Discussion started by: roro
5 Replies

7. Shell Programming and Scripting

Grep yesterday logs from weblogic logs

Hi, I am trying to write a script which would go search and get the info from the logs based on yesterday timestamp and write yesterday logs in new file. The log file format is as follows: """"""""""""""""""""""""""... (3 Replies)
Discussion started by: harish.parker
3 Replies

8. UNIX for Advanced & Expert Users

logs

Hy, I have a question I have a directory in a unix server, Some of my files have a diffrent access time, from the time i accessed them last, I think some one has copied it,it's not an important file,but none the less,it is my file,It mistakenly had a 777 permission( yes ,I know it is a noob's... (1 Reply)
Discussion started by: lordmod
1 Replies

9. Shell Programming and Scripting

Logs

Hey Guys, i am new into shell programming and i have to do one script which have to record all the commands entered by a specific user. Example of that, i have a system running on unix, several users are using this system, i have to create like a databse which will record every user entered that... (5 Replies)
Discussion started by: charbel
5 Replies

10. UNIX for Dummies Questions & Answers

logs

can i include this command into my crontab file > /var/adm/wtmp to clear the contents on a regular basis ? what about file permissions ? (6 Replies)
Discussion started by: cubicle^dweller
6 Replies
Login or Register to Ask a Question