Sponsored Content
Top Forums UNIX for Dummies Questions & Answers command to extract IP data from syslogs Post 64601 by di0de on Tuesday 1st of March 2005 10:15:55 PM
Old 03-01-2005
thanks for your help

Quote:
Originally Posted by zazzybob
Welcome to the forums.

Please post an sample of the log file you wish to extract data from (change IP's and any sensitive information first), and an example of the intended output. Without this, it's impossible to assist. The output of uname -a and echo $SHELL is always a boon too....

Thanks,
ZB
here is an example:

if this is the log file:


192.168.1.100 - - [08/Dec/2005:18:16:07 -0500] "GET /images/welcome.png HTTP/1.1"
200 3032
192.168.1.101 - - [08/Dec/2005:18:16:07 -0500] "GET /images/google.gif HTTP/1.1" 2
00 1680
192.168.1.120 - - [08/Dec/2005:18:16:07 -0500] "GET /images/jeeves.gif HTTP/1.1" 2
00 1976
192.168.1.1 - - [08/Dec/2005:18:16:07 -0500] "GET /images/dogpile.gif HTTP/1.1"
200 985
192.168.100.1 - - [08/Dec/2005:18:16:07 -0500] "GET /images/yahoo.gif HTTP/1.1" 20
0 718
192.168.1.1 - - [08/Dec/2005:18:16:07 -0500] "GET /images/lycos.gif HTTP/1.1" 20
0 1348
192.168.10.1 - - [08/Dec/2005:18:16:07 -0500] "GET /images/av.jpg HTTP/1.1" 200 1
632
192.168.1.11 - - [08/Dec/2005:18:16:07 -0500] "GET /images/valid.gif HTTP/1.1" 20
0 2328
192.168.1.1 - - [08/Dec/2005:18:16:08 -0500] "GET /images/easy.gif HTTP/1.1" 200
1518
192.168.1.1 - - [08/Dec/2005:18:32:18 -0500] "GET / HTTP/1.1" 200 3643
23.34.54.121 - - [18/Dec/2005:08:24:47 -0500] "GET /car/ext-1.jpg HTTP/1.1" 200 4
3611
23.34.54.122 - - [18/Dec/2005:08:25:10 -0500] "GET /car/int-2.jpg HTTP/1.1" 200 4
7963
23.34.54.110 - - [18/Dec/2005:08:25:12 -0500] "GET /car/int-1.jpg HTTP/1.1" 200 5
7079
23.34.54.111 - - [18/Dec/2005:08:31:41 -0500]

[Mon Oct 28 22:02:10 2005] [error] [client 192.168.1.1] File does not exist: /ho
me/httpd/di0de/index.php
[Tue Oct 29 01:23:18 2005] [error] [client 200.81.81.70] client sent HTTP/1.1 re
quest without hostname (see RFC2616 section 14.23): /
[Tue Oct 29 10:27:07 2005] [error] [client 216.39.100.132] File does not exist:
/home/httpd/di0de/scripts/..%5c..%5cwinnt/system32/cmd.exe
[Tue Oct 29 16:30:16 2005] [error] [client 192.168.1.1] File does not exist: /ho
me/httpd/di0de/010102.html

I would like to extract only the IP info into a new file such as:

192.168.1.1
192.168.1.100
192.168.1.11
23.34.54.111
...
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

extract data from a data matrix with filter criteria

Here is what old matrix look like, IDs X1 X2 Y1 Y2 10914061 -0.364613333 -0.362922333 0.001691 -0.450094667 10855062 0.845956333 0.860396667 0.014440333 1.483899333... (7 Replies)
Discussion started by: ssshen
7 Replies

2. Shell Programming and Scripting

how to extract the data ?

Hi, I'm trying to pick out a data field eg. from below. I need the required field as below but they are filled sometimes with weird chars like \-(. or watever. How can I accurately extract the 3rd field in shell? :confused: ID IDNO - REQUIRED FIELD ID 1447 - MAT620BR. ID 1452 -... (13 Replies)
Discussion started by: uxnoob
13 Replies

3. Shell Programming and Scripting

Extract data based on match against one column data from a long list data

My input file: data_5 Ali 422 2.00E-45 102/253 140/253 24 data_3 Abu 202 60.00E-45 12/23 140/23 28 data_1 Ahmad 256 7.00E-45 120/235 140/235 22 data_4 Aman 365 8.00E-45 15/65 140/65 20 data_10 Jones 869 9.00E-45 65/253 140/253 18... (12 Replies)
Discussion started by: patrick87
12 Replies

4. Shell Programming and Scripting

Extract specific data content from a long list of data

My input: Data name: ABC001 Data length: 1000 Detail info Data Direction Start_time End_time Length 1 forward 10 100 90 1 forward 15 200 185 2 reverse 50 500 450 Data name: XFG110 Data length: 100 Detail info Data Direction Start_time End_time Length 1 forward 50 100 50 ... (11 Replies)
Discussion started by: patrick87
11 Replies

5. Solaris

SYSLOGS - Where can I find FTP activities

Greetings to all. I need help from the experts. I have been given a FTP server script that runs all day, looking for files that are FTP'd to our machines. Its hoaky I know, but there are times that files are sent but somehow get lost. Is there a logfile I can view to see when files are received? ... (1 Reply)
Discussion started by: Harleyrci
1 Replies

6. Shell Programming and Scripting

extract the data using AWK command

In a file i have a data like INPUT: no,name,company 1,vivek,hcl 2,senthil,cts 1,narsi,hcl 4,prabhakaran,ibm OUTPUT: 1,vivek,hcl 1,narsi,hcl Using AWK command i want to display the names those having no:1 and company:hcl.Please tell me the command to display above result. (8 Replies)
Discussion started by: katakamvivek
8 Replies

7. Shell Programming and Scripting

Extracting relevant information from syslogs.

I need to analyse some syslogs and I want to print out all the lines containing SSH connections to the inside interface of the firewall and ignore lines where the originating port is 22. So basically I want to print all matches after "to inside:" that contains /22 and ignore lines where /22 occur... (2 Replies)
Discussion started by: lewk
2 Replies

8. AIX

Syslogs permission not getting changed - aix 6.1

Friends, I've tried to modify the syslogs permission by using the perm option in the syslog configuration in AIX 6.1 TL 05. But its not getting applied after the configuration. Have restarted the syslog service also. Need your help!:wall: The below are the conf details and os versions >... (1 Reply)
Discussion started by: novaothers
1 Replies

9. Shell Programming and Scripting

Extract header data from one file and combine it with data from another file

Hi, Great minds, I have some files, in fact header files, of CTD profiler, I tried a lot C programming, could not get output as I was expected, because my programming skills are very poor, finally, joined unix forum with the hope that, I may get what I want, from you people, Here I have attached... (17 Replies)
Discussion started by: nex_asp
17 Replies

10. Solaris

Configuring Centralized Solaris Server to Receive Syslogs

Hello, I am new to Solaris. However, I was able to configure the Syslog server on Solaris. I am using a windows client to send logs to my Solaris syslog server. this is working fine. however, I have 23 servers which are sending their logs to the syslog server and it is very difficult to read all... (2 Replies)
Discussion started by: ravmohun
2 Replies
IpfilterDlfConverter(3pm)				  LogReport's Lire Documentation				 IpfilterDlfConverter(3pm)

NAME
Lire::Firewall::IpfilterDlfConverter - convert ipf (ipmon) logs to firewall DLF DESCRIPTION
Lire::Firewall::IpfilterDlfConverter converts Ipfilter logs into firewall DLF format. Input for this converter is the standard ipf syslog log file as produced by ipmon. IP Filter is shipped with FreeBSD, OpenBSD (up to 2.9) and some other OS's. EXAMPLE
A ipfilter logfile which looks like Oct 30 07:42:29 rolle ipmon[16747]: 07:42:28.585962 ie0 @0:9 b 192.168.48.1,45085 -> 192.168.48.2,22 PR tcp len 20 64 -S OUT Oct 30 07:40:24 rolle ipmon[16747]: 07:40:23.631307 ep1 @0:6 b 192.168.26.5,113 -> 192.168.26.1,3717 PR tcp len 20 40 -AR OUT Oct 30 07:42:29 rolle ipmon[16747]: 07:42:28.585962 ie0 @0:9 b 192.168.48.1,45085 -> 192.168.48.2,22 PR tcp len 20 64 -S OUT Oct 30 07:44:11 rolle ipmon[16747]: 07:44:10.605416 2x ep1 @0:15 b 192.168.26.1,138 -> 192.168.26.255,138 PR udp len 20 257 IN Oct 30 07:44:34 rolle ipmon[16747]: 07:44:33.891869 ie0 @0:10 b 192.168.48.1,23406 -> 192.168.48.2,22 PR tcp len 20 64 -S OUT Oct 30 07:49:13 rolle ipmon[16747]: 07:49:12.554420 ep1 @0:15 b 210.132.100.117 -> 192.168.26.5 PR icmp len 20 56 icmp 3/3 for 192.168.26.5,61915 - 210.132.100.117,53 PR udp len 20 23040 IN Oct 30 07:50:23 rolle ipmon[16747]: 07:50:22.908107 ep1 @0:15 b 210.132.100.117 -> 192.168.26.5 PR icmp len 20 56 icmp 3/3 for 192.168.26.5,4480 - 210.132.100.117,53 PR udp len 20 19712 IN Oct 30 07:56:11 rolle ipmon[16747]: 07:56:11.113029 2x ep1 @0:15 b 192.168.26.1,138 -> 192.168.26.255,138 PR udp len 20 257 IN (that's: .... 'PR' protocol 'len' length_of_ip_headers_saved packetlength direction) will get converted to something like 994398737 denied igmp 100.187.115.1 - ep1 LIRE_NOTAVAIL 224.0.0.2 - 56 994398861 denied igmp 100.187.115.1 - ep1 LIRE_NOTAVAIL 224.0.0.1 - 56 994398862 denied igmp 100.187.115.1 - ep1 LIRE_NOTAVAIL 224.0.0.2 - 56 994406849 denied udp 192.168.26.4 137 ie0 LIRE_NOTAVAIL 192.168.26.255 137 116 994406850 denied udp 192.168.26.4 137 ie0 LIRE_NOTAVAIL 192.168.26.255 137 116 994406866 denied udp 192.168.26.4 137 ie0 LIRE_NOTAVAIL 192.168.26.255 137 98 SEE ALSO
ipl(4) for description of log structure. The ipmon.c source (e.g. on http://www.openbsd.org/cgi-bin/cvsweb/~checkout~/ src/usr.sbin/ipmon/Attic/ipmon.c?rev=1.27& content-type=text/plain&hideattic=0 ) for the specification of the log syntax. The IP Filter webpage on http://coombs.anu.edu.au/~avalon/ip-filter.html AUTHOR
Joost van Baal <joostvb@logreport.org>, Wessel Dankers <wsl@logreport.org> VERSION
$Id: IpfilterDlfConverter.pm,v 1.7 2009/03/15 08:10:55 vanbaal Exp $ COPYRIGHT
Copyright (C) 2001-2003 Stichting LogReport Foundation LogReport@LogReport.org This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with this program (see COPYING); if not, check with http://www.gnu.org/copyleft/gpl.html. Lire 2.1.1 2009-03-15 IpfilterDlfConverter(3pm)
All times are GMT -4. The time now is 10:43 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy