Visit Our UNIX and Linux User Community


command to extract IP data from syslogs


 
Thread Tools Search this Thread
Top Forums UNIX for Dummies Questions & Answers command to extract IP data from syslogs
# 1  
Old 03-01-2005
Data command to extract IP data from syslogs

Hello,

I need to extract IP info from few large files into a single file with IP info only. I guess I can use grep, uniq and redirection but I not sure how. Is there a way to do this with a single command? Your help is greatly appreciated.
# 2  
Old 03-01-2005
Welcome to the forums.

Please post an sample of the log file you wish to extract data from (change IP's and any sensitive information first), and an example of the intended output. Without this, it's impossible to assist. The output of uname -a and echo $SHELL is always a boon too....

Thanks,
ZB
# 3  
Old 03-01-2005
thanks for your help

Quote:
Originally Posted by zazzybob
Welcome to the forums.

Please post an sample of the log file you wish to extract data from (change IP's and any sensitive information first), and an example of the intended output. Without this, it's impossible to assist. The output of uname -a and echo $SHELL is always a boon too....

Thanks,
ZB
here is an example:

if this is the log file:


192.168.1.100 - - [08/Dec/2005:18:16:07 -0500] "GET /images/welcome.png HTTP/1.1"
200 3032
192.168.1.101 - - [08/Dec/2005:18:16:07 -0500] "GET /images/google.gif HTTP/1.1" 2
00 1680
192.168.1.120 - - [08/Dec/2005:18:16:07 -0500] "GET /images/jeeves.gif HTTP/1.1" 2
00 1976
192.168.1.1 - - [08/Dec/2005:18:16:07 -0500] "GET /images/dogpile.gif HTTP/1.1"
200 985
192.168.100.1 - - [08/Dec/2005:18:16:07 -0500] "GET /images/yahoo.gif HTTP/1.1" 20
0 718
192.168.1.1 - - [08/Dec/2005:18:16:07 -0500] "GET /images/lycos.gif HTTP/1.1" 20
0 1348
192.168.10.1 - - [08/Dec/2005:18:16:07 -0500] "GET /images/av.jpg HTTP/1.1" 200 1
632
192.168.1.11 - - [08/Dec/2005:18:16:07 -0500] "GET /images/valid.gif HTTP/1.1" 20
0 2328
192.168.1.1 - - [08/Dec/2005:18:16:08 -0500] "GET /images/easy.gif HTTP/1.1" 200
1518
192.168.1.1 - - [08/Dec/2005:18:32:18 -0500] "GET / HTTP/1.1" 200 3643
23.34.54.121 - - [18/Dec/2005:08:24:47 -0500] "GET /car/ext-1.jpg HTTP/1.1" 200 4
3611
23.34.54.122 - - [18/Dec/2005:08:25:10 -0500] "GET /car/int-2.jpg HTTP/1.1" 200 4
7963
23.34.54.110 - - [18/Dec/2005:08:25:12 -0500] "GET /car/int-1.jpg HTTP/1.1" 200 5
7079
23.34.54.111 - - [18/Dec/2005:08:31:41 -0500]

[Mon Oct 28 22:02:10 2005] [error] [client 192.168.1.1] File does not exist: /ho
me/httpd/di0de/index.php
[Tue Oct 29 01:23:18 2005] [error] [client 200.81.81.70] client sent HTTP/1.1 re
quest without hostname (see RFC2616 section 14.23): /
[Tue Oct 29 10:27:07 2005] [error] [client 216.39.100.132] File does not exist:
/home/httpd/di0de/scripts/..%5c..%5cwinnt/system32/cmd.exe
[Tue Oct 29 16:30:16 2005] [error] [client 192.168.1.1] File does not exist: /ho
me/httpd/di0de/010102.html

I would like to extract only the IP info into a new file such as:

192.168.1.1
192.168.1.100
192.168.1.11
23.34.54.111
...
# 4  
Old 03-01-2005
and i have 3 files to extract this data from

syslog1, syslog2 and syslog3

I can use grep and uniq but I am not sure how to extract the IP info only and redirect it to a new file.

di0de
# 5  
Old 03-02-2005
Code:
grep "GET" syslog* | cut -d"-" -f 1

I used the pattern "GET" in syslogs
If you want to narrow down the lines further use egrep.
# 6  
Old 03-02-2005
This is an awk script to find IP addresses...
Code:
awk '
BEGIN{
    r = "[0-9][0-9]?[0-9]?"
    r = r "\\." r "\\." r "\\." r
}
match ($0, r){
    a[substr($0, RSTART, RLENGTH)] = 1
}
END{
    for (x in a)
        print x
}
' syslog1 syslog2 syslog3 > newlog

# 7  
Old 03-02-2005
thanks

how do I implement and execute this script? thanks for your help.

Quote:
Originally Posted by Ygor
This is an awk script to find IP addresses...
Code:
awk '
BEGIN{
    r = "[0-9][0-9]?[0-9]?"
    r = r "\\." r "\\." r "\\." r
}
match ($0, r){
    a[substr($0, RSTART, RLENGTH)] = 1
}
END{
    for (x in a)
        print x
}
' syslog1 syslog2 syslog3 > newlog

 

Previous Thread | Next Thread
Test Your Knowledge in Computers #717
Difficulty: Medium
A universal Turing machine can calculate any recursive function, decide any recursive language, and accept any recursively enumerable language.
True or False?

10 More Discussions You Might Find Interesting

1. Solaris

Configuring Centralized Solaris Server to Receive Syslogs

Hello, I am new to Solaris. However, I was able to configure the Syslog server on Solaris. I am using a windows client to send logs to my Solaris syslog server. this is working fine. however, I have 23 servers which are sending their logs to the syslog server and it is very difficult to read all... (2 Replies)
Discussion started by: ravmohun
2 Replies

2. Shell Programming and Scripting

Extract header data from one file and combine it with data from another file

Hi, Great minds, I have some files, in fact header files, of CTD profiler, I tried a lot C programming, could not get output as I was expected, because my programming skills are very poor, finally, joined unix forum with the hope that, I may get what I want, from you people, Here I have attached... (17 Replies)
Discussion started by: nex_asp
17 Replies

3. AIX

Syslogs permission not getting changed - aix 6.1

Friends, I've tried to modify the syslogs permission by using the perm option in the syslog configuration in AIX 6.1 TL 05. But its not getting applied after the configuration. Have restarted the syslog service also. Need your help!:wall: The below are the conf details and os versions >... (1 Reply)
Discussion started by: novaothers
1 Replies

4. Shell Programming and Scripting

Extracting relevant information from syslogs.

I need to analyse some syslogs and I want to print out all the lines containing SSH connections to the inside interface of the firewall and ignore lines where the originating port is 22. So basically I want to print all matches after "to inside:" that contains /22 and ignore lines where /22 occur... (2 Replies)
Discussion started by: lewk
2 Replies

5. Shell Programming and Scripting

extract the data using AWK command

In a file i have a data like INPUT: no,name,company 1,vivek,hcl 2,senthil,cts 1,narsi,hcl 4,prabhakaran,ibm OUTPUT: 1,vivek,hcl 1,narsi,hcl Using AWK command i want to display the names those having no:1 and company:hcl.Please tell me the command to display above result. (8 Replies)
Discussion started by: katakamvivek
8 Replies

6. Solaris

SYSLOGS - Where can I find FTP activities

Greetings to all. I need help from the experts. I have been given a FTP server script that runs all day, looking for files that are FTP'd to our machines. Its hoaky I know, but there are times that files are sent but somehow get lost. Is there a logfile I can view to see when files are received? ... (1 Reply)
Discussion started by: Harleyrci
1 Replies

7. Shell Programming and Scripting

Extract specific data content from a long list of data

My input: Data name: ABC001 Data length: 1000 Detail info Data Direction Start_time End_time Length 1 forward 10 100 90 1 forward 15 200 185 2 reverse 50 500 450 Data name: XFG110 Data length: 100 Detail info Data Direction Start_time End_time Length 1 forward 50 100 50 ... (11 Replies)
Discussion started by: patrick87
11 Replies

8. Shell Programming and Scripting

Extract data based on match against one column data from a long list data

My input file: data_5 Ali 422 2.00E-45 102/253 140/253 24 data_3 Abu 202 60.00E-45 12/23 140/23 28 data_1 Ahmad 256 7.00E-45 120/235 140/235 22 data_4 Aman 365 8.00E-45 15/65 140/65 20 data_10 Jones 869 9.00E-45 65/253 140/253 18... (12 Replies)
Discussion started by: patrick87
12 Replies

9. Shell Programming and Scripting

how to extract the data ?

Hi, I'm trying to pick out a data field eg. from below. I need the required field as below but they are filled sometimes with weird chars like \-(. or watever. How can I accurately extract the 3rd field in shell? :confused: ID IDNO - REQUIRED FIELD ID 1447 - MAT620BR. ID 1452 -... (13 Replies)
Discussion started by: uxnoob
13 Replies

10. Shell Programming and Scripting

extract data from a data matrix with filter criteria

Here is what old matrix look like, IDs X1 X2 Y1 Y2 10914061 -0.364613333 -0.362922333 0.001691 -0.450094667 10855062 0.845956333 0.860396667 0.014440333 1.483899333... (7 Replies)
Discussion started by: ssshen
7 Replies

Featured Tech Videos