How to find ip addresses in logfiles?


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting How to find ip addresses in logfiles?
Prev   Next
# 4  
Old 03-20-2011
OK, here's try #2:
Code:
#! /bin/bash

for file in /logs/logfile.*; do
	while read ip; do
		if [[ "${file#${file%??}}" == ".Z" ]]; then
			# This is a compressed file - it ends with .Z - use zgrep
			zgrep ${ip} ${file} >> ~/results/output_log_${ip}
		else
			# Not a .Z file - regular ol' grep
			grep ${ip} ${file} >> ~/results/output_log_${ip}
		fi
	done <iplist
done

In your example, you're redefining the "logfile" variable - only the last one will count. You could set an array or a simple list of files to loop through, though.

See mine above - it gathers the list of logfiles at run time and loops over each one, checks to see if it has a .Z (I assume you use that to mean compressed - it's typical but not necessarily true) to determine if it should run grep or zgrep, appends the output to the output_log_$ip file (append will create if necessary).

I haven't really tested it - does it work on your system with your data?
 
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

How to find the count of IP addresses that belong to different subnets and display the count?

Hi, I have a file with a list of bunch of IP addresses from different VLAN's . I am trying to find the list the number of each vlan occurence in the output Here is how my file looks like 1.1.1.1 1.1.1.2 1.1.1.3 1.1.2.1 1.1.2.2 1.1.3.1 1.1.3.2 1.1.3.3 1.1.3.4 So what I am trying... (2 Replies)
Discussion started by: new2prog
2 Replies

2. UNIX for Advanced & Expert Users

How to find remote IP addresses that applications are scanning them?

Hi, I have a web server running on Debian 6.0.4 in a computer outside my university, but the web URL is blocked by my university, the security group of the university said because it was scanning computers inside university. I could not find any applications in my web server are doing... (3 Replies)
Discussion started by: hce
3 Replies

3. Shell Programming and Scripting

Delete characters & find unique IP addresses with port

Hi, I have a file having following content. <sip:9376507346@97.208.31.7:51088 <sip:9907472291@97.208.31.7:51208 <sip:8103742422@97.208.31.7:51024 <sip:9579892841@97.208.31.7:51080 <sip:9370904222@97.208.31.7:51104 <sip:9327665215@97.208.31.7:51104 <sip:9098364262@97.208.31.7:51024... (2 Replies)
Discussion started by: SunilB2011
2 Replies

4. Red Hat

logrotate httpd logfiles

Hi, I need to logrotate logs in directories in /var/log/httpd/. There are 4 directories in /var/log/httpd/... these directories are /var/log/httpd/access/ /var/log/httpd/debug/ /var/log/httpd/error/ /var/log/httpd/required/ Each of the access, required, error and debug directories have around... (1 Reply)
Discussion started by: renuka
1 Replies

5. Shell Programming and Scripting

Logfiles

Hi All, I have a peculiar problem. I will call a script from another script. Script abc.ksh is called by ABC.ksh as ABC.ksh abc.ksh in abc.ksh I will create and redirect all the statements to log file. ABC.ksh will also has a log file. I want all the logs generated in file abc in ABC... (5 Replies)
Discussion started by: javeed7
5 Replies

6. Shell Programming and Scripting

split monthly logfiles into daily logfiles

Hi, I have a lot of logfiles like fooYYYYMM.log (foo200301.log, foo200810.log) with lines like YYYY-MM-DD TIMESTAMP,text1,text2,text3... but I need (for postprocessing) the form fooYYYYMMDD.log (so foo200402.log becomes foo20040201.log, foo20040202.log...) with unmodified content of lines. ... (1 Reply)
Discussion started by: clzupp
1 Replies

7. Shell Programming and Scripting

Logfiles E-mailed

Hi All There are some cron jobs ,which runs 24 hrs. Log files are generated when one job fails. So I need the log files to be emailed to my personal e-mail id. So that I can see the log files at my home If there is any error. How can I implement this in Unix shell programming. Thanks... (4 Replies)
Discussion started by: deep_kol
4 Replies

8. IP Networking

find computer names from IP addresses?

Arright, here's what I'm trying to do. I want to dig up currently active IP addresses on my subnet, and my present strategy is to ping every address until I find active ones, then ping them more often to verify their status. Next, I want to find the names of the computers associated with those... (1 Reply)
Discussion started by: sladuuch
1 Replies

9. UNIX for Dummies Questions & Answers

Controlling logfiles

I support an app that outputs alert and audit messages to one log file (vendor says they can't be separated). The script that I have written takes a copy (mv cmd) of the file to do the separation and reformatting. I have a problem that I loose records (messages are being written constantly, upto 3+... (5 Replies)
Discussion started by: nhatch
5 Replies

10. UNIX for Dummies Questions & Answers

Logfiles

My server has only has access logs turned on. How do I turn on the other standard logs (i.e. I'd like to see the referring urls). Thanks in advance. (3 Replies)
Discussion started by: pingdom
3 Replies
Login or Register to Ask a Question
NEWSYSLOG(8)						    BSD System Manager's Manual 					      NEWSYSLOG(8)

NAME
newsyslog -- maintain system log files to manageable sizes SYNOPSIS
newsyslog [-CFNPnrsv] [-a directory] [-d directory] [-f config_file] [-S pidfile] [-t timefmt] [[-R tagname] file ...] DESCRIPTION
The newsyslog utility should be scheduled to run periodically by cron(8). When it is executed it archives log files if necessary. If a log file is determined to require archiving, newsyslog rearranges the files so that ``logfile'' is empty, ``logfile.0'' has the last period's logs in it, ``logfile.1'' has the next to last period's logs in it, and so on, up to a user-specified number of archived logs. It is also possible to let archived log filenames be created using the time the log file was archived instead of the sequential number using the -t option. Optionally the archived logs can be compressed to save space. A log can be archived for three reasons: 1. It is larger than the configured size (in kilobytes). 2. A configured number of hours have elapsed since the log was last archived. 3. This is the specific configured hour for rotation of the log. The granularity of newsyslog is dependent on how often it is scheduled to run by cron(8). Since the program is quite fast, it may be sched- uled to run every hour without any ill effects, and mode three (above) assumes that this is so. OPTIONS
The following options can be used with newsyslog: -f config_file Instruct newsyslog to use config_file instead of /etc/newsyslog.conf for its configuration file. -a directory Specify a directory into which archived log files will be written. If a relative path is given, it is appended to the path of each log file and the resulting path is used as the directory into which the archived log for that log file will be written. If an abso- lute path is given, all archived logs are written into the given directory. If any component of the path directory does not exist, it will be created when newsyslog is run. -d directory Specify a directory which all log files will be relative to. To allow archiving of logs outside the root, the directory passed to the -a option is unaffected. -v Place newsyslog in verbose mode. In this mode it will print out each log and its reasons for either trimming that log or skipping it. -n Cause newsyslog not to trim the logs, but to print out what it would do if this option were not specified. This option implies the -r option. -r Remove the restriction that newsyslog must be running as root. Of course, newsyslog will not be able to send a HUP signal to syslogd(8) so this option should only be used in debugging. -s Specify that newsyslog should not send any signals to any daemon processes that it would normally signal when rotating a log file. For any log file which is rotated, this option will usually also mean the rotated log file will not be compressed if there is a dae- mon which would have been signalled without this option. However, this option is most likely to be useful when specified with the -R option, and in that case the compression will be done. -t timefmt If specified newsyslog will create the ``rotated'' logfiles using the specified time format instead of the default sequential file- names. The filename used will be kept until it is deleted. The time format is described in the strftime(3) manual page. If the timefmt argument is set to an empty string or the string ``DEFAULT'', the default built in time format is used. If the timefmt string is changed the old files created using the previous time format will not be automatically removed (unless the new format is very similar to the old format). This is also the case when changing from sequential filenames to time based file names, and the other way around. The time format should contain at least year, month, day, and hour to make sure rotating of old logfiles can select the correct logfiles. -C If specified once, then newsyslog will create any log files which do not exist, and which have the C flag specified in their config file entry. If specified multiple times, then newsyslog will create all log files which do not already exist. If log files are given on the command-line, then the -C or -CC will only apply to those specific log files. -F Force newsyslog to trim the logs, even if the trim conditions have not been met. This option is useful for diagnosing system prob- lems by providing you with fresh logs that contain only the problems. -N Do not perform any rotations. This option is intended to be used with the -C or -CC options when creating log files is the only objective. -P Prevent further action if we should send signal but the ``pidfile'' is empty or does not exist. -R tagname Specify that newsyslog should rotate a given list of files, even if trim conditions are not met for those files. The tagname is only used in the messages written to the log files which are rotated. This differs from the -F option in that one or more log files must also be specified, so that newsyslog will only operate on those specific files. This option is mainly intended for the daemons or programs which write some log files, and want to trigger a rotate based on their own criteria. With this option they can execute newsyslog to trigger the rotate when they want it to happen, and still give the system administrator a way to specify the rules of rotation (such as how many backup copies are kept, and what kind of compression is done). When a daemon does execute newsyslog with the -R option, it should make sure all of the log files are closed before calling newsyslog, and then it should re-open the files after newsyslog returns. Usually the calling process will also want to specify the -s option, so newsyslog will not send a signal to the very process which called it to force the rotate. Skipping the signal step will also mean that newsyslog will return faster, since newsyslog normally waits a few seconds after any signal that is sent. -S pidfile Use pidfile as syslogd(8)'s pidfile. If additional command line arguments are given, newsyslog will only examine log files that match those arguments; otherwise, it will examine all files listed in the configuration file. FILES
/etc/newsyslog.conf newsyslog configuration file /etc/newsyslog.conf.d Each file in this directory will be included by the default newsyslog.conf. /usr/local/etc/newsyslog.conf.d Each file in this directory will be included by the default newsyslog.conf. COMPATIBILITY
Previous versions of the newsyslog utility used the dot (``.'') character to distinguish the group name. Beginning with FreeBSD 3.3, this has been changed to a colon (``:'') character so that user and group names may contain the dot character. The dot (``.'') character is still accepted for backwards compatibility. SEE ALSO
bzip2(1), gzip(1), xz(1), syslog(3), newsyslog.conf(5), chown(8), syslogd(8) HISTORY
The newsyslog utility originated from NetBSD and first appeared in FreeBSD 2.2. AUTHORS
Theodore Ts'o, MIT Project Athena Copyright 1987, Massachusetts Institute of Technology BUGS
Does not yet automatically read the logs to find security breaches. BSD
September 23, 2014 BSD