mail log parsing script in need of makeover Post: 302194917

Sponsored Content

Top Forums Shell Programming and Scripting mail log parsing script in need of makeover Post 302194917 by era on Wednesday 14th of May 2008 02:32:07 AM

05-14-2008

Registered User

Here's an attempt at combining our efforts. The test data you posted is still not very useful; the recipients should be unique, so you can see which ones are being printed, and there should be multiple transactions, some with too many recipients, so we can test that too. But this does seem to do ... something.

Code:

#!/usr/local/bin/perl

use strict;
use warnings;

my %sender_emails = () ;
my %recipient_emails = () ;
my %recipient_count = () ;

while (<>)
{
    # Do this before split for efficiency
    next if (/from=<>/) ;
    next if (/from=<root>/) ;

    my ($msgMon, $msgDay, $msgTime, $msgHost, $msgCmd, $QID, $from_to)
	= split(/\s+/, $_) ;

    if (/qmgr/ && /from=<([^<>]+)>/)
    {
	my $from = $1;
	$sender_emails {$QID} = $from;
    }
    elsif (/smtp/ && /to=<([^<>]+)>/)
    {
	my ($to) = $1;
	# Preformat output the way we want it
	$recipient_emails {$QID} .= $sender_emails{$QID} . "," . $to . "\n";
	$recipient_count {$QID}++;
    }
    elsif (m/: removed/)
    {
	if ($recipient_count{$QID} <= 5)
	{
	    print $recipient_emails{$QID};
	}
	# Flush this key from memory; we're done with it
	delete $sender_emails{$QID};
	delete $recipient_emails{$QID};
	delete $recipient_count{$QID};
    }
}

I took out the hard-coded mail log path; I'd keep it outside the script, just to make it easier to test, or run it on older or just other log files than the main one.

era

View Public Profile for era

Find all posts by era

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Shell script for parsing 300mb log file..

am relatively new to Shell scripting. I have written a script for parsing a big file. The logic is: Apart from lot of other useless stuffs, there are many occurances of <abc> and corresponding </abc> tags. (All of them are properly closed) My requirement is to find a particular tag (say...

2. Shell Programming and Scripting

Help with script parsing a log file

I have a large log file, which I want to first use grep to get the specific lines then send it to awk to print out the specific column and if the result is zero, don't do anything. What I have so far is: LOGDIR=/usr/local/oracle/Transcription/log ERRDIR=/home/edixftp/errors #I want to be...

3. Shell Programming and Scripting

Shell script to parsing log

Hi I Have log like this : 0 234: { 3 2: 04 EE 7 14: '20081114081' 23 1: 00 79 10: '38809' 91 15: '528111510010159' 143 29: 'Streaming/downloading service' 174 3: 'MTV' 179 43: 'rtsp://172.28/MTV2GO-Loop.sdp' 224 1: 05 ...

4. Shell Programming and Scripting

Performance of log parsing shell script very slow

Hello, I am an absolute newbie and whatever I've written in the shell script (below) has all been built with generous help from googling the net and this forum. Please forgive any schoolboy mistakes. Now to the qn, my input file looks like this - 2009:04:03 08:21:41:513,INFO...

5. Shell Programming and Scripting

Script for Parsing Log File

Working on a script that inputs an IP, parses and outputs to another file. A Sample of the log is as follows: I need the script to be able to input IP and print the data in an output file in the following format or something similar: Thanks for any help you can give me!

6. Shell Programming and Scripting

Log parsing script

Hello, I have a script that parses logs and sends the output via digitally signed and encrypted email. This script uses grep -v to exclude patterns in a file. The problem I have is if this is run via cron none of the pattern matching seems to occur. If I run it by hand it runs exactly as it is...

7. Shell Programming and Scripting

Script for parsing vertical log into horizontal

Hi, I have log like this : And i want the output like below : I have try using awk but doesn't work awk ' /ffff /{ts=$1} f && /SectorAntenna\=1/{sa1=$3} f && /SectorAntenna\=2/{sa2=$3} f && /SectorAntenna\=3/{sa3=$3} { s= ts "|" sa1 "|" sa2 "|" sa3 print s f=0 }'

8. Shell Programming and Scripting

Issue with awk script parsing log file

Hello All, I am trying to parse a log file and i got this code from one of the good forum colleagues, However i realised later there is a problem with this awk script, being naive to awk world wanted to see if you guys can help me out. AWK script: awk '$1 ~ "^WRITER_" {p=1;next}...

9. Shell Programming and Scripting

Bash Script - Mail Secure.log

I'm putting together a fairly simple script, to check "secure.log" for content and email the results in a cron, nightly. The script runs fine upon manual execution, it's a problem when ran in cron. This is on a Mac server. Any thoughts? #!bin/bash #Email secure.log, nightly. Subject="Secure...

10. Shell Programming and Scripting

Parsing a log file and creating a report script

The log file is huge and lot of information, i would like to parse and make a report . below is the log file looks like: REPORT DATE: Mon Aug 10 04:16:17 CDT 2017 SYSTEN VER: v1.3.0.9 TERMINAL TYPE: prod SYSTEM: nb11cu51 UPTIME: 04:16AM up 182 days 57 mins min MODEL, TYPE, and SN:...

LEARN ABOUT DEBIAN

comsatd

COMSATD(8)						  System Administration Utilities						COMSATD(8)

NAME

       comsatd - the Comsat daemon

SYNOPSIS

       comsatd [OPTION...]
       comsatd [OPTION...] --test MBOX-URL MSG-QID

DESCRIPTION

       GNU comsatd -- notify users about incoming mail

       -d, --daemon[=NUMBER]
	      runs in daemon mode with a maximum of NUMBER children

       --foreground
	      remain in foreground

       -f, --file=FILE
	      read FILE instead of .biffrc

       -i, --inetd
	      run in inetd mode

       -t, --test
	      run in test mode

	      Common options

       --config-file=FILE, --rcfile=FILE
	      load this configuration file

       --config-help
	      show configuration file summary

       --config-lint, --rcfile-lint
	      check configuration file syntax and exit

       --config-verbose, --rcfile-verbose
	      verbosely log parsing of the configuration files

       --no-site-config, --no-site-rcfile
	      do not load site configuration file

       --no-user-config, --no-user-rcfile
	      do not load user configuration file

       --set=PARAM=VALUE
	      set configuration parameter

       --show-config-options
	      show compilation options

	      Global debugging settings

       --debug-level=LEVEL
	      set Mailutils debugging level

       --debug-line-info
	      show source info with debugging messages

       --log-facility=FACILITY
	      output logs to syslog FACILITY

       -?, --help
	      give this help list

       --usage
	      give a short usage message

       -V, --version
	      print program version

       Mandatory or optional arguments to long options are also mandatory or optional for any corresponding short options.

REPORTING BUGS

       Report bugs to <bug-mailutils@gnu.org>.

COPYRIGHT

       Copyright (C) 2010 Free Software Foundation, inc.  License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
       This is free software: you are free to change and redistribute it.  There is NO WARRANTY, to the extent permitted by law.

SEE ALSO

       The complete GNU mailutils documentation is maintained as a Texinfo manual. If the mailutils-doc package is installed, the command

	      info mailutils

       should give you access to the complete manual.
       You can also find this manual online in the GNU mailutils webpage:
       http://www.gnu.org/software/mailutils/manual/index.html.
       Please  note this manpage was automatically generated by the Debian mailutils packagers. Do not file bugs for its content to the GNU Mailu-
       tils upstream authors.

FSF
								     June 2012								COMSATD(8)