Sponsored Content
Top Forums Shell Programming and Scripting mail log parsing script in need of makeover Post 302194433 by jjamd64 on Tuesday 13th of May 2008 12:20:58 AM
Old 05-13-2008
mail log parsing script in need of makeover

Dear unix forum members,

I'm working on a script that will parse a mail machine's logs and print a list of email addresses in this format:

sender@domain,recipient@domain

The logs look something like this:

06:50:04 0048317AC863: client=localhost.com[127.0.0.1]
06:50:04 0048317AC863: message-id=<user@domain>
06:50:04 0048317AC863: from=<user@domain>,
06:50:04 0048317AC863: to=<user@domain>,
06:50:06 0048317AC863: to=<user@domain>,
06:50:18 0048317AC863: to=<user@domain>,
06:50:18 0048317AC863: to=<user@domain>,
06:50:18 0048317AC863: removed

The "from" and "to" are on different lines and there is another challenge which is that the results should be limited to messages who have 5 or fewer recipients.

I thought it would be easy enough, and I wrote a script that first gets a list of the tag numbers ( 0048317AC863Smilie which belong to messages with 5 or fewer recipients

#!/bin/sh
grep "to=<" /data/log/maillog | grep postfix | grep -vi noqueue | awk '{print $6}' | sort |uniq -c > all_ids

cat all_ids |awk '{print " "$1, $2}' | egrep " 1 | 2 | 3 | 4 | 5 " | cut -f 3 -d " " > ids

Very crude and spaghetti like...and even worse is the FOR loop that follows, which involves grepping through the entire 4000mb maillog file 33,000 times in order to print the sender and recipient addresses.

Needless to say, its not an efficient script, there must be a better way. Please help!! Any responses are appreciated, maybe someone can just point me in the right direction?
Thanks,
JJ
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Shell script for parsing 300mb log file..

am relatively new to Shell scripting. I have written a script for parsing a big file. The logic is: Apart from lot of other useless stuffs, there are many occurances of <abc> and corresponding </abc> tags. (All of them are properly closed) My requirement is to find a particular tag (say... (3 Replies)
Discussion started by: gurpreet470
3 Replies

2. Shell Programming and Scripting

Help with script parsing a log file

I have a large log file, which I want to first use grep to get the specific lines then send it to awk to print out the specific column and if the result is zero, don't do anything. What I have so far is: LOGDIR=/usr/local/oracle/Transcription/log ERRDIR=/home/edixftp/errors #I want to be... (3 Replies)
Discussion started by: mevasquez
3 Replies

3. Shell Programming and Scripting

Shell script to parsing log

Hi I Have log like this : 0 234: { 3 2: 04 EE 7 14: '20081114081' 23 1: 00 79 10: '38809' 91 15: '528111510010159' 143 29: 'Streaming/downloading service' 174 3: 'MTV' 179 43: 'rtsp://172.28/MTV2GO-Loop.sdp' 224 1: 05 ... (10 Replies)
Discussion started by: justbow
10 Replies

4. Shell Programming and Scripting

Performance of log parsing shell script very slow

Hello, I am an absolute newbie and whatever I've written in the shell script (below) has all been built with generous help from googling the net and this forum. Please forgive any schoolboy mistakes. Now to the qn, my input file looks like this - 2009:04:03 08:21:41:513,INFO... (7 Replies)
Discussion started by: sowmitr
7 Replies

5. Shell Programming and Scripting

Script for Parsing Log File

Working on a script that inputs an IP, parses and outputs to another file. A Sample of the log is as follows: I need the script to be able to input IP and print the data in an output file in the following format or something similar: Thanks for any help you can give me! (8 Replies)
Discussion started by: Winsarc
8 Replies

6. Shell Programming and Scripting

Log parsing script

Hello, I have a script that parses logs and sends the output via digitally signed and encrypted email. This script uses grep -v to exclude patterns in a file. The problem I have is if this is run via cron none of the pattern matching seems to occur. If I run it by hand it runs exactly as it is... (2 Replies)
Discussion started by: wpfontenot
2 Replies

7. Shell Programming and Scripting

Script for parsing vertical log into horizontal

Hi, I have log like this : And i want the output like below : I have try using awk but doesn't work awk ' /ffff /{ts=$1} f && /SectorAntenna\=1/{sa1=$3} f && /SectorAntenna\=2/{sa2=$3} f && /SectorAntenna\=3/{sa3=$3} { s= ts "|" sa1 "|" sa2 "|" sa3 print s f=0 }' (7 Replies)
Discussion started by: justbow
7 Replies

8. Shell Programming and Scripting

Issue with awk script parsing log file

Hello All, I am trying to parse a log file and i got this code from one of the good forum colleagues, However i realised later there is a problem with this awk script, being naive to awk world wanted to see if you guys can help me out. AWK script: awk '$1 ~ "^WRITER_" {p=1;next}... (18 Replies)
Discussion started by: Ariean
18 Replies

9. Shell Programming and Scripting

Bash Script - Mail Secure.log

I'm putting together a fairly simple script, to check "secure.log" for content and email the results in a cron, nightly. The script runs fine upon manual execution, it's a problem when ran in cron. This is on a Mac server. Any thoughts? #!bin/bash #Email secure.log, nightly. Subject="Secure... (6 Replies)
Discussion started by: Nvizn
6 Replies

10. Shell Programming and Scripting

Parsing a log file and creating a report script

The log file is huge and lot of information, i would like to parse and make a report . below is the log file looks like: REPORT DATE: Mon Aug 10 04:16:17 CDT 2017 SYSTEN VER: v1.3.0.9 TERMINAL TYPE: prod SYSTEM: nb11cu51 UPTIME: 04:16AM up 182 days 57 mins min MODEL, TYPE, and SN:... (8 Replies)
Discussion started by: amir07
8 Replies
envelopes(5)                                                    File Formats Manual                                                   envelopes(5)

NAME
envelopes - sender/recipient lists attached to messages INTRODUCTION
Electronic mail messages are delivered in envelopes. An envelope lists a sender and one or more recipients. Usually these envelope addresses are the same as the addresses listed in the mes- sage header: (envelope) from djb to root From: djb To: root In more complicated situations, though, the envelope addresses may differ from the header addresses. ENVELOPE EXAMPLES
When a message is delivered to several people at different locations, it is first photocopied and placed into several envelopes: (envelope) from djb to root From: djb Copy #1 of message To: root, god@brl.mil (envelope) from djb to god@brl.mil From: djb Copy #2 of message To: root, god@brl.mil When a message is delivered to several people at the same location, the sender doesn't have to photocopy it. He can instead stuff it into one envelope with several addresses; the recipients will make the photocopy: (envelope) from djb to god@brl.mil, angel@brl.mil From: djb To: god@brl.mil, angel@brl.mil, joe, frde Bounced mail is sent back to the envelope sender address. The bounced mail doesn't list an envelope sender, so bounce loops are impossi- ble: (envelope) from <> to djb From: MAILER-DAEMON To: djb Subject: unknown user frde The recipient of a message may make another copy and forward it in a new envelope: (envelope) from djb to joe From: djb Original message To: joe (envelope) from joe to fred From: djb Forwarded message To: joe A mailing list works almost the same way: (envelope) from djb to sos-list From: djb Original message To: sos-list (envelope) from sos-owner to god@brl.mil From: djb Forwarded message To: sos-list to recipient #1 (envelope) from sos-owner to frde From: djb Forwarded message To: sos-list to recipient #2 Notice that the mailing list is set up to replace the envelope sender with something new, sos-owner. So bounces will come back to sos- owner: (envelope) from <> to sos-owner From: MAILER-DAEMON To: sos-owner Subject: unknown user frde It's a good idea to set up an extra address, sos-owner, like this: the original envelope sender (djb) has no way to fix bad sos-list addresses, and of course bounces must not be sent to sos-list itself. HOW ENVELOPE ADDRESSES ARE STORED
Envelope sender and envelope recipient addresses are transmitted and recorded in several ways. When a user injects mail through qmail-inject, he can supply a Return-Path line or a -f option for the envelope sender; by default the envelope sender is his login name. The envelope recipient addresses can be taken from the command line or from various header fields, depending on the options to qmail-inject. Similar comments apply to sendmail. When a message is transferred from one machine to another through SMTP, the envelope sender is given in a MAIL FROM command, the envelope recipients are given in RCPT TO commands, and the message is supplied separately by a DATA command. When a message is delivered by qmail to a single local recipient, qmail-local records the recipient in Delivered-To and the envelope sender in Return-Path. It uses Delivered-To to detect mail forwarding loops. sendmail normally records the envelope sender in Return-Path. It does not record envelope recipient addresses, on the theory that they are redundant: you received the mail, so you must have been one of the envelope recipients. Note that, if the header doesn't have any recipient addresses, sendmail will move envelope recipient addresses back into the header. This situation occurs if all addresses were originally listed as Bcc, since Bcc is automatically removed. When sendmail sees this, it creates a new Apparently-To header field with the envelope recipient addresses. This has the strange effect that each blind-carbon-copy recipient will see a list of all recipients on the same machine. When a message is stored in mbox format, the envelope sender is recorded at the top of the message as a UUCP-style From (no colon) line. Note that this line is less reliable than the Return-Path line added by qmail-local or sendmail. SEE ALSO
qmail-header(5), qmail-local(8), qmail-inject(8) envelopes(5)
All times are GMT -4. The time now is 09:56 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy