Shell Programming and Scripting

Hi folks,
I'm trying to do a bit of extra monitoring on who is sending mail through our server and as I'm sure you'll understand, the log files aren't exactly small.
As a note, I'm working to sort out a solution to this myself, so I'll keep editing the post when I get a chance.

Here an example exim log line. There are other lines in the file for other types of email that are to be ignored completely.

Code:

2010-03-07 11:34:16 1NoEkR-000AJf-NV <= email@domain.tld H=11.22.33.44 (helo.domain.tld) [11.22.33.44] P=esmtpa A=fixed_login:username S=35249 id=messageid@domain.tld

There are three cases for valid mail through the system being sent with fixed_login.

So what I need to put together is a script that does the following:

1) Scans through the logfile and ignores all lines that do not have "fixed_login" in it. ie "grep fixed_login mainlog" - sorted

We then move onto processing and comparison which I know nothing about, except for how to plan it. Best I can tell, the following would produce a script that would catch all lines that don't match the criteria

2) Search the line for "fixed_login:username" and discard "fixed_login:" giving string A. If string A looks like an email address, discard the "email@" and ".tld" parts.

3) Search the same line for the first instance of anything that looks like an email address. Discard the "email@" and ".tld" parts thus giving string B "domain"

4) Convert strings A and B to lowercase and compare, if they match or a is a substring of B then ignore the line and move onto the next line.

5) If they don't match we then need to take the third variable (message ID) and run a grep on the logfile for that ID. All lines thus relating to the message will be output before moving on to the next line.

6) The final output after all lines are checked will be emailed off (this will be run from cron so just standard output will do and cron will take care of the mail)

I did use awk some years back for basic string processing but can't remember any of it now! I can't use $9 or things like that because there are a few different types of lines that put different bits of information in and change the placement.

So far I have a sed line working on stripping things down but its getting a bit clumsy.

Code:

zgrep fixed_login mainlog.0.gz | sed 's/.* \(.*\) <= \(.*\) H=.*A=fixed_login:\(.*\) S.*/\1 \2 \3/g' | sed 's/\(.*\) .*@\(.*\)\..* /\1 \2 /g' | sed 's/\(.*\) .*@\(.*\)\..*$/\1 \2/g'

I had to go into two sed instances because I needed the first output to also match lines where the second and third variables don't have an email@domain.tld. I then had to add a third sed because parameter 2 ends on a space, 3 in EOL. Sticking a space on the end of the whole string and trying to use one sed produced completely the wrong results!

There are however two problems with this. Firstly, if the domain is email@domain.co.uk or some other double barrel tld, it only strips the last bit (.uk). Secondly, some of the users have used their usernames as email+domain.tld so I need to add @|+ into the 2nd and 3rd seds. I'm having trouble figuring out how to put the pip in and only have it affect the character immediately either side of it rather than the whole expression. I'd rather fix that than go silly and put a couple of extra seds on!

In my opinion, It would be easier if you post a bigger sample of your input data (the exim log) and an example of the desired output.

I'm not sure that would work as the output depends on the content. That is why I have describe the different stages. Maybe I can break apart the different stages into examples of input and output but the main problem is that I want to take the output of some stages and use it to as an input to other stages!