Simple shell script to find and print data

terry2009 · #1 (**permalink**) 09-22-2009

Hi,

I have a log file containing data on emails sent. Looks a bit like this for one email:

Code:

Content-Type: text/plain;
charset="UTF-8"
Date: 12 Jun 2008 14:04:59 +0100
From: from@email.com
Subject: xcf4564xzcv
To: recip@email.co.uk
Size = 364 Jun 12 14:04 smtp_234sldfh.tmp

I need to take the subject, date, time, size and To: and stick it in an output file in the following format

Code:

1,recip@email.co.uk,1,,,1,xcf4564xzcv,1,12 Jun 2008,14:04:59,1,,364

How can I do this in Shell (AIX)... sed? grep? Here is the perl script I wrote:

Code:

open(READLOGFILE,  "C:\\temp\\email.log") or die("Failed to open file");
open(WRITELOGFILE, ">>C:\\temp\\emailstats.log") or die("Failed to open file");
$numRecords  = 0;
$emailSize   = 0;
$foundSize   = 0;
$foundEmail  = 0;
$foundSubject= 0;
$foundDate   = 0;
while($line = <READLOGFILE>) {
 if($line =~ /^Subject\:\s(\S*)/ ) {
  $subject      = $1;
  $foundSubject = 1;
 } 
 if($line =~ /^Date\:\s(.*)\s(\d\d\:\d\d\:\d\d)\s(.*)/) {
  $date      = $1;
  $foundDate = 1;
  $time    = $2;
 } 
 if($line =~ /^To\:\s(\S*\@\S*)/ ) {
  $to         = $1;
  $foundEmail = 1;
 }
 if($line =~ /^Size\s\=\s(\d*)/) {
  $foundSize = 1;
  $emailSize = $1;
 }
 
 if(($foundSubject + $foundEmail + $foundSize + $foundDate) == 4) {
  print WRITELOGFILE "1,".$to.",1,,,1,".$subject.",1,".$date.",".$time.",1,,".$emailSize."\n";
  $numRecords  = $numRecords + 1;
  $totalSize   = $totalSize + $emailSize;
  $emailSize   = 0;
   $foundSize   = 0;
  $foundEmail  = 0;
  $foundSubject= 0;
  $foundDate   = 0;
 }
 
}
print "\nProcessing Complete";

vgersh99 · #2 (**permalink**) 09-22-2009

To keep the forums high quality for all users, please take the time to format your posts correctly.

First of all, use Code Tags when you post any code or data samples so others can easily read your code. You can easily do this by highlighting your code and then clicking on the # in the editing menu. (You can also type code tags [code] and [/code] by hand.)

Second, avoid adding color or different fonts and font size to your posts. Selective use of color to highlight a single word or phrase can be useful at times, but using color, in general, makes the forums harder to read, especially bright colors like red.

Third, be careful when you cut-and-paste, edit any odd characters and make sure all links are working property.

Thank You.

The UNIX and Linux Forums

varontron · #3 (**permalink**) 09-22-2009

Assuming the headers appear every time, you can use multiple calls to 'grep' to output one file for each field, and then use 'paste' to merge them, i.e.,

Code:

grep "^Subject" file.txt > subjects.out
grep "^To" file.txt > tos.out
paste -d"\t" subjects.out tos.out

On the other hand, if there's no guarantee each header will always exist in every set, you have to do basically what you've done already running through the file and setting vars, and echo output, i.e.,

Code:

SUBJECT=
FOUND_SUBJECT=0
cat file.txt | while read line
do
  SUBJECT=`echo $line | grep "^Subject"`
  if [ $SUBJECT ]
     FOUND_SUBJECT=1
  # etc
  . . .
  echo ...
  # reset vars
done
echo "Processing complete"

terry2009 · #4 (**permalink**) 09-22-2009

Hi,

Thanks for you post... but could you make this idiot proof for me please, never used UNIX in my life.

1. How would I get 12 Jun 2008 from the line:

Code:

Date: 12 Jun 2008 14:04:59 +0100

Would AIX/UNIX accept:

Code:

/^Date\:\s(.*)\s(\d\d\:\d\d\:\d\d)\s(.*)/

Thanks
Terry

varontron · #5 (**permalink**) 09-22-2009

Not sure about grep on aix. Check for a -P option. On RHEL 5, this puts grep into Perl mode, which would presumably allow you to keep your regexes intact.

Otherwise, it's very similar. Use quotes to shell-proof your patterns. Drop the slash delimiters.

Since it's a log, you can probs just grep on the keyword at the start of the line, i.e., grep "^Date: " or you could do grep "^Date: [0-9][0-9]:" etc.

To get positional vars out of a line you can use awk, I think like this:
echo $var | awk '{ print $1 $3 }'

Truth be told, though, if you can stick with perl, it's much easier--made for this kind of task.

---------- Post updated at 08:19 AM ---------- Previous update was at 08:17 AM ----------

also, if you're really a newb, you may not know about the man pages. Type 'man awk' or 'man grep' at the command-line. and while you're at it, type 'which perl' . If you get a path to perl, use it.

terry2009 · #6 (**permalink**) 09-22-2009

Nah, client doesn't allow use of perl... Its not one of their strategic software, therefore hasn't got security clearance! Good chance to learn some shell though.

Thanks for you're help, I'll get testing. I'm going to try these:

Code:

grep "^To" email.log > recipients.out|awk '{print $2}'
grep /^Subject\:\s(\S*)/ email.log > subjects.out

Does the sytax look alright?

any ideas how I can paste all the outputs together to get it to look like this:

Code:

1,recip@email.co.uk,1,,,1,xcf4564xzcv,1,12 Jun 2008,14:04:59,1,,364

---------- Post updated at 03:12 PM ---------- Previous update was at 01:35 PM ----------

Resolution for anyone who needs it in future...

Code:

grep "^To" email.log |awk '{print "1,"$2}' > recipients.out ; grep "^Subject" email.log |awk '{print ",1,,,1,"$2}' > subjects.out ; grep "^Date" email.log |awk '{print ",1,"$2" "$3" "$4}' > date.out ; grep "^Date" email.log |awk '{print ","$5}' > time.out ; grep "^Size" email.log |awk '{print ",1,,"$3}' > size.out ; paste recipients.out subjects.out date.out time.out size.out > emailstats.txt ; rm recipients.out ; rm subjects.out ; rm date.out ; rm time.out ; rm size.out

More UNIX and Linux Forum Topics You Might Find Helpful
Thread	Thread Starter	Forum	Replies	Last Post
Simple Find file Script.....	elbombillo	Shell Programming and Scripting	6	02-02-2009 08:49 PM
simple shell - how to get a parameter typed in a shell script	cmitulescu	Shell Programming and Scripting	4	01-09-2009 08:45 PM
simple script detect to find OS version/flavour	fed.linuxgossip	Shell Programming and Scripting	13	08-14-2008 12:04 AM
Linux Shell Question: how to print the shell script name ?	meili100	UNIX for Dummies Questions & Answers	3	07-01-2008 01:55 PM
Script to look for data in a file (not that simple) ...	gseyforth	Shell Programming and Scripting	6	02-16-2006 04:03 PM

View Poll Results: x
x	0	0%
x	1	100.00%
Voters: 1. You may not vote on this poll

Thread Tools	Search this Thread
Show Printable Version Email this Page	Search this Thread: Advanced Search
Display Modes	Rate This Thread
Switch to Linear Mode Hybrid Mode Switch to Threaded Mode	Rate This Thread:


	Powered by