Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Simple shell script to find and print data

View Poll Results: x
x 1 100.00%
x 0 0%
Voters: 1. This poll is closed

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Simple shell script to find and print data
# 1  
Old 09-22-2009
Simple shell script to find and print data
Hi,

I have a log file containing data on emails sent. Looks a bit like this for one email:

Code:
Content-Type: text/plain;
charset="UTF-8"
Date: 12 Jun 2008 14:04:59 +0100
From: from@email.com
Subject: xcf4564xzcv
To: recip@email.co.uk
Size = 364 Jun 12 14:04 smtp_234sldfh.tmp

I need to take the subject, date, time, size and To: and stick it in an output file in the following format

Code:
1,recip@email.co.uk,1,,,1,xcf4564xzcv,1,12 Jun 2008,14:04:59,1,,364

How can I do this in Shell (AIX)... sed? grep? Here is the perl script I wrote:

Code:
open(READLOGFILE,  "C:\\temp\\email.log") or die("Failed to open file");
open(WRITELOGFILE, ">>C:\\temp\\emailstats.log") or die("Failed to open file");
$numRecords  = 0;
$emailSize   = 0;
$foundSize   = 0;
$foundEmail  = 0;
$foundSubject= 0;
$foundDate   = 0;
while($line = <READLOGFILE>) {
 if($line =~ /^Subject\:\s(\S*)/ ) {
  $subject      = $1;
  $foundSubject = 1;
 } 
 if($line =~ /^Date\:\s(.*)\s(\d\d\:\d\d\:\d\d)\s(.*)/) {
  $date      = $1;
  $foundDate = 1;
  $time    = $2;
 } 
 if($line =~ /^To\:\s(\S*\@\S*)/ ) {
  $to         = $1;
  $foundEmail = 1;
 }
 if($line =~ /^Size\s\=\s(\d*)/) {
  $foundSize = 1;
  $emailSize = $1;
 }
 
 if(($foundSubject + $foundEmail + $foundSize + $foundDate) == 4) {
  print WRITELOGFILE "1,".$to.",1,,,1,".$subject.",1,".$date.",".$time.",1,,".$emailSize."\n";
  $numRecords  = $numRecords + 1;
  $totalSize   = $totalSize + $emailSize;
  $emailSize   = 0;
   $foundSize   = 0;
  $foundEmail  = 0;
  $foundSubject= 0;
  $foundDate   = 0;
 }
 
}
print "\nProcessing Complete";

Last edited by terry2009; 09-22-2009 at 07:45 AM.. Reason: code tags, PLEASE!
# 2  
Old 09-22-2009
To keep the forums high quality for all users, please take the time to format your posts correctly.

First of all, use Code Tags when you post any code or data samples so others can easily read your code. You can easily do this by highlighting your code and then clicking on the # in the editing menu. (You can also type code tags [code] and [/code] by hand.)

Second, avoid adding color or different fonts and font size to your posts. Selective use of color to highlight a single word or phrase can be useful at times, but using color, in general, makes the forums harder to read, especially bright colors like red.

Third, be careful when you cut-and-paste, edit any odd characters and make sure all links are working property.

Thank You.

The UNIX and Linux Forums
# 3  
Old 09-22-2009
Assuming the headers appear every time, you can use multiple calls to 'grep' to output one file for each field, and then use 'paste' to merge them, i.e.,
Code:
grep "^Subject" file.txt > subjects.out
grep "^To" file.txt > tos.out
paste -d"\t" subjects.out tos.out

On the other hand, if there's no guarantee each header will always exist in every set, you have to do basically what you've done already running through the file and setting vars, and echo output, i.e.,

Code:
SUBJECT=
FOUND_SUBJECT=0
cat file.txt | while read line
do
  SUBJECT=`echo $line | grep "^Subject"`
  if [ $SUBJECT ]
     FOUND_SUBJECT=1
  # etc
  . . .
  echo ...
  # reset vars
done
echo "Processing complete"

# 4  
Old 09-22-2009
Hi,

Thanks for you post... but could you make this idiot proof for me please, never used UNIX in my life.

1. How would I get 12 Jun 2008 from the line:
Code:
Date: 12 Jun 2008 14:04:59 +0100

Would AIX/UNIX accept:
Code:
/^Date\:\s(.*)\s(\d\d\:\d\d\:\d\d)\s(.*)/

Thanks
Terry
# 5  
Old 09-22-2009
Not sure about grep on aix. Check for a -P option. On RHEL 5, this puts grep into Perl mode, which would presumably allow you to keep your regexes intact.

Otherwise, it's very similar. Use quotes to shell-proof your patterns. Drop the slash delimiters.

Since it's a log, you can probs just grep on the keyword at the start of the line, i.e., grep "^Date: " or you could do grep "^Date: [0-9][0-9]:" etc.

To get positional vars out of a line you can use awk, I think like this:
echo $var | awk '{ print $1 $3 }'

Truth be told, though, if you can stick with perl, it's much easier--made for this kind of task.

---------- Post updated at 08:19 AM ---------- Previous update was at 08:17 AM ----------

also, if you're really a newb, you may not know about the man pages. Type 'man awk' or 'man grep' at the command-line. and while you're at it, type 'which perl' . If you get a path to perl, use it.
# 6  
Old 09-22-2009
Nah, client doesn't allow use of perl... Its not one of their strategic software, therefore hasn't got security clearance! Good chance to learn some shell though.

Thanks for you're help, I'll get testing. I'm going to try these:
Code:
grep "^To" email.log > recipients.out|awk '{print $2}'
grep /^Subject\:\s(\S*)/ email.log > subjects.out

Does the sytax look alright?


any ideas how I can paste all the outputs together to get it to look like this:
Code:
1,recip@email.co.uk,1,,,1,xcf4564xzcv,1,12 Jun 2008,14:04:59,1,,364



---------- Post updated at 03:12 PM ---------- Previous update was at 01:35 PM ----------

Resolution for anyone who needs it in future...
Code:
grep "^To" email.log |awk '{print "1,"$2}' > recipients.out ; grep "^Subject" email.log |awk '{print ",1,,,1,"$2}' > subjects.out ; grep "^Date" email.log |awk '{print ",1,"$2" "$3" "$4}' > date.out ; grep "^Date" email.log |awk '{print ","$5}' > time.out ; grep "^Size" email.log |awk '{print ",1,,"$3}' > size.out ; paste recipients.out subjects.out date.out time.out size.out > emailstats.txt ; rm recipients.out ; rm subjects.out ; rm date.out ; rm time.out ; rm size.out

Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Modifying text file records, find data in one place in the record and print it elsewhere

Hello, I have some text data that is in the form of multi-line records. Each record ends with the string $$$$ and the next record starts on the next line. RDKit 2D 15 14 0 0 0 0 0 0 0 0999 V2000 5.4596 2.1267 0.0000 C 0 0 0 0 0 0 0 0 0 0 0 ... (5 Replies)
Discussion started by: LMHmedchem
5 Replies

2. Shell Programming and Scripting

HELP simple script to find e-mail address on a file

Hello guys, im new to to unix/linux i have a text file like this: person1@test.com iisiiasasas person2@test.com 123w2 3233 sajsja person3@test.com jsajjsa sajsjasaj person4@test.com I want to extract only e-mail address and get rid of all other stuff, i want an output like this ... (4 Replies)
Discussion started by: RazorMX
4 Replies

3. UNIX for Dummies Questions & Answers

Help with simple script to find PID of process

Hi everyone. I've been reading around and am a little bit overwhelmed, hoping to find a kind soul out there to hold my hand through writing my first script. This need has emerged at work and I haven't much experience writing shell scripts, but this is a problem we have with a production environment... (13 Replies)
Discussion started by: thirdcoaster
13 Replies

4. Homework & Coursework Questions

Shell script calling Perl function, sort and find data, write to new files

Use and complete the template provided. The entire template must be completed. If you don't, your post may be deleted! 1. The problem statement, all variables and given/known data: I must write a shell script that calls two external Perl functions--one of which sorts the data in a file, and... (6 Replies)
Discussion started by: kowit010
6 Replies

5. Shell Programming and Scripting

Shell script to find specific file name and load data

I need help as to how to write a script in Unix for the following: We have 3 servers; The mainframe will FTP them to a folder. In that folder we will need the script to look and see if the specific file name is there and load it to the correct table. Can anyone pls help me out with... (2 Replies)
Discussion started by: msrahman
2 Replies

6. Shell Programming and Scripting

Simple script to find common strings in two files

Hi , I want to write a simple script. I have two files file1: BCSpeciality Backend CB CBAPQualDisp CBCimsVFTRCK CBDSNQualDisp CBDefault CBDisney CBFaxMCGen CBMCGeneral CBMCQualDisp file2: CSpeciality Backend (8 Replies)
Discussion started by: ramky79
8 Replies

7. Shell Programming and Scripting

Simple Find file Script.....

Im trying to make a very simple find the first file with the .zip extension in a specific folder and open that file. The folder path and file name will vary every-time and it may contain spaces. If I try to look For this example the folder directory is /Users/username/Desktop/testfolder/abc... (6 Replies)
Discussion started by: elbombillo
6 Replies

8. Shell Programming and Scripting

simple shell - how to get a parameter typed in a shell script

Hi, I am new to unix and using linux 7.2. I would like to create a script that would make it easyer for me to run my java programms. At the moment I have to type java myJavaprogram I am trying to write a script that will allow me to type something like this "myscript myJavaprogram" or maybe... (4 Replies)
Discussion started by: cmitulescu
4 Replies

9. Shell Programming and Scripting

simple script detect to find OS version/flavour

Hi, A newbie question. Following script gives no output. =============================== root@srv # cat /etc/redhat-release | awk {'print $1}' Red root@srv # cat 123.sh if (( `cat /etc/redhat-release | awk {'print $1}'` != CentOS )); then { echo "System runs on Redhat Linux. ... (13 Replies)
Discussion started by: fed.linuxgossip
13 Replies

10. Shell Programming and Scripting

Script to look for data in a file (not that simple) ...

I'm looking for a script or program that would allow me to pass a pattern to it and give me locations on where text appears in a file. I wish it was that straight forward (I would use egrep or something) Say I have the word in my text file "SUDAN" but my user does a search for "SUDANESE". Grep... (6 Replies)
Discussion started by: gseyforth
6 Replies
Login or Register to Ask a Question