parsing email log


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting parsing email log
# 1  
Old 12-20-2007
Question parsing email log

Can anyone give me some examples of how I can parse the following lines of text so that all characters up to and including the @ symbol are deleted? Also, any duplicates would need to be deleted in order to produce the desired output. Any help is much appreciated and explanations of any solutions would be great too.


Dec 25 12:00:01 host1a pop3d: LOGIN: DEBUG: ip=[::ffff:192.168.1.1], username=someuser@domain1.com
Dec 25 12:00:01 host1a pop3d: LOGIN: DEBUG: ip=[::ffff:192.168.1.1], username=someuser@domain2.com
Dec 25 12:00:01 host1a pop3d: LOGIN: DEBUG: ip=[::ffff:192.168.1.1], username=someuser@domain1.com


Desired Output:
domain1.com
domain2.com
# 2  
Old 12-20-2007
in bash
Code:
IFS="@"
sort -u file | while read line
do
    set -- $line
    echo $2
done

in awk:
Code:
# awk -F"@" '{print $2|"sort -u"}' file
# awk -F"@" '{array[$2]}END{for ( i in array ) print i}' file

# 3  
Old 12-20-2007
Code:
awk -F@ '!a[$NF]++ {print $NF}' file


Last edited by vgersh99; 12-20-2007 at 12:06 PM..
# 4  
Old 12-20-2007
Using sed:

sed -e 's/^.*@//g' logfile | sort -u
# 5  
Old 12-20-2007
Question

Thanks guys! Thats very helpful, now can anyone tell me how I would use sed or awk to remove lines that contain a given string of text? Say for example, any lines containing the word "hello", also deleting any duplicate lines (ignoring case).

Bob
hello my name is bob
bob is my name
hello world
bob Hello
hello bob
bob
bob

output:
Bob
bob is my name
# 6  
Old 12-20-2007
It's worth starting a new thread - pls do so in the future.
nawk -f jj.awk myFile

jj.awk:
Code:
{ str=tolower($0)}
str !~ "hello" && !(str in dups) { print; dups[str] }

Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Parsing Log File help

Hi, I am a newbie to scripting. I have multiple log files (saved as .gz) in a directory that looks like this 01-01-2013 10:00 pn: 123 01-01-2013 10:00 sn: 987 01-01-2013 10:00 Test1 01-01-2013 10:00 Result: Pass 01-01-2013 10:00 Time: 5:00 01-01-2013 10:00 Test2 01-01-2013 10:00... (3 Replies)
Discussion started by: linuxnew
3 Replies

2. Shell Programming and Scripting

Log parsing

I have a directory with daily logs that have records like this: Date: 04/17/13 Time: 09:29:15 IP: 123.123.123.123 URL: usr/local/file1 and I want to only count how many times each file was accessed (e.g. file1 in that example above), and I want to also look in all the logs in the current... (3 Replies)
Discussion started by: Jaymz
3 Replies

3. Shell Programming and Scripting

Help Parsing a Log File

Hello all, I am new to scripting and I have written a script that performs an Rsync on my NAS and then moves on to send me an email with the status etc. The problem is that I think Rsync is taking to long to complete and the IF statement is timing out, as it doesn't appear to move on. Here... (1 Reply)
Discussion started by: Mongrel
1 Replies

4. Shell Programming and Scripting

Log parsing script

Hello, I have a script that parses logs and sends the output via digitally signed and encrypted email. This script uses grep -v to exclude patterns in a file. The problem I have is if this is run via cron none of the pattern matching seems to occur. If I run it by hand it runs exactly as it is... (2 Replies)
Discussion started by: wpfontenot
2 Replies

5. Shell Programming and Scripting

Perl log parsing help

Hello, I'm sure this is a very simple problem, but I'm having trouble thinking of an efficient way to do the following: given a large centralized ssh-log, one file on a syslog server, not separated by machines (I wish it were), that looks something like this: Sep 27 16:20:56 machine-name... (1 Reply)
Discussion started by: droog72
1 Replies

6. UNIX for Dummies Questions & Answers

parsing a log file

I need help in parsing the following log files. 10 Apr 2009 0:16:16 * name: Tuna Belly Format: Well done, Price: 999 only 10 Apr 2009 0:16:16 * name: Roast Beef Format: Raw, Price: 55 c 10 Apr 2009 0:16:16 * name: Pasta Format: Dry, Price: 88.43 only etcetc I need to parse this... (8 Replies)
Discussion started by: izuma
8 Replies

7. Shell Programming and Scripting

XML Log Parsing

I have a log file that is around 300 MB of data having continours soap responses as shown below( I have attached only one sample SOAP). I would require to have the following extracted and written onto a new file. timestamp WebPartId bus:block bus:unblock endpt:operation Please help me. ... (3 Replies)
Discussion started by: pk_eee
3 Replies

8. Shell Programming and Scripting

Parsing a large log

I need to parse a large log say 300-400 mb The commands like awk and cat etc are taking time. Please help how to process. I need to process the log for certain values of current date. But I am unbale to do so. (17 Replies)
Discussion started by: asth
17 Replies

9. Shell Programming and Scripting

More complicated log parsing

Hey Guys, I am trying to grep within a file to find and output certain parts of lines to other file(s). The output files need to have a dynamic file name based on a field in the main log. The problem is that every line of the log is not the same, and often not even similar. To explain... (25 Replies)
Discussion started by: sjug
25 Replies

10. Shell Programming and Scripting

Parsing a Log file

Hi All, I'm deffently not a Unix specialist so be Gentel. I need to parse a Log file that looks like that: 2006-06-12 01:00:00,463 ERROR {cleanLoggersFiles} General Error comverse.compas.shared.exceptions.SystemParametersException: Error in reading parameter FileLocation at... (4 Replies)
Discussion started by: tbirenzweig
4 Replies
Login or Register to Ask a Question