Shell Programming and Scripting

If you absolutely must use a shell and nothing else, then Perderabo's solution is probably as good as you're going to get.

However, if you can use standard Unix programs and you have a nawk, this will execute faster (especially if you have 200K records every night).

Code:

#!/usr/bin/nawk -f
BEGIN { FS = "|" }
NF != 18 || !index($9, "@") { errors++ }
END { print errors, "errors"; exit errors != 0 }

You wanted 8 "|" before the email address and 9 after. Changing FS to "|" (a literal string) makes the field count be 18 on lines with 17 of these. But if there is a different number or if the 9th field (following the 8th "|") doesn't contain an "@", count the line as an error.

This doesn't check for all valid forms of email addresses --- that takes a parser. It also doesn't check for possible email addresses in other positions.

You could implement similar solutions in Ruby, Perl or (I presume) Python. You could actually massage the shell into doing something similar, but awk (or one of the others) will execute faster.

I'm sorry, I guess I should of made that clear - awk/nawk is fine..

Yours ran extremely quick against a 685k test file I had. While I don't get files > 200k more then once a night, I'd like to be safe and make sure it works.

We do have an email validator that someone else wrote before I got here - so I'm not worried about that. I was just worried about having the 8 pipes before and 8 pipes after the email address.

Is there an easy way in awk to have it copy the bad lines out to a seperate file?

I'm a junior admin here learning as I go along and would like to automate some of this stuff that my coworkers hadn't gotten a chance to do..

Brian

Try this:

Code:

#!/usr/bin/nawk -f
BEGIN { FS = "|" }
NF != 18 || !index($9, "@") { errors++ ; print }
END { print errors, "errors"; exit errors != 0 }

Call it this way (assuming you have put the above in a file called "findbadlines" and marked it executable).

Code:

findbadlines infile1 infile 2 [...] >badlinesfile

As with all code you find here (or anywhere else), you should understand it before you put your job on the line.

I figured out how to do it and got it working.

I'm making sure I understand exactly what it is the script is doing before I run with it. Makes me explaining it later on a little easier

Thanks for all of your help

Brian