I'm trying to parse out DNS logs from dozens of different domain controllers over a large period of time. The logs are rolled up into individual text files by size, which may contain only a portion of a day's activity or several day's worth (depending on amount of activity). I'm splitting them by date, and transforming the information into a .CSV file to be parsed again later.

Splitting the files up by date is fairly easy, but every once in a while there's errors that will cause the script to fail. Each instance of the script crashing is from special characters being introduced in the results, which cause awk to crash.

I've been trying to find a way to not return lines that have these kinds of characters, but haven't managed to find one yet.

Example: (yes, it has been altered to conceal IP Addresses and domains queried)

Code:

20100207 03:52:40 3F8 PACKET  04073FC0 UDP Rcv 192.168.40.200  a61c   Q [0001   D   NOERROR] A     (7)èˆ5(3)217(3)123(2)11(3)dis(2)ds(4)test(3)com(0)

20100207 03:52:40 924 PACKET  04CCC250 UDP Rcv 192.168.40.200  06d4   Q [0001   D   NOERROR] A     (7)èˆ5(3)217(3)123(2)11(2)ds(4)test(3)com(0)

20100207 03:52:40 EFC PACKET  045E80A0 UDP Rcv 192.168.40.115  35bc   Q [0001   D   NOERROR] PTR   (3)200(2)40(3)138(3)160(7)in-addr(4)test(0)

20100207 03:52:40 DB4 PACKET  03D02080 UDP Rcv 192.168.40.200  265b   Q [0001   D   NOERROR] A     (7)èˆ5(3)217(3)123(2)11(4)test(3)com(0)

Of the 4 lines above, I only want to push the 3rd to awk. Since I don't know where, when, or even if, they'll appear in a file, I can't simply tell it what line to skip, or something along those lines (get 1 file with the special characters, about every 700 or so, out of 90,000 files).

Someone in my office suggested I try sed, but I'm not familiar with it, beyond simple find/replaces, and instances of "sed l" will translate the first line with special characters, then print the line a second time, and crashing.

The code I'm using to find and transform my data is: where "date" = the date I'm searching for and "file" = the domain controller DNS log and "domain-controller" = the name of the domain controller that was the source

Code:

grep.exe -F "date" file | awk "{OFS=\",\"};{print $1,$2,$8,$14,$15,\" >> \"date_domain-controller_DailyRollup.csv\"}"

BTW, yes, that's ugly, but part of it is I have to do it on Windows (yes, I know, believe me).