UNIX for Dummies Questions & Answers

I'm trying to parse out DNS logs from dozens of different domain controllers over a large period of time. The logs are rolled up into individual text files by size, which may contain only a portion of a day's activity or several day's worth (depending on amount of activity). I'm splitting them by date, and transforming the information into a .CSV file to be parsed again later.

Splitting the files up by date is fairly easy, but every once in a while there's errors that will cause the script to fail. Each instance of the script crashing is from special characters being introduced in the results, which cause awk to crash.

I've been trying to find a way to not return lines that have these kinds of characters, but haven't managed to find one yet.

Example: (yes, it has been altered to conceal IP Addresses and domains queried)

Code:

20100207 03:52:40 3F8 PACKET  04073FC0 UDP Rcv 192.168.40.200  a61c   Q [0001   D   NOERROR] A     (7)èˆ5(3)217(3)123(2)11(3)dis(2)ds(4)test(3)com(0)

20100207 03:52:40 924 PACKET  04CCC250 UDP Rcv 192.168.40.200  06d4   Q [0001   D   NOERROR] A     (7)èˆ5(3)217(3)123(2)11(2)ds(4)test(3)com(0)

20100207 03:52:40 EFC PACKET  045E80A0 UDP Rcv 192.168.40.115  35bc   Q [0001   D   NOERROR] PTR   (3)200(2)40(3)138(3)160(7)in-addr(4)test(0)

20100207 03:52:40 DB4 PACKET  03D02080 UDP Rcv 192.168.40.200  265b   Q [0001   D   NOERROR] A     (7)èˆ5(3)217(3)123(2)11(4)test(3)com(0)

Of the 4 lines above, I only want to push the 3rd to awk. Since I don't know where, when, or even if, they'll appear in a file, I can't simply tell it what line to skip, or something along those lines (get 1 file with the special characters, about every 700 or so, out of 90,000 files).

Someone in my office suggested I try sed, but I'm not familiar with it, beyond simple find/replaces, and instances of "sed l" will translate the first line with special characters, then print the line a second time, and crashing.

The code I'm using to find and transform my data is: where "date" = the date I'm searching for and "file" = the domain controller DNS log and "domain-controller" = the name of the domain controller that was the source

Code:

grep.exe -F "date" file | awk "{OFS=\",\"};{print $1,$2,$8,$14,$15,\" >> \"date_domain-controller_DailyRollup.csv\"}"

BTW, yes, that's ugly, but part of it is I have to do it on Windows (yes, I know, believe me).

Try using grep to remove non-ascii character containing lines:

Code:

grep.exe -F "date" file | grep -v '[^A-Za-z0-9!@#$%^&\*\(\)\\\/\.\,\<\>\?+=-] | awk...

Hmmm

I had to add the [] brackets into the string, because it was filtering them out. Sadly, when I added them, it returned all rows.

I then reduced it to only look at the special characters I DO need: ":.[]()"

Code:

grep -F "20100207" crashers.txt | grep -v '[^A-Za-z0-9:\.\[\]\(\)]'

That still returns all four rows, not just the third.

I'll run that past a couple guys in the office here, see if that triggers an idea or three.

Maybe something like this:

grep -F '20100207' temp.dat | grep -v '^.*NOERROR\][[:alnum:]\(\)\-\ ]*$'

Thanks for the replies. Coworker put me on the right path.

Instead of trying to skip over the bad data, just transform it to something usable.

Code:

grep.exe -F "20100207" crashers.hst | tr -dc "[:graph:][:space:]" | awk "{OFS=\",\"};{print $1,$2,$8,$14,$15"}"

this output allows me to get everything instead of having to skip out some of the lines of data, ultimately speeding up the whole process, since I don't have to filter out the errors anymore using multiple grep statements.

Code:

20100207,03:52:40,192.168.40.200,A,(7)A"E+5(3)217(3)123(2)11(3)dis(2)ds(4)test(3)com(0)
20100207,03:52:40,192.168.40.200,A,(7)A"E+5(3)217(3)123(2)11(2)ds(4)test(3)com(0)
20100207,03:52:40,192.168.40.115,PTR,(3)200(2)40(3)138(3)160(7)in-addr(4)test(0)
20100207,03:52:40,192.168.40.200,A,(7)A"E+5(3)217(3)123(2)11(4)test(3)com(0)

Now, if only it would solve my space issues from having the data uncompressed... heh.

Again, thanks.