Hi everyone,
I'm trying to write a shell script that process a log file. The log format is generally:
(8 digit hex of unix time),(system ID),(state)\n
My shell script gets the file from the web, saves it in a local text directory. I then want to change the hex to decimal, convert from unix time to a day/month/year MST format and write out.

I have something that *mostly* works, by downloading the file, opening it with cat, piping the result to sed, using sed to get all the hex values and looping through them.

Unfortunately, there's a bug in the software that produces the log and for some systems the id isn't defined (someone probably forgot to initialize that variable), and it produces a line that looks like: 3B6A7227,››ù√剃,0

When I open this file with cat, the output for lines like that usually just contains a lot of question marks. This is the line I'm using to isolate the hex values:

Code:

cat ~/Downloads/log.txt | sed 's/[^0-9A-Za-z,\n]//g' | sed 's/,.*,[0,1]$//'

Originally I just had the second "sed"; I added the first one in an attempt to remove all the "weird" characters. Unfortunately, when I run this, it comes out as a list of hex numbers EXCEPT for the weird entries. These entries now have their hex number, a comma, then a number of question marks (and sometimes a decimal number), then another comma and the state.

How can I get rid of these? I realize the bug in the logging code needs to be fixed, but I don't have control over that, i'm just trying to clean up the log file.

Thanks!