Escaping non-readable characters using grep, sed or awk
I'm trying to parse out DNS logs from dozens of different domain controllers over a large period of time. The logs are rolled up into individual text files by size, which may contain only a portion of a day's activity or several day's worth (depending on amount of activity). I'm splitting them by date, and transforming the information into a .CSV file to be parsed again later.
Splitting the files up by date is fairly easy, but every once in a while there's errors that will cause the script to fail. Each instance of the script crashing is from special characters being introduced in the results, which cause awk to crash.
I've been trying to find a way to not return lines that have these kinds of characters, but haven't managed to find one yet.
Example: (yes, it has been altered to conceal IP Addresses and domains queried)
Of the 4 lines above, I only want to push the 3rd to awk. Since I don't know where, when, or even if, they'll appear in a file, I can't simply tell it what line to skip, or something along those lines (get 1 file with the special characters, about every 700 or so, out of 90,000 files).
Someone in my office suggested I try sed, but I'm not familiar with it, beyond simple find/replaces, and instances of "sed l" will translate the first line with special characters, then print the line a second time, and crashing.
The code I'm using to find and transform my data is: where "date" = the date I'm searching for and "file" = the domain controller DNS log and "domain-controller" = the name of the domain controller that was the source
BTW, yes, that's ugly, but part of it is I have to do it on Windows (yes, I know, believe me).
Thanks for the replies. Coworker put me on the right path.
Instead of trying to skip over the bad data, just transform it to something usable.
this output allows me to get everything instead of having to skip out some of the lines of data, ultimately speeding up the whole process, since I don't have to filter out the errors anymore using multiple grep statements.
Now, if only it would solve my space issues from having the data uncompressed... heh.
Is there a command for sed and awk that will only sort the line with more characters?
#cat file
123
12345
12
asdgjljhhho
bac
ss
Output:
asdgjljhhho
#cat file2
11.2
12345.00
21.222
12345678.10 (2 Replies)
sed -e "s// /g" old.txt > new.txt
While I do know some control characters need to be escaped, can normal characters also be escaped and still work the same way? Basically I do not know all control characters that have a special meaning, for example, ?, ., % have a meaning and have to be escaped... (11 Replies)
I'm attempting a little hack to get grep to highlight (change foreground color to red) a found string. Assuming a target file "test" consisting of the word "albert":
My executable "algrep" consists of this:
grep $1 $2 | sed "s/$1/\\\033
And when run:
algrep al test
Produces this:... (2 Replies)
I'm looking for SED equivalent for grep -w -f. All I want is to search a list of patterns from a file. Also If the pattern doesn't match I do not want "null returned", rather I would prefer some text as place holder say "BLANK LINE" as I intend to process the output file based on line number.
... (1 Reply)
Hi,
I have a file xy.csv with the following data separated by pipe (|):
BC-NACO|12>ISA43<TEST|
A & A INC|FAMOUS'S AL|
i need to escape the xml characters as below
BC-NACO|12>ISA43<TEST|
A & A INC|FAMOUS'S AL|
Please advise (5 Replies)
Here is my problem.
I have a list of phone numbers that I want to use only the last 4 digits as PINs for something I am working on. I have all the numbers in a file but now I want to be removed all items EXCEPT the last 4 digits.
I have seen sed commands and some grep commands but I am... (10 Replies)
I want to append the following line to /var/spool/cron/root:
*/7 * * * * /root/'Linux CPU (EDF).sh' > /dev/null 2>&1
How to accomplish this using echo?
---------- Post updated at 04:09 PM ---------- Previous update was at 04:07 PM ----------
"Linux CPU (EDF)" is actually stored in a... (11 Replies)
Hi All,
I am having a trouble in passing special characters to a script. As I am new to bash script I dont know how to go and solve this.
mypwd=(a+sdfg!h#
if i pass $mypwd to a bash script, it is not accepting "(,!,+ etc". It would be a great help if some one can help to escape these... (3 Replies)
Hi All,
I wanted to delete all the unwanted characters in the string. ie, to delete all the characters which are not alpha numeric values.
var1="a./bc"
var2='abc/\."123'
like to get the output as
print var1
abc
print var2
abc123
Could you guys help me out pls.
Your help is... (3 Replies)
dear, I would like to rename files in a dir to another format, so I write a bash shell script to handle it. But my problem now is how to handle files having special characters like spaces, (, ):
"a b c (d).doc"
It seems that I need to escape those characters before applying the "mv" command.... (1 Reply)