Sponsored Content
Top Forums UNIX for Advanced & Expert Users cat / sed process weird characters Post 302544387 by bencpeters on Wednesday 3rd of August 2011 03:12:45 PM
Old 08-03-2011
cat / sed process weird characters

Hi everyone,
I'm trying to write a shell script that process a log file. The log format is generally:
(8 digit hex of unix time),(system ID),(state)\n
My shell script gets the file from the web, saves it in a local text directory. I then want to change the hex to decimal, convert from unix time to a day/month/year MST format and write out.

I have something that *mostly* works, by downloading the file, opening it with cat, piping the result to sed, using sed to get all the hex values and looping through them.

Unfortunately, there's a bug in the software that produces the log and for some systems the id isn't defined (someone probably forgot to initialize that variable), and it produces a line that looks like: 3B6A7227,››ù√剃,0

When I open this file with cat, the output for lines like that usually just contains a lot of question marks. This is the line I'm using to isolate the hex values:

Code:
cat ~/Downloads/log.txt | sed 's/[^0-9A-Za-z,\n]//g' | sed 's/,.*,[0,1]$//'

Originally I just had the second "sed"; I added the first one in an attempt to remove all the "weird" characters. Unfortunately, when I run this, it comes out as a list of hex numbers EXCEPT for the weird entries. These entries now have their hex number, a comma, then a number of question marks (and sometimes a decimal number), then another comma and the state.

How can I get rid of these? I realize the bug in the logging code needs to be fixed, but I don't have control over that, i'm just trying to clean up the log file.

Thanks!
 

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

How to get rid of all the weird characters and color on bash shell

Does anyone of you know how to turn off color and weird characters on bash shell when using the command "script"? Everytime users on my server used that command to record their script, they either couldn't print it because lp kept giving the "unknown format character" messages or the print paper... (1 Reply)
Discussion started by: Micz
1 Replies

2. Shell Programming and Scripting

Weird Ascii characters in file names

Hi. I have files in my OS that has weird file names with not-conventional ascii characters. I would like to run them but I can't refer them. I know the ascii # of the problematic characters. I can't change their name since it belongs to a 3rd party program... but I want to run it. is there... (2 Replies)
Discussion started by: yamsin789
2 Replies

3. Shell Programming and Scripting

weird issue about h, g, x in SED

I have a file called merge2.t: Hi Hello how are you. </Endtag> <New> I am fine.</New> This is a test. freelong how Here is the SED: sed -n ' /<\/Endtag>/ !{ H } /<\/Endtag>/ { x p } (4 Replies)
Discussion started by: freelong
4 Replies

4. Shell Programming and Scripting

long process listing with /usr/ucb/ps weird behaves

hello I am trying to run the following script to get the my-progam pid: #!/bin/ksh tt=`/usr/ucb/ps| grep -i $1| grep -v grep | awk '{print $2}'` echo $tt When I run the script I get the more PIDs $./test.sh my-program 12033 15033 15034 Actually my-program's PID is 12033....I... (6 Replies)
Discussion started by: sreeniatbp
6 Replies

5. Shell Programming and Scripting

cat file_list | [script to print last some characters]

Hello guys, I have a list of files. For example: /disk1/mediator_home/tmp/ntest/TSFILE00.8256.GGG1-U.0908250009.unp.20090824P8.is /disk1/mediator_home/tmp/ntest/TSFILE00.8257.GGG1-U.0908250013.unp.20090825P1.is... (2 Replies)
Discussion started by: mr_bold
2 Replies

6. Shell Programming and Scripting

Extra control characters being added when I create a file using cat command

Hi, I am using Cygwin.I created a new file and type into it using cat > newfile. When I open this using vi editor, it contains loads of extra control characters. Whats happening? (1 Reply)
Discussion started by: erora
1 Replies

7. Shell Programming and Scripting

share a shell script which can replace weird characters in directory or file name

I just finish the shell script . This shell can replace weird characters (such as #$%^@!'"...) in file or directory name by "_" I spent long time on replacing apostrophe in file/directory name added: 2012-03-14 the 124th line (/usr/bin/perl -i -e "s#\'#\\'#g" /tmp/rpdir_level$i.tmp) is... (5 Replies)
Discussion started by: begonia
5 Replies

8. Shell Programming and Scripting

Weird ^M characters is disturbing the paste command

Dear all, I have the files: xaa xab xac and I try to paste them using $paste -d, xaa xab xac I see: output 3e-130 ,6e-78 ,5e-74 6e-124 ,0,007 ,0,026 2e-119 When I type: $ paste -d, xaa xab xac |less I see: output 3e-130^M,6e-78^M,5e-74 6e-124^M,0,007^M,0,026 (2 Replies)
Discussion started by: valente
2 Replies

9. Shell Programming and Scripting

Control characters -weird problem

I am using Korn shell on Linux 2.6x platform , and I am suing the following code to capture the lines which contain CONTROL CHARACTERS in my file : awk '/]/ {print NR}' EROLLMENT_INPUT.txt The problem is that this code shows the file has control characters when the file is in folder A ,... (2 Replies)
Discussion started by: kumarjt
2 Replies

10. Shell Programming and Scripting

How to find out the weird blank characters?

I have a text file downloaded from the web, I want to count the unique words used in the file, and a person's speaking length during conversation by counting the words between the opening and closing quotation marks which differ from the standard ASCII code. Also I found out the file contains some... (2 Replies)
Discussion started by: yifangt
2 Replies
sigtool(1)							  Clam AntiVirus							sigtool(1)

NAME
sigtool - signature and database management tool SYNOPSIS
sigtool [options] DESCRIPTION
sigtool can be used to generate MD5 checksums, convert data into hexadecimal format, list virus signatures and build/unpack/test/verify CVD databases and update scripts. OPTIONS
-h, --help Output help information and exit. -V, --version Print version number and exit. --quiet Be quiet - output only error messages. --stdout Write all messages to stdout. --hex-dump Read data from stdin and write hex string to stdout. --md5 [FILES] Generate MD5 checksum from stdin or MD5 sigs for FILES. --mdb [FILES] Generate .mdb signatures for FILES. --html-normalise=FILE Create normalised HTML files comment.html, nocomment.html, and script.html in current working directory. --utf16-decode=FILE Decode UTF16 encoded data. --vba=FILE Extract VBA/Word6 macros from given MS Office document. --vba-hex=FILE Extract Word6 macros from given MS Office document and display the corresponding hex values. -i, --info Print a CVD information and verify MD5 and a digital signature. -b, --build Build a CVD file. -s, --server is required. --server ClamAV Signing Service address (for virus database maintainers only). --unpack FILE, -u FILE Unpack FILE (CVD) to a current directory. --unpack-current Unpack a local CVD file (main or daily) to current directory. --diff=OLD NEW, -d OLD NEW Create a diff file for OLD and NEW CVDs/INCDIRs. --run-cdiff=FILE, -r FILE Execute update script FILE in current directory. --verify-cdiff=FILE, -r FILE Verify DIFF against CVD/INCDIR. -l[FILE], --list-sigs[=FILE] List all signature names from the local database directory (default) or from FILE. -fREGEX, --find-sigs=REGEX Find and display signatures from the local database directory which match the given REGEX. The whole signature body (name, hex string, etc.) is checked. -fREGEX, --decode-sigs=REGEX Decode signatures read from the standard input (eg. piped from --find-sigs) -fREGEX, --test-sigs=DATABASE TARGET_FILE Test all signatures from DATABASE against TARGET_FILE. EXAMPLES
Generate hex string from testfile and save it to testfile.hex: cat testfile | sigtool --hex-dump > testfile.hex CREDITS
Please check the full documentation for credits. AUTHOR
Tomasz Kojm <tkojm@clamav.net> SEE ALSO
freshclam(1), freshclam.conf(5) ClamAV 0.96.1 February 12, 2007 sigtool(1)
All times are GMT -4. The time now is 07:03 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy