Sponsored Content
Top Forums UNIX for Advanced & Expert Users cat / sed process weird characters Post 302544443 by bencpeters on Wednesday 3rd of August 2011 07:47:42 PM
Old 08-03-2011
Methyl - your idea would work, but when I try it, I get an error: 'tr: illegal byte sequence'

Here is a sample data set:
(in plaintext):

Code:
3B698960,eSWPump,0
3B698C36,sHeatPre,1
3B698C36,ePHPump,1
3B698C36,eSWPump,1
3B698CB4,Ô!˛ˇ√8ÏÔ,1
3B698CB4,eHWRPump,1
3B698CB4,››ù√剃,1
3B698CB4,eDownHRV,1
3B698CB4,eUpHRV,1
3B698E91,sHeatPre,0
3B698E91,ePHPump,0
3B698E91,eSWPump,0

and


Code:
$ cat ~/Downloads/log.txt | od -xc
0000000      4233    3936    3938    3036    652c    5753    7550    706d
           3   B   6   9   8   9   6   0   ,   e   S   W   P   u   m   p
0000020      302c    000a    4233    3936    4338    3633    732c    6548
           ,   0  \n  \0   3   B   6   9   8   C   3   6   ,   s   H   e
0000040      7461    7250    2c65    0a31    4233    3936    4338    3633
           a   t   P   r   e   ,   1  \n   3   B   6   9   8   C   3   6
0000060      652c    4850    7550    706d    312c    330a    3642    3839
           ,   e   P   H   P   u   m   p   ,   1  \n   3   B   6   9   8
0000100      3343    2c36    5365    5057    6d75    2c70    0a31    4233
           C   3   6   ,   e   S   W   P   u   m   p   ,   1  \n   3   B
0000120      3936    4338    3442    ef2c    fe21    c3ff    ec38    2cef
           6   9   8   C   B   4   , 357   ! 376 377 303   8 354 357   ,
0000140      0a31    4233    3936    4338    3442    652c    5748    5052
           1  \n   3   B   6   9   8   C   B   4   ,   e   H   W   R   P
0000160      6d75    2c70    0a31    4233    3936    4338    3442    dd2c
           u   m   p   ,   1  \n   3   B   6   9   8   C   B   4   , 032
0000200      dd1a    c39d    e48c    2cc4    0a31    4233    3936    4338
         032 335 235   Ì  ** 344 304   ,   1  \n   3   B   6   9   8   C
0000220      3442    652c    6f44    6e77    5248    2c56    0a31    4233
           B   4   ,   e   D   o   w   n   H   R   V   ,   1  \n   3   B
0000240      3936    4338    3442    652c    7055    5248    2c56    0a31
           6   9   8   C   B   4   ,   e   U   p   H   R   V   ,   1  \n
0000260      4233    3936    4538    3139    732c    6548    7461    7250
           3   B   6   9   8   E   9   1   ,   s   H   e   a   t   P   r
0000300      2c65    0a30    4233    3936    4538    3139    652c    4850
           e   ,   0  \n   3   B   6   9   8   E   9   1   ,   e   P   H
0000320      7550    706d    302c    330a    3642    3839    3945    2c31
           P   u   m   p   ,   0  \n   3   B   6   9   8   E   9   1   ,
0000340      5365    5057    6d75    2c70    0030                        
           e   S   W   P   u   m   p   ,   0                            
0000351

thanks!

Last edited by bencpeters; 08-03-2011 at 08:49 PM.. Reason: wrong option on od...
 

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

How to get rid of all the weird characters and color on bash shell

Does anyone of you know how to turn off color and weird characters on bash shell when using the command "script"? Everytime users on my server used that command to record their script, they either couldn't print it because lp kept giving the "unknown format character" messages or the print paper... (1 Reply)
Discussion started by: Micz
1 Replies

2. Shell Programming and Scripting

Weird Ascii characters in file names

Hi. I have files in my OS that has weird file names with not-conventional ascii characters. I would like to run them but I can't refer them. I know the ascii # of the problematic characters. I can't change their name since it belongs to a 3rd party program... but I want to run it. is there... (2 Replies)
Discussion started by: yamsin789
2 Replies

3. Shell Programming and Scripting

weird issue about h, g, x in SED

I have a file called merge2.t: Hi Hello how are you. </Endtag> <New> I am fine.</New> This is a test. freelong how Here is the SED: sed -n ' /<\/Endtag>/ !{ H } /<\/Endtag>/ { x p } (4 Replies)
Discussion started by: freelong
4 Replies

4. Shell Programming and Scripting

long process listing with /usr/ucb/ps weird behaves

hello I am trying to run the following script to get the my-progam pid: #!/bin/ksh tt=`/usr/ucb/ps| grep -i $1| grep -v grep | awk '{print $2}'` echo $tt When I run the script I get the more PIDs $./test.sh my-program 12033 15033 15034 Actually my-program's PID is 12033....I... (6 Replies)
Discussion started by: sreeniatbp
6 Replies

5. Shell Programming and Scripting

cat file_list | [script to print last some characters]

Hello guys, I have a list of files. For example: /disk1/mediator_home/tmp/ntest/TSFILE00.8256.GGG1-U.0908250009.unp.20090824P8.is /disk1/mediator_home/tmp/ntest/TSFILE00.8257.GGG1-U.0908250013.unp.20090825P1.is... (2 Replies)
Discussion started by: mr_bold
2 Replies

6. Shell Programming and Scripting

Extra control characters being added when I create a file using cat command

Hi, I am using Cygwin.I created a new file and type into it using cat > newfile. When I open this using vi editor, it contains loads of extra control characters. Whats happening? (1 Reply)
Discussion started by: erora
1 Replies

7. Shell Programming and Scripting

share a shell script which can replace weird characters in directory or file name

I just finish the shell script . This shell can replace weird characters (such as #$%^@!'"...) in file or directory name by "_" I spent long time on replacing apostrophe in file/directory name added: 2012-03-14 the 124th line (/usr/bin/perl -i -e "s#\'#\\'#g" /tmp/rpdir_level$i.tmp) is... (5 Replies)
Discussion started by: begonia
5 Replies

8. Shell Programming and Scripting

Weird ^M characters is disturbing the paste command

Dear all, I have the files: xaa xab xac and I try to paste them using $paste -d, xaa xab xac I see: output 3e-130 ,6e-78 ,5e-74 6e-124 ,0,007 ,0,026 2e-119 When I type: $ paste -d, xaa xab xac |less I see: output 3e-130^M,6e-78^M,5e-74 6e-124^M,0,007^M,0,026 (2 Replies)
Discussion started by: valente
2 Replies

9. Shell Programming and Scripting

Control characters -weird problem

I am using Korn shell on Linux 2.6x platform , and I am suing the following code to capture the lines which contain CONTROL CHARACTERS in my file : awk '/]/ {print NR}' EROLLMENT_INPUT.txt The problem is that this code shows the file has control characters when the file is in folder A ,... (2 Replies)
Discussion started by: kumarjt
2 Replies

10. Shell Programming and Scripting

How to find out the weird blank characters?

I have a text file downloaded from the web, I want to count the unique words used in the file, and a person's speaking length during conversation by counting the words between the opening and closing quotation marks which differ from the standard ASCII code. Also I found out the file contains some... (2 Replies)
Discussion started by: yifangt
2 Replies
yaml(n) 							  YAML processing							   yaml(n)

__________________________________________________________________________________________________________________________________________________

NAME
yaml - YAML Format Encoder/Decoder SYNOPSIS
package require Tcl 8.4 package require yaml ?0.3.5? ::yaml::yaml2dict ?options? txt ::yaml::yaml2huddle ?options? txt ::yaml::setOption ?options? ::yaml::dict2yaml dict ?indent? ?wordwrap? ::yaml::list2yaml list ?indent? ?wordwrap? ::yaml::huddle2yaml huddle ?indent? ?wordwrap? _________________________________________________________________ DESCRIPTION
The yaml package provides a simple Tcl-only library for parsing the YAML http://www.yaml.org/ data exchange format as specified in http://www.yaml.org/spec/1.1/. The yaml package returns data as a Tcl dict. Either the dict package or Tcl 8.5 is required for use. COMMANDS
::yaml::yaml2dict ?options? txt ::yaml::yaml2huddle ?options? txt Parse yaml formatted text txt into a Tcl dict/huddle and return the value. -file txt is a filename of YAML-stream. -stream txt is just a YAML-stream. -types list The list is a type list for the yaml-scalar types.(e.g. !!str !!timestamp !!integer !!true ...) -types {timestamp integer null true false} In this case, if a string matched "timestamp", converted to the TCL internal timestamp.(e.g. "2001-12-15T02:59:43.1Z" => 1008385183) -m:true param The param is two elements of list for the value of true, and considered strings. -m:true {1 {true on + yes y}} In this case, the string "yes" found in YAML Stream, automatically converted 1. -m:false param The param is two elements of list for the value of false, and considered strings. -m:false {0 {false off - no n}} -m:null param The param is two elements of list for the value of null, and considered strings. -m:null {"" {null nil "" ~}} -validate Experiment,old: Output stream contains YAML's-tag, each node. % puts [::yaml::load -validate {[aaa, bbb]}] => !!seq {{!!str aaa} {!!str bbb}} ::yaml::setOption ?options? Change implicit options for the library. Now, the params are the same as ::yaml::yaml2dict. Arguments of::yaml::yaml2dict is more priority than this setting. ::yaml::dict2yaml dict ?indent? ?wordwrap? ::yaml::list2yaml list ?indent? ?wordwrap? ::yaml::huddle2yaml huddle ?indent? ?wordwrap? Convert a dict/list/huddle object into YAML stream. indent spaces indent of each block node. currently default is 2. wordwrap word wrap for YAML stream. currently default is 40. EXAMPLES
An example of a yaml stream converted to Tcl. A yaml stream is returned as a single item with multiple elements. { --- !<tag:clarkevans.com,2002:invoice> invoice: 34843 date : 2001-01-23 bill-to: &id001 given : Chris family : Dumars address: lines: | 458 Walkman Dr. Suite #292 city : Royal Oak state : MI postal : 48046 ship-to: *id001 product: - sku : BL394D quantity : 4 description : Basketball price : 450.00 - sku : BL4438H quantity : 1 description : Super Hoop price : 2392.00 tax : 251.42 total: 4443.52 comments: Late afternoon is best. Backup contact is Nancy Billsmer @ 338-4338. } => invoice 34843 date 2001-01-23 bill-to {given Chris family Dumars address {lines {458 Walkman Dr. Suite #292 } city {Royal Oak} state MI postal 48046}} ship-to {given Chris family Dumars address {lines {458 Walkman Dr. Suite #292 } city {Royal Oak} state MI postal 48046}} product {{sku BL394D quantity 4 description Basketball price 450.00} {sku BL4438H quantity 1 description {Super Hoop} price 2392.00}} tax 251.42 total 4443.52 comments {Late afternoon is best. Backup contact is Nancy Billsmer @ 338-4338.} An example of a yaml object converted to Tcl. A yaml object is returned as a multi-element list (a dict). { --- - [name , hr, avg ] - [Mark McGwire, 65, 0.278] - [Sammy Sosa , 63, 0.288] - Mark McGwire: {hr: 65, avg: 0.278} Sammy Sosa: { hr: 63, avg: 0.288} } => {name hr avg} {{Mark McGwire} 65 0.278} {{Sammy Sosa} 63 0.288} {{Mark McGwire} {hr 65 avg 0.278} {Sammy Sosa} {hr 63 avg 0.288}} LIMITATIONS
tag parser not implemented. currentry, tags are merely ignored. Only Anchor => Aliases ordering. back alias-referring is not supported. Too many braces, or too few braces. Not enough character set of line feeds. Please use only " " as line breaks. BUGS, IDEAS, FEEDBACK This document, and the package it describes, will undoubtedly contain bugs and other problems. Please report such in the category yaml of the Tcllib SF Trackers [http://sourceforge.net/tracker/?group_id=12883]. Please also report any ideas for enhancements you may have for either package and/or documentation. SEE ALSO
base64, huddle, json KEYWORDS
data exchange, huddle, parsing, text processing, yaml COPYRIGHT
Copyright (c) 2008 KATO Kanryu <kanryu6@users.sourceforge.net> yaml 0.3.5 yaml(n)
All times are GMT -4. The time now is 01:16 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy