Replace hex values using sed command


 
Thread Tools Search this Thread
Top Forums UNIX for Advanced & Expert Users Replace hex values using sed command
# 8  
Old 01-15-2018
Of course there is always 'printf' and could easily be in a 'read' loop...
OSX 10.13.2, default bash terminal.
Longhand to show it working...
Code:
Last login: Mon Jan 15 16:11:31 on ttys000
AMIGA:amiga~> bash --version
GNU bash, version 3.2.57(1)-release (x86_64-apple-darwin17)
Copyright (C) 2007 Free Software Foundation, Inc.
AMIGA:amiga~> printf "\xc1Bc Aa \xc1 BC \xc1a\xc1\n" > lalo.txt
AMIGA:amiga~> hexdump -C lalo.txt
00000000  c1 42 63 20 41 61 20 c1  20 42 43 20 c1 61 c1 0a  |.Bc Aa . BC .a..|
00000010
AMIGA:amiga~> string=$( cat lalo.txt )
AMIGA:amiga~> printf "${string//$'\xc1'/A}\n" > lalo1.txt
AMIGA:amiga~> cat lalo1.txt
ABc Aa A BC AaA
AMIGA:amiga~> _

# 9  
Old 01-19-2018
Hi,
With perl (>= 5.8) :
Code:
$ echo 'çéèàùÁ' | perl -MUnicode::Normalize -pe 'BEGIN{binmode STDIN, ":encoding(utf-8)"};$_ = NFD $_; y/[^x00-xFF]//cd'
ceeauA

But not work with character as '€' or 'œ' (no translate but remove by perl).
# 10  
Old 01-20-2018
Although some versions of sed accept backslash escapes for octal, hexadecimal, and common character escapes like \n for <newline> and \t for <tab>; none of these are present in "standard" sed.

The sed command needed to change Latin capital letter A with acute to Latin capital letter A is simple:
Code:
sed 's/Á/A/' lalo.txt

but this works if, and only if, your current locale is using the same codeset that is used to encode Latin capital letter A with acute that is present in lalo.txt. Based on the od output you showed us in post #1, we can say that Latin capital letter A with acute is encoded correctly for the ISO/IEC 8859-1 codeset in lalo.txt. And based on the other posts you have made in this thread, my guess would be that your current locale is using UTF-8 as its underlying codeset. Therefore, the following printfj command should create a sed command that will do what you want:
Code:
printf "LC_ALL=C sed 's/\xc1/A/' lalo.txt > lalo2.txt" > lalo.ksh

If you run the above command and then run the command:
Code:
ksh lalo.ksh

you should end up with the file lalo2.txt with the output you want, as shown by the command:
Code:
$ od -bc lalo2.ksh
0000000   101 012                                                        
           A  \n                                                        
0000002
$

The LC_ALL=C is probably needed because the ISO/IEC 8859-1 encoding of Á found in lalo.txt is not a valid character in the codeset used by your current locale. The LC_ALL=C will cause that sed command to be run in a locale where every single-byte value is a valid character. This should avoid the errors like:
Code:
sed: 1: "s/?/A/": RE error: illegal byte sequence

that you might get without it.
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

How to replace with "sed" some hex values by other hex values?

Assume I have a file \usr\home\\somedir\myfile123.txt and I want to replace all occurencies of the two (concatenated) hex values x'AD' x'A0' bytwo other (concatenated) hex values x'20' x'6E' How can I achieve this with the gnu sed tool? Additional question: Is there a way to let sed show... (1 Reply)
Discussion started by: pstein
1 Replies

2. Shell Programming and Scripting

Help search and replace hex values only in specific files

perl -pi -e 's/\x00/\x0d\x0a/g' `grep -l $'GS' filelist` This isn't working :confused:, it's not pulling the files that contain the regex. Please help me rewrite this :wall:. Ideally for this to work on 9K of 20K files in the directory, I've tried this but I don't know enough about awk... (7 Replies)
Discussion started by: verge
7 Replies

3. Shell Programming and Scripting

sed replace values of column with ordered numbering

Hello, I am trying to write a shell script that will create a gnuplot file. My main problem is that I have a data file with two columns: 1.05929120E+09 5.0000701214792 1.05930096E+09 5.00006386985764 1.05930584E+09 5.00019465404908 1.05931072E+09 5.00031960589719 ... (2 Replies)
Discussion started by: pau
2 Replies

4. Shell Programming and Scripting

Convert hex values to displayable characters

Hi, I am a bit stuck with displaying characters. I am having values like below in the proper displayable characters. which I would want to print the actual value on the right hand side. I dont want to create an array because I would have to create 255 different values. isnt there another way of... (17 Replies)
Discussion started by: ahmedwaseem2000
17 Replies

5. Programming

searching files for hex or oct values

I have a set of files without extensions. How can I programatically tell if a file is in gzip format? The gzip file format spec RFC 1952 GZIP File Format Specification version 4.3 states that gzip files have certain hex/oct values at the beginning of the file. 1st byte = 0x1f in hex,... (2 Replies)
Discussion started by: daflore
2 Replies

6. Shell Programming and Scripting

Replace blank values for string using sed

Hi everyone, I was wondering how could from a file where each row is separated by tabulations, the row values where are in blank replace them by a string or value. My file has this form: 26/01/09 13:45:00 0 0 26/01/09 14:00:00 1495.601318 0 26/01/09 14:15:00 1495.601318 0 ... (4 Replies)
Discussion started by: tonet
4 Replies

7. UNIX for Dummies Questions & Answers

using sed to replace values...

I have a file: $somevar=somevalue $anothervar= $someothervar=45 I'd like to be able to replace the values in the file. I just don't know how exactly to use sed. I was thinking along the lines of: cat file | sed "s/$somevar=/anotherval/g" I was hoping this would make the... (2 Replies)
Discussion started by: mrwatkin
2 Replies

8. Shell Programming and Scripting

Loop with sed command to replace line with sed command in it

Okay, title is kind of confusion, but basically, I have a lot of scripts on a server that I need to replace a ps command, however, the new ps command I'm trying to replace the current one with pipes to sed at one point. So now I am attempting to create another script that replaces that line. ... (1 Reply)
Discussion started by: cbo0485
1 Replies

9. Shell Programming and Scripting

How do i use sed to replace string with values from a dictionary file

I have file 1 with one million rows. one of the fields is "FIRSTNAME" (the string) I have a second file with about 20 first names. JUDE DAVID HOMER CANE ABEL MARTY CARL SONNY STEVE BERT OSCAR MICKY JAMES JOHN GLENN DOUG (3 Replies)
Discussion started by: yoyolandre
3 Replies

10. UNIX for Advanced & Expert Users

Modifying binary file by editing Hex values ?

Hi , i'm using special binary file (lotus notes) and modifying an hexadecimal address range with windows hex editor and it works fine ! The file is an AIX one and i'm forced to transfert (ftp) it before modifying it and re-transfert ! NOW i would do this directly under AIX ! I can... (4 Replies)
Discussion started by: Nicol
4 Replies
Login or Register to Ask a Question