Regex and backreference to replace in binary file


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Regex and backreference to replace in binary file
# 1  
Old 10-30-2013
Regex and backreference to replace in binary file

Hello to all,

I have this sed script that replaces hex strins within a binary file.

As you can see, I want to replace all bytes 4X with 2X (where X could take values 0 to F).

Code:
sed -e 's/\x40/\x20/g' -e 's/\x41/\x21/g' -e 's/\x42/\x22/g' -e 's/\x43/\x23/g' -e 's/\x44/\x24/g' -e 's/\x46/\x26/g' -e 's/\x47/\x27/g' file

I would like to know if it is possible to use regex and backreference for this in sed or perl, etc?

I'd like to be able to do something like this, but is not working:

Code:
sed -e 's/\x4\(.\)/\x2\1/g' file

Thanks in advance for any help.
# 2  
Old 10-30-2013
If you show input data and expected output it will be helpful for us.
# 3  
Old 10-30-2013
You can try simple loop in Perl:
Code:
perl -pe 'for ($i=0;$i<16;$i++){$x=sprintf "%X",$i; s/\x4$x/\x2$x/g}' file

# 4  
Old 10-30-2013
Hello Akshay,

It could be any binary file, an image for example like attached. Only want to replace hex strings with another. In this case the string is only one byte 41 or 42 or 4X... with 21 or 22 or 2X using regex and backreference.

Hello bartus11,

Is not working, when I check with hexdump -C , it seems that actually is removing or deleting bytes.

I'd like to replace more than one hex string, in sed is easy only do multiple replacements using -e 's///g' .. -e 's///g' .. -e 's///g'.

Thanks for the help.
# 5  
Old 10-30-2013
Although some versions of sed may work on binary files, the standards only require sed to work on text files. A simple way that should work on most current systems is to use tr:
Code:
LC_ALL=POSIX tr '\100-\117' '\040-\057' <in_file >out_file

Some implementations of the tr utility may accept hex ranges, but the standards only specify octal as shown above. If your current locale has an underlying code set that only has single-byte eight-bit characters (such as ASCII or EBCDIC; but not UTF-8), you can skip the LC_ALL=POSIX.

Note, however, that if your image includes geolocation, timestamp, or camera settings data in addition to the actual image, you might not like the results unless you can extract just the image data you want to be processed.

Last edited by Don Cragun; 10-30-2013 at 05:33 PM.. Reason: fix typo
# 6  
Old 10-30-2013
Hello Don,

Thanks for your help. It seems to work!

Is possible to do multiple replacements for example this to '\220-\233' '\300-\313' and '\100-\117' to '\040-\057' in a single command?

is possible to replace an hex string of more than one byte? for example ff4567 with 3A0013

Thanks again.
# 7  
Old 10-30-2013
Quote:
Originally Posted by Ophiuchus
Hello Don,

Thanks for your help. It seems to work!

Is possible to do multiple replacements for example this to '\220-\233' '\300-\313' and '\100-\117' to '\040-\057' in a single command?

is possible to replace an hex string of more than one byte? for example ff4567 with 3A0013

Thanks again.
To do multiple ranges:
Code:
LC_ALL=POSIX tr '\100-\117\220-\233' '\040-\057\300-\313'<in_file >out_file

The tr utility translates characters or bytes; not strings or multi-byte sequences, and as I said before, tr isn't required to recognize hex. Look at the man page for tr on your system to determine if your tr utility has additional options or formats that may help you.
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

Regex Expression Replace

I have a XML file where there is a tag with like <wd:address_line_1>1234 Street</wd:address_line_1> I want to replace the values "1234 Street" with "Test Data". Different people have different address lines and i want to replace with a fixed value to mask the file. I was trying to use sed... (7 Replies)
Discussion started by: dr46014
7 Replies

2. UNIX for Advanced & Expert Users

Segregate file content using sed backreference

I have some text like EU1BTDAT:ASSGNDD filename='$SEQFILES/SUNIA.PJ008202.CARDLIB/DATECARD' EU1BTDATEST:ASSGNDD filename='$SEQFILES/SUNIA.PJ008202.CARDLIB/DATECARD' EU1CLOSEDATES:ASSGNDD filename='$SEQFILES/SUNIA.PJ008202.CARDLIB/DATECARD' EU1DATED:ASSGNDD... (8 Replies)
Discussion started by: gotamp
8 Replies

3. UNIX for Dummies Questions & Answers

Binary search and replace

Hello again. I have two problems - is it possible to solve them? 1. I want to replace a few bytes after specific hex-string. i.e.: I want to replace two bytes after AA AB AC: AA AB AC 00 00 AA AA AA so the expected result should be: AA AB AC FF FF AA AA AA 2. I want to replace three bytes... (9 Replies)
Discussion started by: useretail
9 Replies

4. Shell Programming and Scripting

Regex - search and replace

I have file which contains data in the following format all in a single line: BDW_PUBLN_ID DECIMAL(18:0) NOT NULL PRIMARY INDEX ARGO_ACCT_DEP_PI ( OFC_ID ,CSHBX_ID ,TRXN_SEQ_NUM ,PROCG_DT ) PARTITION BY RANGE_N(PROCG_DT BETWEEN DATE '2012-03-01' AND DATE '2014-12-31' EACH INTERVAL '1' MONTH );... (4 Replies)
Discussion started by: ysvsr1
4 Replies

5. Shell Programming and Scripting

Convert binary file to csv and then back to the binary format

Hello *nix specialists, Im working for a non profit organisation in Germany to transport DSL over WLAN to people in areas without no DSL. We are using Linksys WRT 54 router with DD-WRT firmware There are at the moment over 180 router running but we have to change some settings next time. So my... (7 Replies)
Discussion started by: digidax
7 Replies

6. UNIX for Dummies Questions & Answers

read regex from ID file, print regex and line below from source file

I have a file of protein sequences with headers (my source file). Based on a list of IDs (which are included in some of the headers), I'd like to print out only the specified sequences, with only the ID as header. In other words, I'd like to search source.txt for the terms in IDs.txt, and print... (3 Replies)
Discussion started by: pathunkathunk
3 Replies

7. Shell Programming and Scripting

sed regex backreference replacement

Hello, I want to rename multiple files and catch some points about backreference within sed and regex. Here is a part of my file list. Input: S92A.fa S92B.fa ... S96Z.fa S921.fa S922.fa ... S997.fa Note: The file names are not necessarily continuous from A~Z or 921 ~ 997, as some of the... (3 Replies)
Discussion started by: yifangt
3 Replies

8. Emergency UNIX and Linux Support

search replace regex question

Hi, I need to run a search and replace on a large database, what I need to change is all instances of #### (eg. 1764 or 1964) to (####) (eg. (1764) or (1964)) But there might be other numbers in there such as (1764) and I do not need those changed to ((1764)) How can I... (7 Replies)
Discussion started by: lawstudent
7 Replies

9. UNIX for Dummies Questions & Answers

replace line after a regex

Hi, I am trying to write a script which will modify a given account's settings by searching for a line in a file and then replacing the line after it. Here is a portion of my input file: type=friend username=0002 secret=password host=dynamic dtmfmode=rfc2833 mailbox=0002 context=sip... (2 Replies)
Discussion started by: the1armedcoder
2 Replies

10. Shell Programming and Scripting

replace string in binary file

Dear all, How can replace special string with another string in binary file? Thanks&Best Regards (2 Replies)
Discussion started by: sg1207
2 Replies
Login or Register to Ask a Question