Another binary manipulation thread.


 
Thread Tools Search this Thread
Top Forums UNIX for Advanced & Expert Users Another binary manipulation thread.
# 8  
Old 08-23-2013
Hi DGPickett...

I like what I see...

Ooooh, I never thought about binary file reading using "read" as the prime mover as I can easily work around hexdump, (or od), for my binary file handling.

Let me delve deeper and find a back door. <wink>

If I am defeated I will let you know...

I must be allowed the two character "\0" for any binary/byte zero however...

This will be really interesting for me - thanks...
# 9  
Old 08-23-2013
Moderator's Comments:
Mod Comment While i appreciate the interesting discussion i think this is hardly "Unix for Dummies" stuff. I transfer this thread over to "Unix Experts". I hope you feel suitably honoured. ;-))


bakunin
This User Gave Thanks to bakunin For This Post:
# 10  
Old 08-24-2013
Hi bakunin...

Thanks a lot; if only I was that good...

Hi DGPickett...

This is the best I can do. Although there is a lot of superfluous code the important bits can be seen. This uses a default Macbook Pro system using bash...

Code:
#!/bin/bash --posix

ls -l /tmp/binary.file

text=""
newtext=""
count=0

# Active part...
while read
do
	text=$(xxd -p /tmp/binary.file | tr -d '\n')
done < /tmp/binary.file

echo "*****************************************************"
echo -n "$text"
echo ""
echo "*****************************************************"

# Conversion part...
for count in $( seq 0 2 2047 )
do
	if [ "${text:$count:2}" == "00" ]
	then
		newtext=$newtext'\\0'
	else
		newtext=$newtext'\x'${text:$count:2}
	fi
done

# Binary string part...
text=$(echo -e -n "$newtext")
echo "*****************************************************"
echo -n "$text" > /tmp/newbinary.file
echo -n "$text"
echo ""
echo "*****************************************************"
hexdump -C /tmp/newbinary.file

The results on screen...
Code:
Last login: Sat Aug 24 21:32:46 on ttys000
AMIGA:barrywalker~> ./readtest.sh
-rw-r--r--  1 barrywalker  wheel  1024 24 Aug 21:18 /tmp/binary.file
*****************************************************
e8c28d4dc08d47000a19c075d84174527600d71dd80100b4e8e87cff8300070f00c99c01ffeb3d030009b70f78e78d00000300ff0000000538c748a304070d3f050aeb00300a00cb031100eb00f7c0551c890000895be5c7405378ab0083c40022d85689f0cd480fb880c34cf855538d184948434845004908485d644131c3c0484d540f48ff757d007248cd4d8bff5de80105ffff892483c2b14106ff078b854c53430e0d74ff4841895e8905f2750f0048e54c4848000000e874eb89858b0f3800003c76f62274c68375ffc375e8000048005b006448dec406e80600745148ff05f80989f6019e060077f41185e98d29ebebefff000048d93d8dd7c34cc7c60d8d00fac015ff41df3500c33349217589065effce8900090048276335c7bf000641008cc748002489484141410089854841310500e8ffbfff22c0059aba8dff3500ef00eff8003eac8a00455c8948488575e822170001c1ff0f460289594875ff756306c400633d89030848858989808bff48ff1cbc0fff4548ffff740089f9c7e97541fff8008d41c63c00c6ff161048c048495e0440f600850f03fd00c08309640000538b01c60055455c4c4010894885a248aadb834800ff48ac484148df74ec00ff1fcc00224930ec004808e848f7ed04fd488d54004c494c894848c07400488df706fb05058d4914c889feff0004d5ff01004c45038b00754583670089ff627fff00084c8dff8de7e7d985e7ff7c00853e008b31ff55bec3fdc0850100050009e85d574900481389f205488385eb4b00ea004889030074e8000089e7008900c05b4803db0005480cc700ee41c700fbffebb28b0048065be70a034c00831b80ac838318c1c0ff008d418bfe3b44e848e883e88dffadb27dd0742674f8820b29ffec0089c68b4945745d76001d00d14800dd8b483063413f09ba0538001ec69500080085251f85090048ff8a19e8458b030005003de8e675544c253d785541cade8d412b0e097474fffd00e8003b00000a888d18ffc0fe8874040000fe8d85ffe9090fff83484c91c0b5de06c3c0080a1651458daf8983c100c0fbc08989007440ff242d7400c4484c48d8010805e08b45c889ed8d94c64841480400f300551806e85d01e9a40000207c46f6ff00005338608323f800c1ff028901c0be4c00481c4889404555e80007884c3405730808ca6344e90063f65745f6ebe882540125fffb785341f7241018ebe8e87e850000ff48a3485d9045413b3fff48fe45ecffbf0548ff45050a488dfd39531c97088b8688634874009cdfb8859c004844bf3dbe8d00858a057848e8bec9e84889af0848ff52484870ffbd021fd6770800c7040e002b00b50589fc8900fc024529ff8b0000fc0100ff4c8b0000ac04003d108b0575e8ff7989e8bd007d08ff8dfe1900418d00488d85080fff067485fb83
*****************************************************
*****************************************************
?M??G\0
?u?AtRv\0??\0???|??\0\0ɜ??=\0	?x?\0\0\0?\0\0\08?H?
?\00
\0?\0?\0??U?\0\0?[??@Sx?\0??\0"?V???H???L?US?IHCHE\0H]dA1??HMTH?u}\0rH?M??]????$t?HA?^??u\0H?LHH\0\0\0?t뉅?8\0\0<v?"tƃu??u?\0\0H\0[\0dH???\0tQH??	???\0w???\0???A?5\0?3I!u?^?Ή\0	\0H'c5ǿ\0A\0??H\0$?HAAA\0??HA1\0????"?????5\0?\0??\0>??\0E\?HH?u?"\0??F?YHu?uc?\0c=H??????H???EH??t\0????uA??\0?A?<\0??H?HI^@?\0??\0??	d\0\0S??\0UE\L@?H??H?ۃH\0?H?HAH?t?\0??\0"I0?\0?H???H?T\0LIL?HH?t\0H????Iȉ??\0??\0LE?\0uE?g\0??b?\L?????م??|\0?>\0?1?U?????\0\0	?]WI\0H??H???K\0?\0H?\0t?\0\0??\0?\0?[H?\0H
           ?\0?A?\0??벋\0H[?
L\0???????\0?A??;D?H?????}?t&t??
                                )??\0?ƋIEt]v\0\0?H\0݋H0cA?	?8\0ƕ\\0?%?	\0H???E?\0\0=??uTL%=xUA?ލA+	tt??\0?\0;\0\0
??????t\0\0?????	??HL??????
QE?????\0?????\0t@?$-t\0?HLH??Eȉ퍔?HAH\0?\0U?]?\0\0 |F??\0\0S8`?#?\0?????L\0HH?@EU?\0?L?cD?\0c?WE???T%??xSA?$???~?\0\0?H?H]?EA;??H?E???H?E
H??9S???cHt\0?߸??\0HD?=??\0??xH???H?H?RHHp???\0?\0+\0????\0?E)??\0\0?\0?L?\0\0?\0=?u??y??\0???\0A?\0H??t???
*****************************************************
00000000  e8 c2 8d 4d c0 8d 47 5c  30 0a 19 c0 75 d8 41 74  |...M..G\0...u.At|
00000010  52 76 5c 30 d7 1d d8 01  5c 30 b4 e8 e8 7c ff 83  |Rv\0....\0...|..|
00000020  5c 30 07 0f 5c 30 c9 9c  01 ff eb 3d 03 5c 30 09  |\0..\0.....=.\0.|
00000030  b7 0f 78 e7 8d 5c 30 5c  30 03 5c 30 ff 5c 30 5c  |..x..\0\0.\0.\0\|
00000040  30 5c 30 05 38 c7 48 a3  04 07 0d 3f 05 0a eb 5c  |0\0.8.H....?...\|
00000050  30 30 0a 5c 30 cb 03 11  5c 30 eb 5c 30 f7 c0 55  |00.\0...\0.\0..U|
00000060  1c 89 5c 30 5c 30 89 5b  e5 c7 40 53 78 ab 5c 30  |..\0\0.[..@Sx.\0|
00000070  83 c4 5c 30 22 d8 56 89  f0 cd 48 0f b8 80 c3 4c  |..\0".V...H....L|
00000080  f8 55 53 8d 18 49 48 43  48 45 5c 30 49 08 48 5d  |.US..IHCHE\0I.H]|
00000090  64 41 31 c3 c0 48 4d 54  0f 48 ff 75 7d 5c 30 72  |dA1..HMT.H.u}\0r|
000000a0  48 cd 4d 8b ff 5d e8 01  05 ff ff 89 24 83 c2 b1  |H.M..]......$...|
000000b0  41 06 ff 07 8b 85 4c 53  43 0e 0d 74 ff 48 41 89  |A.....LSC..t.HA.|
000000c0  5e 89 05 f2 75 0f 5c 30  48 e5 4c 48 48 5c 30 5c  |^...u.\0H.LHH\0\|
000000d0  30 5c 30 e8 74 eb 89 85  8b 0f 38 5c 30 5c 30 3c  |0\0.t.....8\0\0<|
000000e0  76 f6 22 74 c6 83 75 ff  c3 75 e8 5c 30 5c 30 48  |v."t..u..u.\0\0H|
000000f0  5c 30 5b 5c 30 64 48 de  c4 06 e8 06 5c 30 74 51  |\0[\0dH.....\0tQ|
00000100  48 ff 05 f8 09 89 f6 01  9e 06 5c 30 77 f4 11 85  |H.........\0w...|
00000110  e9 8d 29 eb eb ef ff 5c  30 5c 30 48 d9 3d 8d d7  |..)....\0\0H.=..|
00000120  c3 4c c7 c6 0d 8d 5c 30  fa c0 15 ff 41 df 35 5c  |.L....\0....A.5\|
00000130  30 c3 33 49 21 75 89 06  5e ff ce 89 5c 30 09 5c  |0.3I!u..^...\0.\|
00000140  30 48 27 63 35 c7 bf 5c  30 06 41 5c 30 8c c7 48  |0H'c5..\0.A\0..H|
00000150  5c 30 24 89 48 41 41 41  5c 30 89 85 48 41 31 05  |\0$.HAAA\0..HA1.|
00000160  5c 30 e8 ff bf ff 22 c0  05 9a ba 8d ff 35 5c 30  |\0...."......5\0|
00000170  ef 5c 30 ef f8 5c 30 3e  ac 8a 5c 30 45 5c 89 48  |.\0..\0>..\0E\.H|
00000180  48 85 75 e8 22 17 5c 30  01 c1 ff 0f 46 02 89 59  |H.u.".\0....F..Y|
00000190  48 75 ff 75 63 06 c4 5c  30 63 3d 89 03 08 48 85  |Hu.uc..\0c=...H.|
000001a0  89 89 80 8b ff 48 ff 1c  bc 0f ff 45 48 ff ff 74  |.....H.....EH..t|
000001b0  5c 30 89 f9 c7 e9 75 41  ff f8 5c 30 8d 41 c6 3c  |\0....uA..\0.A.<|
000001c0  5c 30 c6 ff 16 10 48 c0  48 49 5e 04 40 f6 5c 30  |\0....H.HI^.@.\0|
000001d0  85 0f 03 fd 5c 30 c0 83  09 64 5c 30 5c 30 53 8b  |....\0...d\0\0S.|
000001e0  01 c6 5c 30 55 45 5c 4c  40 10 89 48 85 a2 48 aa  |..\0UE\L@..H..H.|
000001f0  db 83 48 5c 30 ff 48 ac  48 41 48 df 74 ec 5c 30  |..H\0.H.HAH.t.\0|
00000200  ff 1f cc 5c 30 22 49 30  ec 5c 30 48 08 e8 48 f7  |...\0"I0.\0H..H.|
00000210  ed 04 fd 48 8d 54 5c 30  4c 49 4c 89 48 48 c0 74  |...H.T\0LIL.HH.t|
00000220  5c 30 48 8d f7 06 fb 05  05 8d 49 14 c8 89 fe ff  |\0H.......I.....|
00000230  5c 30 04 d5 ff 01 5c 30  4c 45 03 8b 5c 30 75 45  |\0....\0LE..\0uE|
00000240  83 67 5c 30 89 ff 62 7f  ff 5c 30 08 4c 8d ff 8d  |.g\0..b..\0.L...|
00000250  e7 e7 d9 85 e7 ff 7c 5c  30 85 3e 5c 30 8b 31 ff  |......|\0.>\0.1.|
00000260  55 be c3 fd c0 85 01 5c  30 05 5c 30 09 e8 5d 57  |U......\0.\0..]W|
00000270  49 5c 30 48 13 89 f2 05  48 83 85 eb 4b 5c 30 ea  |I\0H....H...K\0.|
00000280  5c 30 48 89 03 5c 30 74  e8 5c 30 5c 30 89 e7 5c  |\0H..\0t.\0\0..\|
00000290  30 89 5c 30 c0 5b 48 03  db 5c 30 05 48 0c c7 5c  |0.\0.[H..\0.H..\|
000002a0  30 ee 41 c7 5c 30 fb ff  eb b2 8b 5c 30 48 06 5b  |0.A.\0.....\0H.[|
000002b0  e7 0a 03 4c 5c 30 83 1b  80 ac 83 83 18 c1 c0 ff  |...L\0..........|
000002c0  5c 30 8d 41 8b fe 3b 44  e8 48 e8 83 e8 8d ff ad  |\0.A..;D.H......|
000002d0  b2 7d d0 74 26 74 f8 82  0b 29 ff ec 5c 30 89 c6  |.}.t&t...)..\0..|
000002e0  8b 49 45 74 5d 76 5c 30  1d 5c 30 d1 48 5c 30 dd  |.IEt]v\0.\0.H\0.|
000002f0  8b 48 30 63 41 3f 09 ba  05 38 5c 30 1e c6 95 5c  |.H0cA?...8\0...\|
00000300  30 08 5c 30 85 25 1f 85  09 5c 30 48 ff 8a 19 e8  |0.\0.%...\0H....|
00000310  45 8b 03 5c 30 05 5c 30  3d e8 e6 75 54 4c 25 3d  |E..\0.\0=..uTL%=|
00000320  78 55 41 ca de 8d 41 2b  0e 09 74 74 ff fd 5c 30  |xUA...A+..tt..\0|
00000330  e8 5c 30 3b 5c 30 5c 30  0a 88 8d 18 ff c0 fe 88  |.\0;\0\0........|
00000340  74 04 5c 30 5c 30 fe 8d  85 ff e9 09 0f ff 83 48  |t.\0\0.........H|
00000350  4c 91 c0 b5 de 06 c3 c0  08 0a 16 51 45 8d af 89  |L..........QE...|
00000360  83 c1 5c 30 c0 fb c0 89  89 5c 30 74 40 ff 24 2d  |..\0.....\0t@.$-|
00000370  74 5c 30 c4 48 4c 48 d8  01 08 05 e0 8b 45 c8 89  |t\0.HLH......E..|
00000380  ed 8d 94 c6 48 41 48 04  5c 30 f3 5c 30 55 18 06  |....HAH.\0.\0U..|
00000390  e8 5d 01 e9 a4 5c 30 5c  30 20 7c 46 f6 ff 5c 30  |.]...\0\0 |F..\0|
000003a0  5c 30 53 38 60 83 23 f8  5c 30 c1 ff 02 89 01 c0  |\0S8`.#.\0......|
000003b0  be 4c 5c 30 48 1c 48 89  40 45 55 e8 5c 30 07 88  |.L\0H.H.@EU.\0..|
000003c0  4c 34 05 73 08 08 ca 63  44 e9 5c 30 63 f6 57 45  |L4.s...cD.\0c.WE|
000003d0  f6 eb e8 82 54 01 25 ff  fb 78 53 41 f7 24 10 18  |....T.%..xSA.$..|
000003e0  eb e8 e8 7e 85 5c 30 5c  30 ff 48 a3 48 5d 90 45  |...~.\0\0.H.H].E|
000003f0  41 3b 3f ff 48 fe 45 ec  ff bf 05 48 ff 45 05 0a  |A;?.H.E....H.E..|
00000400  48 8d fd 39 53 1c 97 08  8b 86 88 63 48 74 5c 30  |H..9S......cHt\0|
00000410  9c df b8 85 9c 5c 30 48  44 bf 3d be 8d 5c 30 85  |.....\0HD.=..\0.|
00000420  8a 05 78 48 e8 be c9 e8  48 89 af 08 48 ff 52 48  |..xH....H...H.RH|
00000430  48 70 ff bd 02 1f d6 77  08 5c 30 c7 04 0e 5c 30  |Hp.....w.\0...\0|
00000440  2b 5c 30 b5 05 89 fc 89  5c 30 fc 02 45 29 ff 8b  |+\0.....\0..E)..|
00000450  5c 30 5c 30 fc 01 5c 30  ff 4c 8b 5c 30 5c 30 ac  |\0\0..\0.L.\0\0.|
00000460  04 5c 30 3d 10 8b 05 75  e8 ff 79 89 e8 bd 5c 30  |.\0=...u..y...\0|
00000470  7d 08 ff 8d fe 19 5c 30  41 8d 5c 30 48 8d 85 08  |}.....\0A.\0H...|
00000480  0f ff 06 74 85 fb 83                              |...t...|
00000487
AMIGA:barrywalker~>

# 11  
Old 08-28-2013
Code:
    e8c28d4dc08d47000a19c075d84174
became
    00000000  e8 c2 8d 4d c0 8d 47 5c  30 0a 19 c0 75 d8 41 74  |...M..G\0...u.At|

Did you want to make the null more text friendly?

It looks like xxd was doing the lifting, not bash, with no reference to $REPLY (since you did 'while read' with no variable). Can you read, recognize and write a null in bash alone?
# 12  
Old 08-28-2013
Hi DGPickett...
Quote:
Did you want to make the null more text friendly?
Yes and no. It is the only way I can get byte value zero into a string variable using only this default OSX install, as bash is not byte value zero friendly...

As for the xxd command, try and remove the file pointer after "done" and see what happens.
Try replacing it with say /dev/null too. I have absolutely no idea why /tmp/binary.file IS needed, but it is...

If you enter:-
Code:
echo -e -n "$text" > /tmp/somefilename

It will end up the same as /tmp/binary.file...

I have just had another idea.
Let me experiment and see if this idea works...

I will explain after some experimention what the idea is/was, whether successful or not, after I have tried it...
# 13  
Old 08-30-2013
There is a wealth of possibilities, between the readline options, stty raw, and IFS, but so many tools do not deal with null, it's hard to even test. If you bash "read =n 1" a null into a variable, is it hard to tell the variable is the empty string '' or ""?

To write it out, one might use the printf character option, treating it as a special case and providing the null as a zero char (int) value. Once you get to treating it as a special case, you can write it with something like "echo '\0\c' ".

$IFS characters are another challenge, but if you use "read -n 1" you could do your own $IFS processing if any, setting $IFS to '' or unsetting it.
# 14  
Old 08-30-2013
Getting byte zero, in pseudo_zero mode, into the shell is easy as the Arduino Voltmeter DEMO below shows.

However I have attempted to put byte zero into a string variable as a single character and I now concede defeat. It is not possible with default commands to force value of byte zero into any variable, so a minimum of "\0", 2 bytes, is required to _emulate_ it...

Code:
#!/bin/bash --posix
# $VER: Arduino.sh_Version_0.00.10_(C)2013_B.Walker_G0LCU.

Arduino_Device="Unknown!"
DATA="0"

> /tmp/usbdata.raw

echo "Remove the Arduino board from the USB port ,IF, it is connected."
read -p "Press <CR> to continue:- " -e Arduino_Device
echo "The error report IS expected!"
ls /dev/cu.usb*
echo "Now connect the Arduino board to the USB port."
sleep 5
read -p "Press <CR> to continue:- " -e Arduino_Device

Arduino_Device="`ls /dev/cu.usb*`"
clear
while true
do
	if [ ${#DATA} -le 0 ]
	then
		echo "ERROR! Unsuccessful data aquisition..."
		break
	fi
	DATA=$[ ( $DATA * 20 ) ]
	printf "\\x1B[11;33f0.000 Volts DC.\n\n"
	if [ ${#DATA} -eq 2 ]
	then
		printf "\\x1B[11;36f$DATA\n\n"
	fi
	if [ ${#DATA} -eq 3 ]
	then
		printf "\\x1B[11;35f$DATA\n\n"
	fi
	if [ ${#DATA} -eq 4 ]
	then
		DATA="${DATA:0:1}.${DATA:1:2}"
		printf "\\x1B[11;33f$DATA\n\n"
	fi
	dd if="$Arduino_Device" of=/tmp/usbdata.raw bs=1 count=1
	DATA=`hexdump -n1 -s0 -v -e '1/1 "%u"' /tmp/usbdata.raw`
done

Display like this:-
Code:
Last login: Fri Aug 30 21:34:09 on ttys000
AMIGA:barrywalker~> ./Arduino.sh
Remove the Arduino board from the USB port ,IF, it is connected.
Press <CR> to continue:- 
The error report IS expected!
ls: /dev/cu.usb*: No such file or directory
Now connect the Arduino board to the USB port.
Press <CR> to continue:- 

                                0.000 Volts DC.

1+0 records in
1+0 records out
1 bytes transferred in 0.007705 secs (130 bytes/sec)c)

# AND...

                                2.580 Volts DC.

1+0 records in
1+0 records out
1 bytes transferred in 0.007620 secs (131 bytes/sec)))

# AND...

                                5.100 Volts DC.

1+0 records in
1+0 records out
1 bytes transferred in 0.007678 secs (130 bytes/sec)))

Login or Register to Ask a Question

Previous Thread | Next Thread

4 More Discussions You Might Find Interesting

1. Forum Support Area for Unregistered Users & Account Problems

Not able to post thread/reply to thread

Dear Moderator I am not able to post any new thread or post reply to mine old thread. Kindly help as i am stuck on one problem and needed suggestion. Regards Jaydeep (1 Reply)
Discussion started by: jaydeep_sadaria
1 Replies

2. Shell Programming and Scripting

Another Building Block, Binary File Manipulation...

Apologies for any typos, and IF this has been done before... This is yet another building block. The code generates a 256 byte binary file of _characters_ 0x00 to 0xFF for general usage and generates another binary file manipulated in a basic way. I need this facility for a kids project I am... (0 Replies)
Discussion started by: wisecracker
0 Replies

3. Shell Programming and Scripting

Convert binary file to csv and then back to the binary format

Hello *nix specialists, Im working for a non profit organisation in Germany to transport DSL over WLAN to people in areas without no DSL. We are using Linksys WRT 54 router with DD-WRT firmware There are at the moment over 180 router running but we have to change some settings next time. So my... (7 Replies)
Discussion started by: digidax
7 Replies

4. Programming

How to cancel a thread safely from the initial thread?

how about asynchronous canceling? or with signal? if with signal whether it effects the process? my english so badly :( :( (1 Reply)
Discussion started by: alan.zhao
1 Replies
Login or Register to Ask a Question