I have been doing binary experiments yet again and came across a superb piece of code...
I extracted a very small piece and re-wrote to suit my needs:-
Code:
#!/bin/bash --posix
# bash-hexdump
# Open the file $1 to be read with an fd 3.
exec 3<"$1"
saveIFS="$IFS"
IFS=""
char="00"
val="FF"
position=0
while read -s -u 3 -d '' -r -n 1 char
do
# """If the leading character is a single-quote or double-quote,
# the value shall be the numeric value in the underlying codeset
# of the character following the single-quote or double-quote."""
printf -v val "%02X" "'$char"
if [ ${#val} -gt 2 ]
then
position=$[ ( ${#val} - 2 ) ]
val="${val:$position:2}"
fi
echo -n " $val "
done
echo ""
IFS="$saveIFS"
# Finally ensure fd 3 is closed.
exec 3<&-
Results:-
Code:
Last login: Mon Oct 14 11:27:22 on ttys000
AMIGA:barrywalker~> ./bin_import.sh BinaryFile.dat
00 01 02 03 04 05 06 07 08 09 0A 0B 0C 0D 0E 0F 10 11 12 13 14 15 16
17 18 19 1A 1B 1C 1D 1E 1F 20 21 22 23 24 25 26 27 28 29 2A 2B 2C 2D
2E 2F 30 31 32 33 34 35 36 37 38 39 3A 3B 3C 3D 3E 3F 40 41 42 43 44
45 46 47 48 49 4A 4B 4C 4D 4E 4F 50 51 52 53 54 55 56 57 58 59 5A 5B
5C 5D 5E 5F 60 61 62 63 64 65 66 67 68 69 6A 6B 6C 6D 6E 6F 70 71 72
73 74 75 76 77 78 79 7A 7B 7C 7D 7E 7F 80 81 82 83 84 85 86 87 88 89
8A 8B 8C 8D 8E 8F 90 91 92 93 94 95 96 97 98 99 9A 9B 9C 9D 9E 9F A0
A1 A2 A3 A4 A5 A6 A7 A8 A9 AA AB AC AD AE AF B0 B1 B2 B3 B4 B5 B6 B7
B8 B9 BA BB BC BD BE BF C0 C1 C2 C3 C4 C5 C6 C7 C8 C9 CA CB CC CD CE
CF D0 D1 D2 D3 D4 D5 D6 D7 D8 D9 DA DB DC DD DE DF E0 E1 E2 E3 E4 E5
E6 E7 E8 E9 EA EB EC ED EE EF F0 F1 F2 F3 F4 F5 F6 F7 F8 F9 FA FB FC
FD FE FF
AMIGA:barrywalker~>
NOTE: Binary 0, (zero), has been accepted...
QUESTION:
I understand how "exec" is creating a file descriptor from "$1" but how does this affect "read"'s ability to accept binary 0, (zero), for manipulation by "printf"...
This is just a wild guess: i suppose read itself doesn't have anything to do with it. Probably the shell itself "cooks away" binary zeroes as it digests input and read only gets what was put through the shell. As the shell never gets to see the file (at least not as "input", that is, from stdin) read is served the whole story and not only parts of it.
I hope this helps.
bakunin
Moderator's Comments:
ahh, forgot: i move the thread over to the Experts forum. This is clearly not "Dummy" stuff but pretty advanced.
The -n 1 makes read only one character instead of the line (up to a \n character).
Further the -r tells to not discard leading space characters (that only fits for line mode).
IMHO the stuff can be simplified to
Code:
#!/bin/bash --posix
# bash-hexdump
char="00"
val="FF"
position=0
while IFS="" read -r -n 1 char
do
# """If the leading character is a single-quote or double-quote,
# the value shall be the numeric value in the underlying codeset
# of the character following the single-quote or double-quote."""
printf -v val "%02X" "'$char"
if [ ${#val} -gt 2 ]
then
position=$[ ( ${#val} - 2 ) ]
val="${val:$position:2}"
fi
echo -n " $val "
done <"$1"
echo ""
Surprisingly seems a little different!?
This User Gave Thanks to MadeInGermany For This Post:
QUESTION:
I understand how "exec" is creating a file descriptor from "$1" but how does this affect "read"'s ability to accept binary 0, (zero), for manipulation by "printf"...
exec has no effect whatsoever on the handling of the nullbyte.
Quote:
Originally Posted by wisecracker
NOTE: Binary 0, (zero), has been accepted...
No, it hasn't. That code is simply the beneficiary of serendipitous default behavior. The null byte is not in the variable, but printf fills any vacuums with zeroes/nullstrings, which in this case gives the correct result.
Arguments to non-string format specifiers are treated as C language constants, except that a leading plus or minus sign is allowed, and if the leading character is a single or double quote, the value is the ASCII value of the following character.
The format is reused as necessary to consume all of the arguments. If the format requires more arguments than are supplied, the extra format specifications behave as if a zero value or null string, as appropriate, had been supplied. The return value is zero on success, non-zero on failure.
In short, it's a lucky break (but I don't think it's an unsafe dependence).
The code that you posted will not work under all circumstances. read works with characters but in a hexdump, bytes are what matter. If the locale specifies a multibyte encoding, the results will be incorrect.
I observed such breakage on one of my machines with the following environment:
The preceding result, with its string of incorrect zeroes, occurred on a Windows machine using Cygwin. In a Linux (Ubuntu) virtual machine, bytes 128-255 were also mangled, but instead of double zeroes the output consisted of 16-digit hexadecimal numbers.
Multibyte character issues are also a concern when generating the binary data with a locale-aware awk implementation (in this case, gawk):
OK a few tests, OSX 10.7.5, default bash terminal:-
The first block is MadeInGermany's, the last three characters should be newlines not NULLs.
Second is alister's and only allows KB input much akin to INKEY$ on this tool.
Third is mine unchanged from the original post, last three characters correct.
Lastly a real hexdump command showing the last three characters.
Alister, the original code I found did include LANG=C but I removed it. Thanks for the info...
The test binary code is attached as a "txt" file...
Code:
Last login: Mon Oct 14 18:32:36 on ttys000
AMIGA:barrywalker~> ./hex_dump3.sh /tmp/bin.dat
20 00 20 09 00 20 00 00 01 02 03 04 05 06 07 08 09 00 0B 0C
0D 0E 0F 10 11 12 13 14 15 16 17 18 19 1A 1B 1C 1D 1E 1F 20
21 22 23 24 25 26 27 28 29 2A 2B 2C 2D 2E 2F 30 31 32 33 34
35 36 37 38 39 3A 3B 3C 3D 3E 3F 40 41 42 43 44 45 46 47 48
49 4A 4B 4C 4D 4E 4F 50 51 52 53 54 55 56 57 58 59 5A 5B 5C
5D 5E 5F 60 61 62 63 64 65 66 67 68 69 6A 6B 6C 6D 6E 6F 70
71 72 73 74 75 76 77 78 79 7A 7B 7C 7D 7E 7F 80 81 82 83 84
85 86 87 88 89 8A 8B 8C 8D 8E 8F 90 91 92 93 94 95 96 97 98
99 9A 9B 9C 9D 9E 9F A0 A1 A2 A3 A4 A5 A6 A7 A8 A9 AA AB AC
AD AE AF B0 B1 B2 B3 B4 B5 B6 B7 B8 B9 BA BB BC BD BE BF C0
C1 C2 C3 C4 C5 C6 C7 C8 C9 CA CB CC CD CE CF D0 D1 D2 D3 D4
D5 D6 D7 D8 D9 DA DB DC DD DE DF E0 E1 E2 E3 E4 E5 E6 E7 E8
E9 EA EB EC ED EE EF F0 F1 F2 F3 F4 F5 F6 F7 F8 F9 FA FB FC
FD FE FF 00 00 00
AMIGA:barrywalker~> ./hex_dump4.sh /tmp/bin.dat
71
77
65
72
74
79
75
69
6F
70
AMIGA:barrywalker~> ./hex_dump5.sh /tmp/bin.dat
20 0A 20 09 0A 20 0A 00 01 02 03 04 05 06 07 08 09 0A 0B 0C
0D 0E 0F 10 11 12 13 14 15 16 17 18 19 1A 1B 1C 1D 1E 1F 20
21 22 23 24 25 26 27 28 29 2A 2B 2C 2D 2E 2F 30 31 32 33 34
35 36 37 38 39 3A 3B 3C 3D 3E 3F 40 41 42 43 44 45 46 47 48
49 4A 4B 4C 4D 4E 4F 50 51 52 53 54 55 56 57 58 59 5A 5B 5C
5D 5E 5F 60 61 62 63 64 65 66 67 68 69 6A 6B 6C 6D 6E 6F 70
71 72 73 74 75 76 77 78 79 7A 7B 7C 7D 7E 7F 80 81 82 83 84
85 86 87 88 89 8A 8B 8C 8D 8E 8F 90 91 92 93 94 95 96 97 98
99 9A 9B 9C 9D 9E 9F A0 A1 A2 A3 A4 A5 A6 A7 A8 A9 AA AB AC
AD AE AF B0 B1 B2 B3 B4 B5 B6 B7 B8 B9 BA BB BC BD BE BF C0
C1 C2 C3 C4 C5 C6 C7 C8 C9 CA CB CC CD CE CF D0 D1 D2 D3 D4
D5 D6 D7 D8 D9 DA DB DC DD DE DF E0 E1 E2 E3 E4 E5 E6 E7 E8
E9 EA EB EC ED EE EF F0 F1 F2 F3 F4 F5 F6 F7 F8 F9 FA FB FC
FD FE FF 0A 0A 0A
AMIGA:barrywalker~> hexdump -C /tmp/bin.dat
00000000 20 0a 20 09 0a 20 0a 00 01 02 03 04 05 06 07 08 | . .. ..........|
00000010 09 0a 0b 0c 0d 0e 0f 10 11 12 13 14 15 16 17 18 |................|
00000020 19 1a 1b 1c 1d 1e 1f 20 21 22 23 24 25 26 27 28 |....... !"#$%&'(|
00000030 29 2a 2b 2c 2d 2e 2f 30 31 32 33 34 35 36 37 38 |)*+,-./012345678|
00000040 39 3a 3b 3c 3d 3e 3f 40 41 42 43 44 45 46 47 48 |9:;<=>?@ABCDEFGH|
00000050 49 4a 4b 4c 4d 4e 4f 50 51 52 53 54 55 56 57 58 |IJKLMNOPQRSTUVWX|
00000060 59 5a 5b 5c 5d 5e 5f 60 61 62 63 64 65 66 67 68 |YZ[\]^_`abcdefgh|
00000070 69 6a 6b 6c 6d 6e 6f 70 71 72 73 74 75 76 77 78 |ijklmnopqrstuvwx|
00000080 79 7a 7b 7c 7d 7e 7f 80 81 82 83 84 85 86 87 88 |yz{|}~..........|
00000090 89 8a 8b 8c 8d 8e 8f 90 91 92 93 94 95 96 97 98 |................|
000000a0 99 9a 9b 9c 9d 9e 9f a0 a1 a2 a3 a4 a5 a6 a7 a8 |................|
000000b0 a9 aa ab ac ad ae af b0 b1 b2 b3 b4 b5 b6 b7 b8 |................|
000000c0 b9 ba bb bc bd be bf c0 c1 c2 c3 c4 c5 c6 c7 c8 |................|
000000d0 c9 ca cb cc cd ce cf d0 d1 d2 d3 d4 d5 d6 d7 d8 |................|
000000e0 d9 da db dc dd de df e0 e1 e2 e3 e4 e5 e6 e7 e8 |................|
000000f0 e9 ea eb ec ed ee ef f0 f1 f2 f3 f4 f5 f6 f7 f8 |................|
00000100 f9 fa fb fc fd fe ff 0a 0a 0a |..........|
0000010a
AMIGA:barrywalker~>
EDIT:
I forgot to add that I want to use the bash builtins only and not rely on transient commands like od,
xxd or hexdump to get my required results purely as an exercise to learn more about the limitations
of bash, (or any shell for that matter), scripting... You guys are a godsend to people like me and
certainly have put me on the straight and narrow...
Hi 2 all,
i have had AIX 7.2
:/# /usr/IBMAHS/bin/apachectl -v
Server version: Apache/2.4.12 (Unix)
Server built: May 25 2015 04:58:27
:/#:/# /usr/IBMAHS/bin/apachectl -M
Loaded Modules:
core_module (static)
so_module (static)
http_module (static)
mpm_worker_module (static)
... (3 Replies)
Hello.
System : opensuse leap 42.3
I have a bash script that build a text file.
I would like the last command doing :
print_cmd -o page-left=43 -o page-right=22 -o page-top=28 -o page-bottom=43 -o font=LatinModernMono12:regular:9 some_file.txt
where :
print_cmd ::= some printing... (1 Reply)
How to use "mailx" command to do e-mail reading the input file containing email address, where column 1 has name and column 2 containing “To” e-mail address
and column 3 contains “cc” e-mail address to include with same email.
Sample input file, email.txt
Below is an sample code where... (2 Replies)
I want to print any matching IP addresse in List1 with List 2;
List 1
List of IP addresses;
161.85.58.210
250.57.15.129
217.23.162.249
74.76.129.101
30.221.177.237
3.147.200.59
170.58.142.64
127.65.109.33
150.167.242.146
223.3.20.186
25.181.180.99
2.55.199.32 (3 Replies)
Hi,
I have line in input file as below:
3G_CENTRAL;INDONESIA_(M)_TELKOMSEL;SPECIAL_WORLD_GRP_7_FA_2_TELKOMSEL
My expected output for line in the file must be :
"1-Radon1-cMOC_deg"|"LDIndex"|"3G_CENTRAL|INDONESIA_(M)_TELKOMSEL"|LAST|"SPECIAL_WORLD_GRP_7_FA_2_TELKOMSEL"
Can someone... (7 Replies)
Hallo,
i need a Prompting read in my script:
read -p "Enter your command: " command
But i always get this Error:
-p: is not an identifier
When I run these in c-shell i get this error
/usr/bin/read: read: bad option(s)
How can I use a Prompt in the read command? (9 Replies)
Hi Friends,
Can any of you explain me about the below line of code?
mn_code=`env|grep "..mn"|awk -F"=" '{print $2}'`
Im not able to understand, what exactly it is doing :confused:
Any help would be useful for me.
Lokesha (4 Replies)
Hi
I have installed solaris 10 on an intel machine. Logged in as root. In CDE, i open terminal session, type login alex (normal user account) and password and i get this message
No utpmx entry: you must exec "login" from lowest level "shell" :confused:
What i want is: open various... (0 Replies)
Dears,
I've written a script which allows me to send mails in different formats with different attaches. Now I still want to add a feature to this script. My users would like to be able to receive a "read" or "delivered" receipt for their mails.
The script send mails on behalve of an specific... (1 Reply)
10-14-2013
