How to parse a binary file?


 
Thread Tools Search this Thread
Top Forums UNIX for Advanced & Expert Users How to parse a binary file?
# 8  
Old 05-20-2013
Hello wisecracker,

Sorry, the below is a correct sample. I was doing some tests with the first sample and I changed bad the hex digits when I pasted here.

I read you code in the link you shared me. I see the code converts from hex each byte to decimal value to get the ascii char. But in
my file, some byte sequences BCD coded and should be interpreted as literal decimal values. For example the values highlighted in red
are the numbers 874401013008290 and 3742050001. Others are hexadecimal bytes sequences.

So, regarding the hexdump line below, how can I modifiy the hexdump command to print decimal values without convert it to hex value?,
I mean, when those sequences must be treated as decimal values literally.
Code:
number=`hexdump -n1 -s$subscript -v -e '1/1 "%u"' SomeBinaryFile.dat`

If there is way to put in a template the rules defined in description of the file in how must be treated the sequences of bytes, I'll be able
to parse the binary file.

Thanks for any help.

Regards.

Binary file
Code:
00000000  31 41 42 43 5f 4a 50 51  5f 50 54 30 38 30 33 30  |1ABC_JPQ_PT08030|
00000010  39 32 36 30 30 33 31 07  ff ff ff ff ff ff ff ff  |9260031.........|
00000020  32 00 00 01 87 44 01 01  30 08 29 0f 37 42 05 00  |2....D..0.).7B..|
00000030  01 ff ff ff 00 15 00 0a  48 00 01 33 00 01 36 00  |........H..3..6.|
00000040  01 37 00 01 66 00 01 65  00 01 77 00 01 78 00 01  |.7..f..e..w..x..|
00000050  69 00 01 79 00 04 93 00  01 22 00 00 21 00 02 09  |i..y....."..!...|
00000060  00 01 26 00 01 0f 00 01  11 00 01 13 00 01 08 00  |..&.............|
00000070  01 2b 00 00 2c 00 00 2d  00 00 2e 00 00 55 00 01  |.+..,..-.....U..|
00000080  56 00 07 2a 00 00 2f 00  00 30 00 00 31 00 00 ff  |V..*../..0..1...|
00000090  34 00 80 19 32 c9 06 00  00 08 88 a0 e0 ca 0e 54  |4...2..........T|
000000a0  00 91 0b 37 42 39 90 10  8f ff ff 00 14 80 09 35  |...7B9.........5|
000000b0  c9 06 00 00 08 88 00 00  03 81 0f 01 02 00 00 00  |................|
000000c0  01 37 42 99 01 05 ff ff  ff 00 83 10 01 0c 00 00  |.7B.............|
000000d0  00 01 37 42 99 01 05 ff  ff ff 00 01 86 1d 02 0c  |..7B............|
000000e0  00 00 00 04 37 42 99 01  09 ff ff ff 00 0e 00 00  |....7B..........|
000000f0  00 04 37 42 99 01 09 ff  ff ff 00 87 0f 01 01 00  |..7B............|
00000100  00 00 03 37 42 99 01 08  ff ff ff 00 84 0e 00 00  |...7B...........|
00000110  00 00 00 00 01 01 ff ff  00 00 00 00 85 06 00 00  |................|
00000120  00 00 00 00 ff 32 00 00  02 87 44 01 01 30 08 29  |.....2....D..0.)|
00000130  2f 37 42 05 00 03 ff ff  ff 00 15 00 0a 48 00 01  |/7B..........H..|
00000140  33 00 01 36 00 01 37 00  01 66 00 01 65 00 01 77  |3..6..7..f..e..w|
00000150  00 01 78 00 01 69 00 01  79 00 04 93 00 01 22 00  |..x..i..y.....".|
00000160  00 21 00 02 09 00 01 26  00 01 0f 00 01 11 00 01  |.!.....&........|
00000170  13 00 01 08 00 01 2b 00  00 2c 00 00 2d 00 00 2e  |......+..,..-...|
00000180  00 00 55 00 01 56 00 07  2a 00 00 2f 00 00 30 00  |..U..V..*../..0.|
00000190  00 31 00 00 ff 34 00 80  19 32 c9 06 00 00 08 88  |.1...4...2......|
000001a0  a0 e0 ca 0e 54 00 91 0b  37 42 39 90 10 8f ff ff  |....T...7B9.....|
000001b0  00 14 80 09 35 c9 06 00  00 08 88 00 00 03 81 0f  |....5...........|
000001c0  01 02 00 00 00 01 37 42  99 01 05 ff ff ff 00 83  |......7B........|
000001d0  10 01 0c 00 00 00 01 37  42 99 01 05 ff ff ff 00  |.......7B.......|
000001e0  01 86 1d 02 0c 00 00 00  04 37 42 99 01 09 ff ff  |.........7B.....|
000001f0  ff 00 0e 00 00 00 04 37  42 99 01 09 ff ff ff 00  |.......7B.......|
00000200  87 0f 01 01 00 00 00 03  37 42 99 01 08 ff ff ff  |........7B......|
00000210  00 84 0e 00 00 00 00 00  00 01 01 ff ff 00 00 00  |................|

# 9  
Old 05-20-2013
Think of your problem!

Code:
hextext=`hexdump -n1 -s$subscript -v -e '1/1 "%02x"' SomeBinaryFile.dat`

This should give you the _hex_ byte at position "$subscript" as a 2 byte text string.
This should correspond with the "values" you see but auto-changed to ASCII.

Just use a loop and concatenate that which you need...
# 10  
Old 05-21-2013
Hello wisecracker,

Thank you for your help.

I've been trying your code. But I'm not sure what happens when using the hexdump line you shared me in last post.

I want to interpret sequence of bytes as decimals (literal decimals). I'm using to test the code below:
Code:
start_offset=36
jump=1
last_byte_limit=43

for subscript in $( seq $start_offset $jump $last_byte_limit )
do
    hextext=`hexdump -n1 -s$subscript -v -e '1/1 "%02x"' BinSample.dat`
    char=$char`printf "%d" '0x'$hextext`
done
printf $char

The decimal sequence I would like to extract from the Binary sample in my previous post (in red) is from byte 36 to 43. The desired
result is: 87 44 01 01 30 08 29 0f. Exactly how it looks in hexdump, but the code is giving me another output (13568114884115)
and I don't know why.

May you please correct me.

Thanks in advance.
# 11  
Old 05-21-2013
(How is it that I, a mere amateur, is showing a professional the way?)

Once again think of your problem!

Did you ACTUALLY read my other reply correctly?

Code:
start_offset=36
jump=1
last_byte_limit=43

subscript=0
hextext=""
char=""

for subscript in $( seq $start_offset $jump $last_byte_limit )
do
    hextext=`hexdump -n1 -s$subscript -v -e '1/1 "%02x"' BinSample.dat`
    # NOTE! A (white)space is added as this is YOUR requirement.
    char=$char" "$hextext
done
printf "$char"

I have not tested it but it should work...

EDIT:
Now tested on this Macbook Pro 22:42pm local UK time using the binary file generated in the original code:-
Code:
Last login: Tue May 21 18:19:33 on ttys000
Barrys-MacBook-Pro:~ barrywalker$ /Users/barrywalker/hextext.sh
 db da d9 d8 d7 d6 d5 d4Barrys-MacBook-Pro:~ barrywalker$ _

NOTE:- There is NO newline...

Last edited by wisecracker; 05-21-2013 at 06:46 PM.. Reason: Changed the comment in the code from "a" to "YOUR". Now tested...
This User Gave Thanks to wisecracker For This Post:
# 12  
Old 05-22-2013
Hello wisecracker,

Thank your for the help.
Quote:
Originally Posted by wisecracker
(How is it that I, a mere amateur, is showing a professional the way?)
Actually I'm not a proffesional in this area of programming or unix, in fact I'm a newbie in bash scripts. I only asking in the forum where experts could help.

Now my problem would be how to decode based on description, because sequences of bytes are interpreted sometimes as iso coded, sometimes as decimal values, sometimes as binary coded. I think here is where a template for parsing is needed but I don't know how to do it helping me with the hexdump command.

With hexdump, is possible convert an hex number like "A" into binary?
A=1010.

Thanks for help so far.

Regards
# 13  
Old 05-22-2013
I really have no idea how to parse NIBBLES directly but this as a byte transfer should be a starter and splitting into two halve should be very easy...

Search the forums for more info on adding leading zeros, lower to upper case, etc...
It's all on here...

Research the bc command for more info:-
Code:
#!/bin/bash
# NOTE! Uppercase required...
for bits in 00 FF 55 AA
do
        BITS=`echo "obase=2; ibase=16; $bits" | bc`
        printf "$BITS\n"
done

Test using this Macbook Pro...
Code:
Last login: Wed May 22 07:12:22 on ttys000
Barrys-MacBook-Pro:~ barrywalker$ ./hextobit.sh
0
11111111
1010101
10101010
Barrys-MacBook-Pro:~ barrywalker$


Last edited by wisecracker; 05-22-2013 at 04:23 AM..
# 14  
Old 05-23-2013
Hello again wisecracker,

Thank for your last sample script, It will help me in this issue as the other code you shared will be very useful for me.

Thanks again, I've learned several things from you in this thread.

Best regards.
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Convert binary file to csv and then back to the binary format

Hello *nix specialists, Im working for a non profit organisation in Germany to transport DSL over WLAN to people in areas without no DSL. We are using Linksys WRT 54 router with DD-WRT firmware There are at the moment over 180 router running but we have to change some settings next time. So my... (7 Replies)
Discussion started by: digidax
7 Replies

2. Shell Programming and Scripting

Python Binary File Read and Parse

Hi to everyone :), i have a challenge right now in python that for now needs a bit of help in one part of the c0de. The task is create a new file with the name of the file defined by the ASCII content between the 3 byte and the 16 byte that is parsed from the binary file, the file is over 20 Mb i... (0 Replies)
Discussion started by: drd0spt
0 Replies

3. UNIX for Dummies Questions & Answers

[AIX] Binary file warning for text file.

Hello guys, We had to move from a DC to another, and we are now facing an "issue" with some text files. Looks like that some of our log files are set as binary: file TuxConnectorURA.20121012 TuxConnectorURA.20121012: data or International Language text less TuxConnectorURA.20121012... (2 Replies)
Discussion started by: EnioMarques
2 Replies

4. Shell Programming and Scripting

Output redirection of c binary file to a file in shell script is failing

I am struck up with a problem and that is with output redirection. I used all the ways for the redirection of the output of c binary to a file, still it is failing. Here are the different ways which I have used: ./a.out | tee -a /root/tmp.txt 2>&1 ./a.out | tee -a /root/tmp.txt 1>&1 ./a.out |... (2 Replies)
Discussion started by: Maya29988
2 Replies

5. UNIX for Advanced & Expert Users

How to copy a binary file while the file is being written to by another process

Hello, Can I copy a binary file while the file is being written to by another process? Another process (program) “P1” creates and opens (for writing) binary file “ABC” on local disk. Process P1 continuously write into ABC file every couple of seconds, adding 512-byte blocks of data. ABC file... (1 Reply)
Discussion started by: mbuki
1 Replies

6. UNIX for Dummies Questions & Answers

Pipe binary file matches grep results to file

I am using grep to match a pattern, but the output is strange. $ grep -r -o "pattern" * Gives me: Binary file foo1 matches Binary file foo2 matches Binary file foo3 matches To find the lines before/after, I then have to use the following on each file: $ strings foo1 | grep -A1 -B1... (0 Replies)
Discussion started by: chipperuga
0 Replies

7. Shell Programming and Scripting

To log binary file output to a txt file

Hi, I wrote a small script whose function is to execute the postemsg provided if the threshold breaches. I want to log this postemsg messages to a log file. But I am not able to do. Can someone throw some light on how to log the output of this. I am pasting a snippet of that code. ... (2 Replies)
Discussion started by: dbashyam
2 Replies

8. UNIX for Dummies Questions & Answers

Binary file

How can we see the contents of a binary file? (1 Reply)
Discussion started by: siba.s.nayak
1 Replies

9. Shell Programming and Scripting

dd - binary file

I have a task that says: make a file (called binaryfile ) that contains 4 bytes of NULL data 6 bytes of random data 8 bytes of 1 10 bytes of 5 and 12 bytes of 9. For the first 2, I can used : dd if=/dev/null of=binaryfile bs=8 count =4 and dd if=/dev/urandom of=binaryfile bs=8... (3 Replies)
Discussion started by: spiriad
3 Replies

10. Solaris

compiled binary file gives "cannot execute binary file"

Hi, I have two Solaris machines. 1. SunOS X 5.8 Generic_108528-29 sun4u sparc SUNW,Sun-Blade-1500 2. SunOS Y 5.8 Generic_108528-13 sun4u sparc SUNW,Ultra-60 I am trying to buiild a project on both these machines. The Binary output file compiled on machine 2 runs on both the machines. Where... (0 Replies)
Discussion started by: scgupta
0 Replies
Login or Register to Ask a Question