Sponsored Content
Top Forums UNIX for Dummies Questions & Answers Split binary file every occurrence of a group of characters Post 302787515 by PatrickE on Friday 29th of March 2013 06:58:31 PM
Old 03-29-2013
I realized that example is mostly in order where they are mixed more at some point.
Here let me try this.

File.mpg
00 00 01 BA 44 00 04 00 04 01 01 89 C3 F8 00 00 01 BB 00 12 80 C4 E1 00 E1 7F B9 E0 E8 B8 C0 20 BD E0 3A BF E0 02 00 00 01 BF 03 D4 00 00 00 00 00 00 00 00 00 01 E0 40 00 00 00 47 40 00 00 D9 ~ 39KB of random data ~ 00 00 01 BA 44 00 04 56 DD 0D 01 89 C3 F8 00 00 01 BB 00 12 80 C4 E1 00 E1 7F B9 E0 E8 B8 C0 20 BD E0 3A BF E0 02 00 00 01 BF 03 D4 00 00 00 00 13 00 00 00 00 01 E0 40 00 00 00 D9 E1 00 01 89 ~ 43KB of random data ~ 00 00 01 BA 44 BF EF 12 1C B1 01 89 C3 F8 00 00 01 BB 00 12 80 C4 E1 00 E1 7F B9 E0 E8 B8 C0 20 BD E0 3A BF E0 02 00 00 01 BF 03 D4 00 00 0D 39 C8 00 00 00 00 01 C0 40 00 0B FF 81 EB 0C 00 89 ~ 584KB of random data ~ 00 00 01 BA 44 BF FF D9 CC 05 01 89 C3 F8 00 00 01 BB 00 12 80 C4 E1 00 E1 7F B9 E0 E8 B8 C0 20 BD E0 3A BF E0 02 00 00 01 BF 03 D4 00 00 0D 3A E5 00 00 00 00 01 C0 40 00 0C 00 89 DA 0C 01 91 ~ 580KB of random data ~ 00 00 01 BA 46 97 97 06 26 1D 00 AF CB F8 00 00 01 BB 00 12 80 57 E5 04 E1 7F B9 E0 E8 B8 C0 20 BD E0 3A BF E0 02 00 00 01 E0 07 D4 80 C1 0D 31 A5 E9 E9 FB 11 A5 E9 A3 99 1E 60 E8 00 00 01 B3 ~ 205KB of random data ~ 00 00 01 BA 44 00 0F B4 7E 39 02 3B 4B F8 00 00 01 BB 00 12 81 1D A5 06 E1 7F B9 E0 E8 B8 C0 20 BD E0 3A BF E0 02 00 00 01 E0 07 D4 80 C1 0D 31 00 05 79 79 11 00 05 33 17 1E 60 E8 00 00 01 B3 ~ 791KB of random data ~

Split on content

01e0-0e17-440004000401.mpg
00 00 01 BA 44 00 04 00 04 01 01 89 C3 F8 00 00 01 BB 00 12 80 C4 E1 00 E1 7F B9 E0 E8 B8 C0 20 BD E0 3A BF E0 02 00 00 01 BF 03 D4 00 00 00 00 00 00 00 00 00 01 E0 40 00 00 00 47 40 00 00 D9 ~ 39KB of random data ~
~ Split ~
01e0-0e17-44000456DD0D.mpg
00 00 01 BA 44 00 04 56 DD 0D 01 89 C3 F8 00 00 01 BB 00 12 80 C4 E1 00 E1 7F B9 E0 E8 B8 C0 20 BD E0 3A BF E0 02 00 00 01 BF 03 D4 00 00 00 00 13 00 00 00 00 01 E0 40 00 00 00 D9 E1 00 01 89 ~ 43KB of random data ~
~ Split ~
01c0-0e17-44BFEF121CB1.mpg
00 00 01 BA 44 BF EF 12 1C B1 01 89 C3 F8 00 00 01 BB 00 12 80 C4 E1 00 E1 7F B9 E0 E8 B8 C0 20 BD E0 3A BF E0 02 00 00 01 BF 03 D4 00 00 0D 39 C8 00 00 00 00 01 C0 40 00 0B FF 81 EB 0C 00 89 ~ 584KB of random data ~
~ Split ~
01c0-0e17-44BFFFD9CC05.mpg
00 00 01 BA 44 BF FF D9 CC 05 01 89 C3 F8 00 00 01 BB 00 12 80 C4 E1 00 E1 7F B9 E0 E8 B8 C0 20 BD E0 3A BF E0 02 00 00 01 BF 03 D4 00 00 0D 3A E5 00 00 00 00 01 C0 40 00 0C 00 89 DA 0C 01 91 ~ 580KB of random data ~
~ Split ~
8057-4e17-46979706261D.mpg
00 00 01 BA 46 97 97 06 26 1D 00 AF CB F8 00 00 01 BB 00 12 80 57 E5 04 E1 7F B9 E0 E8 B8 C0 20 BD E0 3A BF E0 02 00 00 01 E0 07 D4 80 C1 0D 31 A5 E9 E9 FB 11 A5 E9 A3 99 1E 60 E8 00 00 01 B3 ~ 205KB of random data ~
~ Split ~
811d-6e17-44000FB47E39.mpg
00 00 01 BA 44 00 0F B4 7E 39 02 3B 4B F8 00 00 01 BB 00 12 81 1D A5 06 E1 7F B9 E0 E8 B8 C0 20 BD E0 3A BF E0 02 00 00 01 E0 07 D4 80 C1 0D 31 00 05 79 79 11 00 05 33 17 1E 60 E8 00 00 01 B3 ~ 791KB of random data ~
~ Split ~


After that I combine them and get.
01e0-0e17.mpg
01c0-0e17.mpg
8057-4e17.mpg
811d-6e17.mpg

---------- Post updated at 05:56 PM ---------- Previous update was at 05:56 PM ----------

Correct.

---------- Post updated at 05:58 PM ---------- Previous update was at 05:56 PM ----------

Yes that is correct. ahamed
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Split a binary file into 2 basing on 2 delemiter string

Hi all, I have a binary file (orig.dat) and two special delimiter strings 'AAA' and 'BBB'. My binary file's content is as follow: <Data1.1>AAA<Data1.2>BBB <Data2.1>AAA<Data2.2>BBB ... <DataN.1>AAA<DataN.2>BBB DataX.Y might have any length, and contains any kind of special/printable... (1 Reply)
Discussion started by: Averell
1 Replies

2. Shell Programming and Scripting

Split these into many ...(/etc/group)!!

Guys Following input line is from /etc/group file.As we know last entry in a line of /etc/group is userlist (all the users belonging to that group). I need to splilt this one line into 3 lines as shown below (3 because userlist has 3 names in it). Input: lp:!:11:root,lp,printq ... (13 Replies)
Discussion started by: ak835
13 Replies

3. Shell Programming and Scripting

Split file by data group

Hi all, I'm having a little trouble solving a file split I need to get done. I have the following data: 1. Light 1A. Light Soft texture: it's soft color: the color value is that of something light vital statistics: srm: 23 og: 1.035 sp: 1.065 comment: this is nice if you like... (8 Replies)
Discussion started by: mkastin
8 Replies

4. Shell Programming and Scripting

Split binary file with pattern

Hello! Have some problem with extract files from saved session. File contains any kind of special/printable characters. DATA NumberA DATA DATA Begin DATA1.1 DATA1.2 NumberB1 DATA1.3 DATA1.4 End DATA DATA DATA Begin DATA2.1 DATA2.2 NumberB2 DATA2.3 DATA2.4 End DATA DATA ... (4 Replies)
Discussion started by: vvild
4 Replies

5. Shell Programming and Scripting

remove last characters after %EOF (pdf binary file)

Hi, I want to know how I can remove the last characters of ANY pdf file. I read it under "od" in the command shell to see which were the last characters: $od corruptedfile.pdf -c When I see the file, I need to keep only the last characters, or "end of the file": %EOF (obviously keeping all... (1 Reply)
Discussion started by: diegugawa
1 Replies

6. Shell Programming and Scripting

split file based on group count

Hi, can some one please help me to split the file based on groups. like in the below scenario x indicates the begining of the group and the file should be split each with 2 groups below there are 10 groups it should create 5 files. could you please help? (4 Replies)
Discussion started by: hitmansilentass
4 Replies

7. Shell Programming and Scripting

Deleting all characters before the last occurrence of /

Hi All, I have a text file with the following text in it: file:///About/accessibility.html file:///About/disclaimer.html file:///About/disclaimer.html#disclaimer file:///pubmed?term=%22Dacre%20I%22%5BAuthor%5D file:///pubmed?term=%22Madigan%20J%22%5BAuthor%5D... (8 Replies)
Discussion started by: shoaibjameel123
8 Replies

8. Shell Programming and Scripting

split a string and convert to binary

Hi All, Iam new to unix scripting and I want a split a string into 4 characters each, leaving the last two characters and convert the splitted values into binary. For example: string='ffd80012ffe20000ffebfffeffea0007fff0ffd70014fff1fff0fff0fff201' this should split as ffd8 0012 ffe2 . .... (5 Replies)
Discussion started by: srinivasayedla
5 Replies

9. UNIX for Dummies Questions & Answers

counting occurrence of characters in a string

Hello, I have a string like this 0:1:2:0:2:2:4:0:0:0:-200:500...... what i want is to break down how many different characters are there and their count. For example for above string it should display 0 - 5 times 1 - 1 times 2 - 3 times 4 - 1 times . . . I am stuck in writing... (8 Replies)
Discussion started by: exit86
8 Replies

10. Shell Programming and Scripting

Split a big file into multiple files based on first four characters

I have a requirement to split a huge file to smaller text files based on first four characters which look like ABCD 1234 DFGH RREX : : : : : 0000 Each of these records are OF EQUAL bytes with a different internal layout based on the above first digit identifier.. Any help to start... (5 Replies)
Discussion started by: etldev
5 Replies
par2(1) 						      General Commands Manual							   par2(1)

NAME
par2 - PAR 2.0 compatible file verification and repair tool. SYNOPSIS
par2 c(reate) [options] <par2 file> [files] par2 v(erify) [options] <par2 file> [files] par2 r(epair) [options] <par2 file> [files] Or: par2create [options] <par2 file> [files] par2verify [options] <par2 file> [files] par2repair [options] <par2 file> [files] DESCRIPTION
par2 is a program for creating and using PAR2 files to detect damage in data files and repair them if necessary. It can be used with any kind of file. OPTIONS
-b<n> Set the Block-Count -s<n> Set the Block-Size (Don't use both -b and -s) -r<n> Level of Redundancy (%) -c<n> Recovery block count (don't use both -r and -c) -f<n> First Recovery-Block-Number -u Uniform recovery file sizes -l Limit size of recovery files (Don't use both -u and -l) -n<n> Number of recovery files (Don't use both -n and -l) -m<n> Memory (in MB) to use -v [-v] Be more verbose -q [-q] Be more quiet (-qq gives silence) -- Treat all remaining CommandLine as filenames EXAMPLES
With PAR 2.0 you can create PAR2 recovery files for as few as 1 or as many as 32768 files. If you wanted to create PAR1 recovery files for a single file you are forced to split the file into muliple parts and RAR is frequently used for this purpose. You do NOT need to split files with PAR 2.0. To create PAR 2 recovery files for a single data file (e.g. one called test.mpg), you can use the following command: par2 create test.mpg If test.mpg is an 800 MB file, then this will create a total of 8 PAR2 files with the following filenames (taking roughly 6 minutes on a PC with a 1500MHz CPU): test.mpg.par2 - This is an index file for verification only test.mpg.vol00+01.par2 - Recovery file with 1 recovery block test.mpg.vol01+02.par2 - Recovery file with 2 recovery blocks test.mpg.vol03+04.par2 - Recovery file with 4 recovery blocks test.mpg.vol07+08.par2 - Recovery file with 8 recovery blocks test.mpg.vol15+16.par2 - Recovery file with 16 recovery blocks test.mpg.vol31+32.par2 - Recovery file with 32 recovery blocks test.mpg.vol63+37.par2 - Recovery file with 37 recovery blocks The test.mpg.par2 file is 39 KB in size and the other files vary in size from 443 KB to 15 MB. These par2 files will enable the recovery of up to 100 errors totalling 40 MB of lost or damaged data from the original test.mpg file when it and the par2 files are posted on UseNet. When posting on UseNet it is recommended that you use the "-s" option to set a blocksize that is equal to the Article size that you will use to post the data file. If you wanted to post the test.mpg file using an article size of 300 KB then the command you would type is: par2 create -s307200 test.mpg This will create 9 PAR2 files instead of 8, and they will be capable of correcting up to 134 errors totalling 40 MB. It will take roughly 8 minutes to create the recovery files this time. In both of these two examples, the total quantity of recovery data created was 40 MB (which is 5% of 800 MB). If you wish to create a greater or lesser quantity of recovery data, you can use the "-r" option. To create 10% recovery data instead of the default of 5% and also to use a block size of 300 KB, you would use the following command: par2 create -s307200 -r10 test.mpg This would also create 9 PAR2 files, but they would be able to correct up to 269 errors totalling 80 MB. Since twice as much recovery data is created, it will take about 16 minutes to do so with a 1500MHz CPU. The "-u" and "-n" options can be used to control exactly how many recovery files are created and how the recovery blocks are distributed amoungst them. They do not affect the total quantity of recovery data created. The "-f" option is used when you create additional recovery data. e.g. If you have already created 10% and want another 5% then you migh use the following command: par2 create -s307200 -r5 -f300 test.mpg This specifies the same block size (which is a requirement for additional recovery files), 5% recovery data, and a first block number of 300. The "-m" option controls how much memory par2 uses. It defaults to 16 MB unless you override it. CREATING PAR2 FILES FOR MULTIPLE DATA FILES When creating PAR2 recovery files form multiple data files, you must specify the base filename to use for the par2 files and the names of all of the data files. If test.mpg had been split into multiple RAR files, then you could use: par2 create test.mpg.rar.par2 test.mpg.part*.rar The files filename "test.mpg.rar.par2" says what you want the par2 files to be called and "test.mpg.part*.rar" should select all of the RAR files. VERIFYING AND REPAIRING When using par2 recovery files to verify or repair the data files from which they were created, you only need to specify the filename of one of the par2 files to par2. e.g.: par2 verify test.mpg.par2 This tells par2 to use the information in test.mpg.par2 to verify the data files. par2 will automatically search for the other par2 files that were created and use the information they contain to determine the filenames of the original data files and then to verify them. If all of the data files are ok, then par2 will report that repair will not be required. If any of the data files are missing or damaged, par2 will report the details of what it has found. If the recovery files contain enough recovery blocks to repair the damage, you will be told that repair is possible. Otherwise you will be told exactly how many recovery blocks will be required in order to repair. To carry out a repair use the following command: par2 repair test.mpg.par2 This tells par2 to verify and if possible repair any damaged or missing files. If a repair is carried out, then each file which is repaired will be re-verified to confirm that the repair was successful. MISSNAMED AND INCOMPLETE DATA FILES If any of the recovery files or data files have the wrong filename, then par2 will not automatically find and scan them. To have par2 scan such files, you must include them on the command line when attempting to verify or repair. e.g.: par2 r test.mpg.par2 other.mpg This tells par2 to scan the file called other.mpg to see if it contains any data belonging to the original data files. If one of the extra files specified in this way is an exact match for a data file, then the repair process will rename the file so that it has the correct filename. Because par2 is designed to be able to find good data within a damaged file, it can do the same with incomplete files downloaded from UseNet. If some of the articles for a file are missing, you should still download the file and save it to disk for par2 to scan. If you do this then you may find that you can carry out a repair in a situation where you would not otherwise have sufficient recovery data. You can have par2 scan all files that are in the current directory using a command such as: par2 r test.mpg.par2 * WHAT TO DO WHEN YOU ARE TOLD YOU NEED MORE RECOVERY BLOCKS If par2 determines that any of the data files are damaged or missing and finds that there is insufficient recovery data to effect a repair, you will be told that you need a certain number of recovery blocks. You can obtain these by downloading additional recovery files. In order to make things easy, par2 files have filenames that tell you exactly how many recovery blocks each one contains. Assuming that the following command was used to create recovery data: par2 c -b1000 -r5 test.mpg Then the recovery files that are created would be called: test.mpg.par2 test.mpg.vol00+01.par2 test.mpg.vol01+02.par2 test.mpg.vol03+04.par2 test.mpg.vol07+08.par2 test.mpg.vol15+16.par2 test.mpg.vol31+19.par2 The first file in this list does not contain any recovery data, it only contains information sufficient to verify the data files. Each of the other files contains a different number of recovery blocks. The number after the '+' sign is the number of recovery blocks and the number preceding the '+' sign is the block number of the first recovery block in that file. If par2 told you that you needed 10 recovery blocks, then you would need "test.mpg.vol01+02.par2" and "test.mpg.vol07+08.par". You might of course choose to fetch "test.mpg.vol15+16.par2" instead (in which case you would have an extra 6 recovery blocks which would not be used for the repair). NOTES
This version of par2 does not support recording path information for files. Whilst you can create recovery files for files from multiple locations, it will expect all files to be in the current directory when verifying and repairing. This limitation will be corrected in an update. REED SOLOMON CODING
PAR2 uses Reed Solomon Coding to perform its calculations. For details of this coding technique try the following link: ``A Tutorial on Reed-Solomon Coding for Fault-Tolerance in RAID-like Systems'' <http://www.cs.utk.edu/~plank/plank/papers/CS-96-332.html> AUTHOR
par2 was written by Peter Brian Clements <peterbclements@users.sourceforge.net>. This man page was contributed by Andres Salomon <dilinger@voxel.net> for the Debian GNU/Linux system (but may be used by others). May 2004 par2(1)
All times are GMT -4. The time now is 11:58 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy