Shell Programming and Scripting

I was wondering if anyone could explain to me how to split a variable length EBCDIC file into seperate files based on the record key. I have the COBOL layout, and so I need to split the file into 13 different EBCDIC files so that I can run each one through a C++ converter I have, and get the corresponding csv output file to put into a database. The records are:

Record Key Segment Name
01 GRROOT
02 GRCYCLE
. .
. .
. .
13 GRR3RMKS

If it helps at all, the PDF that comes with the EBCDIC file showing the COBOL layout states that the record length of the file is 422, the blocking factor is 77 and the blocksize is 32,494. There is additional information such as GRROOT length is 150 bytes, GRCYCLE is 72 bytes, etc.

Thanks for the help

There is an awful lot that is unspecified here:If you're trying to process EBCDIC files on an ASCII based system, the dd utility will probably be at the base of your processing. Look at the dd man page on your system and see if something like the following would be a good start to getting a file you can then split awk or grep:

Code:

dd if=YourEBCDICInputFileName of=YourASCIIOutputFileName ibs=422x77 cbs=422 conv=ascii,unblock,sync

and, after splitting YourASCIIOutputFileName into the files you want based on your keys, you could convert them back into fixed-length, blocked, EBCDIC files using dd again with obs=422x77 instead of ibs=422x77 and conv=ebcdic,block... and appropriate if= and of= parameters.

Thanks for the reply Don. I am doing this for a little side work project. Here are some of the specifics:

1) Any OS. I have a machine that runs ubuntu, and my work computer is windows 10. It sounds like ubunutu would be my preference here.
2) There is binary data that is being processed, packed decimal fields if that sounds right.
3) Reading through the cobol the record key is the first two bytes (1,2) of each record
4) The 422 and 77 bytes were numbers that appear in the front of the PDF, but then later it says that each record is of variable length, and gives me the length of each record. The total number of records would change each month, since this is a monthly dataset.

As I am typing this, it sounds like I would need to use the dd command and be able to change the number of bytes that is read each time based on what the record key is. So lets say I use dd and I want to read the first two bytes. If the ASCII conversion of those bytes = 01, then I know that the record length is 150 bytes, so I want to read the 150 and write them to a new EBCDIC file, that will later be sent through a program that unpacks the fields and converts to a csv. Then I would want to skip 150 bytes and read the next two bytes. Lets say those = 02, so I know that the record is 72 bytes. So on and so forth

You can convert the entire EBCDIC file to ASCII just using:

Code:

dd if=EBCDICfile of=ASCIIfile conv=ascii

but, with packed decimal fields in the COBOL input file, you may end up with null bytes in the output file. And, you can't have null bytes in a text file. If you aren't working on text files, many of the standard utilities produce undefined results.

But, you can use cut and paste even if the files being processed are not text files. So, after converting your file to ASCII, you could walk through the file in a loop, starting with offset=1:As long as you don't modify the parts of the record that contain packed decimal data, converting back from ASCII to EBCDIC should still have correct packed decimal data in the resulting EBCDIC file.

why not install gnucobol and write a cobol program.

Alright Don, so I have spent the last few days playing with this now and have run into a couple quirks. First off are some things about the file. We will call the original EBCDIC file with all of the data data.ebc. I go ahead and do the simple conversion using dd to get a new file, data.ascii. Running the wc command gives me

0 lines in data.ebc, with 64454170 bytes
5948 lines in data.ascii with 64454170 bytes

Then I use the tr command to get rid of newlines so that I have one line in my new.ascii file. Then I go through new.ascii and cut the first two bytes, get 01, and write that to a file, increment and repeat. This works perfectly until I get to bytes 16880, in which the program then gets thrown off. Interestingly in data.ascii, there are 16507 bytes in the first line. So somehow I need to make it so that I have either a file that has only one line (since using tr to delete '\n' seems to be causing issues) or I need a file that has 422 bytes on each line, so that the first two bytes of each line correspond to either 01,02,03,...,12,13.

OK. This is good! You translated EBCDIC bytes to the corresponding ASCII bytes and no bytes were added or lost. But, even though this is an ASCII file, it is not a text file; the <newline> characters are just binary data in your file; not line terminators.
Ouch. No! Don't remove ANY bytes from data.ascii. Those <newline> characters you're seeing in that file are probably the ASCII byte values corresponding to some of the binary packed decimal data bytes in your input.

The data in data.ascii is just a stream of bytes containing the records in your data; there are no record separators in data.ebc nor in data.ascii. In addition to <newline> characters, there are probably also <nul> (all bits 0) bytes that should not appear in a text file. But, we aren't going to treat data.ascii (or data.ebc) as a text file.

Can you show us a table where the 1st column gives us the 1st two characters of your records (the two bytes that specify the record type), the 2nd column gives us the length in bytes of records of that type (either with or without the two bytes specifying the record type, but tell us whether or not the record size given includes those bytes), and the 3rd column gives us the name of the file that to which records of this type should be appended? (Are these output files supposed to be ASCII or EBCDIC? On first read of your requirements, I thought wanted to feed ASCII data to your C++ converter and then take the output from your C++ converter and translate that back to EBCDIC. Reading your first post again, it isn't clear to me whether the C++ converter wants EBCDIC input or ASCII input.)