Shell Programming and Scripting

Hi,

I am performing a tail command on EBCDIC coded files and am getting the following error message when I try to process very large files.

Code:

tail: There is not enough memory available now.

The file in question has more than 10 millions lines.

I would like to ask for advice for getting around this issue. The reason why tail is used is because I am inserting new bytes in this file and concatenate the new elements with the rest of the file. The tail is performed on the total number of lines minus 1.

I am currently trying to make it work by using the split command on a 100 MB basis, running the tail command on the leading piece to build my workfile, then I do concatenate the result with the rest of the pieces. But I would like to ask for more alternatives as well in case I am proven wrong in my approach.

Any suggestions?
Thanks

Hmmm first of all I have strong doubts about the performance of the while read loop, secondly the cat command will give me a corrupted file since the file is not ASCII encoded which will make most lines unreadable by the cat command.

Code:

cat: Cannot open KßÓÂðòððððHX.

Thanks for the input though.

There is no while read loop.

The cat command will read whatever is in the file; its encoding makes no difference. It handles binary files in exactly the same manner as text files: byte by byte.

You didn't use the script I posted; in it, there is no file to be opened by cat; it is using redirection.

Sorry for misunderstanding your point, I had my head around so many things that I missed it completely...

What you posted seems to work and performance is similar to my alternative solution using split/tail/cat, but in a much lighter format.

As far as I could see there are no difference between our output files so I may opt for the most simplistic solution aka yours.

Cheers