Shell Programming and Scripting

hi all,

I have millions of zip files about 200M each, and i'm trying to add a couple of small files(jpg txt etc.) into these zip files, less than 100k.

when i'm doing this, is it gonna unzip all 200M large files, add files and re zip again? or just add new small ones in?

i'm concerned with the processing time, and want this add processing instant.

thanks for anyone who can clarify for me

Depends what you mean by "zip file".

If you mean zip file as in an honest-to-goodness, pkware-compatible, old-fashioned .zip file as created by the zip and unzip commands, then yes. Files in .zip archives are compressed individually, not collectively, and the .zip contains a table of contents which can be quickly and easily modified and extended after the file is created.

If you mean a "zip file" as in some sort of compressed stream/blob like a .tar.gz or .tar.bz2, then no, you'll have to decompress it first.

These different approaches have advantages and disadvantages. .zip's ability to be modified is useful in situations like yours, but makes it impossible to stream in a pipe chain. It's also poor at preserving file permissions. tar on the other hand does an excellent job of preserving ownership and permissions, and can be streamed over pipes and network sockets.

In general, there's nothing really wrong with .zip if you don't need streaming and don't care about preserving file permissions.

thanks for your reply.
i mean the old fashion .zip file

The only worry I'd have about modifying the same file repeatedly is that the table of contents must be changed, every time. It's not simply adding to the end. Mess up the table of contents and the file's entire contents could be unrecoverable. Be extremely careful to prevent more than one thing from accessing the file simultaneously. This even includes reads, because the table of contents changing in mid-read could cause confusion or worse to the program reading it.

Splitting among a few zips to reduce the potential for damage, and of course keeping backups, may be wise.