Help with zipping files


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Help with zipping files
# 1  
Old 02-21-2011
Help with zipping files

Hi,

I have come across a requirement in which I need to zip files. This is fine but the requriement has one conditions like below:

One .z file can not have more than 10,000 files

Now in the directory I have several files liek below:

aaa_file_10_00001.txt
aaa_file_10_00002.txt
aaa_file_20_00001.txt
aaa_file_30_00001.txt
aaa_file_10_00003.txt
...........................
...........................
...........................
aaa_file_30_00750.txt
aaa_file_30_10005.txt

If i have to zip the above files, I will have 4 .z files. One for files like aaa_file_20_*.txt, one for files like aaa_file_30_*.txt and two for files like aaa_file_10_*.txt I am planning to move files of different type (diferentiated by 3rd field in the file name) to a temporary directory and zip them. But the problem is I can not zip more than 10,000 files together. How do I move files of type aaa_file_10_*.txt from 1-10000 (this is the value in 4th field of the file name )first, zip them and then move the reamining 10001-10005.

Using a loop is not a good option as per my understanding. Is there any other way of doing it which will not have impact on performance ?

Thanks
Angshuman
# 2  
Old 02-21-2011
In bash,
Code:
for i in {00000..10000}
do
 echo aaa_file_10_${i}.txt ## take whatever action you want to perform (eg moving)
done

you can also use 'seq' command or even just normal incremental count.
# 3  
Old 02-21-2011
Hi Anchal,

Thank you for your reply. I understand that we can achieve this using loop. But my concern is performance. If I use a loop when there is actually more than 10,000 files of same type, it will take lot of time. I was thinking if there is any one liner using awk or any other way if I can do that.

Thanks
Angsuman

---------- Post updated at 08:33 PM ---------- Previous update was at 03:51 PM ----------

Hi All,

Any one can help on this?

Thanks
Angsuman
# 4  
Old 02-26-2011
Any one can help me on this?
# 5  
Old 02-26-2011
You agreed not to bump posts when you registered; nobody here is "on call". If you really, really need help RIGHT NOW there's even an "emergency support" forum you didn't use too.

It would help a lot to know what your system and shell is.

---------- Post updated at 12:13 PM ---------- Previous update was at 11:47 AM ----------

Run this in its own empty directory to see what it does. It should work in most shells. You can probably adapt it to do what you want.

Code:
#!/bin/sh

# Create some example files to work on.
# it will create directories 00 through 09.
# 00/ will contain files 00-00 through 00-99,
# 01/ will contain files 01-00 through 01-99, etc.
for DIR in 00 01 02 03 04 05 06 07 08 09
do
        mkdir -p $DIR
        for A in 0 1 2 3 4 5 6 7 8 9
        do
                for B in 0 1 2 3 4 5 6 7 8 9
                do
                        touch "$DIR/$DIR-$A$B"
                done
        done
done

# Let's list all the 00-* files, and split the list apart at 20 lines each.
# The 'split' command will create files "list-aa", "list-ab", and so forth.
# the 'sort' makes the filenames come out in order from 00 to 99, if
# you don't care about that you can leave it out.
find ./ -type f -name "00-*" | sort | split -l 20 - list-
# Now we use each list to create its own zip file.
for LIST in list-*
do
        ZIPFILE="${LIST}.zip"
        zip -r "${ZIPFILE}" -@ < "${LIST}" && rm "${LIST}"
done

# show our zipfiles
ls *.zip
# how many files in the first zip?
unzip -l list-aa.zip

# 6  
Old 02-27-2011
performance issues? zipping 10,000 files will never be quick.
maybe in 2015 it will.

since when is a loop slow? it's all the zipping which will
be the bottleneck. not any loop.

you can use shell expansion if you know the file names, and they adhere
to a numbering sequence


mv blah_[0][0-9][0-9][0-9][[0-9].txt


no loop,
hmm, but I wonder if internally bash would express the above as a loop?

or, you could list the names to a file, and use split(1) to get separate lists of names.
# 7  
Old 03-01-2011
Quote:
Originally Posted by bigearsbilly
hmm, but I wonder if internally bash would express the above as a loop?
Most sane shells would probably express it as a "too many arguments" error...
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

Find and removing the old files and zipping the files using shell script

Hi, I am trying to removing the old files which were older than 10 days and same g zipping the files using the shell script. script was return as follows. find /jboss7_homes/JBOSS7/SKYLIV??/SKYLIV??_CRM/jboss-eap-7.0/standalone/log -mtime +10 -type f | xargs rm -f find /cer_skyliv??/log... (6 Replies)
Discussion started by: venkat918
6 Replies

2. Shell Programming and Scripting

Zipping the files with data

I have files a_cd_1.csv a_cd_2.csv a_cd_3.csv I need to zip these files into one zip file a_cd.zip but if no data is there in any of these csv then only csv with data should be zipped. Also these files will always have header record if data is present or not. Please let me know how this... (9 Replies)
Discussion started by: weknowd
9 Replies

3. Shell Programming and Scripting

Zipping files

Hi Guys, The script below works but it creates a zip folder under 123_arch.zip -- test1 -- orig1.txt -- orig2.txt -- orig3.txt -- orig4.txt I don't want the sub directory test1 but everything under the base *arch name I can not create a long name with... (4 Replies)
Discussion started by: GaryP1973
4 Replies

4. UNIX for Dummies Questions & Answers

Zipping files

Hi All, I have a scenario where in am using uuencode to send a txt file as an excel to end users( email attachment).I have 7 different files and these files are sent as emails 7 times... So my question is, can i not zip all the 7 files at once and attach those files in a single... (9 Replies)
Discussion started by: saggiboy10
9 Replies

5. Shell Programming and Scripting

zipping files then remove the folders

Hi, i am using the below script to zip the files with respect to their timestamp of the files.It is working fine. For example lets take I have two files in this directory /path/folder1 with the timestamp as this month and previous month. When i run this script first time it is creating two... (7 Replies)
Discussion started by: velava
7 Replies

6. Shell Programming and Scripting

zipping files

Hi, Is there any difference if files are individually zipped and archived in a directory or if files are moved into archiving directory and zipping that directory. (3 Replies)
Discussion started by: swathich
3 Replies

7. UNIX for Dummies Questions & Answers

Zipping files - Please help me its urgent

Dear all, I have thousands of log files in my log directory which I need to zip them and archive. I tried using zip command. But it is not allowing me to archive it more that 4GB of file size. So how to archive them. If it is not possible how to zip all files in to multiple archive files which... (3 Replies)
Discussion started by: tvbhkishore
3 Replies

8. UNIX for Dummies Questions & Answers

Zipping files?

how would i zip a file? what does zip mean? (4 Replies)
Discussion started by: trob
4 Replies

9. Solaris

Error while zipping files

I was trying to remove & archive all the files in the current directory and embed them into the zip file, i mean i don't want to list the files any more after zipping those files. For this i wrote the following code though operation was succeeded zipping all the files and creating a zip file i am... (2 Replies)
Discussion started by: Ariean
2 Replies

10. Shell Programming and Scripting

zipping files

Dear Experts, I need a script which will zipped the files older than 2 days. but i dont want to use find . * -mtime 2. Is there is any other method to achive this task. i will ececute the script daily. Regards, (3 Replies)
Discussion started by: shary
3 Replies
Login or Register to Ask a Question