Compress multiple files at one go Post: 302300309

Sponsored Content

Top Forums UNIX for Dummies Questions & Answers Compress multiple files at one go Post 302300309 by TonyFullerMalv on Monday 23rd of March 2009 08:24:12 PM

03-23-2009

Registered User

If you have a "too many files" problem then break it down so that you put a certain number of files into a tar file, gzip the tar file, delete those files then repeat: I would script it with something like this:

Code:

i=1
DATESTR=`date '+%m%d'`
EXIT="no"
cd <desired_directory> && \
while [ "${EXIT}" = "no" ]; do
  find . -type f -mtime +1176 | head -16000 > /tmp/filelist
  if [ -s /tmp/filelist ]; then
    tar cvf tarfile.${i}.${DATESTR}.tar `cat /tmp/filelist` && \
    gzip tarfile.${i}.${DATESTR}.tar && \
    sed -e 's/^/rm /' /tmp/filelist > /tmp/rmfiles && \
    sh /tmp/rmfiles
  else
    EXIT="yes"
  fi
  i=`expr $i + 1` 
done

I have not been able to test this (not having thousands of files to hand!) so I suggest putting an "ls" in place of the "rm" to start with and you may have to reduce the 16000 to get the tar to work.
I hope you do not have filenames with spaces in!
If this were running on Solaris I could do tar cvf tarfile -I /tmp/listfile and filenames with spaces in would not be a problem...

Last edited by TonyFullerMalv; 03-23-2009 at 09:34 PM..

TonyFullerMalv

View Public Profile for TonyFullerMalv

Find all posts by TonyFullerMalv

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Compress all INACTIVE Files

Hi gurus, I have a batch job running daily night automatically. The job produces file with extension '.0' like a.0, b.0 etc. Now due to file space constraints, most of the time, the job fails with insufficient disk space and then we have to manually start the job again and keep running the...

2. Shell Programming and Scripting

Compress multiple files

Hi Friends, Can anyone help me out with compressing multiple files. I have multiple files in directory , I have to compress these into a single file, I tried using gzip -r outfile.gz file1 file2 file3. It is not working Thanks in advance for your help S :)

3. UNIX for Dummies Questions & Answers

Compress files

Hi All, I would like to archive some of the scripts below(USFINUM042006_01.CSV USFINUM042006_02.CSV and USFINUM042006_03.CSV )and also use a wildcard e.g. <command> USFINUM*.CSV. Also there are a lot of similar files but I want only the three latest files to be compressed. Which is the best...

4. Shell Programming and Scripting

compress files

Could someone give me an idea how to compress all files from a given directory that are not of type .z (compressed). Please help.

5. UNIX for Dummies Questions & Answers

find new files and compress them

Hi! First off I'd like to stress that I'm a true dummy :) I have a website with SSH access and since it has user generated content I want to backup my website every day end send it through FTP to a different server. I got it working for my mysql database, so the only thing remaining are the...

6. UNIX for Dummies Questions & Answers

How to compress files without extension

Could someone please help? I'm trying to compress all the files in a directory without extension. I know for typical files with extension, the command is something like: tar -zcvf file.tar.gz *.doc What is the command for files without extension? Thanks.

7. UNIX for Dummies Questions & Answers

Issue: Compress in unix server and FTP to windows and open the compress file using Winzip

Hi All ! We have to compress a big data file in unix server and transfer it to windows and uncompress it using winzip in windows. I have used the utility ZIP like the below. zip -e <newfilename> df2_test_extract.dat but when I compress files greater than 4 gb using zip utility, it...

8. Shell Programming and Scripting

Compress files as per timestamp in multiple subdirectories

I'd really appreciate if anyone could assist me with this code A directory with multiple subdirectories has multiple files which are timestamp'ed. We need to - compress files as per timestamp - save compressed file/s in the respective folder - delete the source files ============...

9. Shell Programming and Scripting

Compress Files in Multiple Directories

I would like to compress the files in multiple directories. For some reason, it only compress the first directory (/Sanbox/logs1) but not the rest of the other directories ("/Sanbox/logs2" "/Sanbox/logs3" "/Sanbox/logs4" ). Any help would be appreciated. Here's my code: #!/bin/bash...

10. UNIX for Beginners Questions & Answers

Compress multiple gzip files

Good afternoon friends. I wanted to make a query, how to compress several files and leave them all in 1, for example flat text files: filename_1.csv filename_2.csv filename_3.csv expected result filename_end.gzip = (filename_1.csv filename_2.csv filename_3.csv) please

LEARN ABOUT DEBIAN

bup-join

bup-join(1)						      General Commands Manual						       bup-join(1)

NAME

       bup-join - concatenate files from a bup repository

SYNOPSIS

       bup join [-r host:path] [refs or hashes...]

DESCRIPTION

       bup join  is  roughly the opposite operation to bup-split(1).  You can use it to retrieve the contents of a file from a local or remote bup
       repository.

       The supplied list of refs or hashes can be in any format accepted by git(1), including branch names, commit ids, tree ids, or blob ids.

       If no refs or hashes are given on the command line, bup join reads them from stdin instead.

OPTIONS

       -r, --remote=host:path
	      Retrieves objects from the given remote repository instead of the local one.  path may be blank, in which case  the  default  remote
	      repository is used.  The connection to the remote server is made with SSH.  If you'd like to specify which port, user or private key
	      to use for the SSH connection, we recommend you use the ~/.ssh/config file.

EXAMPLE

	      # split and then rejoin a file using its tree id
	      TREE=$(tar -cvf - /etc | bup split -t)
	      bup join $TREE | tar -tf -

	      # make two backups, then get the second-most-recent.
	      # mybackup~1 is git(1) notation for the second most
	      # recent commit on the branch named mybackup.
	      tar -cvf - /etc | bup split -n mybackup
	      tar -cvf - /etc | bup split -n mybackup
	      bup join mybackup~1 | tar -tf -

SEE ALSO

       bup-split(1), bup-save(1), ssh_config(5)

BUP

       Part of the bup(1) suite.

AUTHORS

       Avery Pennarun <apenwarr@gmail.com>.

Bup unknown-															       bup-join(1)

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Compress all INACTIVE Files

Discussion started by: super_duper_guy

2. Shell Programming and Scripting

Compress multiple files

Discussion started by: sbasetty

3. UNIX for Dummies Questions & Answers

Compress files

Discussion started by: indira

4. Shell Programming and Scripting

compress files

Discussion started by: lesstjm