Archiving big ammount of files.


 
Thread Tools Search this Thread
Top Forums UNIX for Dummies Questions & Answers Archiving big ammount of files.
# 1  
Old 03-27-2008
Question Archiving big ammount of files.

Hello All.

I have problem archiving files.
The problem isSmilie
I have about 10000 files in one directory, all this file aproximately the same size, i need to gzip them and write on DVD. But all this files take about 15 GB of space (already gzipped). So i need DVD Blue-Ray Smilie or i need to split this files on 5 parts and upload them on DVD. I choose second)
Please help me on this issue.
I somehow need to tar this files in 5 archives, and then gzip tar archive. But i cannot understand the way i can tar them on different archives.
Or tar them in one archive and then split this archive on many, but will all files will be readable? if i will open only one part of such archive?

It will be realy nice if someone can help me. And this is not a Homework Smilie
# 2  
Old 03-27-2008
If they are already gzipped, tarring and gzipping again is likely to bring only marginal gains, unless the files are very similar. Might be worth it to just burn one DVD more and put them as regular files without tarring and gzipping -- you don't want to risk making all of them useless if one of the DVDs is damaged. Also the added benefit of random access is muuuuch nicer than having to read through four DVDs of gzipped tar just to extract a single file on the last DVD.
# 3  
Old 03-27-2008
Thank you for fast reply.
But the problem is that windows machine is hanging when i'm trying to open folder where this file storred.
About DVD's i'm always making double copy of DVD in this cases Smilie.
Unix reading them perfectly. i just need to tar and gzip them in different files.
Somehow i need to create list of files, split this file on 2000 rows, tar files according list, and then gzip this tar. I know how to tar and zip but i don't know how to tar files according to the previously created list, or in the best way make tars which have only 2000 files.
# 4  
Old 03-27-2008
Windows I believe mainly has issues with directories with many files in them. If you create a deeper directory tree with a smaller amount of files in each, even poor Windows should be able to cope. It might even bring an opportunity to clean up the file organization a bit.
 
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Archiving files using shell script

Dear Team, I am looking for transferring files to and from the local and remote servers using SFTP commands. Currently the script is using the mget and mput commands to do the copying of the files. While I am trying to move the files from local to remote server, I would also like to archive... (21 Replies)
Discussion started by: Rads
21 Replies

2. Shell Programming and Scripting

Archiving the files

hi, Am trying to acrhive a bunch of files on some ftp site and somehow managed to come out with the below logic. I'm getting "syntax error: unexpected end of file" error. Interestingly this below snipeet works fine if run for the first time but the subsequent runs fail! Anybody has any idea... (3 Replies)
Discussion started by: Amee5
3 Replies

3. Shell Programming and Scripting

Archiving older files

Hello Group, I would request your help to build a shell script in order to find files older than 90 days then create the same directory structure under the second disk (/archive directory) and move the file preserving the same timestamps (ownership, etc). Also keep the log of files moved... (4 Replies)
Discussion started by: csierra
4 Replies

4. Shell Programming and Scripting

Moving and renaming large ammount of files

Hey, I'm kinda new to the shell scripting and I don't wanna mess things up yet :) Looking for a solution to the following: I need to move all the files like "filename.failed.dateandtime" to another directory also renaming them "filename.ready". I can't figure how to do this with multiple files... (4 Replies)
Discussion started by: sg3
4 Replies

5. UNIX for Dummies Questions & Answers

Archiving the files in a .txt file

HI , I have a file abc.txt, which has some .csv files listed. example. abc.txt 1.csv 2.csv 3.csv 4.csv 5.csv I want to move all the files listed in abc.txt to a archive directory,and zip the moved files. Can anyone help me with the script. Thanks,sai (1 Reply)
Discussion started by: saii
1 Replies

6. Shell Programming and Scripting

Issue while archiving the files

Hi, In our current process we are reading the file, (which is placed by external vendor)from one particular folder and processing those files through ETL(informatica). We are reading these file as " ls -ltr *.txt" Once the process is finish these files are moved to archived script by "mv"... (1 Reply)
Discussion started by: Amey Joshi
1 Replies

7. Shell Programming and Scripting

Archiving the Files in a folder

My requirement is to put all the files from output directory(ATT) to archive directory(archive\) creating a new folder with datetimestamp(20100212_120014) every time it runs. where ${IMF_TARGET_DIR} is my base directory. ${IMF_ARCHIVE_DIR} is my Archive directory... (1 Reply)
Discussion started by: vsmeruga
1 Replies

8. Shell Programming and Scripting

Archiving the files

Hi, Suppose I have 2 files of yesterday's. And today I have received 3 files. Before processing anything I want to archieve the 2 files of yesterday's into a different folder. How can this be done? Regards, Sunitha (1 Reply)
Discussion started by: Sunitha_edi82
1 Replies

9. Shell Programming and Scripting

Archiving and moving the files

hi all i have a requirement where in i have to zip all the files with "*.bkp" after 14 days and move the zip files to Archive directory .... i am able to achieve the first functionality but not able to achive the second one ...here is my code find ${LOG_DIR} -name "*.bkp" -mtime +14 | xargs -i... (1 Reply)
Discussion started by: nvuradi
1 Replies

10. UNIX for Dummies Questions & Answers

Backing up or Archiving files in UNIX

Hi All, I am very new to the UNIX world and find myself in a new position at work that requires me to archive large CADD files based in both UNIX and Windows environments on CD's. I have one engineer that wants to export these files as a table (I guess) and it appears to have a lot of paper... (2 Replies)
Discussion started by: Dsartelle
2 Replies
Login or Register to Ask a Question