Sponsored Content
Top Forums UNIX for Advanced & Expert Users How to zip/tar millions of files? Post 302478674 by verdepollo on Wednesday 8th of December 2010 02:20:25 PM
Old 12-08-2010
First of all, thanks all for your suggestions...

Well, even though I have a lot of files, each one of them is pretty small.

I'd say that the whole ~5 million text files do not take more than 50 GB of disk space*

Some details about my system:

Code:
RHEL 5.5 (Tikanga)
Kernel 2.6.18 x86_64
Local ext3 LVM (3 Physical Volumes)
4.5 GB RAM

My Disks:
Code:
SCSI device sdb: 31457280 512-byte hdwr sectors
sd 0:0:0:0: Attached scsi disk sdb
  Type:   Direct-Access                      ANSI SCSI revision: 02
 target0:0:1: FAST-40 WIDE SCSI 80.0 MB/s ST (25 ns, offset 127)

Code:
[root@atlas ~]# df -i
Filesystem            Inodes   IUsed   IFree IUse% Mounted on
/dev/mapper/volAvg-A1lv
                     15466496 4464503 11001993   29% /export
					 
[root@atlas ~]# df -k
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/mapper/volAvg-A1lv
                     121790564  89066984  26538244  78% /export

*According to df -k I'm using around 80 GB but that's because I have other files in the same filesystem that are eating up close to 30 GB.

Quote:
When you say somewhere else, I guess you mean a different server ?
A different server or a different filesystem within the same server; whatever approach is faster.

Quote:
Is it safe to assume that the filesystem will be quiescent?
Not exactly quiescent but with very little disk usage since I will run this process at night when nobody uses the server.

Quote:
Do you have spare discs equivalent to say twice the existing space?
Yes, I can attach more disks if necessary.

Last edited by verdepollo; 12-08-2010 at 03:33 PM..
 

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

unzip .zip file and list the files included in the .zip archive

Hello, I am trying to return the name of the resulting file from a .zip archive file using unix unzip command. unzip c07212007.cef7081.zip Archive: c07212007.cef7081.zip SecureZIP for z/OS by PKWARE inflating: CEP/CEM7080/PPVBILL/PASS/G0063V00 I used the following command to unzip in... (5 Replies)
Discussion started by: oracledev
5 Replies

2. UNIX Desktop Questions & Answers

Using Tar Zip

Hi, I want to backup my SQL database using tar zip but I'm paranoid that I will archive it. What I mean is I want the files to stay where they are but make a zipped copy of the files as well, I don't want to delete the originals. Is the command? tar -cvzf databasename.tar.gz... (1 Reply)
Discussion started by: chickenhouse
1 Replies

3. Shell Programming and Scripting

To write a shell script which groups files with certain pattern, create a tar and zip

Hi Guru's, I have to write a shell script which groups file names based upon the certain matching string pattern, then creates the Tar file for that particular group of files and then zips the Tar file created for the respective group of files. For example, In the given directory these files... (3 Replies)
Discussion started by: rahu_sg
3 Replies

4. Shell Programming and Scripting

Need script to remove millions of tmp files in /html/cache/ directory

Hello, I just saw that on my vps (centOS) my oscommerce with a seo script has created millions of tmp files inside the /html/cache/ directory. I would need to remove all those files (millions), I tried via shell but the vps loads goes to very high and it hangs, is there some way to do a... (7 Replies)
Discussion started by: andymc1
7 Replies

5. UNIX for Dummies Questions & Answers

TAR and ZIP files

Hi, I need a help with zip and tar. I have no done any scripts before with zip command. What I need to achieve is list files in a directory with a specific name (ID_DATE format- given examples) and then zip (or gunzip which I need to use, I am not sure) with timestamp on the file name and then... (15 Replies)
Discussion started by: Vijay81
15 Replies

6. Shell Programming and Scripting

help with tar & zip only last months(say,Sep) files

Need to 1. archive all the files in a directory from the previous month into a tar/gz file, ignoring all already archived 'tar.gz' files 2. Check created .tar.gz file isnt corrupted and has all the required files in it. and then remove the original files. I am using a function to get the... (1 Reply)
Discussion started by: Prev
1 Replies

7. Shell Programming and Scripting

Zip Multiple files to One .zip file in AIX system

Hi I have a requirement in unix shell where I need to zip multiple files on server to one single .zip file. I dont see zip command in AIX and gzip command not doing completely what I want. One I do .zip file, I should be able to unzip in my local Computer. Here is example what I want... (9 Replies)
Discussion started by: RAMA PULI
9 Replies

8. Shell Programming and Scripting

How to create zip/gz/tar files for if the files are older than particular days in UNIX or Linux?

I need a script file for backup (zip or tar or gz) of old log files in our unix server (causing the space problem). Could you please help me to create the zip or gz files for each log files in current directory and sub-directories also? I found one command which is to create gz file for the... (4 Replies)
Discussion started by: Mallikgm
4 Replies

9. UNIX for Beginners Questions & Answers

How can we Zip multiple files created on the same date into one single zip file.?

Hi all i am very new to shell scripting and need some help from you to learn 1)i have some log files that gets generated on daily basis example: i have abc_2017_01_30_1.log ,2017_01_30_2.log like wise so i want to zip this 4 logs which are created on same date into one zip folder. 2)Post zipping... (2 Replies)
Discussion started by: b.saipriyanka
2 Replies

10. Shell Programming and Scripting

How can we Zip multiple files created on the same date into one single zip file.?

Hi all i am very new to shell scripting and need some help from you to learn 1)i have some log files that gets generated on daily basis example: i have abc_2017_01_30_1.log ,2017_01_30_2.log like wise so i want to zip this 4 logs which are created on same date into one zip folder. 2)Post zipping... (1 Reply)
Discussion started by: b.saipriyanka
1 Replies
FSCK.MINIX(8)						       System Administration						     FSCK.MINIX(8)

NAME
fsck.minix - check consistency of Minix filesystem SYNOPSIS
fsck.minix [options] device DESCRIPTION
fsck.minix performs a consistency check for the Linux MINIX filesystem. The program assumes the filesystem is quiescent. fsck.minix should not be used on a mounted device unless you can be sure nobody is writ- ing to it. Remember that the kernel can write to device when it searches for files. The device name will usually have the following form: /dev/hda[1-63] IDE disk 1 /dev/hdb[1-63] IDE disk 2 /dev/sda[1-15] SCSI disk 1 /dev/sdb[1-15] SCSI disk 2 If the filesystem was changed, i.e., repaired, then fsck.minix will print "FILE SYSTEM HAS CHANGED" and will sync(2) three times before exiting. There is no need to reboot after check. WARNING
fsck.minix should not be used on a mounted filesystem. Using fsck.minix on a mounted filesystem is very dangerous, due to the possibility that deleted files are still in use, and can seriously damage a perfectly good filesystem! If you absolutely have to run fsck.minix on a mounted filesystem, such as the root filesystem, make sure nothing is writing to the disk, and that no files are "zombies" waiting for deletion. OPTIONS
-l, --list List all filenames. -r, --repair Perform interactive repairs. -a, --auto Perform automatic repairs. This option implies --repair and serves to answer all of the questions asked with the default. Note that this can be extremely dangerous in the case of extensive filesystem damage. -v, --verbose Be verbose. -s, --super Output super-block information. -m, --uncleared Activate MINIX-like "mode not cleared" warnings. -f, --force Force a filesystem check even if the filesystem was marked as valid. Marking is done by the kernel when the filesystem is unmounted. -V, --version Display version information and exit. -h, --help Display help text and exit. DIAGNOSTICS
There are numerous diagnostic messages. The ones mentioned here are the most commonly seen in normal usage. If the device does not exist, fsck.minix will print "unable to read super block". If the device exists, but is not a MINIX filesystem, fsck.minix will print "bad magic number in super-block". EXIT CODES
The exit code returned by fsck.minix is the sum of the following: 0 No errors 3 Filesystem errors corrected, system should be rebooted if filesystem was mounted 4 Filesystem errors left uncorrected 7 Combination of exit codes 3 and 4 8 Operational error 16 Usage or syntax error AUTHORS
Linus Torvalds <torvalds@cs.helsinki.fi> Error code values by Rik Faith <faith@cs.unc.edu> Added support for filesystem valid flag: Dr. Wettstein <greg%wind.uucp@plains.nodak.edu>. Check to prevent fsck of mounted filesystem added by Daniel Quinlan <quinlan@yggdrasil.com>. Minix v2 fs support by Andreas Schwab <schwab@issan.informatik.uni-dortmund.de>, updated by Nicolai Langfeldt <janl@math.uio.no>. Portability patch by Russell King <rmk@ecs.soton.ac.uk>. SEE ALSO
fsck(8), fsck.ext2(8), mkfs(8), mkfs.ext2(8), mkfs.minix(8), reboot(8) AVAILABILITY
The fsck.minix command is part of the util-linux package and is available from Linux Kernel Archive <https://www.kernel.org/pub/linux/utils /util-linux/>. util-linux June 2015 FSCK.MINIX(8)
All times are GMT -4. The time now is 09:50 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy