04-11-2012
Speed problems with tar'ing a 500Gb directory on an eSATA drive
I'm trying to compress a directory structure on an external hard drive, connected by eSATA cable to my linux (Ubuntu 10.04) desktop. The total volume is 500Gb with half a million files, ranging from Kb to Mb in size. The drive is 2Tb, with 0.8Tb free space before compression.
running "tar -pcf directory.tar directory" worked for a previous, entirely analogous, 400Gb set of data in about 10 hours.
This time, the command has been running for 7 days, and the tar file is now only growing at 2 Gb/hour - estimated another 50+ days for completion.
I've run it twice now (the cable fell out the first time after two days) and the lack of results is reproducible. Deleting some of the other data from the external drive made no difference.
I'm about to try installing a large RAID0 system in the linux desktop (current drive is almost full), do a straight cp of the directory to there, and repeat the tar locally.
But if anyone has any ideas why this process might be so painfully slow it would be appreciated!
Thanks.
Simon
10 More Discussions You Might Find Interesting
1. Solaris
I'm trying to tar a bunch of files off to a tape, but for one specific file (it is fairly large, roughly 10Gb) I get the error:
too large to archive
Does tar have a limit of the size of file it can write off to tape? I'm using SunOS 5.8.
Thanks!
-Fred (6 Replies)
Discussion started by: FredSmith
6 Replies
2. UNIX for Dummies Questions & Answers
If I have a directory /directory1 and want to tar and zip everything in it into a file new_tar.tar.gz on disk (not tape)
How can I do it?
I tried tar -cv /new_tar.tar /directory1/*
But I got an error: tar: /dev/rmt/0: No such device or address (4 Replies)
Discussion started by: FredSmith
4 Replies
3. UNIX for Advanced & Expert Users
Recently we brought up a Spectralogic 2K Tape Library that had been out of service for about 3 years to replace a DDS-4 tape drive unit as our main backup device.
Everything seemed to go fine but now I have run into a little problem.
System details:
FBSD 6.1
SpectraLogic 2K library with a... (1 Reply)
Discussion started by: thumper
1 Replies
4. UNIX for Dummies Questions & Answers
Hi,
How do I tar all but a specific set of files in a directory? Is it possible to use regular expressions in the tar command? I want to tar all files except those beginning with D. I tried this
tar -cvf files.tar ^
but this didn't work. Anyone any ideas.
Thanks (2 Replies)
Discussion started by: sirbrian
2 Replies
5. UNIX for Dummies Questions & Answers
Hi all, my directory structure is as follows /a/b/c.
I would like to tar the /a directory including the subdirectories b and c.
i intend to use the command tar -cvfz a.tgz a/ My question is where do i execute the command? do i execute it at the '/' prompt or at '/a' prompt ? My concern at... (1 Reply)
Discussion started by: new2ss
1 Replies
6. UNIX for Advanced & Expert Users
Hi all. I was able to set up an IBM Ultrium LTO 4 tape drive to use iSCSI (using open-iscsi drivers) to communicate with Red Hat, but it's going really slow, maxing out in tar and dd tests at like 16 MB/s (using a block size of 128k). The thing is rated for 30MB/s. I feel like even though I have... (1 Reply)
Discussion started by: jeriryan87
1 Replies
7. Shell Programming and Scripting
I have a file that is 20 - 80+ MB in size that is a certain type of log file.
It logs one of our processes and this process is multi-threaded. Therefore the log file is kind of a mess. Here's an example:
The logfile looks like: "DATE TIME - THREAD ID - Details", and a new file is created... (4 Replies)
Discussion started by: elinenbe
4 Replies
8. Shell Programming and Scripting
HI,
if I have a tarfile called pmapdata.tar that contains
tar -tvf pmapdata.tar
-rw-r--r-- 0/0 21 Oct 15 11:00 2009 /var/tmp/pmapdata/pmap4628.txt
-rw-r--r-- 0/0 21 Oct 14 20:00 2009 /var/tmp/pmapdata/pmap23752.txt
-rw-r--r-- 0/0 1625 Oct 13 20:00 2009... (1 Reply)
Discussion started by: borderblaster
1 Replies
9. UNIX for Dummies Questions & Answers
How do I create individual tars of a all the directories in a directory? I have a directory called 'patients', each patient has a directory in the patients directory. I want to create tars such that each patient has their own tar file.
Thanks! (5 Replies)
Discussion started by: HappyPhysicist
5 Replies
10. Shell Programming and Scripting
The below bash will untar each tar.bz2 folder in the directory, then remove the tar.bz2.
Each of the tar.bz2 folders ranges from 40-75GB and currently takes ~2 hours to extract. Is there a way to speed up the extraction process?
I am using a xeon processor with 12 cores. Thank you :).
... (7 Replies)
Discussion started by: cmccabe
7 Replies
pbput(1) bikeshed pbput(1)
NAME
pbput - compress and encode arbitrary files to pastebin.com
pbputs - compress, encrypt, encode arbitrary files to pastebin.com
pbget - decode and decompress arbitrary files from pastebin.com
SYNOPSIS
pbput [FILENAME]
cat foo | pbput
pbputs [FILENAME] [GPG_USER]
cat foo | pbputs [GPG_USER]
pbget URL [DIRECTORY]
DESCRIPTION
pbput is a program that can upload text files, binary files or entire directory structures to a pastebin, such as pastebin.com.
pbget is a program that be used to retrieve content uploaded to a pastebin by pbput.
pbputs operates exactly like pbput, except it encrypts the data. An optional GPG_USER argument is allowed, which will sign and encrypt the
data to the target user in one's keyring (which could be oneself!). Otherwise, the user is prompted for a symmetric passphrase for
encrypting the content with gpg(1) before uploading. pbget will automatically prompt the receiving user for the pre-shared passphrase.
pbput and pbputs can take its input either on STDIN, or as a FILENAME argument.
- If STDIN is used, then the receiving user's pbget will simply paste the input on STDOUT.
- If a FILENAME or DIRECTORY is passed as an argument, then it is first archived using tar(1) to preserve the file and directory
attributes
pbget takes a URL as its first, mandatory argument. Optionally, it takes a DIRECTORY as a second parameter. If the incoming data is in
fact a file or file structure in a tar(1) archive, then that data will be extracted in the specified DIRECTORY. If no DIRECTORY is speci-
fied, then a temporary directory is created using mktemp(1).
In any case the uploaded/downloaded data is optionally tar(1) archived, always lzma(1) compressed, optionally gpg(1) encrypted, and always
base64(1) encoded. http://pastebin.com is used by default.
EXAMPLES
$ pbput /sbin/init
http://pastebin.com/BstNzasK
$ pbget http://pastebin.com/BstNzasK
sbin/init
INFO: Output is in [/tmp/pbget.bG67DwY6Zl]
$ cat /etc/lsb-release | pbput
http://pastebin.com/p43gJv6Z
$ pbget http://pastebin.com/p43gJv6Z
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=11.04
DISTRIB_CODENAME=natty
DISTRIB_DESCRIPTION="Ubuntu 11.04"
$ pbputs /etc/shadow
Enter passphrase:
http://pastebin.com/t2ZaCYr3
$ pbget http://pastebin.com/t2ZaCYr3
Enter passphrase:
root:09cc6d2d9d63371a425076e217f77698:15096:0:99999:7:::
daemon:*:15089:0:99999:7:::
bin:*:15089:0:99999:7:::
sys:*:15089:0:99999:7:::
....
SEE ALSO
pastebinit(1), lzma(1), base64(1), tar(1), gpg(1), mktemp(1)
AUTHOR
This manpage and the utility was written by Dustin Kirkland <kirkland@ubuntu.com> for Ubuntu systems (but may be used by others). Permis-
sion is granted to copy, distribute and/or modify this document under the terms of the GNU General Public License, Version 2 or later pub-
lished by the Free Software Foundation.
On Debian systems, the complete text of the GNU General Public License can be found in /usr/share/common-licenses/GPL, or on the web at
http://www.gnu.org/licenses/gpl.txt.
bikeshed 6 Oct 2010 pbput(1)