Sponsored Content
Top Forums UNIX for Advanced & Expert Users Speed problems with tar'ing a 500Gb directory on an eSATA drive Post 302623095 by omnisppot on Friday 13th of April 2012 06:28:56 AM
Old 04-13-2012
[QUOTE=methyl;302622215]I don't think that tar or cp are the right commands.

To make a straight copy to another mounted filesystem and preserve permissions:
Code:
cd /filesystem_to_copy
find . -xdev -print | cpio -pdumv /new_filesystem

Ps. I have never used tar to back up anything. It is sometimes useful for moving files to alien systems.


Thanks for the input, but the goal is to move the 500Gb of data from the external drive to an offsite compute cluster. I believe the only way I can do this is ftp, and ftp only supports moving single files, not directories. GUIs like Filezilla don't work as they prompt for a new password every time the token-generated one expires.
I don't think it's possible to mount the external hard drive from a cluster that's behind a firewall - I can only connect to the cluster, not from it Smilie

---------- Post updated at 05:25 AM ---------- Previous update was at 05:20 AM ----------

Quote:
Originally Posted by Corona688
What bus speeds would you expect from your disks, omnisppot? Could you be having southbridge issues -- perhaps the bus is saturated?
Sorry, I don't know how to precisely answer that question! I know I can (if I had enough internal hard drive space) "cp -r" all the data down the SATA cable in a few hours without any issues. It certainly seems I/O on the external drive is the bottleneck with tar. Hopefully this is at the external disk end and not the mobo bus end. Hopefully (I'll find out next week) doing it on a 4-drive RAID0 will overcome that!

---------- Post updated at 05:28 AM ---------- Previous update was at 05:25 AM ----------

Quote:
Originally Posted by methyl
I agree with Corona688. Probably Hardware problem.
However I have seen a modern tar (i.e. one which can deal with files larger than 2Gb) crawl when it demands more memory.
My linux box has 16Gb RAM, but whilst doing this system usage didn't exceed 3Gb (including running the O/S and everything else).
 

10 More Discussions You Might Find Interesting

1. Solaris

Error tar'ing files to tape

I'm trying to tar a bunch of files off to a tape, but for one specific file (it is fairly large, roughly 10Gb) I get the error: too large to archive Does tar have a limit of the size of file it can write off to tape? I'm using SunOS 5.8. Thanks! -Fred (6 Replies)
Discussion started by: FredSmith
6 Replies

2. UNIX for Dummies Questions & Answers

tar'ing and zipping files

If I have a directory /directory1 and want to tar and zip everything in it into a file new_tar.tar.gz on disk (not tape) How can I do it? I tried tar -cv /new_tar.tar /directory1/* But I got an error: tar: /dev/rmt/0: No such device or address (4 Replies)
Discussion started by: FredSmith
4 Replies

3. UNIX for Advanced & Expert Users

tar problems using Sony AIT drive

Recently we brought up a Spectralogic 2K Tape Library that had been out of service for about 3 years to replace a DDS-4 tape drive unit as our main backup device. Everything seemed to go fine but now I have run into a little problem. System details: FBSD 6.1 SpectraLogic 2K library with a... (1 Reply)
Discussion started by: thumper
1 Replies

4. UNIX for Dummies Questions & Answers

tar'ing and regular expressions

Hi, How do I tar all but a specific set of files in a directory? Is it possible to use regular expressions in the tar command? I want to tar all files except those beginning with D. I tried this tar -cvf files.tar ^ but this didn't work. Anyone any ideas. Thanks (2 Replies)
Discussion started by: sirbrian
2 Replies

5. UNIX for Dummies Questions & Answers

Tar-ing the correct directories

Hi all, my directory structure is as follows /a/b/c. I would like to tar the /a directory including the subdirectories b and c. i intend to use the command tar -cvfz a.tgz a/ My question is where do i execute the command? do i execute it at the '/' prompt or at '/a' prompt ? My concern at... (1 Reply)
Discussion started by: new2ss
1 Replies

6. UNIX for Advanced & Expert Users

iSCSI speed problems

Hi all. I was able to set up an IBM Ultrium LTO 4 tape drive to use iSCSI (using open-iscsi drivers) to communicate with Red Hat, but it's going really slow, maxing out in tar and dd tests at like 16 MB/s (using a block size of 128k). The thing is rated for 30MB/s. I feel like even though I have... (1 Reply)
Discussion started by: jeriryan87
1 Replies

7. Shell Programming and Scripting

grep'ing and sed'ing chunks in bash... need help on speeding up a log parser.

I have a file that is 20 - 80+ MB in size that is a certain type of log file. It logs one of our processes and this process is multi-threaded. Therefore the log file is kind of a mess. Here's an example: The logfile looks like: "DATE TIME - THREAD ID - Details", and a new file is created... (4 Replies)
Discussion started by: elinenbe
4 Replies

8. Shell Programming and Scripting

tar command dont tar to original directory

HI, if I have a tarfile called pmapdata.tar that contains tar -tvf pmapdata.tar -rw-r--r-- 0/0 21 Oct 15 11:00 2009 /var/tmp/pmapdata/pmap4628.txt -rw-r--r-- 0/0 21 Oct 14 20:00 2009 /var/tmp/pmapdata/pmap23752.txt -rw-r--r-- 0/0 1625 Oct 13 20:00 2009... (1 Reply)
Discussion started by: borderblaster
1 Replies

9. UNIX for Dummies Questions & Answers

Tar-ing folders in a folder

How do I create individual tars of a all the directories in a directory? I have a directory called 'patients', each patient has a directory in the patients directory. I want to create tars such that each patient has their own tar file. Thanks! (5 Replies)
Discussion started by: HappyPhysicist
5 Replies

10. Shell Programming and Scripting

Speed up extraction od tar.bz2 files using bash

The below bash will untar each tar.bz2 folder in the directory, then remove the tar.bz2. Each of the tar.bz2 folders ranges from 40-75GB and currently takes ~2 hours to extract. Is there a way to speed up the extraction process? I am using a xeon processor with 12 cores. Thank you :). ... (7 Replies)
Discussion started by: cmccabe
7 Replies
CPIO(1L)																  CPIO(1L)

NAME
cpio - copy files to and from archives SYNOPSIS
cpio {-o|--create} [-0acvABLV] [-C bytes] [-H format] [-M message] [-O [[user@]host:]archive] [-F [[user@]host:]archive] [--file=[[user@]host:]archive] [--format=format] [--message=message] [--null] [--reset-access-time] [--verbose] [--dot] [--append] [--block-size=blocks] [--dereference] [--io-size=bytes] [--quiet] [--force-local] [--rsh-command=command] [--help] [--version] < name-list [> archive] cpio {-i|--extract} [-bcdfmnrtsuvBSV] [-C bytes] [-E file] [-H format] [-M message] [-R [user][:.][group]] [-I [[user@]host:]archive] [-F [[user@]host:]archive] [--file=[[user@]host:]archive] [--make-directories] [--nonmatching] [--preserve-modification-time] [--numeric-uid- gid] [--rename] [-t|--list] [--swap-bytes] [--swap] [--dot] [--unconditional] [--verbose] [--block-size=blocks] [--swap-halfwords] [--io- size=bytes] [--pattern-file=file] [--format=format] [--owner=[user][:.][group]] [--no-preserve-owner] [--message=message] [--force-local] [--no-absolute-filenames] [--sparse] [--only-verify-crc] [--quiet] [--rsh-command=command] [--help] [--version] [pattern...] [< archive] cpio {-p|--pass-through} [-0adlmuvLV] [-R [user][:.][group]] [--null] [--reset-access-time] [--make-directories] [--link] [--quiet] [--pre- serve-modification-time] [--unconditional] [--verbose] [--dot] [--dereference] [--owner=[user][:.][group]] [--no-preserve-owner] [--sparse] [--help] [--version] destination-directory < name-list DESCRIPTION
This manual page documents the GNU version of cpio. cpio copies files into or out of a cpio or tar archive, which is a file that contains other files plus information about them, such as their file name, owner, timestamps, and access permissions. The archive can be another file on the disk, a magnetic tape, or a pipe. cpio has three operating modes. In copy-out mode, cpio copies files into an archive. It reads a list of filenames, one per line, on the standard input, and writes the ar- chive onto the standard output. A typical way to generate the list of filenames is with the find command; you should give find the -depth option to minimize problems with permissions on directories that are unwritable or not searchable. In copy-in mode, cpio copies files out of an archive or lists the archive contents. It reads the archive from the standard input. Any non-option command line arguments are shell globbing patterns; only files in the archive whose names match one or more of those patterns are copied from the archive. Unlike in the shell, an initial `.' in a filename does match a wildcard at the start of a pattern, and a `/' in a filename can match wildcards. If no patterns are given, all files are extracted. In copy-pass mode, cpio copies files from one directory tree to another, combining the copy-out and copy-in steps without actually using an archive. It reads the list of files to copy from the standard input; the directory into which it will copy them is given as a non-option argument. cpio supports the following archive formats: binary, old ASCII, new ASCII, crc, HPUX binary, HPUX old ASCII, old tar, and POSIX.1 tar. The binary format is obsolete because it encodes information about the files in a way that is not portable between different machine architec- tures. The old ASCII format is portable between different machine architectures, but should not be used on file systems with more than 65536 i-nodes. The new ASCII format is portable between different machine architectures and can be used on any size file system, but is not supported by all versions of cpio; currently, it is only supported by GNU and Unix System V R4. The crc format is like the new ASCII format, but also contains a checksum for each file which cpio calculates when creating an archive and verifies when the file is extracted from the archive. The HPUX formats are provided for compatibility with HPUX's cpio which stores device files differently. The tar format is provided for compatability with the tar program. It can not be used to archive files with names longer than 100 charac- ters, and can not be used to archive "special" (block or character devices) files. The POSIX.1 tar format can not be used to archive files with names longer than 255 characters (less unless they have a "/" in just the right place). By default, cpio creates binary format archives, for compatibility with older cpio programs. When extracting from archives, cpio automati- cally recognizes which kind of archive it is reading and can read archives created on machines with a different byte-order. Some of the options to cpio apply only to certain operating modes; see the SYNOPSIS section for a list of which options are allowed in which modes. OPTIONS -0, --null In copy-out and copy-pass modes, read a list of filenames terminated by a null character instead of a newline, so that files whose names contain newlines can be archived. GNU find is one way to produce a list of null-terminated filenames. -a, --reset-access-time Reset the access times of files after reading them, so that it does not look like they have just been read. -A, --append Append to an existing archive. Only works in copy-out mode. The archive must be a disk file specified with the -O or -F (--file) option. -b, --swap In copy-in mode, swap both halfwords of words and bytes of halfwords in the data. Equivalent to -sS. Use this option to convert 32-bit integers between big-endian and little-endian machines. -B Set the I/O block size to 5120 bytes. Initially the block size is 512 bytes. --block-size=BLOCK-SIZE Set the I/O block size to BLOCK-SIZE * 512 bytes. -c Identical to "-H newc", use the new (SVR4) portable format. If you wish the old portable (ASCII) archive format, use "-H odc" instead. -C IO-SIZE, --io-size=IO-SIZE Set the I/O block size to IO-SIZE bytes. -d, --make-directories Create leading directories where needed. -E FILE, --pattern-file=FILE In copy-in mode, read additional patterns specifying filenames to extract or list from FILE. The lines of FILE are treated as if they had been non-option arguments to cpio. -f, --nonmatching Only copy files that do not match any of the given patterns. -F, --file=archive Archive filename to use instead of standard input or output. To use a tape drive on another machine as the archive, use a filename that starts with `HOSTNAME:'. The hostname can be preceded by a username and an `@' to access the remote tape drive as that user, if you have permission to do so (typically an entry in that user's `~/.rhosts' file). --force-local With -F, -I, or -O, take the archive file name to be a local file even if it contains a colon, which would ordinarily indicate a remote host name. -H FORMAT, --format=FORMAT Use archive format FORMAT. The valid formats are listed below; the same names are also recognized in all-caps. The default in copy-in mode is to automatically detect the archive format, and in copy-out mode is "bin". bin The obsolete binary format. odc The old (POSIX.1) portable format. newc The new (SVR4) portable format, which supports file systems having more than 65536 i-nodes. crc The new (SVR4) portable format with a checksum added. tar The old tar format. ustar The POSIX.1 tar format. Also recognizes GNU tar archives, which are similar but not identical. hpbin The obsolete binary format used by HPUX's cpio (which stores device files differently). hpodc The portable format used by HPUX's cpio (which stores device files differently). -i, --extract Run in copy-in mode. -I archive Archive filename to use instead of standard input. To use a tape drive on another machine as the archive, use a filename that starts with `HOSTNAME:'. The hostname can be preceded by a username and an `@' to access the remote tape drive as that user, if you have permission to do so (typically an entry in that user's `~/.rhosts' file). -k Ignored; for compatibility with other versions of cpio. -l, --link Link files instead of copying them, when possible. -L, --dereference Dereference symbolic links (copy the files that they point to instead of copying the links). -m, --preserve-modification-time Retain previous file modification times when creating files. -M MESSAGE, --message=MESSAGE Print MESSAGE when the end of a volume of the backup media (such as a tape or a floppy disk) is reached, to prompt the user to insert a new volume. If MESSAGE contains the string "%d", it is replaced by the current volume number (starting at 1). -n, --numeric-uid-gid In the verbose table of contents listing, show numeric UID and GID instead of translating them into names. Also extracts tar ar- chives using the numeric UID and GID instead of the user/group names. (cpio archives are always extracted using the numeric UID and GID.) --no-absolute-filenames In copy-in mode, create all files relative to the current directory, even if they have an absolute file name in the archive. --no-preserve-owner In copy-in mode and copy-pass mode, do not change the ownership of the files; leave them owned by the user extracting them. This is the default for non-root users, so that users on System V don't inadvertantly give away files. -o, --create Run in copy-out mode. -O archive Archive filename to use instead of standard output. To use a tape drive on another machine as the archive, use a filename that starts with `HOSTNAME:'. The hostname can be preceded by a username and an `@' to access the remote tape drive as that user, if you have permission to do so (typically an entry in that user's `~/.rhosts' file). --only-verify-crc When reading a CRC format archive in copy-in mode, only verify the CRC's of each file in the archive, don't actually extract the files. -p, --pass-through Run in copy-pass mode. --quiet Do not print the number of blocks copied. -r, --rename Interactively rename files. -R [user][:.][group], --owner [user][:.][group] In copy-out and copy-pass modes, set the ownership of all files created to the specified user and/or group. Either the user or the group, or both, must be present. If the group is omitted but the ":" or "." separator is given, use the given user's login group. Only the super-user can change files' ownership. --rsh-command=COMMAND Notifies mt that it should use COMMAND to communicate with remote devices instead of /usr/bin/ssh or /usr/bin/rsh. --sparse In copy-in and copy-pass modes, write files with large blocks of zeros as sparse files. -s, --swap-bytes In copy-in mode, swap the bytes of each halfword (pair of bytes) in the files. -S, --swap-halfwords In copy-in mode, swap the halfwords of each word (4 bytes) in the files. -t, --list Print a table of contents of the input. -u, --unconditional Replace all files, without asking whether to replace existing newer files with older files. -v, --verbose List the files processed, or with -t, give an `ls -l' style table of contents listing. In a verbose table of contents of a ustar archive, user and group names in the archive that do not exist on the local system are replaced by the names that correspond locally to the numeric UID and GID stored in the archive. -V --dot Print a "." for each file processed. --version Print the cpio program version number and exit. CPIO(1L)
All times are GMT -4. The time now is 04:17 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy