Sponsored Content
Top Forums UNIX for Advanced & Expert Users Speed problems with tar'ing a 500Gb directory on an eSATA drive Post 302623241 by rbatte1 on Friday 13th of April 2012 10:09:48 AM
Old 04-13-2012
Would you consider:-
Code:
# cd source_directory
# tar -cvf - . | rsh target_server "cd target_directory ; tar -xvf -"

I'm assuming it's rsh not remsh for your OS.

If the server is remote or the network is the bottleneck, you could consider:-
Code:
# cd source_directory
# tar -cvf - . | compress | rsh target_server "cd target_directory ; uncompress | tar -xvf -"

Of course, this latter option costs on CPU and is best on multi-proc servers so that the tar and compress are not competing.
I've shovelled 200Gb between remote sites over 2M link in a weekend with something like the above, although the syntax will need to be checked. I must have got pretty good compression I suppose. I can't really test it at the moment.

You will need to ensure that the local server can remote shell to the target. An entry in /.rhosts should suffice, but if this seems a good plan but you can't get remote shell working, let us know.


I hope that this helps
Robin
Liverpool/Blackburn
UK
This User Gave Thanks to rbatte1 For This Post:
 

10 More Discussions You Might Find Interesting

1. Solaris

Error tar'ing files to tape

I'm trying to tar a bunch of files off to a tape, but for one specific file (it is fairly large, roughly 10Gb) I get the error: too large to archive Does tar have a limit of the size of file it can write off to tape? I'm using SunOS 5.8. Thanks! -Fred (6 Replies)
Discussion started by: FredSmith
6 Replies

2. UNIX for Dummies Questions & Answers

tar'ing and zipping files

If I have a directory /directory1 and want to tar and zip everything in it into a file new_tar.tar.gz on disk (not tape) How can I do it? I tried tar -cv /new_tar.tar /directory1/* But I got an error: tar: /dev/rmt/0: No such device or address (4 Replies)
Discussion started by: FredSmith
4 Replies

3. UNIX for Advanced & Expert Users

tar problems using Sony AIT drive

Recently we brought up a Spectralogic 2K Tape Library that had been out of service for about 3 years to replace a DDS-4 tape drive unit as our main backup device. Everything seemed to go fine but now I have run into a little problem. System details: FBSD 6.1 SpectraLogic 2K library with a... (1 Reply)
Discussion started by: thumper
1 Replies

4. UNIX for Dummies Questions & Answers

tar'ing and regular expressions

Hi, How do I tar all but a specific set of files in a directory? Is it possible to use regular expressions in the tar command? I want to tar all files except those beginning with D. I tried this tar -cvf files.tar ^ but this didn't work. Anyone any ideas. Thanks (2 Replies)
Discussion started by: sirbrian
2 Replies

5. UNIX for Dummies Questions & Answers

Tar-ing the correct directories

Hi all, my directory structure is as follows /a/b/c. I would like to tar the /a directory including the subdirectories b and c. i intend to use the command tar -cvfz a.tgz a/ My question is where do i execute the command? do i execute it at the '/' prompt or at '/a' prompt ? My concern at... (1 Reply)
Discussion started by: new2ss
1 Replies

6. UNIX for Advanced & Expert Users

iSCSI speed problems

Hi all. I was able to set up an IBM Ultrium LTO 4 tape drive to use iSCSI (using open-iscsi drivers) to communicate with Red Hat, but it's going really slow, maxing out in tar and dd tests at like 16 MB/s (using a block size of 128k). The thing is rated for 30MB/s. I feel like even though I have... (1 Reply)
Discussion started by: jeriryan87
1 Replies

7. Shell Programming and Scripting

grep'ing and sed'ing chunks in bash... need help on speeding up a log parser.

I have a file that is 20 - 80+ MB in size that is a certain type of log file. It logs one of our processes and this process is multi-threaded. Therefore the log file is kind of a mess. Here's an example: The logfile looks like: "DATE TIME - THREAD ID - Details", and a new file is created... (4 Replies)
Discussion started by: elinenbe
4 Replies

8. Shell Programming and Scripting

tar command dont tar to original directory

HI, if I have a tarfile called pmapdata.tar that contains tar -tvf pmapdata.tar -rw-r--r-- 0/0 21 Oct 15 11:00 2009 /var/tmp/pmapdata/pmap4628.txt -rw-r--r-- 0/0 21 Oct 14 20:00 2009 /var/tmp/pmapdata/pmap23752.txt -rw-r--r-- 0/0 1625 Oct 13 20:00 2009... (1 Reply)
Discussion started by: borderblaster
1 Replies

9. UNIX for Dummies Questions & Answers

Tar-ing folders in a folder

How do I create individual tars of a all the directories in a directory? I have a directory called 'patients', each patient has a directory in the patients directory. I want to create tars such that each patient has their own tar file. Thanks! (5 Replies)
Discussion started by: HappyPhysicist
5 Replies

10. Shell Programming and Scripting

Speed up extraction od tar.bz2 files using bash

The below bash will untar each tar.bz2 folder in the directory, then remove the tar.bz2. Each of the tar.bz2 folders ranges from 40-75GB and currently takes ~2 hours to extract. Is there a way to speed up the extraction process? I am using a xeon processor with 12 cores. Thank you :). ... (7 Replies)
Discussion started by: cmccabe
7 Replies
XON(1)							      General Commands Manual							    XON(1)

NAME
xon - start an X program on a remote machine SYNOPSIS
xon remote-host [-access] [-debug] [-name window-name] [-nols] [-screen screen-no] [-user user-name] [command ...] DESCRIPTION
Xon runs the specified command (default xterm -ls) on the remote machine using rsh, remsh, or rcmd. Xon passes the DISPLAY, XAUTHORITY and XUSERFILESEARCHPATH environment variables to the remote command. When no command is specified, xon runs 'xterm -ls'. It additionally specifies the application name to be 'xterm-remote-host' and the win- dow title to be '-fIremote-host'. Xon can only work when the remote host will allow you to log in without a password, by having an entry in the .rhosts file permitting access. OPTIONS
Note that the options follow the remote host name (as they do with rlogin). -access Runs xhost locally to add the remote host to the host access list in the X server. This won't work unless xhost is given permission to modify the access list. -debug Normally, xon disconnects the remote process from stdin, stdout and stderr to eliminate the daemon processes which usually connect them across the network. Specifying the -debug option leaves them connected so that error messages from the remote execution are sent back to the originating host. -name window-name This specifies a different application name and window title for the default command (xterm). -nols Normally xon passes the -ls option to the remote xterm; this option suspends that behaviour. -screen screen-no This changes the screen number of the DISPLAY variable passed to the remote command. -user user-name By default, xon simply uses rsh/remsh/rcmd to connect to the remote machine using the same user name as on the local machine. This option cause xon to specify an alternative user name. This will not work unless you have authorization to access the remote account, by placing an appropriate entry in the remote users .rhosts file. BUGS
Xon can get easily confused when the remote-host, user-name or various environment variable values contain white space. Xon has no way to send the appropriate X authorization information to the remote host. X Version 11 Release 6.6 XON(1)
All times are GMT -4. The time now is 08:04 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy