The UNIX and Linux Forums  

Go Back   The UNIX and Linux Forums > Top Forums > Shell Programming and Scripting
.
google unix.com



Shell Programming and Scripting Post questions about KSH, CSH, SH, BASH, PERL, PHP, SED, AWK and OTHER shell scripts and shell scripting languages here.

More UNIX and Linux Forum Topics You Might Find Helpful
Thread Thread Starter Forum Replies Last Post
Split large file and add header and footer to each file ashish4422 Shell Programming and Scripting 1 04-15-2008 06:12 AM
Strange difference in file size when copying LARGE file.. 0ktalmagik Filesystems, Disks and Memory 1 06-03-2006 07:34 PM
Importing dump file anushilrai Shell Programming and Scripting 2 03-20-2006 05:56 AM
making copy of 0 level dump via ufsdump vishalsngh SUN Solaris 2 01-27-2006 10:25 AM
help, what is the difference between core dump and panic dump? aileen UNIX for Dummies Questions & Answers 1 06-11-2001 08:08 PM

Reply
 
Submit Tools LinkBack Thread Tools Search this Thread Rate Thread Display Modes
  #1 (permalink)  
Old 04-07-2008
Registered User
 

Join Date: Jan 2008
Posts: 35
Copy large dump file

Hi Experts..

Could anyone please let me know the easier way to copy large dump of files from one server to another. I am trying to copy a set of dump files on two different servers placed in different geographic locations.. Though there are other factors such as latency, etc., slowing up the process, i am curious to know if there are any other method to copy it fast.. I am using scp functionality to copy at present..

Thanks in advance !!
Reply With Quote
Sponsored Links
  #2 (permalink)  
Old 04-07-2008
in2nix4life's Avatar
Registered User
 

Join Date: Oct 2007
Location: East Coast
Posts: 58
Not sure what OS you're using, but you could try using the 'split' command to break the files down to smaller chunks, then use the 'cat' command on the otherside to join them back together:

i.e.
split -b 1m largefile FILE_

then to rejoin:

cat FILE_* > largefile

Hope this helps.
Reply With Quote
  #3 (permalink)  
Old 04-07-2008
Registered User
 

Join Date: Jan 2008
Posts: 35
thanks much for ur reply.. We are using AIX..
Reply With Quote
  #4 (permalink)  
Old 04-07-2008
era era is offline Forum Advisor  
Herder of Useless Cats (On Sabbatical)
 

Join Date: Mar 2008
Location: /there/is/only/bin/sh
Posts: 3,652
If you are not using scp compression, add that.

Maybe look at rsync as well. It can use scp as the channel, but add some intelligence to avoid needless transfers.
Reply With Quote
  #5 (permalink)  
Old 04-07-2008
Registered User
 

Join Date: Apr 2008
Posts: 3
You could look at rsync+ssh, and give ssh the "blowfish" encryption option as it is supposed to be faster than the default.

Like so:
Code:
rsync -avz --ignore-existing -e  'ssh -oConnectTimeout=10 -c blowfish -ax' source-file destserver:/dest/path
Advantage of rsync is that it can resume a broken transfer.

You could also try regular scp with:
Code:
 -c blowfish -C
[ -C is for compression ]

You could also try piping the large file through tar+gzip as described here: here and here

If you can download using HTTP, aget is another option.

All this may not make much of an impact if your file cannot be compressed, or if your network is too slow.
Reply With Quote
Google The UNIX and Linux Forums
Reply

Bookmarks

Tags
None

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes Rate This Thread
Rate This Thread:




All times are GMT -4. The time now is 12:25 PM.


Powered by: vBulletin, Copyright ©2000 - 2006, Jelsoft Enterprises Limited.
vBCredits v1.4 Copyright ©2007 - 2008, PixelFX Studios
The UNIX and Linux Forums Content Copyright ©1993-2009. All Rights Reserved.Ad Management by RedTyger

Content Relevant URLs by vBSEO 3.2.0

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66