Visit Our UNIX and Linux User Community


Script to multi-transfer splitted files via scp


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Script to multi-transfer splitted files via scp
# 1  
Old 12-14-2012
Code Script to multi-transfer splitted files via scp

Hey :3
I am moving some stuff between different servers.

I do it like this:
Code:
scp -r -P 22 -i ~/new.ppk /var/www/bigfile.tar.gz user@123.123.123.123:/var/www/bigfile.tar.gz

Lets say, this file is 50 GiB. I would like to know, if its possible to split the file in different parts, automatically, and open multiple scp streams to upload it on the remote server.

Is that possible?
# 2  
Old 12-14-2012
I'm not sure if we can split file and perform transfer, but I found an interesting article about using pigz for improving transfer speeds here
I hope this helps.
# 3  
Old 12-14-2012
If your server is on a local network the limitation is probably network speed and CPU time, so running more scp's simultaneously won't really make it faster.

Previous Thread | Next Thread
Test Your Knowledge in Computers #543
Difficulty: Easy
A global variable can be accessed and referenced on every line of code.
True or False?

10 More Discussions You Might Find Interesting

1. UNIX for Advanced & Expert Users

SCP File Transfer

On unix AIX server, when I am trying to transfer file from one directory to another directory on the same server through a program(where i call the script) it gives error "Lost Connection". (5 Replies)
Discussion started by: Pash
5 Replies

2. UNIX for Advanced & Expert Users

SCP File Transfer

I have 3 AIX server namely - Server 1 , Server 2 and Server 3. And have done SCP setup between Server 1 and Server 2 so that i dont have to give password when i transfer file from Server 1 to Server 2 by setting public key between the server. Q1. If the unix password of the target server... (3 Replies)
Discussion started by: Pash
3 Replies

3. UNIX for Advanced & Expert Users

SCP - File transfer message

Whenever I transfer file through SCP between two server it gives below given message.....Is there a way to avoid it.... Target server : newyork $ scp ABC27801.iue newyork:./iABC/x0017801.iue Message ===== This system is for the use of authorized users only. Individuals using this... (1 Reply)
Discussion started by: Pash
1 Replies

4. UNIX for Advanced & Expert Users

perl script to transfer newly generated files by scp

Hi all, I have root directory on server 1 say A and having sub directory B now my application generates output files and put in sub directory B. now i need to transfer these files from server1 to server2 by scp which is having same directory structure A and sub directory B I have tried... (2 Replies)
Discussion started by: tushar_spatil
2 Replies

5. UNIX for Advanced & Expert Users

Transfer only modified files by SCP

Hi all, I want to transfer only modified files from one unix server to another unix sever ,tried hard but not succeeded please help me out to resolve it. The script is as below :wall: ssh user@hostname <<EOT >files.txt cd /x/y find . -mtime -0 -type f |\ while read file do echo $file... (8 Replies)
Discussion started by: tushar_spatil
8 Replies

6. UNIX for Advanced & Expert Users

scp command for file transfer

I am not able to throw a file from server173 to server067 i.e. wlsuser@server173> scp /tmp/harsha.txt wlsuser@server067:/tmp fails However, I am able to pull a file from server173 onto server067's /tmp dir wlsuser@server067> scp wlsuser@server173:/tmp/harsha.txt /tmp... (2 Replies)
Discussion started by: shifahim
2 Replies

7. Shell Programming and Scripting

SCP During Transfer to SunOS

Hi, I'd like to seek your help in finding the cause of this problem. I utilize SCP command to transfer files from one server to another. I utilize the return code that SCP return so as to verify if transfer is ok or not. Below is a sample code that I created. if scp... (9 Replies)
Discussion started by: padi
9 Replies

8. Shell Programming and Scripting

Need Header for all splitted files - awk

Input file: i have a file and need to split into multiple files based on first column. i need the header for all the splitted files. I'm unable to get the header. $ cat log.txt id,mailtype,value 1252468812,yahoo,3.5 1252468812,hotmail,2.4 1252468819,yahoo,1.2 1252468812,msn,8.9... (6 Replies)
Discussion started by: mannefromdetroi
6 Replies

9. UNIX for Dummies Questions & Answers

File transfer using SCP

I have a shell script which uses SCP command to transfer the files from one server to another server. The files are getting transferred successfully, but the problem is the files transferred to the destination server didnot have the permissions as that of the files on the source server. Command... (5 Replies)
Discussion started by: kumarm
5 Replies

10. Shell Programming and Scripting

file transfer using scp..

Hi Frdz I have a problem like. I need to transfer a file from source to destination (different systems with different IPs) using "scp" command and before transfer the file i have to check the file is available in destination or not, if it is there no need to transfer, otherwise we have to... (5 Replies)
Discussion started by: KiranKumarKarre
5 Replies

Featured Tech Videos