Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

How can we overcome Broken pipe error during scp,SFTP,Rsync while transferring big files.?

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting How can we overcome Broken pipe error during scp,SFTP,Rsync while transferring big files.?
# 1  
Old 09-14-2015
How can we overcome Broken pipe error during scp,SFTP,Rsync while transferring big files.?
Hello All,

Hope all are doing well.

We use scp (some times sftp and rsync also) for transferring big files (around 2GB each ) from 1 Network to another Network.
The Issues which we face :-

During transfer some times( Once in 1 week (or twice)) , the speed of transfer gets down to 30 kb/s, which is usually 500kb/s during normal days.And most times when speed become low after few min. it failed with BROKEN PIPE ERROR.

Is there any way we can transfer files by avoiding above error
OR
any way in unix we can transfer the file(usually jars ) by breaking them into parts
OR is there any way ,if we can continue the file copy from the same point where it failed last time.

Many Thanks in Advance.
# 2  
Old 09-14-2015
scp shows average speeds, which makes me suspect the upload is not actually slow during these times but completely stalled.

As for how to solve this it depends what's causing it. Some sort of intermittent network issue is likely the culprit.
This User Gave Thanks to Corona688 For This Post:
jim mcnamara
# 3  
Old 09-14-2015
corona is right - the network needs fixing, not the process.

jfsplit - tool to split jar files into smaller files, and join small files back into a single large file:
JFSplit - File Splitter and Joiner

Use the split command to chop text and data files into smaller pieces, cat command to join them back together.
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

SFTP error between UNIX to iSeries (Write Failed Broken Pipe)

Hi, I am iseries resource and having Issue in one of the sFTP failures between one of my job (Unix --> iSeries). The response to sending machine (Unix) is "Write Failed Broken Pipe". Appreciate any help Available on why this Issue happens, how can we replicate the same, what fixes can... (1 Reply)
Discussion started by: hamelchauhan
1 Replies

2. UNIX for Dummies Questions & Answers

broken pipe error

I'm new to scripting, and this forum has been invaluable in helping me out. I'm hoping I can get some personal help now though. I have a korn script that takes a list of servers and either telnets or sshs into it (only some are set up for ssh). What I'm doing now is trying to telnet first, and... (10 Replies)
Discussion started by: aimeet
10 Replies

3. Shell Programming and Scripting

Broken Pipe error

Hello while doing sftp over server "A" , i am getting a broken pipe error i.e cat: write error: Broken pipe what does that mean? please let me know if you want any other info on this.. (3 Replies)
Discussion started by: urfrnddpk
3 Replies

4. UNIX for Advanced & Expert Users

sftp error:- Couldn't send packet: Broken pipe

Hi, I am trying to sftp a large file(appx. 2 GB size) from solaris 10 to a Linux server. While sftping(with no options), connection is terminated after some time with broken pipe error. sftp with option -R1 or -B1500 is successful. Linux server ssh Version: OpenSSH_3.9p1, OpenSSL 0.9.7a... (3 Replies)
Discussion started by: brij123
3 Replies

5. Shell Programming and Scripting

Error copying files from Unix (Solaris10) to Windows Server 2003 using scp/sftp

Hi, I have generated a Public/Private Key Pair in Solaris Unix (source) server and deployed the Public key in Windows 2003(target) server .ssh directory of user profile. When i try to connect(ssh, scp, sftp) from Unix, i'm getting below error message. Sun_SSH_1.1, SSH protocols 1.5/2.0,... (0 Replies)
Discussion started by: ajaykumarb
0 Replies

6. Programming

Broken Pipe error

All, I am using the below code The C code : if ((fp2=fopen(szout_fname,"r"))==NULL) { sprintf(stream_ptr1,"cat %s | sort -t, -rn -k 11,11 | awk -F\",\" '{ \ if ( \$3 ==\"%s\" ) {print... (0 Replies)
Discussion started by: arunkumar_mca
0 Replies

7. Emergency UNIX and Linux Support

rsync transferring multiple files issue

Hi, I want to specify multiple remote directories but want to transfer them in a single command with one connection with remote server. This avoids entering passwords repeatedly and is also efficient. e.g. rsync -vrt --size-only --delete user@host:/home/user1/dir1... (9 Replies)
Discussion started by: sardare
9 Replies

8. AIX

broken pipe error

Hi, I am working on AIX 5.3 . I have client-server program which is in ProC.while sending packet to server i am getting error as broken pipe and program exiting. please help?/? (1 Reply)
Discussion started by: ajaysahoo
1 Replies

9. UNIX for Dummies Questions & Answers

Transferring hidden files with scp

Is there a way to scp hidden files only ( with regex ) without specifying the name of the hidden files ? Many thanks in advance ! :) (13 Replies)
Discussion started by: matrixmadhan
13 Replies

10. Shell Programming and Scripting

Error transferring large file using sftp protocol

Hello All, I am not able to transfer large file completely (more than 60 MB) through automated SFTP script. It transfers only the partial file (approximately 55 MB) and it stops. I use "expect" command to transfer file in a automatic mode through script. The file transfer rate i get is around... (0 Replies)
Discussion started by: prashant.patel
0 Replies
Login or Register to Ask a Question