Sponsored Content
Top Forums Shell Programming and Scripting How can we overcome Broken pipe error during scp,SFTP,Rsync while transferring big files.? Post 302954977 by Upendra Bhushan on Monday 14th of September 2015 03:33:00 AM
Old 09-14-2015
How can we overcome Broken pipe error during scp,SFTP,Rsync while transferring big files.?

Hello All,

Hope all are doing well.

We use scp (some times sftp and rsync also) for transferring big files (around 2GB each ) from 1 Network to another Network.
The Issues which we face :-

During transfer some times( Once in 1 week (or twice)) , the speed of transfer gets down to 30 kb/s, which is usually 500kb/s during normal days.And most times when speed become low after few min. it failed with BROKEN PIPE ERROR.

Is there any way we can transfer files by avoiding above error
OR
any way in unix we can transfer the file(usually jars ) by breaking them into parts
OR is there any way ,if we can continue the file copy from the same point where it failed last time.

Many Thanks in Advance.
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Error transferring large file using sftp protocol

Hello All, I am not able to transfer large file completely (more than 60 MB) through automated SFTP script. It transfers only the partial file (approximately 55 MB) and it stops. I use "expect" command to transfer file in a automatic mode through script. The file transfer rate i get is around... (0 Replies)
Discussion started by: prashant.patel
0 Replies

2. UNIX for Dummies Questions & Answers

Transferring hidden files with scp

Is there a way to scp hidden files only ( with regex ) without specifying the name of the hidden files ? Many thanks in advance ! :) (13 Replies)
Discussion started by: matrixmadhan
13 Replies

3. AIX

broken pipe error

Hi, I am working on AIX 5.3 . I have client-server program which is in ProC.while sending packet to server i am getting error as broken pipe and program exiting. please help?/? (1 Reply)
Discussion started by: ajaysahoo
1 Replies

4. Emergency UNIX and Linux Support

rsync transferring multiple files issue

Hi, I want to specify multiple remote directories but want to transfer them in a single command with one connection with remote server. This avoids entering passwords repeatedly and is also efficient. e.g. rsync -vrt --size-only --delete user@host:/home/user1/dir1... (9 Replies)
Discussion started by: sardare
9 Replies

5. Programming

Broken Pipe error

All, I am using the below code The C code : if ((fp2=fopen(szout_fname,"r"))==NULL) { sprintf(stream_ptr1,"cat %s | sort -t, -rn -k 11,11 | awk -F\",\" '{ \ if ( \$3 ==\"%s\" ) {print... (0 Replies)
Discussion started by: arunkumar_mca
0 Replies

6. Shell Programming and Scripting

Error copying files from Unix (Solaris10) to Windows Server 2003 using scp/sftp

Hi, I have generated a Public/Private Key Pair in Solaris Unix (source) server and deployed the Public key in Windows 2003(target) server .ssh directory of user profile. When i try to connect(ssh, scp, sftp) from Unix, i'm getting below error message. Sun_SSH_1.1, SSH protocols 1.5/2.0,... (0 Replies)
Discussion started by: ajaykumarb
0 Replies

7. UNIX for Advanced & Expert Users

sftp error:- Couldn't send packet: Broken pipe

Hi, I am trying to sftp a large file(appx. 2 GB size) from solaris 10 to a Linux server. While sftping(with no options), connection is terminated after some time with broken pipe error. sftp with option -R1 or -B1500 is successful. Linux server ssh Version: OpenSSH_3.9p1, OpenSSL 0.9.7a... (3 Replies)
Discussion started by: brij123
3 Replies

8. Shell Programming and Scripting

Broken Pipe error

Hello while doing sftp over server "A" , i am getting a broken pipe error i.e cat: write error: Broken pipe what does that mean? please let me know if you want any other info on this.. (3 Replies)
Discussion started by: urfrnddpk
3 Replies

9. UNIX for Dummies Questions & Answers

broken pipe error

I'm new to scripting, and this forum has been invaluable in helping me out. I'm hoping I can get some personal help now though. I have a korn script that takes a list of servers and either telnets or sshs into it (only some are set up for ssh). What I'm doing now is trying to telnet first, and... (10 Replies)
Discussion started by: aimeet
10 Replies

10. UNIX for Dummies Questions & Answers

SFTP error between UNIX to iSeries (Write Failed Broken Pipe)

Hi, I am iseries resource and having Issue in one of the sFTP failures between one of my job (Unix --> iSeries). The response to sending machine (Unix) is "Write Failed Broken Pipe". Appreciate any help Available on why this Issue happens, how can we replicate the same, what fixes can... (1 Reply)
Discussion started by: hamelchauhan
1 Replies
SFTP(1) 						    BSD General Commands Manual 						   SFTP(1)

NAME
sftp -- Secure file transfer program SYNOPSIS
sftp [-vC1] [-b batchfile] [-o ssh_option] [-s subsystem | sftp_server] [-B buffer_size] [-F ssh_config] [-P sftp_server path] [-R num_requests] [-S program] host sftp [[user@]host[:file [file]]] sftp [[user@]host[:dir[/]]] DESCRIPTION
sftp is an interactive file transfer program, similar to ftp(1), which performs all operations over an encrypted ssh(1) transport. It may also use many features of ssh, such as public key authentication and compression. sftp connects and logs into the specified host, then enters an interactive command mode. The second usage format will retrieve files automatically if a non-interactive authentication method is used; otherwise it will do so after successful interactive authentication. The last usage format allows the sftp client to start in a remote directory. The options are as follows: -b batchfile Batch mode reads a series of commands from an input batchfile instead of stdin. Since it lacks user interaction it should be used in conjunction with non-interactive authentication. sftp will abort if any of the following commands fail: get, put, rename, ln, rm, mkdir, chdir, lchdir and lmkdir. -o ssh_option Can be used to pass options to ssh in the format used in ssh_config(5). This is useful for specifying options for which there is no separate sftp command-line flag. For example, to specify an alternate port use: sftp -oPort=24. -s subsystem | sftp_server Specifies the SSH2 subsystem or the path for an sftp server on the remote host. A path is useful for using sftp over protocol ver- sion 1, or when the remote sshd does not have an sftp subsystem configured. -v Raise logging level. This option is also passed to ssh. -B buffer_size Specify the size of the buffer that sftp uses when transferring files. Larger buffers require fewer round trips at the cost of higher memory consumption. The default is 32768 bytes. -C Enables compression (via ssh's -C flag). -F ssh_config Specifies an alternative per-user configuration file for ssh. This option is directly passed to ssh(1). -P sftp_server path Connect directly to a local sftp-server (rather than via ssh) This option may be useful in debugging the client and server. -R num_requests Specify how many requests may be outstanding at any one time. Increasing this may slightly improve file transfer speed but will increase memory usage. The default is 16 outstanding requests. -S program Name of the program to use for the encrypted connection. The program must understand ssh(1) options. -1 Specify the use of protocol version 1. INTERACTIVE COMMANDS
Once in interactive mode, sftp understands a set of commands similar to those of ftp(1). Commands are case insensitive and pathnames may be enclosed in quotes if they contain spaces. bye Quit sftp. cd path Change remote directory to path. lcd path Change local directory to path. chgrp grp path Change group of file path to grp. grp must be a numeric GID. chmod mode path Change permissions of file path to mode. chown own path Change owner of file path to own. own must be a numeric UID. exit Quit sftp. get [flags] remote-path [local-path] Retrieve the remote-path and store it on the local machine. If the local path name is not specified, it is given the same name it has on the remote machine. If the -P flag is specified, then the file's full permission and access time are copied too. help Display help text. lls [ls-options [path]] Display local directory listing of either path or current directory if path is not specified. lmkdir path Create local directory specified by path. ln oldpath newpath Create a symbolic link from oldpath to newpath. lpwd Print local working directory. ls [flags] [path] Display remote directory listing of either path or current directory if path is not specified. If the -l flag is specified, then dis- play additional details including permissions and ownership information. lumask umask Set local umask to umask. mkdir path Create remote directory specified by path. put [flags] local-path [local-path] Upload local-path and store it on the remote machine. If the remote path name is not specified, it is given the same name it has on the local machine. If the -P flag is specified, then the file's full permission and access time are copied too. pwd Display remote working directory. quit Quit sftp. rename oldpath newpath Rename remote file from oldpath to newpath. rmdir path Remove remote directory specified by path. rm path Delete remote file specified by path. symlink oldpath newpath Create a symbolic link from oldpath to newpath. ! command Execute command in local shell. ! Escape to local shell. ? Synonym for help. AUTHORS
Damien Miller <djm@mindrot.org> SEE ALSO
scp(1), ssh(1), ssh-add(1), ssh-keygen(1), ssh_config(5), sftp-server(8), sshd(8) T. Ylonen and S. Lehtinen, SSH File Transfer Protocol, draft-ietf-secsh-filexfer-00.txt, January 2001, work in progress material. BSD
February 4, 2001 BSD
All times are GMT -4. The time now is 10:13 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy