Sponsored Content
Top Forums UNIX for Dummies Questions & Answers Copying a list of files from one 'server' to another? Post 302234861 by awolbach on Wednesday 10th of September 2008 02:44:48 PM
Old 09-10-2008
use symlinks

Quote:
Originally Posted by cfajohnson

What shell are you using? If it's bash or ksh put the list of files in an array, e.g.:

Code:
filelist=( "file 1" "file 2" etc. )

scp "${filelist[@]}" remote.server:path

Or use the list directly on the command line:

Code:
scp "file 1" "file 2" etc. remote.server:path

In these situations you run the risk of exhausting the maximum length of arguments to a process. I ran into this problem today when I needed to randomly divide two massive sets of files evenly between two servers. Create a new directory for each server and fill it with symlinks to the files you wish to copy to that server. If it's a lot of files, automate it by creating a script:

Code:
cat file_list | sed 's/^.*$/ln -s & /' > symlink_script.sh

Execute the script inside of your new directory. Then, just copy that directory over to the target server, and scp will follow the symlinks.
 

10 More Discussions You Might Find Interesting

1. UNIX for Advanced & Expert Users

Copying files between 2 Unix server

Is there a simple way to copy data from one server to a different server? Seems that if 2 servers are on the same network, there should be a simple way to copy between the two. Not just one file. I need to copy a whole directory with subdirectories from one server to a different one. I... (9 Replies)
Discussion started by: Docboyeee
9 Replies

2. UNIX for Dummies Questions & Answers

copying files from one location to another based on a list

I have a text list of about 3,000 file names (image files), which exist on a server and that I want to copy over to another location. I understand the Unix cp code, but what's the string to have it copy multiple files based on an external list? Many thanks! (4 Replies)
Discussion started by: rebornhonest
4 Replies

3. Shell Programming and Scripting

copying a list of files from one machine to another with the same path

Hello All, I would like to download a list of files from a remote server to my local machine preserving the path i.e, create a similar directory tree. ex: files on the remote machine with path /remoteserver/005_S_0221/mpr/m1/I72128/dat/06A515T.dat... (3 Replies)
Discussion started by: avatar_007
3 Replies

4. Shell Programming and Scripting

Copying files from one server to other.

Any one advice here- need to copy some files from one server to other server thru shell programming. (8 Replies)
Discussion started by: samkhu
8 Replies

5. UNIX for Dummies Questions & Answers

Copying files from different server

Is there any option to copy files from a different server other than using an SCP command??? thanks,:b: Arun manas (2 Replies)
Discussion started by: arunmanas
2 Replies

6. Shell Programming and Scripting

Copying List of Files Under Different File Names ?!

Hi to all here , Excuse me for the lamer question but I need UNIX command for copying List of Files Under Different File Names ! for example I have this list: new_001.jpg new_002.jpg new_003.jpg ..... I want to copy all files that start with "new_" to the same directory but under... (7 Replies)
Discussion started by: boboweb_
7 Replies

7. UNIX for Dummies Questions & Answers

Copying files from one server to another

I need to copy huge set of data(files & floder) from one server to other. Can any one guide me.I have tried throught ftp mget but it is vain...floder are not moving...... Do suggest me any good method (12 Replies)
Discussion started by: kkalyan
12 Replies

8. Shell Programming and Scripting

Copying the files to Windows server from UNIX server

Hi Team, I had a requirement to write a shell script which automatically transfer the files from unix server to windows server. I can able to unix to unix using Scp command. I am not sure how to do unix to windows. I am very new on this concept. Could you please help me or guide in... (4 Replies)
Discussion started by: gvkumar25
4 Replies

9. Shell Programming and Scripting

Copying files to remote server

I am trying to copy code to remote server, doing something like this However it is copying one file and than the script exits i=0; j=0 while read fn; do dir=${fn%/*} # Gets directory path fnm=${fn##*/} # Gets filename excl. path rgx_nwk="s/${nwk}/${nwk}.sac/g" ... (2 Replies)
Discussion started by: kristinu
2 Replies

10. UNIX for Beginners Questions & Answers

List all files and copying to different location

Hello All, I want to copy all files from the list below to a path /qa0/repo/AS/CPY (lnx17:qa0:/qa0/repo/AS>) ls -lrt | grep 'Feb 14' -rw-rw-r-- 1 Ks1 qa0 39 Feb 14 20:11 HHH.dat -rw-rw-r-- 1 Ks1 qa0 32 Feb 14 20:11 HHH1.dat -rw-rw-r-- 1 Ks1 qa0 29 Feb 14 20:11... (3 Replies)
Discussion started by: arunkumar_mca
3 Replies
SFTP(1) 						    BSD General Commands Manual 						   SFTP(1)

NAME
sftp -- Secure file transfer program SYNOPSIS
sftp [-vC1] [-b batchfile] [-o ssh_option] [-s subsystem | sftp_server] [-B buffer_size] [-F ssh_config] [-P sftp_server path] [-R num_requests] [-S program] host sftp [[user@]host[:file [file]]] sftp [[user@]host[:dir[/]]] DESCRIPTION
sftp is an interactive file transfer program, similar to ftp(1), which performs all operations over an encrypted ssh(1) transport. It may also use many features of ssh, such as public key authentication and compression. sftp connects and logs into the specified host, then enters an interactive command mode. The second usage format will retrieve files automatically if a non-interactive authentication method is used; otherwise it will do so after successful interactive authentication. The last usage format allows the sftp client to start in a remote directory. The options are as follows: -b batchfile Batch mode reads a series of commands from an input batchfile instead of stdin. Since it lacks user interaction it should be used in conjunction with non-interactive authentication. sftp will abort if any of the following commands fail: get, put, rename, ln, rm, mkdir, chdir, lchdir and lmkdir. -o ssh_option Can be used to pass options to ssh in the format used in ssh_config(5). This is useful for specifying options for which there is no separate sftp command-line flag. For example, to specify an alternate port use: sftp -oPort=24. -s subsystem | sftp_server Specifies the SSH2 subsystem or the path for an sftp server on the remote host. A path is useful for using sftp over protocol ver- sion 1, or when the remote sshd does not have an sftp subsystem configured. -v Raise logging level. This option is also passed to ssh. -B buffer_size Specify the size of the buffer that sftp uses when transferring files. Larger buffers require fewer round trips at the cost of higher memory consumption. The default is 32768 bytes. -C Enables compression (via ssh's -C flag). -F ssh_config Specifies an alternative per-user configuration file for ssh. This option is directly passed to ssh(1). -P sftp_server path Connect directly to a local sftp-server (rather than via ssh) This option may be useful in debugging the client and server. -R num_requests Specify how many requests may be outstanding at any one time. Increasing this may slightly improve file transfer speed but will increase memory usage. The default is 16 outstanding requests. -S program Name of the program to use for the encrypted connection. The program must understand ssh(1) options. -1 Specify the use of protocol version 1. INTERACTIVE COMMANDS
Once in interactive mode, sftp understands a set of commands similar to those of ftp(1). Commands are case insensitive and pathnames may be enclosed in quotes if they contain spaces. bye Quit sftp. cd path Change remote directory to path. lcd path Change local directory to path. chgrp grp path Change group of file path to grp. grp must be a numeric GID. chmod mode path Change permissions of file path to mode. chown own path Change owner of file path to own. own must be a numeric UID. exit Quit sftp. get [flags] remote-path [local-path] Retrieve the remote-path and store it on the local machine. If the local path name is not specified, it is given the same name it has on the remote machine. If the -P flag is specified, then the file's full permission and access time are copied too. help Display help text. lls [ls-options [path]] Display local directory listing of either path or current directory if path is not specified. lmkdir path Create local directory specified by path. ln oldpath newpath Create a symbolic link from oldpath to newpath. lpwd Print local working directory. ls [flags] [path] Display remote directory listing of either path or current directory if path is not specified. If the -l flag is specified, then dis- play additional details including permissions and ownership information. lumask umask Set local umask to umask. mkdir path Create remote directory specified by path. put [flags] local-path [local-path] Upload local-path and store it on the remote machine. If the remote path name is not specified, it is given the same name it has on the local machine. If the -P flag is specified, then the file's full permission and access time are copied too. pwd Display remote working directory. quit Quit sftp. rename oldpath newpath Rename remote file from oldpath to newpath. rmdir path Remove remote directory specified by path. rm path Delete remote file specified by path. symlink oldpath newpath Create a symbolic link from oldpath to newpath. ! command Execute command in local shell. ! Escape to local shell. ? Synonym for help. AUTHORS
Damien Miller <djm@mindrot.org> SEE ALSO
scp(1), ssh(1), ssh-add(1), ssh-keygen(1), ssh_config(5), sftp-server(8), sshd(8) T. Ylonen and S. Lehtinen, SSH File Transfer Protocol, draft-ietf-secsh-filexfer-00.txt, January 2001, work in progress material. BSD
February 4, 2001 BSD
All times are GMT -4. The time now is 10:14 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy