Login or Register to Ask a Question and Join Our Community

Sponsored Content
Full Discussion: Copying a list of files from one 'server' to another?
Top Forums UNIX for Dummies Questions & Answers Copying a list of files from one 'server' to another? Post 302234861 by awolbach on Wednesday 10th of September 2008 02:44:48 PM
Old 09-10-2008
use symlinks
Quote:
Originally Posted by cfajohnson

What shell are you using? If it's bash or ksh put the list of files in an array, e.g.:

Code:
filelist=( "file 1" "file 2" etc. )

scp "${filelist[@]}" remote.server:path

Or use the list directly on the command line:

Code:
scp "file 1" "file 2" etc. remote.server:path

In these situations you run the risk of exhausting the maximum length of arguments to a process. I ran into this problem today when I needed to randomly divide two massive sets of files evenly between two servers. Create a new directory for each server and fill it with symlinks to the files you wish to copy to that server. If it's a lot of files, automate it by creating a script:

Code:
cat file_list | sed 's/^.*$/ln -s & /' > symlink_script.sh

Execute the script inside of your new directory. Then, just copy that directory over to the target server, and scp will follow the symlinks.
 

10 More Discussions You Might Find Interesting

1. UNIX for Advanced & Expert Users

Copying files between 2 Unix server

Is there a simple way to copy data from one server to a different server? Seems that if 2 servers are on the same network, there should be a simple way to copy between the two. Not just one file. I need to copy a whole directory with subdirectories from one server to a different one. I... (9 Replies)
Discussion started by: Docboyeee
9 Replies

2. UNIX for Dummies Questions & Answers

copying files from one location to another based on a list

I have a text list of about 3,000 file names (image files), which exist on a server and that I want to copy over to another location. I understand the Unix cp code, but what's the string to have it copy multiple files based on an external list? Many thanks! (4 Replies)
Discussion started by: rebornhonest
4 Replies

3. Shell Programming and Scripting

copying a list of files from one machine to another with the same path

Hello All, I would like to download a list of files from a remote server to my local machine preserving the path i.e, create a similar directory tree. ex: files on the remote machine with path /remoteserver/005_S_0221/mpr/m1/I72128/dat/06A515T.dat... (3 Replies)
Discussion started by: avatar_007
3 Replies

4. Shell Programming and Scripting

Copying files from one server to other.

Any one advice here- need to copy some files from one server to other server thru shell programming. (8 Replies)
Discussion started by: samkhu
8 Replies

5. UNIX for Dummies Questions & Answers

Copying files from different server

Is there any option to copy files from a different server other than using an SCP command??? thanks,:b: Arun manas (2 Replies)
Discussion started by: arunmanas
2 Replies

6. Shell Programming and Scripting

Copying List of Files Under Different File Names ?!

Hi to all here , Excuse me for the lamer question but I need UNIX command for copying List of Files Under Different File Names ! for example I have this list: new_001.jpg new_002.jpg new_003.jpg ..... I want to copy all files that start with "new_" to the same directory but under... (7 Replies)
Discussion started by: boboweb_
7 Replies

7. UNIX for Dummies Questions & Answers

Copying files from one server to another

I need to copy huge set of data(files & floder) from one server to other. Can any one guide me.I have tried throught ftp mget but it is vain...floder are not moving...... Do suggest me any good method (12 Replies)
Discussion started by: kkalyan
12 Replies

8. Shell Programming and Scripting

Copying the files to Windows server from UNIX server

Hi Team, I had a requirement to write a shell script which automatically transfer the files from unix server to windows server. I can able to unix to unix using Scp command. I am not sure how to do unix to windows. I am very new on this concept. Could you please help me or guide in... (4 Replies)
Discussion started by: gvkumar25
4 Replies

9. Shell Programming and Scripting

Copying files to remote server

I am trying to copy code to remote server, doing something like this However it is copying one file and than the script exits i=0; j=0 while read fn; do dir=${fn%/*} # Gets directory path fnm=${fn##*/} # Gets filename excl. path rgx_nwk="s/${nwk}/${nwk}.sac/g" ... (2 Replies)
Discussion started by: kristinu
2 Replies

10. UNIX for Beginners Questions & Answers

List all files and copying to different location

Hello All, I want to copy all files from the list below to a path /qa0/repo/AS/CPY (lnx17:qa0:/qa0/repo/AS>) ls -lrt | grep 'Feb 14' -rw-rw-r-- 1 Ks1 qa0 39 Feb 14 20:11 HHH.dat -rw-rw-r-- 1 Ks1 qa0 32 Feb 14 20:11 HHH1.dat -rw-rw-r-- 1 Ks1 qa0 29 Feb 14 20:11... (3 Replies)
Discussion started by: arunkumar_mca
3 Replies

LEARN ABOUT CENTOS

dh_compress

DH_COMPRESS(1)							     Debhelper							    DH_COMPRESS(1)

NAME

       dh_compress - compress files and fix symlinks in package build directories

SYNOPSIS

       dh_compress [debhelperoptions] [-Xitem] [-A] [file...]

DESCRIPTION

       dh_compress is a debhelper program that is responsible for compressing the files in package build directories, and makes sure that any
       symlinks that pointed to the files before they were compressed are updated to point to the new files.

       By default, dh_compress compresses files that Debian policy mandates should be compressed, namely all files in usr/share/info,
       usr/share/man, files in usr/share/doc that are larger than 4k in size, (except the copyright file, .html and other web files, image files,
       and files that appear to be already compressed based on their extensions), and all changelog files. Plus PCF fonts underneath
       usr/share/fonts/X11/

FILES

       debian/package.compress
	   These files are deprecated.

	   If this file exists, the default files are not compressed. Instead, the file is ran as a shell script, and all filenames that the shell
	   script outputs will be compressed. The shell script will be run from inside the package build directory. Note though that using -X is a
	   much better idea in general; you should only use a debian/package.compress file if you really need to.

OPTIONS

       -Xitem, --exclude=item
	   Exclude files that contain item anywhere in their filename from being compressed. For example, -X.tiff will exclude TIFF files from
	   compression.  You may use this option multiple times to build up a list of things to exclude.

       -A, --all
	   Compress all files specified by command line parameters in ALL packages acted on.

       file ...
	   Add these files to the list of files to compress.

CONFORMS TO

       Debian policy, version 3.0

SEE ALSO

       debhelper(7)

       This program is a part of debhelper.

AUTHOR

       Joey Hess <joeyh@debian.org>

11.1.6ubuntu2							    2018-05-10							    DH_COMPRESS(1)
All times are GMT -4. The time now is 04:57 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy