04-10-2008
I'm missing some of the bigger-picture elements
Quote:
Originally Posted by
vijaykrc
can i delete each file when the copy is done?
If i use SCP by looping i know it can be done but it takes lot of time if i have more files. If i use the * to transfer multiple files it goes in quickly as it dont have to connect every time to the server.
Well, are you generating all the files once, then moving all the files, then deleting all the files, with nothing touching the directory during the transfer? If so, then if your scp /$sourcepath/* has finished with exit code 0, it's reporting success, so the files got to the target machine. As long as nothing has generated more source files, then you could rm /$sourcepath/* and be happy, or depending on your available space, tar them and save them for a week.
How much time are you losing during the connection process? Aren't you ultimately going to automate this and just drink coffee while the move script runs from crontab and then sends you an email of the log when it's done? Is the time you lose actually a critical factor?
If you're having to move, daily, a million tiny files from server A to server B, something else is wrong with that picture. Would server B accept a tar.gz of each group of files you want to send?
I am sure Era can find several other angles to improve this. A little more information about the timeline of file creation might be useful.
10 More Discussions You Might Find Interesting
1. Shell Programming and Scripting
I need to transfer multiple files using SCP between two solaris machines. Can somebody explain how to achieve that ? (3 Replies)
Discussion started by: parthum
3 Replies
2. Shell Programming and Scripting
Hi Frdz
I have a problem like.
I need to transfer a file from source to destination (different systems with different IPs) using "scp" command and before transfer the file i have to check the file is available in destination or not, if it is there no need to transfer, otherwise we have to... (5 Replies)
Discussion started by: KiranKumarKarre
5 Replies
3. UNIX for Dummies Questions & Answers
I have a shell script which uses SCP command to transfer the files from one server to another server. The files are getting transferred successfully, but the problem is the files transferred to the destination server didnot have the permissions as that of the files on the source server.
Command... (5 Replies)
Discussion started by: kumarm
5 Replies
4. UNIX for Advanced & Expert Users
I am not able to throw a file from server173 to server067
i.e.
wlsuser@server173> scp /tmp/harsha.txt wlsuser@server067:/tmp
fails
However, I am able to pull a file from server173 onto server067's /tmp dir
wlsuser@server067> scp wlsuser@server173:/tmp/harsha.txt /tmp... (2 Replies)
Discussion started by: shifahim
2 Replies
5. UNIX for Advanced & Expert Users
Whenever I transfer file through SCP between two server it gives below given message.....Is there a way to avoid it....
Target server : newyork
$ scp ABC27801.iue newyork:./iABC/x0017801.iue
Message
=====
This system is for the use of authorized users only. Individuals using this... (1 Reply)
Discussion started by: Pash
1 Replies
6. UNIX for Advanced & Expert Users
I have 3 AIX server namely - Server 1 , Server 2 and Server 3.
And have done SCP setup between Server 1 and Server 2 so that i dont have to give password when i transfer file from Server 1 to Server 2 by setting public key between the server.
Q1. If the unix password of the target server... (3 Replies)
Discussion started by: Pash
3 Replies
7. UNIX for Advanced & Expert Users
On unix AIX server, when I am trying to transfer file from one directory to another directory on the same server through a program(where i call the script) it gives error "Lost Connection". (5 Replies)
Discussion started by: Pash
5 Replies
8. AIX
I am testing the following command to transfer a file from my server (AIX 5.2) to another server. I was able to generate the keys and sent them the public key.
scp -v -P 4030 /home/lawson/.ssh/jimtest.txt someuser@some.ftpsite.net:/Inbound/jimtest.txt > jimtest_out.txt 2>&1
Based on... (3 Replies)
Discussion started by: jyoung
3 Replies
9. Shell Programming and Scripting
I am trying to transfer a zip file of around 30 MB in my automation script using scp from system A to B.
When I manually do scp, file is complete transferred but when automation shell script runs it, zip file is not completely transferred.
Stack Trace while doing manual :
Executing:... (1 Reply)
Discussion started by: Shaishav Shah
1 Replies
10. UNIX for Beginners Questions & Answers
Hi All,
Need some help regarding file transfer between server.
Suppose we have system-A and system-B. To transfer file from system-A to system-B we usually share the
public keys of system-A to system-B and do scp/sftp to transfer a file.
Is it possible that public key of system-B can be... (3 Replies)
Discussion started by: abhi_123
3 Replies
LEARN ABOUT DEBIAN
parallel-scp
PARALLEL-SCP(1) PARALLEL-SCP(1)
NAME
parallel-scp - parallel versions of scp
SYNOPSIS
parallel-scp [OPTIONS] -h hosts.txt local remote
DESCRIPTION
pssh provides a number of commands for executing against a group of computers, using SSH. It's most useful for operating on clusters of
homogenously-configured hosts.
parallel-scp copy files in parallel to a set of machines.
OPTIONS
-r --recursive
recusively copy directories (OPTIONAL)
-h --hosts
hosts file (each line "host[:port] [user]")
-l --user
username (OPTIONAL)
-p --par
max number of parallel threads (OPTIONAL)
-o --outdir
output directory for stdout files (OPTIONAL)
-e --errdir
output directory for stderr files (OPTIONAL)
-t --timeout
timeout (secs) (-1 = no timeout) per host (OPTIONAL)
-O --options
SSH options (OPTIONAL)
-v --verbose
turn on warning and diagnostic messages (OPTIONAL)
EXAMPLE
The following example runs hostname on three machines (IPs or hostnames) specified in the file ips.txt using login irb2 and saves the
output in /tmp/foo.
# parallel-scp -h ips.txt -l irb2 /etc/hosts /tmp/hosts
Success on 128.112.152.122:22
Success on 18.31.0.190:22
Success on 128.232.103.201:22
ENVIRONMENT
All four programs take similar sets of options. All of these options can be set using the following environment variables:
o PSSH_HOSTS
o PSSH_USER
o PSSH_PAR
o PSSH_OUTDIR
o PSSH_VERBOSE
o PSSH_OPTIONS
SEE ALSO
parallel-ssh(1), parallel-slurp(1), parallel-nuke(1), parallel-rsync(1), ssh(1), scp(1),
AUTHOR
Brent N. Chun <bnc@theether.org>
COPYING
Copyright: 2003, 2004, 2005, 2006, 2007 Brent N. Chun
NOTES
1. bnc@theether.org
mailto:bnc@theether.org
03/30/2009 PARALLEL-SCP(1)