Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

ftp multiple files at the same time

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting ftp multiple files at the same time
# 1  
Old 08-31-2011
Question ftp multiple files at the same time
Hi All,

I am using ncftpput in one of my c-shell script to upload files to a remote location. The source files keep coming continuosly and to upload each file ncftpput opens a new connection everytime. It means ncftp uploads the file1 than file2 than file3 .... There is gap 20-25 secs between each of the files and some times when its a peak time the backlog is getting too much high approx 100 source file Smilie
I have seen a project called filezilla, where you have the option to choose multiple files and its starts processing almost 2-3 files at a time. Its a great time saving tool, but I am unable to find that for AIX and there is no command line like ncftp Smilie.
Could any one among you may able to give me a hand please ?

Regards - Sraj
# 2  
Old 08-31-2011
Looking at ncftp's man page, it seems to support what you want already.

Code:
ncftpput [options] remote-host remote-directory local-files...

See the plural.

As for how to incorporate that into your script, that depends on how your script works, can we see it?
# 3  
Old 09-01-2011
ftp multiple files at the same time
Hi Corno,
Here is the one line code with output....

Code:
ncftpput -A -u <usname> -p <pwd> <000.000.000.00> /remote_dir *

anandv.txt:                                             21.72 kB   10.79 kB/s
arorapoly.txt:                                          10.90 kB    5.37 kB/s
asian.txt:                                               2.85 kB    2.99 kB/s
asr4.txt:                                                4.62 kB    3.58 kB/s
avihar.txt:                                             25.67 kB    6.17 kB/s
beawer.txt:                                              3.97 kB    3.12 kB/s
bhatinda4.txt:                                           9.66 kB    5.43 kB/s
bikaner5.txt:                                            9.52 kB    5.13 kB/s
crpark.txt:                                             13.31 kB    7.49 kB/s
defence.txt:                                            24.14 kB   12.53 kB/s
dlf1.txt:                                                6.99 kB    4.62 kB/s
durgap.txt:                                             19.30 kB    9.75 kB/s
dwarka3.txt:                                            11.82 kB    7.38 kB/s
dwarka6.txt:                                           255.35 kB   65.93 kB/s
farid21.txt:                                             2.52 kB    2.62 kB/s
faridabad.txt:                                           5.20 kB    3.83 kB/s
faridkot2.txt:                                           8.95 kB    5.78 kB/s
farukh.txt:                                             12.12 kB    6.71 kB/s
ganga4.txt:                                              4.47 kB    3.45 kB/s
ggn2.txt:                                              145.24 kB   44.80 kB/s
ggncolumb.txt:                                           8.37 kB    6.55 kB/s
gnoida.txt:                                             10.67 kB    2.98 kB/s
greenp.txt:                                             20.84 kB    9.88 kB/s
hamirpur.txt:                                            2.05 kB    1.87 kB/s
hpur3.txt:                                              11.61 kB    6.11 kB/s
indira2.txt:                                            51.92 kB   22.35 kB/s
indore2.txt:                                             6.19 kB    1.74 kB/s
jabal4.txt:                                              9.60 kB    2.69 kB/s
jagraon.txt:                                            17.64 kB    9.22 kB/s
jangpurab.txt:                                          15.10 kB    8.65 kB/s
jlndr1.txt:                                              4.49 kB    3.21 kB/s
kakinada.txt:                      ETA:   0:00    4.00/ 10.52 kB  799.84 kB/s

ncftp is capable to upload one file at a time here. Could you help me to understand how it can make a thread.

Regards.
Last edited by radoulov; 09-01-2011 at 08:41 AM.. Reason: Code tags!
# 4  
Old 09-01-2011
Oh, you want to open dozens or hundreds of network connections simultaneously. Not usually a good idea. Too many connections can overwhelm a router, and the host you're uploading to might even consider it an attack.

Do you have ssh access to the machine? Can you upload an archive of some sort and unpack it there? Uploading one big file would avoid some overhead.
# 5  
Old 09-01-2011
Can You pack the files? Simple tar with no compression should be sufficient. Than You would have to send one file instead of "many".
Besides, most ftp configurations doesn't allow more that 2-5 connections for the same user/IP/whatever.
# 6  
Old 09-01-2011
Thanks both Sulti and Corona,

No, I don't wat want to open tons of connections, but 2-3 threads would definately be ok. in my script a while loop continusly monitor the source forder. When a file arrives in the folder the program hand ot over to ncftpput. So any mechanism to open at least 2-3 thread would be ok any how.

Any comments please ?

Regards

---------- Post updated at 05:04 PM ---------- Previous update was at 12:33 PM ----------

Is there any body else to know the solution ?

Regards
# 7  
Old 09-01-2011
You could invoke ncftpput as a background process and start multiple sessions within your while-loop.
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Ftp multiple files one at a time

Hi, I have a problem while ftp'ing zip files one after the other from linux source directory to a ftp host machine. here is the shell script: #!/bin/ksh dir=data/dir1/dir2 # this dir is linux source directory where zip files located. rmtdir='/home/' echo $dir for i in /$dir/*; do if ;... (7 Replies)
Discussion started by: raj78
7 Replies

2. Shell Programming and Scripting

Transfer multiple different files in FTP

Hi All, i am doing ftp how can i transfer multiple different files example- 03-21-13 06:33AM 46800 CATT_HOURS.pgp 03-21-13 06:33AM 266392 CATT_TAX.txt 03-21-13 06:33AM 91448 CATT_STATUS.txt 03-21-13 06:33AM 468 ... (3 Replies)
Discussion started by: krupasindhu18
3 Replies

3. Shell Programming and Scripting

FTP multiple files from multiple directories

I have multiple files that starts as TRADE_LOG spread across multiple folders in the given structure.. ./dir1/1/TRADE_LOG*.gz ./dir2/10/TRADE_LOG*.gz ./dir11/12/TRADE_LOG*.gz ./dir12/13/TRADE_LOG*.gz when I do ftp uisng mput from the "." dir I am getting the below given error mput... (1 Reply)
Discussion started by: prasperl
1 Replies

4. Shell Programming and Scripting

To remove multiple files in FTP

We have a files in FTP server..... after getting the files from FTP by mget *.* i hav to remove all files (multiple files) at once... is there any command to delete multiple files at once (2 Replies)
Discussion started by: nani1984
2 Replies

5. Shell Programming and Scripting

ftp multiple files from same directory

Hi there Gurus, I have the following ftp script: $ more ftp_dump_arch4.sh #! /usr/bin/ksh # Constant variables HOST='xx.xx.xx.xx' USER='user' PASSWD='password' dir='/export/file' ftp_log='/tmp' ftp -n $HOST > $ftp_log/ftp.log << END user $USER $PASSWD verbose lcd $dir bin (3 Replies)
Discussion started by: lweegp
3 Replies

6. Shell Programming and Scripting

Same time ftp download in perl multiple sites.

I currently have a perl script that have to retreive a single file from 20+ sites every 10 min. Right now it will ftp to site1 and download and continue up until site20. I am trying to get this to run all the ftp d/l at the same time. This is where I have my problem, I can't get it to work. ... (5 Replies)
Discussion started by: kofs79
5 Replies

7. Shell Programming and Scripting

Help in FTP'ing multiple files

Hi, I have written the following FTP script to get the multiple files from remote server to local server. My problem is that 'mget *' is not working in the script. I also tried with 'mget *.txt', 'mget *.*' etc. without any success. It do not copy any file to local server. In the script, Prompt... (10 Replies)
Discussion started by: berlin_germany
10 Replies

8. IP Networking

Automated ftp for Multiple files

I have seen the script posted yesterday for automated ftp Can we do some thing like ftp ing multiple files in one script Example input.txt has all files names to be ftped input.txt ------ a.tar b.ccp c.perl i need to ftp all the files present in input.txt i tried something like... (0 Replies)
Discussion started by: pbsrinivas
0 Replies

9. HP-UX

how to ftp multiple files

Hi, I have to write a ftp script which transfers multiple files from one unix server to another. When I try to transfer single file it goes through successfully. But, When I try to do multiple files none of the files get ftp'd. And also, even the single file goes transferred successfully, I... (4 Replies)
Discussion started by: isingh786
4 Replies

10. Shell Programming and Scripting

Need to send multiple files during ftp

Hi guys... I'm working on #!/bin/sh script in a Solaris 7 box that should send several files over to another machine via FTP. Here's what the script looks like: # This script will send the daily MSP customer counts # to the Crystal Reports server located at 192.168.2.106 cd... (2 Replies)
Discussion started by: cdunavent
2 Replies
Login or Register to Ask a Question