Sponsored Content
Top Forums Shell Programming and Scripting shell script - ftp downloading serveral files without mget Post 302345489 by macProgger23 on Wednesday 19th of August 2009 11:06:22 AM
Old 08-19-2009
ok ok we're getting closer Smilie

Quote:
#!/bin/bash
function send_ftp
{
ftp -n -i -v > $2 <<EOF
open X X
user X X
epsv4
$1
bye
EOF
}
file_log=log.$$
file_list=tmp.$$

dir_remote=.
#Put the max file that you can copy in one instrucction of mget
max_file=5

#call to generate the list of files in remote host
send_ftp "dir $dir_remote" $file_log

#in my case de first 12lines is the "head" of the log file, besure in you case
#the grep filter is because the lines of ftp begin with number and the files no
sed "1,12 d" $file_log | grep -v ^[0-9] | sed "s/ */#/g"| cut -f9 -d"#" >$file_list
#rm -f $file_log
echo "The file log:$file_log"

#to optimice the ftp comands i make a buffer with the max number of files that we
#can pass to de mget instruction
buffer=""
conta=0
while read file
do
echo "DEBUG:File Name=[$file]"
buffer="$buffer $file"
conta=$(expr $conta + 1)
if [ $max_file -eq $conta ]
then
send_ftp "mget $buffer" "/dev/null"
buffer=""
conta=0
fi
done <$file_list

if [ "$buffer" ]
then
send_ftp "mget $buffer" "/dev/null"
fi
#rm -f $file_list
echo "The file with the list:$file_list"

console gives back something like this:

Quote:
DEBUG:File Name=[XXXX]
DEBUG:File Name=[XXXX]
DEBUG:File Name=[XXXX]
DEBUG:File Name=[XXXX]
DEBUG:File Name=[XXXX]
ftp: No control connection for command.
ftp: No control connection for command.
ftp: No control connection for command.
ftp: No control connection for command.

i needed to add the epsv4 command to get "dir" working
and mget will only work in extended passive mode too

is there any part in your script that got problems with epsv-mode? can't see through this with my low shell script knowledge

i hope there's a solution cause i just failed downloading from this ftp in java

Last edited by macProgger23; 08-19-2009 at 01:09 PM..
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

ftp files inside a shell script

I have a shell script where I am trying to ftp some files but I get the error message "EOF unclosed" every time the script reaches the ftp section. Here is how my script is written. #more code up here rm -f $object >> $LOG_FILE 2>&1 fi #end of if done #end of for loop ... (5 Replies)
Discussion started by: matrix1067
5 Replies

2. Shell Programming and Scripting

Random files do not FTP in the shell script

The following script is used to loop through files in the /tmp directory and transfer those files onto another server. However, some of the files do not transfer. It is very random when the transferring is done (i.e. one of the files won't transfer then next time, that one will transfer and... (1 Reply)
Discussion started by: RLatham20
1 Replies

3. Shell Programming and Scripting

Problem in Downloading one day old files from FTP site

HI, I'm downloading one day old files from ftp site. Below is my script ---------------------------- printf "open $HOST \n" > ftp.cmd printf "user $USER $PASSWD\n" >> ftp.cmd printf "bin\n" >> ftp.cmd #printf "cd /Models/\n" >> ftp.cmd printf "prompt\n" >> ftp.cmd printf "for oldfile... (4 Replies)
Discussion started by: shekhar_v4
4 Replies

4. Shell Programming and Scripting

No return code in ftp mget script

I have a automated FTP script that gets a file using mget. I am using mget because the date will change on the file frequently. The mget works, however if I incorrectly type the file (e.g. if I want to get /dog123 and I enter /dg*) I do not receive and error code from in the FTP session. The... (1 Reply)
Discussion started by: dog123
1 Replies

5. UNIX for Dummies Questions & Answers

Using FTP mget from Visual Basic Shell

Hi calling a batch file with FTP commands from Visual basing with Shell, it basically to copy files then delete them. The delete seems to work but the copy, it looks like the mget command doesn't have the complete information on where to copy the files. Here is the batch file content. Open... (2 Replies)
Discussion started by: Mario120560
2 Replies

6. UNIX for Dummies Questions & Answers

ftp files from one server to another using shell script

Hi Guys Any Help I have created a spool file that i need to copy onto another server using FTP in a shell script both servers are linux (3 Replies)
Discussion started by: itai
3 Replies

7. Shell Programming and Scripting

shell script to ftp multiple files

Hi, i use the below script to send a single file to remote server from linux. ftp -nvi <<!EOF open $Host_name user $USER_ID $PWD binary mput $file_name quit !EOF (where i... (2 Replies)
Discussion started by: pradebban
2 Replies

8. Shell Programming and Scripting

Downloading FTP Files

Hi every one, I have the requirement to download the files from FTP and move those files to unix box. Once after coping the files, i need to remove the files in FTP. I'm a newbie in Unix script. Can you please suggest a script for this.. Thanks in advance.. (2 Replies)
Discussion started by: Murali4u
2 Replies

9. UNIX for Advanced & Expert Users

Help with get/mget from FTP server with files older than 10 minutes

Hi! I am new to unix and this forum as well.. Can someone please help me : I want to "get/mget" files which are older than 10 minutes from a remote FTP server like "ftp.com". After getting the files to local unix server say "Prod.com" , i need to delete only those files from ftp.com which... (4 Replies)
Discussion started by: SravsJaya
4 Replies

10. UNIX for Dummies Questions & Answers

FTP mget will only grab files not folders

Hey All, first post :rolleyes: So I am writting a script to pull down files from an ftp that will be called from a bat file on windows. This seems pretty straight forward, and grabs all of the "files" in the cd location, but I am running into some permission issue that will not allow me to... (1 Reply)
Discussion started by: mpatton
1 Replies
AXEL(1) 						      General Commands Manual							   AXEL(1)

NAME
Axel - A light download accelerator for Linux. SYNOPSIS
axel [OPTIONS] url1 [url2] [url...] DESCRIPTION
Axel is a program that downloads a file from a FTP or HTTP server through multiple connection, each connection downloads its own part of the file. Unlike most other programs, Axel downloads all the data directly to the destination file, using one single thread. It just saves some time at the end because the program doesn't have to concatenate all the downloaded parts. OPTIONS
One argument is required, the URL to the file you want to download. When downloading from FTP, the filename may contain wildcards and the program will try to resolve the full filename. Multiple URL's can be specified as well and the program will use all those URL's for the download. Please note that the program does not check whether the files are equal. Other options: --max-speed=x, -s x You can specify a speed (bytes per second) here and Axel will try to keep the average speed around this speed. Useful if you don't want the program to suck up all of your bandwidth. --num-connections=x, -n x You can specify an alternative number of connections here. --output=x, -o x Downloaded data will be put in a local file with the same name, unless you specify a different name using this option. You can spec- ify a directory as well, the program will append the filename. --search[=x], -S[x] Axel can do a search for mirrors using the filesearching.com search engine. This search will be done if you use this option. You can specify how many different mirrors should be used for the download as well. The search for mirrors can be time-consuming because the program tests every server's speed, and it checks whether the file's still available. --no-proxy, -N Don't use any proxy server to download the file. Not possible when a transparent proxy is active somewhere, of course. --verbose If you want to see more status messages, you can use this option. Use it more than once if you want to see more. --quiet, -q No output to stdout. --alternate, -a This will show an alternate progress indicator. A bar displays the progress and status of the different threads, along with current speed and an estimate for the remaining download time. --header=x, -H x Add an additional HTTP header. This option should be in the form "Header: Value". See RFC 2616 section 4.2 and 14 for details on the format and standardized headers. --user-agent=x, -U x Set the HTTP user agent to use. Some websites serve different content based upon this parameter. The default value will include "Axel", its version and the platform. --help, -h A brief summary of all the options. --version, -V Get version information. NOTE
Long (double dash) options are supported only if your platform knows about the getopt_long call. If it does not (like *BSD), only the short options can be used. RETURN VALUE
The program returns 0 when the download was succesful, 1 if something really went wrong and 2 if the download was interrupted. If something else comes back, it must be a bug.. EXAMPLES
axel ftp://ftp.{be,nl,uk,de}.kernel.org/pub/linux/kernel/v2.4/linux-2.4.17.tar.bz2 This will use the Belgian, Dutch, English and German kernel.org mirrors to download a Linux 2.4.17 kernel image. axel -S4 ftp://ftp.kernel.org/pub/linux/kernel/v2.4/linux-2.4.17.tar.bz2 This will do a search for the linux-2.4.17.tar.bz2 file on filesearching.com and it'll use the four (if possible) fastest mirrors for the download. (Possibly including ftp.kernel.org) (Of course, the commands are a single line, but they're too long to fit on one line in this page.) FILES
/etc/axelrc System-wide configuration file. Note that development versions place this file in /usr/local/etc. ~/.axelrc Personal configuration file These files are not documented in a man-page, but the example file which comes with the program contains enough information, I hope. The position of the system-wide configuration file might be different. COPYRIGHT
Axel is Copyright 2001-2002 Wilmer van der Gaast. BUGS
Please report bugs at https://alioth.debian.org/tracker/?group_id=100070&atid=413085. AUTHORS
Wilmer van der Gaast. <wilmer@gaast.net> AXEL(1)
All times are GMT -4. The time now is 07:15 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy