shell script - ftp downloading serveral files without mget


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting shell script - ftp downloading serveral files without mget
# 8  
Old 08-19-2009
thank you for helping so far chipcmc

Quote:
#!/bin/bash
function send_ftp
{
ftp -n -i -v > $2 <<EOF
open proxyHost proxyPort
user user_ftp@HOSTADRESS password
cd $dir_remote
$1
bye
EOF
}
file_log=log.$$
file_list=tmp.$$


dir_remote=/.
#Put the max file that you can copy in one instrucction of mget
max_file=10

#call to generate the list of files in remote host
send_ftp "dir $dir_remote" $file_log

#in my case de first 12lines is the "head" of the log file, besure in you case
#the grep filter is because the lines of ftp begin with number and the files no
sed "1,12 d" $file_log | grep -v ^[0-9] | sed "s/ */#/g"| cut -f9 -d"#" >$file_list
rm -f $file_log

#to optimice the ftp comands i make a buffer with the max number of files that we
#can pass to de mget instruction
buffer=""
conta=0
while read file
do
buffer="$buffer $file"
conta=$(expr $conta + 1)
if [ $max_file -eq $conta ]
then
send_ftp "mget $buffer" "/dev/null"
buffer=""
conta=0
fi
done <$file_list

if [ "$buffer" ]
then
send_ftp "mget $buffer" "/dev/null"
fi
rm -f $file_list
i'm still trying to make this work, but no success yet Smilie

i'm getting no error, script is working for 1-2 minutes and then the shell is free to type again but no new files in the local folder or any other folder and no log files like in trys before....

even tried with different dir_remote or without a folder behind dir (cause there are no folders only files on the server, so its the default folder....) but i get the same result.... working without error or result...
# 9  
Old 08-19-2009
chage this code:
Code:
rm -f $file_list

and 

rm -f $file_log

for this one:
Code:
#rm -f $file_list
echo "The file with the list:$file_list"

#rm -f $file_log
echo "The file log:$file_log"

When the script finish now they say you where are the files and you can edit Smilie.


And you can add this:
Code:
echo "DEBUG:File Name=[$file]"

just under the instruction do:
Code:
while read file
do
echo "DEBUGFile Name=[$file]"
....the rest of the code

And when the script run they print out the name of every file that is going to copy.

Onething: why you put "/." in dir_remote? this means for that they go to de / of the server.
be sure the name of the directori. If the directory of work is the HOME directori of the user ftp you need to change and put dir_remote=.
Other way is to delete the "cd $dir_remote" of the script
# 10  
Old 08-19-2009
ok ok we're getting closer Smilie

Quote:
#!/bin/bash
function send_ftp
{
ftp -n -i -v > $2 <<EOF
open X X
user X X
epsv4
$1
bye
EOF
}
file_log=log.$$
file_list=tmp.$$

dir_remote=.
#Put the max file that you can copy in one instrucction of mget
max_file=5

#call to generate the list of files in remote host
send_ftp "dir $dir_remote" $file_log

#in my case de first 12lines is the "head" of the log file, besure in you case
#the grep filter is because the lines of ftp begin with number and the files no
sed "1,12 d" $file_log | grep -v ^[0-9] | sed "s/ */#/g"| cut -f9 -d"#" >$file_list
#rm -f $file_log
echo "The file log:$file_log"

#to optimice the ftp comands i make a buffer with the max number of files that we
#can pass to de mget instruction
buffer=""
conta=0
while read file
do
echo "DEBUG:File Name=[$file]"
buffer="$buffer $file"
conta=$(expr $conta + 1)
if [ $max_file -eq $conta ]
then
send_ftp "mget $buffer" "/dev/null"
buffer=""
conta=0
fi
done <$file_list

if [ "$buffer" ]
then
send_ftp "mget $buffer" "/dev/null"
fi
#rm -f $file_list
echo "The file with the list:$file_list"

console gives back something like this:

Quote:
DEBUG:File Name=[XXXX]
DEBUG:File Name=[XXXX]
DEBUG:File Name=[XXXX]
DEBUG:File Name=[XXXX]
DEBUG:File Name=[XXXX]
ftp: No control connection for command.
ftp: No control connection for command.
ftp: No control connection for command.
ftp: No control connection for command.

i needed to add the epsv4 command to get "dir" working
and mget will only work in extended passive mode too

is there any part in your script that got problems with epsv-mode? can't see through this with my low shell script knowledge

i hope there's a solution cause i just failed downloading from this ftp in java

Last edited by macProgger23; 08-19-2009 at 01:09 PM..
# 11  
Old 08-19-2009
Every that you put inside <<EOF ..... EOF is like you type when you make a conection by yourself, if serv suport the epsv4 command and you need you can put, no problems xD
[CODE]
ftp -n -i -v > $2 <<EOF
open ------- -----
user ---@------ -----
epsv4
$1
bye
EOF
[CODE]
The otput is exactly:
DEBUG:File Name=[XXXX]
if you see XXXXX i think there are a problem to generate the list of files... please edit the 2files and sent me the content (think is better you send me a private with the content)

Last edited by chipcmc; 08-19-2009 at 01:26 PM..
# 12  
Old 08-19-2009
pls erase the uncensored server-data in your quote, i forgot to censore it...

and no i am not seeing XXXXX ... i see the real filename, so no problem with listing.... at the unimport part i don't forgot spoiling omg^^

i think theres a problem with the connection breaking up.... before mget
cause of: "ftp: No control connection for command."
# 13  
Old 08-19-2009
i think now is a problem of ftp...
to probe if is problem of ftp you can make the ftp by yourself and see if when you make then ftp you have the same problem.
# 14  
Old 08-20-2009
i succeeded in downloading 1 file , with the code posted in the 1 post (with get Filename in it)

so the ftp is fine, but there's a problem with the connection like we establish it...... could probably the mget with a buffer have problems with epsv4 ???

certainly the ftp-proxy does only work with passive mode (our admin told me), cause in active mode it would open another channel for every action and that will brought our connection to the firewall

i think it is the mget $buffer command which leads to the error "ftp: No control connection for command."

and because of the output, it seems that for every max file package, the while do loop runs first for every file, and all the send_ftp were executed at last, so that i get this output...

Quote:
DEBUG:File Name=[XXXX]
DEBUG:File Name=[XXXX]
DEBUG:File Name=[XXXX]
DEBUG:File Name=[XXXX]
DEBUG:File Name=[XXXX]
ftp: No control connection for command.
ftp: No control connection for command.
ftp: No control connection for command.
ftp: No control connection for command.
i would except something like this:

Quote:
DEBUG:File Name=[XXXX]
ftp: No control connection for command.
DEBUG:File Name=[XXXX]
ftp: No control connection for command.
DEBUG:File Name=[XXXX]
ftp: No control connection for command.
DEBUG:File Name=[XXXX]
ftp: No control connection for command.
or is this function send_ftp executed in another thread, so that it is trying to connect and failing after the while do loop is finished for 1 package?

oh man scripting for an ftp behind an ftp-proxy really sucks :X
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

FTP mget will only grab files not folders

Hey All, first post :rolleyes: So I am writting a script to pull down files from an ftp that will be called from a bat file on windows. This seems pretty straight forward, and grabs all of the "files" in the cd location, but I am running into some permission issue that will not allow me to... (1 Reply)
Discussion started by: mpatton
1 Replies

2. UNIX for Advanced & Expert Users

Help with get/mget from FTP server with files older than 10 minutes

Hi! I am new to unix and this forum as well.. Can someone please help me : I want to "get/mget" files which are older than 10 minutes from a remote FTP server like "ftp.com". After getting the files to local unix server say "Prod.com" , i need to delete only those files from ftp.com which... (4 Replies)
Discussion started by: SravsJaya
4 Replies

3. Shell Programming and Scripting

Downloading FTP Files

Hi every one, I have the requirement to download the files from FTP and move those files to unix box. Once after coping the files, i need to remove the files in FTP. I'm a newbie in Unix script. Can you please suggest a script for this.. Thanks in advance.. (2 Replies)
Discussion started by: Murali4u
2 Replies

4. Shell Programming and Scripting

shell script to ftp multiple files

Hi, i use the below script to send a single file to remote server from linux. ftp -nvi <<!EOF open $Host_name user $USER_ID $PWD binary mput $file_name quit !EOF (where i... (2 Replies)
Discussion started by: pradebban
2 Replies

5. UNIX for Dummies Questions & Answers

ftp files from one server to another using shell script

Hi Guys Any Help I have created a spool file that i need to copy onto another server using FTP in a shell script both servers are linux (3 Replies)
Discussion started by: itai
3 Replies

6. UNIX for Dummies Questions & Answers

Using FTP mget from Visual Basic Shell

Hi calling a batch file with FTP commands from Visual basing with Shell, it basically to copy files then delete them. The delete seems to work but the copy, it looks like the mget command doesn't have the complete information on where to copy the files. Here is the batch file content. Open... (2 Replies)
Discussion started by: Mario120560
2 Replies

7. Shell Programming and Scripting

No return code in ftp mget script

I have a automated FTP script that gets a file using mget. I am using mget because the date will change on the file frequently. The mget works, however if I incorrectly type the file (e.g. if I want to get /dog123 and I enter /dg*) I do not receive and error code from in the FTP session. The... (1 Reply)
Discussion started by: dog123
1 Replies

8. Shell Programming and Scripting

Problem in Downloading one day old files from FTP site

HI, I'm downloading one day old files from ftp site. Below is my script ---------------------------- printf "open $HOST \n" > ftp.cmd printf "user $USER $PASSWD\n" >> ftp.cmd printf "bin\n" >> ftp.cmd #printf "cd /Models/\n" >> ftp.cmd printf "prompt\n" >> ftp.cmd printf "for oldfile... (4 Replies)
Discussion started by: shekhar_v4
4 Replies

9. Shell Programming and Scripting

Random files do not FTP in the shell script

The following script is used to loop through files in the /tmp directory and transfer those files onto another server. However, some of the files do not transfer. It is very random when the transferring is done (i.e. one of the files won't transfer then next time, that one will transfer and... (1 Reply)
Discussion started by: RLatham20
1 Replies

10. Shell Programming and Scripting

ftp files inside a shell script

I have a shell script where I am trying to ftp some files but I get the error message "EOF unclosed" every time the script reaches the ftp section. Here is how my script is written. #more code up here rm -f $object >> $LOG_FILE 2>&1 fi #end of if done #end of for loop ... (5 Replies)
Discussion started by: matrix1067
5 Replies
Login or Register to Ask a Question