I'm having trouble with some automated sftp pulls. I'm using expect inside bash scripts and spawning SFTP. Some times the expect seems bog down. I have tried to put sleeps in my code to give everything time to work before I move on to next step but I till continue to get issues. For example when the function removeFlagFiles is called a for loop is used to get the name of each flag file downloaded. The expect part of the loop is supposed to delete the flag file. I can watch the output and all of a sudden the interaction seems to slow way down and sometimes the file is not deleted. Can someone suggest better ways of doing this?
Code:
#!/bin/bash
#
# Author: Gene Osteen
# Date: 12/11/2014
# Description: Pull performant HUMA files directory(s) and flag file(s).
# Hard-coded directories that have to be in the current (running) directory.
FILTER="HUMA*.flag"
DOWNLOAD_DIR="/ftpdata/HUMA/incoming/HUMA.dir"
REMOTE_DIR=""
function pullDataFile {
echo "In pullDataFile" >> $4
echo "remote dir = $1"
echo "file name = $2"
echo "dir = $3"
echo "remote dir = $1 file name = $2 dir = $3\n" >> $4
maxiter=5
iter=1
filestate="trypull"
while [ $filestate = "trypull" ]
do
echo "================= Start SFTP pull of $2 iteration $iter ===================" >> $4
/usr/bin/expect<<EOF
spawn sftp somesftp@10.10.10.10
sleep 1
expect "(yes/no)?"
sleep 1
send "yes \r"
sleep 1
expect "password:"
sleep 1
send "somepassword\n"
sleep 1
expect "sftp>"
log_file $4
send "cd $1 \r"
expect "sftp>"
send "lcd $3 \r"
expect "sftp>"
send "get $2 \r"
expect {
#Check for progress, note does not work with all versions of SFTP
#If a match is found restart expect loop
-re "\[0-9]*%" {
set percent $expect_out(0,string)
#puts $logf "File transfer at $percent, continuing..."
exp_continue
}
#Check for common errors, by no means all of them
-re "Couldn't|(.*)disconnect|(.*)stalled" {
#puts $logf "Unable to transfer file"
exit 1
}
#OK continue
"sftp>" {
#puts $logf "File transfer completed"
}
}
log_file
send "quit \r"
EOF
sleep 3
if [ -f ./$3/$2 ]; then
echo "File ./$3/$2 Arrived" >> $4
filestate="Arrived"
else
echo "File ./$3/$2 HAS NOT ARRIVED" >> $4
iter=$[$iter+1]
if [ $iter -gt $maxiter ]; then
filestate="Too Many Tries"
else
sleep 10
fi
fi
done
if [ $filestate = "Arrived" ]; then
echo "chmod ./$3/$2 to 755" >> $4
chmod 775 ./$3/$2
# Below must be formatted with no leading white space
/usr/bin/expect<<EOF
spawn sftp somesftp@10.10.10.10
sleep 1
expect "(yes/no)?"
send "yes \r"
sleep 1
expect "password:"
send "somepassword\n"
sleep 1
expect "sftp>"
log_file $4
send "cd $1 \r"
expect "sftp>"
send "rm $2 \r"
expect "sftp>"
log_file
send "quit \r"
EOF
else
echo "Send Email" >> $4
#Send email when you can
fi
sleep 4
}
function removeFlagFiles {
echo "======================= Remove Flag Files =======================================\n\n" >> $1
for f in $FILTER;
do
echo "Remove $f" >> $1
delfile="$f"
delfile2="${delfile//\#/\\\#}"
echo "delfile2 = $delfile2"
echo "delfile = $f delfile2 = $v"
sleep 3
/usr/bin/expect<<EOF
spawn sftp somesftp@10.10.10.10
sleep 1
expect "(yes/no)?"
sleep 1
send "yes \r"
sleep 1
expect "password:"
sleep 1
send "somepassword\n"
expect "sftp>"
sleep 1
log_file $1
send "cd outgoing \r"
sleep 1
expect "sftp>"
send "rm $delfile2 \r"
sleep 1
expect "sftp>"
log_file
send "quit \r"
EOF
sleep 4
done
echo "======================= Finished Removing Flag Files =======================================\n\n\n" >> $1
}
function pullFlagFiles {
echo "======================== Pull existing flag files===========================================" >> $1
echo "Start of SFTP" >> $1
/usr/bin/expect<<EOF
spawn sftp somesftp@10.10.10.10
expect "(yes/no)?"
send "yes \r"
expect "password:"
send "somepassword\n"
expect "sftp>"
log_file $1
send "cd outgoing \r"
expect "sftp>"
send "get $FILTER \r"
expect "sftp>"
log_file
send "quit \r"
EOF
echo "======================== Finished Pulling existing flag files===========================================\n\n\n" >> $1
sleep 4
}
function processFiles {
for f in $FILTER;
do
echo "======================== Process File $f =================================\n" >> $1
arr=($(echo $f | tr "#" "\n"))
dir=${arr[@]:1:1}
fname=${arr[@]:2:2}
fname2="${fname/.flag/}"
remote_dir="outgoing/HUMA.dir/$dir"
echo $dir
echo $fname2
echo $remote_dir
newdir="$DOWNLOAD_DIR/$dir"
echo "newdir = $newdir"
echo "newdir = $newdir" >> $1
# Create the directory if it's not there already.
echo "Try to create $newdir"
echo "Try to create $newdir" >> $1
mkdir -p $newdir
status=$?
echo "status from mkdir $newdir = $status" >> $1
if [ -d $newdir ];
then
echo "$newdir exists."
echo "$newdir exists." >> $1
else
echo "File $newdir does not exist."
echo "File $newdir does not exist." >> $1
return
fi
chmod 775 $newdir
echo "Target Directory = $newdir\n" >> $1
# Get data file
pullDataFile $remote_dir $fname2 $dir $1
if [ -f $dir/$1 ];
then
echo "File $dir/$1 exists." >> $1
else
echo "File $dir/$1 does not exist." >> $1
fi
# Change the permission and Move the file into the directory where it's wanted.
chmod 775 $f
echo "mv $f $newdir" >> $1
mv $f $newdir
if [ -f $fname2 ];
then
echo "File $newdir/$fname2 exists." >> $1
else
echo "File $newdir/$$fname2 does not exist." >> $1
fi
echo "==================== Finished File $f ===========================================\n\n\n" >> $1
done
}
###################
####### MAIN ######
###################
cd /ftpdata/HUMA/incoming/HUMA.dir
DATE=`date +%Y%m%d`
echo "DATE = $DATE"
logfile="./Logs/ftp_log_$DATE.log"
echo "Log_file = $logfile"
if [ -f ./HUMA_running.txt ]
then
echo "Already running"
exit
fi
echo "Let's start"
touch HUMA_running.txt
echo "Start HUMA search : $(date)" >> $logfile
# Now do a full cycle of pull, rename, and process.
echo "pullFlagFiles" >> $logfile
pullFlagFiles $logfile
ls -l $FILTER > /dev/null 2>&1
if [ "$?" = "0" ]; then
removeFlagFiles $logfile
processFiles $logfile
else
echo "No files to process" >> $logfile
fi
echo "" >> $logfile
rm ./HUMA_running.txt
Hello all,
I've written an automated SFTP script to work with the Expect command. It recently occurred to me however, that if the client side box does not have the known host entry for the server, it will not work correctly. So I have added an expect for the known host prompt, and that part... (2 Replies)
Hi all, i got a really strange problem
i wrote a script, when i run this script manually everything works fine
but when i make a cronjob for it, with the same user, the EXPECT script will not work.
Only the first line will be executed
this is the SHell bash script
#!/bin/sh;
php -q... (2 Replies)
Hi All,
Here is my Expect script, I don't get any error message when I run it. But the file never goes to other system? I also paste the output screen below.
When I run the script, the script runs so fast. But when I do it manually, it takes about 10 minutes for the file to transfer.
... (1 Reply)
Hi Experts,
I am learning expect and wrote the below script for automatic sftp into a server:
#!/usr/local/bin/expect -f -d
spawn sftp -v test@mumux503 # logs into mumux503 as test user
expect "password:"
sleep 20
send "test\r"; # sending the password for test... (3 Replies)
hello All,
I am doing SFTP using expect. We just change our server from sun solaris 8 to sun solaris 10.
The script was working good on sun solaris 8.
But it is giving problem on 10. from shell, SFTP is working fine.Please help me.
What can be the problem.
LIB_sftp_get()
{
... (0 Replies)
How to automate sftp with out using expect script?
My batch file has the password but it is not taking. Please see below.
I want to use this sftp connection in a loop for pushing new files in a directory one at a time. Hence I can not use an expect script.
bash-2.05$... (5 Replies)
I am trying to use a for loop in my expect cmdFile that I am calling.
I want to be able to call either one file name or a series of file names in the working directory (that I won't know the names before hand) and then pass the names to the sftp program.
Something like for i in (ls *txt)
do (0 Replies)
Hi,
I am writing a script to enter into a list of servers and take the backup of the cronjobs on that servers...
==============================
-bash-3.00$ expect spawn.exp ssh rcdn6-vm59-33
spawn ssh servers1 /CODE]====================
-bash-3.00$ cat spawn.exp
#!/usr/bin/expect
eval... (1 Reply)
I have to send few gzipped files from local server to SFTP server.
My Server Info
Distributor ID: Ubuntu Description: Ubuntu 12.04.4 LTS Release: 12.04 Codename: precise
Created a bash script and could able to send files to sftp, but i want to send email if transfer is successful.
... (1 Reply)
Hi,
I am trying to sftp using expect, but not getting through as it prompt for password is coming
Following is the code
#/usr/bin/expect > output.log
sftp medcdr@10.130.254.50
expect "password:"
send "Med@Cdr12\n"
expect "sftp>"
send "put ZTE_*201505*\r"
expect "sftp>"
send "bye\r"... (7 Replies)