Automated scp using shell & expect


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Automated scp using shell & expect
# 1  
Old 10-27-2016
Oracle Automated scp using shell & expect

Hi All,

I have written a script to automate scp of files. Most of the times it works fine except few cases. I want your help and suggestions to fix these failures.

I have used expect & shell to do the automated scp. Below is code


Code:
$ cat scp.ksh
#!/bin/ksh
inputfile=$1
fdest_sid=$2
echo -n "Enter Fully Qualified Source Server Name:"
read -r rserver
echo -n "Enter userid of source server:"
read -r ruser
echo -n "Enter password for source server:"
stty -echo
read -r rpassword
stty echo
echo  ""
export rserver
export ruser
export rpassword

export logfile=file_transfer_$(date +%Y%M%d_%H%M%S).log
> $logfile

cnt=$(cat $inputfile|wc -l)
var=$(cat $inputfile|tr '\n' ' '|sed -e 's/.$//g')
scp_cmd=$(echo "scp $ruser@$rserver:'${var}' $fdest_sid")
./expect_scp "$scp_cmd" "$cnt" >> $logfile 2>&1




$ cat expect_scp
#!/usr/bin/expect
set scp_cmd [lindex $argv 0]
set cnt [lindex $argv 1]
set rpass "$env(rpassword)\r"
set prompt "(%|#|\$) $"
set j 0
spawn ksh
send "$scp_cmd\r"
expect "Are you sure you want to continue connecting (yes/no)?" {send "yes\r"}
expect "*assword:"  {send "$rpass\r"}
while {$j < $cnt} {
        expect {
                "ETA" {
                        puts "still transferring..."
                        exp_continue;        
                        }
                "100%" {
                        puts "done"
                        }
                }
        incr j
}
expect "*$"
send {"exit\r"}


In file transfer log, i can see the shell command that i pass to expect script are send to the spawned shell with some control characters.

eg:

command passed to expect script:
Code:
scp oracle@server.domain.com:'/FS1/DB1/ETL_STAGING_A01.dbf /FS1/DB1/KALIDO_IN_D03.dbf /FS2/DB1/ETL_STAGING_A02.dbf /FS2/DB1/KALIDO_IN_D04.dbf /FS1/DB1/ETL_STAGING_A03.dbf /FS1/DB1/KALIDO_IN_D05.dbf /FS2/DB1/ETL_STAGING_A04.dbf /FS2/DB1/KALIDO_IN_D06.dbf /FS1/DB1/ETL_STAGING_A05.dbf' /FS3/DB2


Expect Script sending below command to shell:
Code:
$ scp oracle@server.domain.com:'/FS1/DB1/ETL ^H_STAGING ^H_A01.dbf /FS1/DB1/KALIDO ^H_IN ^H_D03.dbf /dbbcnprod/or^M$ oradata02/DB1/ETL ^H_STAGING ^H_A02.dbf /FS2/DB1/KALIDO ^H_IN ^H_D04.d                                                                                 <^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^Hbf /FS1/DB1/ETL ^H_STAGING ^H_A03.dbf /FS1/DB1/                                                                                ^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^M$ WH/KALIDO ^H_IN ^H_D05.dbf /FS2/DB1/ETL ^H_STAGING ^H_A04.dbf /dbbcnprod/o                                                                                ^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^Hradata02/DB1/KALIDO ^H_IN ^H_D06.dbf /FS1/DB1/ETL ^H_STAGING ^H_A05.^M$  ^H_D06.dbf /FS1/DB1/ETL ^H_STAGING ^H_A05.dbf' /u02/oradata03/DB2                                                                                ^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^M


These control charcters are causing scp failures some times.
How do i get rid of these control characters?


Regards,
Veeresham
# 2  
Old 10-27-2016
First of all, and without reading the rather wide code/output, I would respectfully suggest that you should not try to force your way through scp with expect. Tools built like scp, ssh, sftp etc. have security designed in to them and you are trying to break/bypass them in an insecure manner.

You would be far better generating keys to automate the process. It will also make your code far more sensible. Is there a reason not to use the tools they way they are designed? If your script is interactive, why not get the tools to do the work for you?



Robin
# 3  
Old 10-27-2016
I'd rather check the input file if it contains unusual characters that, via tr, sed, $var and an unnecessary "command substitution", end up in the $scp_cmd.

And, I second rbatte1, in that that is an overly complicated approach. Did you, on top of the suggestions he made, consider using sftp for multi file transfer?
# 4  
Old 10-27-2016
Thanks Robin & Rudi for your comments. I will think of using key generation technique to do this. But still want to know from where these junk characters are coming.

Rudi,

Here is the input file. I generate this file by querying a DB table.

Code:
/FS1/DB1/ETL_STAGING_A01.dbf
/FS1/DB1/KALIDO_IN_D03.dbf 
/FS2/DB1/ETL_STAGING_A02.dbf 
/FS2/DB1/KALIDO_IN_D04.dbf 
/FS1/DB1/ETL_STAGING_A03.dbf 
/FS1/DB1/KALIDO_IN_D05.dbf 
/FS2/DB1/ETL_STAGING_A04.dbf 
/FS2/DB1/KALIDO_IN_D06.dbf 
/FS1/DB1/ETL_STAGING_A05.dbf

Regards,
Veeresham


Moderator's Comments:
Mod Comment Please use CODE tags as required by forum rules!

Last edited by RudiC; 10-27-2016 at 12:12 PM.. Reason: Added CODE tags.
# 5  
Old 10-27-2016
If you install keys, the procedure for copying without a password is:

Code:
scp localfile username@hostname:destinationfile

...that's it.

It's that simple, because that's the way it's intended to be used -- not forcing it with a third-party brute force hacking tool.
# 6  
Old 10-27-2016
Unfortunately, control chars are not that easily seen in test posts. Show us the result of
Code:
od -ctx1 $inputfile

and/or
Code:
echo $var | od -ctx1

.
# 7  
Old 10-27-2016
Hi Rudi,

Below is the output of od command

Code:
$ od -ctx1 2.dfscp_saa
0000000   /   d   b   b   c   n   p   r   o   d   /   o   r   a   d   a
         2f  64  62  62  63  6e  70  72  6f  64  2f  6f  72  61  64  61
0000020   t   a   0   2   /   P   R   O   D   D   W   H   /   E   T   L
         74  61  30  32  2f  50  52  4f  44  44  57  48  2f  45  54  4c
0000040   _   S   T   A   G   I   N   G   _   A   0   6   .   d   b   f
         5f  53  54  41  47  49  4e  47  5f  41  30  36  2e  64  62  66
0000060  \n   /   d   b   b   c   n   p   r   o   d   /   o   r   a   d
         0a  2f  64  62  62  63  6e  70  72  6f  64  2f  6f  72  61  64
0000100   a   t   a   0   2   /   P   R   O   D   D   W   H   /   K   A
         61  74  61  30  32  2f  50  52  4f  44  44  57  48  2f  4b  41
0000120   L   I   D   O   _   I   N   _   D   0   8   .   d   b   f  \n
         4c  49  44  4f  5f  49  4e  5f  44  30  38  2e  64  62  66  0a
0000140   /   d   b   b   c   n   p   r   o   d   /   o   r   a   d   a
         2f  64  62  62  63  6e  70  72  6f  64  2f  6f  72  61  64  61
0000160   t   a   0   1   /   P   R   O   D   D   W   H   /   E   T   L
         74  61  30  31  2f  50  52  4f  44  44  57  48  2f  45  54  4c
0000200   _   S   T   A   G   I   N   G   _   A   0   7   .   d   b   f
         5f  53  54  41  47  49  4e  47  5f  41  30  37  2e  64  62  66
0000220  \n   /   d   b   b   c   n   p   r   o   d   /   o   r   a   d
         0a  2f  64  62  62  63  6e  70  72  6f  64  2f  6f  72  61  64
0000240   a   t   a   0   1   /   P   R   O   D   D   W   H   /   K   A
         61  74  61  30  31  2f  50  52  4f  44  44  57  48  2f  4b  41
0000260   L   I   D   O   _   I   N   _   D   0   9   .   d   b   f  \n
         4c  49  44  4f  5f  49  4e  5f  44  30  39  2e  64  62  66  0a
0000300   /   d   b   b   c   n   p   r   o   d   /   o   r   a   d   a
         2f  64  62  62  63  6e  70  72  6f  64  2f  6f  72  61  64  61
0000320   t   a   0   2   /   P   R   O   D   D   W   H   /   E   T   L
         74  61  30  32  2f  50  52  4f  44  44  57  48  2f  45  54  4c
0000340   _   S   T   A   G   I   N   G   _   A   0   8   .   d   b   f
         5f  53  54  41  47  49  4e  47  5f  41  30  38  2e  64  62  66
0000360  \n   /   d   b   b   c   n   p   r   o   d   /   o   r   a   d
         0a  2f  64  62  62  63  6e  70  72  6f  64  2f  6f  72  61  64
0000400   a   t   a   0   2   /   P   R   O   D   D   W   H   /   K   A
         61  74  61  30  32  2f  50  52  4f  44  44  57  48  2f  4b  41
0000420   L   I   D   O   _   I   N   _   D   1   0   .   d   b   f  \n
         4c  49  44  4f  5f  49  4e  5f  44  31  30  2e  64  62  66  0a
0000440   /   d   b   b   c   n   p   r   o   d   /   o   r   a   d   a
         2f  64  62  62  63  6e  70  72  6f  64  2f  6f  72  61  64  61
0000460   t   a   0   1   /   P   R   O   D   D   W   H   /   E   T   L
         74  61  30  31  2f  50  52  4f  44  44  57  48  2f  45  54  4c
0000500   _   S   T   A   G   I   N   G   _   A   0   9   .   d   b   f
         5f  53  54  41  47  49  4e  47  5f  41  30  39  2e  64  62  66
0000520  \n   /   d   b   b   c   n   p   r   o   d   /   o   r   a   d
         0a  2f  64  62  62  63  6e  70  72  6f  64  2f  6f  72  61  64
0000540   a   t   a   0   1   /   P   R   O   D   D   W   H   /   K   A
         61  74  61  30  31  2f  50  52  4f  44  44  57  48  2f  4b  41
0000560   L   I   D   O   _   I   N   _   D   1   1   .   d   b   f  \n
         4c  49  44  4f  5f  49  4e  5f  44  31  31  2e  64  62  66  0a
0000600   /   d   b   b   c   n   p   r   o   d   /   o   r   a   d   a
         2f  64  62  62  63  6e  70  72  6f  64  2f  6f  72  61  64  61
0000620   t   a   0   2   /   P   R   O   D   D   W   H   /   E   T   L
         74  61  30  32  2f  50  52  4f  44  44  57  48  2f  45  54  4c
0000640   _   S   T   A   G   I   N   G   _   A   1   0   .   d   b   f
         5f  53  54  41  47  49  4e  47  5f  41  31  30  2e  64  62  66
0000660  \n   /   d   b   b   c   n   p   r   o   d   /   o   r   a   d
         0a  2f  64  62  62  63  6e  70  72  6f  64  2f  6f  72  61  64
0000700   a   t   a   0   2   /   P   R   O   D   D   W   H   /   K   A
         61  74  61  30  32  2f  50  52  4f  44  44  57  48  2f  4b  41
0000720   L   I   D   O   _   I   N   _   D   1   2   .   d   b   f  \n
         4c  49  44  4f  5f  49  4e  5f  44  31  32  2e  64  62  66  0a
0000740   /   d   b   b   c   n   p   r   o   d   /   o   r   a   d   a
         2f  64  62  62  63  6e  70  72  6f  64  2f  6f  72  61  64  61
0000760   t   a   0   1   /   P   R   O   D   D   W   H   /   E   T   L
         74  61  30  31  2f  50  52  4f  44  44  57  48  2f  45  54  4c
0001000   _   S   T   A   G   I   N   G   _   A   1   1   .   d   b   f
         5f  53  54  41  47  49  4e  47  5f  41  31  31  2e  64  62  66
0001020  \n
         0a
0001021

Code:
$ cat 2.dfscp_saa
/dbbcnprod/oradata02/PRODDWH/ETL_STAGING_A06.dbf
/dbbcnprod/oradata02/PRODDWH/KALIDO_IN_D08.dbf
/dbbcnprod/oradata01/PRODDWH/ETL_STAGING_A07.dbf
/dbbcnprod/oradata01/PRODDWH/KALIDO_IN_D09.dbf
/dbbcnprod/oradata02/PRODDWH/ETL_STAGING_A08.dbf
/dbbcnprod/oradata02/PRODDWH/KALIDO_IN_D10.dbf
/dbbcnprod/oradata01/PRODDWH/ETL_STAGING_A09.dbf
/dbbcnprod/oradata01/PRODDWH/KALIDO_IN_D11.dbf
/dbbcnprod/oradata02/PRODDWH/ETL_STAGING_A10.dbf
/dbbcnprod/oradata02/PRODDWH/KALIDO_IN_D12.dbf
/dbbcnprod/oradata01/PRODDWH/ETL_STAGING_A11.dbf

Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Using Expect and scp. Not moving directories

An Expect script using scp to copy directories is not moving the subdirectories as wanted. The directory structure is: computerA (user 'test') /home/test/parentDir/subdir1 /home/test/parentDir/subdir2 and I want them copied to computerB (user 'archive') /home/archive/subdir1... (1 Reply)
Discussion started by: nohj ordpl
1 Replies

2. Shell Programming and Scripting

scp script getting timed out with expect

Hi, I have an expect script where in i am trying to scp a folder but it is getting timed out. Any help will be appreciated. (I don't have the option for sharing keys) expect -c 2> /dev/null " spawn scp -r -o NumberOfPasswordPrompts=1 -o StrictHostKeyChecking=no root@10.10.10.10:test_dir... (2 Replies)
Discussion started by: temp_user
2 Replies

3. Shell Programming and Scripting

How to use 'expect' to pass UID & Password to a "for loop" in shell script?

Friends, Need someone's help in helping me with the below requirement for a script: > For a list of servers(over 100+), I need to login into each of them(cannot configure password-less ssh) & grab few configuration details < I know, this is possible through expect programming in a simple... (14 Replies)
Discussion started by: thisissouvik
14 Replies

4. AIX

How to use 'expect' to pass UID & Password to a "for loop" in shell script?

Friends, Need someone's help in helping me with the below requirement for a script: > For a list of servers(over 100+), I need to login into each of them(cannot configure password-less ssh) & grab few configuration details < I know, this is possible through expect programming in a simple... (2 Replies)
Discussion started by: thisissouvik
2 Replies

5. Shell Programming and Scripting

scp not working in expect script

Hi All, I run the scp command in shell prompt without issue, but when on expect script as below: #!/usr/bin/expect spawn scp /var/spool/sms/failed.tar.gz abc@10.10.12.2:/home/abc expect "abc@10.10.12.2's password: " send "abcfef\r" exit 0 It looks not working at all and the... (3 Replies)
Discussion started by: elingtey
3 Replies

6. Shell Programming and Scripting

Automated SCP from a job?

Hi all - I have a script which runs on the OS level, but refuses to run as a cron or as an Oracle job. The Script is pretty straight forward: #!/bin/bash username="MyUsername" host="Remote.server" path="Remote.directory/files/*.*" password="MyPassword" expect -c " spawn /usr/bin/scp... (3 Replies)
Discussion started by: danimaltex
3 Replies

7. UNIX for Advanced & Expert Users

Automated SCP script passing password to preserve source file timestamp

Hi My requirement is i want to copy files from remote server to the local server and also i need to preserve the timestamp of the remote file. By using scp -p , it is working fine in the interactive call but it is not preserving he file timestamp when i use it in the non interactive scp call... (1 Reply)
Discussion started by: skumar75
1 Replies

8. Shell Programming and Scripting

How to put scp in background inside expect

Gents, I have a wrapper script passing couple of information to an expect script mainly responsible for spawning scp and providing the password (which is transmitted down from the main script). the main script prepare the transfer to couple of servers, idea being to transfer the files in... (3 Replies)
Discussion started by: luc3004
3 Replies

9. Solaris

SCP & SSH errors

Hi I am trying to scp a file between to servers (both on same subnet and can see each other). However, whenever I try I get the following error: ld.so.1: ssh: fatal: relocation error: file /usr/local/bin/ssh: symbol EVP_CIPHER_CTX_key_length: referenced symbol not found lost connection I... (4 Replies)
Discussion started by: skewbie
4 Replies

10. UNIX for Advanced & Expert Users

scp automated script

Hi Unix gurus, I am trying to create a script to automate the copying of files daily from one server to another using the scp command. --> #!/bin/ksh KEY="$HOME/.ssh/SSHKEY" if ;then echo "Private key not found at $KEY" >> $LOGFILE echo "* Please create it with \"ssh-keygen -t dsa\" *"... (5 Replies)
Discussion started by: gholdbhurg
5 Replies
Login or Register to Ask a Question