Visit Our UNIX and Linux User Community


sftping continously with out breaking the connection


 
Thread Tools Search this Thread
Top Forums UNIX for Advanced & Expert Users sftping continously with out breaking the connection
# 1  
Old 11-08-2007
Question sftping continously with out breaking the connection real [emergency.......!!!!!!!!]

Hello Every one,

I have a situation where I want to sftp the files to third part server and I came up with below script where in I am using "expect" utility for sftping the files.

.....................................................
#!/usr/local/bin/expect --
#
set timeout 120
set password [lindex $argv 0]
set host [lindex $argv 1]
set username [lindex $argv 2]
set filename [lindex $argv 3]
spawn sftp $username@$host
expect -nocase password: {send "$password\r"}
expect "sftp>"
send "mput $filename\n"
expect "sftp>"
sleep 2
send "exit \n"
-----------------------------------------

Here whats happens now...
every time there is a file my java code calls this script and sftps the files to remote server. As I have realized that trafic of files going out to the remote sever is very large and it is very expensive to open a new connection and sftp a small file and then closing a connection ; again opening a connection which is causing lots of performance issues.
Also I cannot keep connection active forever....

What I would like to happen is keep the connection active for lets see 1 hrs or some defined time and then keep using the same connection to sftping the files and whenever there is no connection avaliable make a new connection.

Is any one came across this kind of situation or any body can give more inputs on this one.

Cheers,
Rex

please this is real emergency.......!!!!!!!!

Last edited by nilesrex; 11-08-2007 at 11:29 PM.. Reason: please this is real emergency.......!!!!!!!!
# 2  
Old 11-08-2007
Quote:
Originally Posted by nilesrex
please this is real emergency.......!!!!!!!!
Don't bother with "sftp", use "scp".
# 3  
Old 11-08-2007
Error

Thanks Porter,

My real requirement is to use sftp as both parties have agreed to use both we are doing EDI transactions. Is there any thing which I can add to the script to make it continous....

even if i use scp i need to open a connection and close it....so I need a solution other than using scp iinstead sftp

Cheers,
nilesh
# 4  
Old 11-08-2007
trivially you might think you want to do something like

Code:
drive_sftp()
{
     while true
     do
         echo "mget blah.*"
        sleep 2
     done
}

drive_sftp | sftp user@host

but that is not going to fly because there is no synchronisation between knowing when you have downloaded the files.

You would need to have a client program to sftp which will understand it's responses and connects to it's stdin/stdout and stderr.
# 5  
Old 11-08-2007
Hey Porter,

Actually I copied a incorrect thing I am sftping the files to end system. so it is MPUT so I have control with me for the files.

What I am thinking is whenever files are ready to be sftp'ed they will call this script and check if connection is active and if yes then do the sftp else open a new connection and send the files.... I agree there would be problems because of multithreading where connection can be a seamaphor or some thing however if I have some thing which I can modify and work on would be greate....

Today I did profiling of this processes what I have seen is at times I get 40 files in a min to do sftping... so it very bad that I open & close connections 40 times in a single min....

What do you think??

Cheers,
rex

Last edited by nilesrex; 11-08-2007 at 11:53 PM.. Reason: MPUT....and not MGET sorry my mistake
# 6  
Old 11-09-2007
Quote:
Originally Posted by nilesrex
What do you think??
How are they dealing with the files at the remote end, for example do they just look in a directory, if so how do they know the file is complete?

I can see what you are doing, with ftp/sftp what I suggest is you put the file up in one directory, then move it to another directory on the server once it's complete.

Code:
put file   in-transit/file
move in-transit/file ready/file

and rather than use mput, use put explicitly so you know what you are sending, and once you have sent the file, move it to a "sent" directory.

How are you going to handle the case where you start to send a file and the comms link drops half way through?

How are you going to avoid overwriting files on the server?
# 7  
Old 11-09-2007
Data

Those are my concerns are .... I googled a lot to find out such a utility or script or program which would do this for me.....

Actually I can collect the bunch of file and send it to sftp server once but SLA is to send the files immedieatly....

I think there should be something out of the box functionality provided in some tool.....

I dont know how to proceed more on this one....

Cheers,
rex

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Windows & DOS: Issues & Discussions

How to play video files one after the other continously?

Hi, There are many MP4 files in a folder say 50 files . All these files are video clipping files.Instead of playing the video one by one , is it possible to play all video clipping files into single shot ? Say for example when i play one video file it gets over after sometime and to view... (5 Replies)
Discussion started by: Maddy123
5 Replies

2. Shell Programming and Scripting

Redirecting stdout continously to a file

I have a C program that continously outputs info to stdout. The problem is that I am redirecting the stdout and stderr to a file and stdout is written at the end of the problem rather than continously to the file. This could be a problem if for example the program is killed and the stdout output is... (3 Replies)
Discussion started by: igurov
3 Replies

3. Shell Programming and Scripting

SFTPing non-interactive mode

Hi All, I need to sftp without any intervention.(LInux,Ksh) I have tried some ideas listed here...but its not working... Please advise on it. (1 Reply)
Discussion started by: jesu
1 Replies

4. Solaris

Solaris 10 ftp connection problem (connection refused, connection timed out)

Hi everyone, I am hoping anyone of you could help me in this weird problem we have in 1 of our Solaris 10 servers. Lately, we have been having some ftp problems in this server. Though it can ping any server within the network, it seems that it can only ftp to a select few. For most servers, the... (4 Replies)
Discussion started by: labdakos
4 Replies

5. Solaris

Link status turning up and down continously

Hi All, Happy New Year. I have a weird situation in one of my servers.The OS is solaris 10 We don't have bcme1 interface configured/plumbed on that system but every 5 second i get this message in /var/adm/messages Jan 4 20:22:45 XXXX bcme: NOTICE: bcme1 : Link is Up (1000Mbps, Full... (3 Replies)
Discussion started by: Jartan
3 Replies

6. Shell Programming and Scripting

Polling continously for presence of a file

Hi, My os is sun solaris 5.10 and Korn shell scripting. I have a file name like CCNA_EARLY_SWP.w062309 where 062309 is date in mmddyy .This is the value date of the file.(will I need to check continously from 5.00 - 7.00 am daily for this file . If the file has not come at 5 am or 7am... (4 Replies)
Discussion started by: manoj39
4 Replies

7. Shell Programming and Scripting

How to append the output continously from a script

Hi All, Am using the below script to produce some statistics. Currently it send the results to a log file and sends the contents of the log to a mail ID. Next time when it runs it erases the previous log and writes the latest output to the log file. I want the output to be appended to... (2 Replies)
Discussion started by: nirmal84
2 Replies

8. UNIX for Advanced & Expert Users

Running the job continously.

I run a job, before completion of that job, another one started the same job in the middle of running the same job. What will happen? (6 Replies)
Discussion started by: rajesh08
6 Replies

9. UNIX for Advanced & Expert Users

after 30 seconds breaking the internet connection

Please help me to solve this problem. after every 30 second or 1 minute i m getting the folowing error in browser, which works fine after pressing the reload tab. #################################################### While trying to retrieve the URL: Ubuntu Forums - Search Forums The... (2 Replies)
Discussion started by: jagnikam
2 Replies

10. UNIX for Dummies Questions & Answers

Script should run continously

Hi I have a small req. I have a script called as abc.sh I want to execute this script continously for every 1 minute even if i exit from the server i.e., it should keeps on running for every one minute even if i logged off Can any one send me the sample code or procedure to work... (3 Replies)
Discussion started by: pssandeep
3 Replies

Featured Tech Videos