We are receiving data feed files in SFTP location daily. so the folder structure in SFTP location is abc/def/studyname_1/outbound/zipped files
So we will be getting different studies and for each study a folder is created abc/def/studyname_2/outbound/zipped files , abc/def/studyname_3/outbound/zipped files
Script needs to go to each study name and then fetch the zipped files to another Windows location.
Here I would want to have a reference table created in SQL where I have listed all the incoming zipped files and its SFTP location. So the script also needs to cross verify with the sql table and then pick-up the files.
Or
we can fetch zip filenames and then insert into sql table and mark it as processed.
I understand we can use MGET to fetch all the files in SFTP location. I am confused on how I can achieve the tracking of the zip files and how to go through each study and fetch files.
Moderator's Comments:
Please use CODE tags as required by forum rules!
Last edited by RudiC; 12-06-2017 at 04:08 PM..
Reason: Added CODE tags.
The "sleep 120" does nothing since sftp has no such command, by the way.
Quote:
I understand we can use MGET to fetch all the files in SFTP location. I am confused on how I can achieve the tracking of the zip files and how to go through each study and fetch files.
What do you mean 'tracking'?
You can examine the contents of zip files with unzip -l
You should not be using expect to cram passwords into sftp. You should be using shared keys. That way sftp will work as intended with no need to force-feed it via expect.
Rather than bludgeon your way past good security and causing yourself more complications with expect, could you not use SSH keys to get password-less authentication in place? This will simplify thing tremendously and you would end up with code more like this:-
No need for a sleep attempt or anything. The process will exit when the mget completes.
I have added pushd before to set the required local directory & popd afterwards to put you back where you were beforehand, but these may be OS/shell specific and you haven't told us either. Another way would be OLDPWD=$PWD ; cd $localdir before and cd $OLDPWD afterwards, but it looks messier to me.
There are plenty of thread about setting up password-less authentication. Basically the client making the sftp request has to create a key-pair with ssh-keygen and send the *.pub file created to the server it wants to connect to. The content has to be added to the correct place on the server and then the connection can be opened. Have a search and let us know how you get on.
Thank you for your answer.. I would need help in picking zipped files in subdirectories.
so the folder structure in SFTP location is abc/def/studyname_1/outbound/zipped files
abc/def/studyname_2/outbound/zipped files , abc/def/studyname_3/outbound/zipped files
So the script needs to go to abc/def/ and then pick the zipped files from each sub-directory which are placed in SFTP server and move to different directory in Unix.
- What is your OS and its version? (os that we can anticipate which tools will be there and which may be their quiirks)
- What is your shell and its version? (for pretty much the same reasons)
Some project-related questions:
- Does the directory structure of the source dirs need to be preserved? i.e. when you want to "pick the zipped files from each sub-directory which are placed in SFTP server and move to different directory" as you said, will the files
go to
or should they go to
If the latter, what should be done about identical filenames?
- What should be done about different versions? i.e. suppose there is a file
and in the target directory there is already a file named
What should the script do in this case? Use the newer file? use the file in the source directory, regardless? Leave the file in the target directory alone? None of the above?
Hi All,
Please suggest me how to make script for file transfer from server X to another server Y.
I have some directory path in server X as below:
/home/directory_1/
.
.
/home/directory_n/
where some text files are available in each directory where some files records zero and some files... (6 Replies)
Hi,
I want to delete a file on the local server, while connected to remote server through FTP.
I am using the below code for this
$FTP_CMD -v -n $HOST <<*! >> $LOGFILE 2>&1
user $USER $PASSWORD
cd $DIR
... (11 Replies)
I need to create a script which needs to transfer the "<filename>+yyyymmdd".dat file from a unix machine to a linux machine.. daily. we can use an active batch to schedule the script. however, the script needs to detect if the new file is present, if present, then transfer.
any help ?? (1 Reply)
Hi,
I need a shell script to transfer a file from one server(unix box) to another server(windows box).
I have the details of the source and destination Ip's.
source path : /home/UNIX/server
filename:abc.txt
Destination folder: D:/UNIX/test
I am using AIX server.
Type of shell :... (1 Reply)
I have a file having n number of records .i want first 1000 records from the file and store in temporary file to process on sunday.
I want script should should automatically take the next 1000 records for processing on next sunday.
can we do it using head and tail
head -1000 | tail... (2 Replies)
I have made a script which transfers some files of the entire week , but the script fails when the next month is started.
For e.g; if i run the script on 5th may , but i need to transfer files of its previous week which is from 24th April to 30th april ,the script fails, i have this loop in the... (2 Replies)
I use expect to spawn an sftp file transfer. On occasion the transfer fails or doesn't complete correctly, resulting in:
"sftp> cd /returns
sftp> rename /returns/TESTFILE.TXT /returns/archive/TESTFILE.TXT
Couldn't rename file "/returns/TESTFILE.TXT" to "/returns/archive/TESTFILE.TXT":
Failure... (0 Replies)
Hi
My requirement is i want to copy files from remote server to the local server and also i need to preserve the timestamp of the remote file.
By using scp -p , it is working fine in the interactive call but it is not preserving he file timestamp when i use it in the non interactive scp call... (1 Reply)
Hi,
Firstly, I have no experience (at all) with shell scripting. So please, go easy on me :)
I did a search for what I was looking for although there were a few things available here and on the net I couldn't find anything tailored to what I am looking for.
Simply put, I have two servers.... (2 Replies)
I want to automate the file transfer from a remote site to my pc over a WAN. Can anyone please suggest an industry standard secure file transfer tool for a windows environment?
I am looking for an industry accepted, more secure than traditional FTP.
Thanks! (1 Reply)