Visit Our UNIX and Linux User Community


File transfer from remote to local


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting File transfer from remote to local
# 1  
Old 03-16-2018
File transfer from remote to local

Hi,
I came across the scenario, that I need to copy files from the remote server to my local. The files in the remote server are created by another job and its keep on generating the files in that remote folder.

We can't able to use SCP command and we're using SFTP to connect the server and mget to copy the files.

Once we copied the files we're giving rm command to wipe the directory of the remote server.

For example mget command moving 100 files and few more files are created in that folder bcoz of the job and while giving rm command it will remove the 100 files + newly created files from the remote server.

Is there any other best way to do this without losing the newly created files.

Thanks
Janarthan
# 2  
Old 03-16-2018
Make a list of the files that you have processed on the receiving machine, and remove only those files from the sending machine just prior to the next mget.
# 3  
Old 03-16-2018
How about renameing the remote files to a temp directory, then geting and thereafter deleting them? The newly created files will stay untouched in the original directory.
# 4  
Old 03-16-2018
I think you are asking: 'how to stop deletion of files on a file server until after the file has been copied'

One way: rename each file right after you copy it over.
You can do this several ways example using your current pid
Code:
# create a special local directory
pid=$$
local_dir=./myfiles/${pid}
test -d  $local_dir || mkdir $local_dir
cd $local_dir
sftp someplace.com
username password
cd /path/to/files
mget files*
bye
find . -type f -name |
while read fname 
do
         sftp someplace.com <<EOF
         username password
         cd /path/to/files
         rename "$fname" "$pid"
         bye
EOF
        mv "$fname" /path/to/permanent_local_storage/directory
        rm "$fname"  # get rid of local duplicate. 
done
sftp someplace.com <<EOF
  username password
  cd /path/to/files
  rm "$pid"
bye
EOF

Your remote directory has one file left because you kept renaming every file you copied to a pid number. Remove it when you are done. This can be improved in lots of ways.

Note: the EOF that ends a heredoc in this example has to be LEFT i.e., column 1

EDIT: I see Rudi had the same idea.
It can float if the first one for the heredoc is <<-EOF
This User Gave Thanks to jim mcnamara For This Post:

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Advanced & Expert Users

Best way to transfer files to remote FTPS server instead of local FTPS server

Hi, I am working on an application which runs on an Informatica Red-Hat 5.10 Linux Server. The application involves several Informatica ETL workflows which generate 100s of Text files with lot of data. Many of the files will each be up to 5 GB in size. Currently the Informatica server itself... (7 Replies)
Discussion started by: waavman
7 Replies

2. UNIX for Dummies Questions & Answers

Remote file transfer between Linux and Windows

Hello, I have a file, say details.txt on my windows machine. I want to copy it to linux machine through a shell script and edit the file and transfer the file back to windows machine. (All I want to do is to edit the file on windows machine from linux machine by a script.) I have tried scp... (1 Reply)
Discussion started by: Devendra Hupri
1 Replies

3. UNIX for Dummies Questions & Answers

Transfer image file from local to remote with sftp

Hi. I have managed to transfer a file from remote to my raspberry pi, but I have the camera mounted on the Raspbien so I would like to transfer the image the other way. I use this line: sshpass -p 'PASSWORD' scp -- USER@ssh.servername.com:/www/cam/image.jpg /home/pi/shared/web/image.jpg (4 Replies)
Discussion started by: brickglow
4 Replies

4. Shell Programming and Scripting

How to transfer file from Local PC to UNIX Directory without FTP?

Dear All, i am trying to get the file from windows location to unix location without using FTP and neither thru entering the user id and password. I have one unix form which is running on web application and user is entering the location and file name there now i know the file name and path. So i... (8 Replies)
Discussion started by: ripudaman.singh
8 Replies

5. UNIX for Dummies Questions & Answers

How to transfer file from Local PC to Unix Directory without FTP!!!

Dear Friends, How to transfer files from my local PC to Unix directory without using FTP. Scenario: Transfer/Upload a file from PC to unix using web browser without using FTP technologies. I heard something like sendunix and sendpc used to transfer files from unix to Desktop and... (1 Reply)
Discussion started by: kk_c2il2
1 Replies

6. HP-UX

How to execute a remote file with local command

Hello, I know this is somewhat strange, but please let me know if possible. I want to execute a program file in the remote machine with command on the local machine. Let me make things more clear. Suppose I have a cc on my local system and do not have that on the remote system. I want to use... (2 Replies)
Discussion started by: Veera_Raghav
2 Replies

7. UNIX for Dummies Questions & Answers

File System - Remote or Local??

Is there a way to find if the file systems mounted on a AIX/Linux box is local or remote? (1 Reply)
Discussion started by: Un1xNewb1e
1 Replies

8. Shell Programming and Scripting

Transfer file from local unix server to remote server

want to remove this thread. thanks (2 Replies)
Discussion started by: indira
2 Replies

9. HP-UX

Transfer file from local unix server to remote server

want to remove the thread thanks (2 Replies)
Discussion started by: indira
2 Replies

10. UNIX for Dummies Questions & Answers

SCO Openserver 6 Remote File Transfer

Hi, I have a server which has SCO Openserver 6 on it and also kermit installed, unfortunately I can't seem to get kermit to send or receive files, is there another way I can use to transfer files remotely? I only have remote access to the server via a dial up modem Any comments/suggestions... (1 Reply)
Discussion started by: Martyn
1 Replies

Featured Tech Videos