Is it possible - SCP script ignoring certain files
Hey everyone. First, let me preface this post by stating that there are some bad things I'm doing. I'm using python to work around SCP's refusal to take standard input for a password. The reason for doing this as opposed to using keys is because the servers are on lock down due to them being in our test lab, and they're not to be touched it it doesn't mirror production. So because of the amount of beaurocracy that I would have to go through, even though I already have access to the data I need, I just need a workaround - a way to copy the data to a processing server that doesn't involve me being the man in the middle.
Here is the script as it stands:
Now, this currently grabs every CSV file in the directory on the remote server, which is fine for grabbing lab data. However, if a similar solution needs to be used for production, we'll be dealing with significantly larger files... each one upwards of 100MB, so the overhead traffic could become immense.
What I would like to do is find a way to ignore files that have already been downloaded, and only download new/updated files. Is this possible? Any help would be appreciated.
You can fetch the list of *.csv files on server the same way you're doing: instead of scp, invoke this
Then capture the output and check for existence of each file on the local machine.
You may also consider the -C option (compression) of scp (ssh).
The single quotes are essential to prevent shell globbing.
Hey :3
I am moving some stuff between different servers.
I do it like this:
scp -r -P 22 -i ~/new.ppk /var/www/bigfile.tar.gz user@123.123.123.123:/var/www/bigfile.tar.gz
Lets say, this file is 50 GiB. I would like to know, if its possible to split the file in different parts,... (2 Replies)
Hello,
I tried to write a bash script (code is below) that does scp files that contain a certain string, and that subsequently deletes only those files that have been copied (in my case new files are created every second so it is important to only delete those that have been copied). The key is... (0 Replies)
Hi everyone,
I'd like to know if is there a way to list files but ignoring some according to their modification time (or creation, access time, etc.) with the command 'ls' alone.
I know the option -I exist, but it seems to only looking in the file name..
Thank you in advance for the... (8 Replies)
Hi all,
I have root directory on server 1 say A and having sub directory B now my application generates output files and put in sub directory B.
now i need to transfer these files from server1 to server2 by scp which is having same directory structure A and sub directory B
I have tried... (2 Replies)
Hi Scripting experts,
I am new to the unix scripting. Please help me out for solving the condition given below
I am trying to develop a script for Copying files which are getting generated in server A to server B using scp. In serverA files are generating as for eg abc1.txt, abc2.txt,... (5 Replies)
Hi guys,
I am searching some files not equal to the pattern with this command
find ! -name "PATTERN" -type f
but my problem is the find command because he also search inside subdirectories and that's the thing i don't want that.
Is there any comand to ignore the directories... (4 Replies)
Hi!
Letīs say I want copy dump-files to a location.
These dump-files vary between 80 and 280MB and will be produced in about 1min or less.
I have a cronjob which (not only) copies those.
So how can I find out whether a file is currently produced or not?
Because if my script works with these... (2 Replies)
I'm almost brand new to UNIX, so I have no idea if how easy or difficult this would be, or if it's even possible.
I've been using FTP to copy a total of about 150gb of files to a remote drive. Since the directory being copied is so large, I've been trying to break it up into smaller chunks based... (0 Replies)
Hi Guys,
How can I compare 2 text files and show the differences between the 2 files but have some of the text ignored?
Example
File1 File2
Thomas Thomass !1st name!
David Davidd !2nd name!
John John !3rd name!
So,... (1 Reply)