Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

HELP! Need to compare 2 folders on 2 different systems, and copy unmatched filenames to other folder

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting HELP! Need to compare 2 folders on 2 different systems, and copy unmatched filenames to other folder
# 1  
Old 01-19-2012
HELP! Need to compare 2 folders on 2 different systems, and copy unmatched filenames to other folder
This has been tearing my hair out.

I need to:
1: compare server1:/data/archive/ to server2:/data/archive/ (through rsync, ssh, etc)
2: filenames that don't match, get copied (scp) to server2:/data/

server1 and server2 have ssh, scp, rsync access between eachother.

Is there any option in rsync to check/compare server1:/data/archive/ to server2:/data/archive/ and have the files synced to another destination such as server2:/data/ ??

TIA
# 2  
Old 01-20-2012
This assumes you set up ssh keys for the user in question
Code:
ssh server1 'cd /data/archive && ls | cksum | grep -v -i directory' > server1.lis
ssh server2 'cd /data/archive && ls | cksum | grep -v -i directory' > server2.lis
awk '{arr[$0]++} END{ for (i in arr){if(arr[i]==1){print arr[i]} } }' server1.lis server2.lis >uniq.txt

This gives you a list of files that are different, either in content and/or in name.
You can then scp the files to server3. I'm tired right now so I will opt for not worrying about failures
too much.
Code:
awk '{print $(NF)}' uniq.txt |
while read fname 
do
    scp server1:/data/archive/${fname} server3:/data/archive/${fname}  || 
    scp server2:/data/archive/${fname} server3:/data/archive/${fname}
done < uniq.txt

TEST this first.
# 3  
Old 01-20-2012
PS: rsync can do a compare and not move files. But, it cannot sync to a third directory based on files not existing in duplicate directories somewhere else.
# 4  
Old 01-20-2012
if need compare the content:
Code:
for server in server1 server2
do 
    ssh $server 'find /data/archive -type f -exec md5sum {} \;| sort >$server.list
done

only check the filename
Code:
ssh server1 'find /data/archive -type f -print| sort >server1.lis

The rest should be same as above
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Script to compare partial filenames in two folders and delete duplicates

Background: I use a TV tuner card to capture OTA video files (.mpeg) and then my Plex Media Server automatically optimizes the files (transcodes for better playback) and places them in a new directory. I have another Plex Library pointing to the new location for the optimized .mp4 files. This... (2 Replies)
Discussion started by: shaky
2 Replies

2. Shell Programming and Scripting

How do you compare two folders and copy the difference to a third folder in a remote server?

How do you compare one local folder and a remote folder and copy the difference to a third folder in a remote folder.e.g. Folder A -- Is in a remote server and it has the following files TEST1.OUT TEST2.OUT TEST3.OUT Folder B --Is in a local server and it has the following files ... (5 Replies)
Discussion started by: cumeh1624
5 Replies

3. Shell Programming and Scripting

How to copy files with the same filenames as those in another folder to that same folder?

Hello All A similar question like this was asked before but I need to change part of the question. I've two folders, Folder A contains some image files in 150 subfolders; Folder B contains text files in 350 subfolders. All image files in Folder A have the same filename as the text... (5 Replies)
Discussion started by: chlade
5 Replies

4. Shell Programming and Scripting

Linux Script to compare two folders and copy missing files

Hi, I need help in shell scripting. If someone can help me, that would be great! Problem. I want Linux Script to compare two folders and copy missing files. Description. I have two directories /dir1 /dir2 I need to copy all distinct/new/unique/missing files from /dir1 and that... (1 Reply)
Discussion started by: S.Praveen Kumar
1 Replies

5. Shell Programming and Scripting

How best to remove certain characters from filenames and folders recursively

hello, I'm trying to figure out which tool is best for recursively renaming and files or folders using the characters \/*?”<>| in their name. I've tried many examples that use Bash, Python and Perl, but I'm not much of a programmer I seem to have hit a roadblock. Does anyone have any... (15 Replies)
Discussion started by: prometheon123
15 Replies

6. Shell Programming and Scripting

Change all filenames under different folders...

Hi, all: I'd love to use shell script to change all filenames under different folders once for all: I've got over 100 folders, in each of them, there is a file named "a.ppm". I wanna change all these "a.ppm" to "b.ppm", and still . Visually, the directory structure looks like: and hope... (1 Reply)
Discussion started by: jiapei100
1 Replies

7. Shell Programming and Scripting

Find files of type within folder copy multiple results to different folders

Ok question number two: I'd like to search a directory for multiple file types (rar, txt, deb) and depending on what's found, copy those files to folders named Rar, TextFiles, and Debs. I'm looking for speed here so the faster the script the better. I want it to be a function that I pass 1 argument... (4 Replies)
Discussion started by: DC Slick
4 Replies

8. Shell Programming and Scripting

copy all files with the same filenames as those in another folder

Hi, all: I've got two folders, folder A contains some image files (say, 100 files) in .jpg format; folder B contains all description files (say, 500 files) in .txt format. All image files in folder A are able to find their corresponding description files in folder B. That is to say,... (3 Replies)
Discussion started by: jiapei100
3 Replies

9. Shell Programming and Scripting

copy some files from users home folders to my folder

i have users home directories in /home all the users have some files starting with character e and i want to copy all these files in a folder in my (root) home using a script i tried the script for i in m5 do cd m5 cp e1* /home/pc/exam cd .. done but get these... (3 Replies)
Discussion started by: pcrana
3 Replies

10. UNIX for Advanced & Expert Users

Auto copy for files from folder to folder upon instant writing

Hello all, I'm trying to accomplish that if a file gets written to folder /path/to/a/ it gets automatically copied into /path/to/b/ the moment its get written. I thought of writing a shell script and cron it that every X amount of minutes it copies these files over but this will not help me... (2 Replies)
Discussion started by: Bashar
2 Replies
Login or Register to Ask a Question