Login or Register to Ask a Question and Join Our Community

Sponsored Content
Full Discussion: HELP! Need to compare 2 folders on 2 different systems, and copy unmatched filenames to other folder
Top Forums Shell Programming and Scripting HELP! Need to compare 2 folders on 2 different systems, and copy unmatched filenames to other folder Post 302591564 by jim mcnamara on Thursday 19th of January 2012 11:49:36 PM
Old 01-20-2012
This assumes you set up ssh keys for the user in question
Code:
ssh server1 'cd /data/archive && ls | cksum | grep -v -i directory' > server1.lis
ssh server2 'cd /data/archive && ls | cksum | grep -v -i directory' > server2.lis
awk '{arr[$0]++} END{ for (i in arr){if(arr[i]==1){print arr[i]} } }' server1.lis server2.lis >uniq.txt

This gives you a list of files that are different, either in content and/or in name.
You can then scp the files to server3. I'm tired right now so I will opt for not worrying about failures
too much.
Code:
awk '{print $(NF)}' uniq.txt |
while read fname 
do
    scp server1:/data/archive/${fname} server3:/data/archive/${fname}  || 
    scp server2:/data/archive/${fname} server3:/data/archive/${fname}
done < uniq.txt

TEST this first.
 

10 More Discussions You Might Find Interesting

1. UNIX for Advanced & Expert Users

Auto copy for files from folder to folder upon instant writing

Hello all, I'm trying to accomplish that if a file gets written to folder /path/to/a/ it gets automatically copied into /path/to/b/ the moment its get written. I thought of writing a shell script and cron it that every X amount of minutes it copies these files over but this will not help me... (2 Replies)
Discussion started by: Bashar
2 Replies

2. Shell Programming and Scripting

copy some files from users home folders to my folder

i have users home directories in /home all the users have some files starting with character e and i want to copy all these files in a folder in my (root) home using a script i tried the script for i in m5 do cd m5 cp e1* /home/pc/exam cd .. done but get these... (3 Replies)
Discussion started by: pcrana
3 Replies

3. Shell Programming and Scripting

copy all files with the same filenames as those in another folder

Hi, all: I've got two folders, folder A contains some image files (say, 100 files) in .jpg format; folder B contains all description files (say, 500 files) in .txt format. All image files in folder A are able to find their corresponding description files in folder B. That is to say,... (3 Replies)
Discussion started by: jiapei100
3 Replies

4. Shell Programming and Scripting

Find files of type within folder copy multiple results to different folders

Ok question number two: I'd like to search a directory for multiple file types (rar, txt, deb) and depending on what's found, copy those files to folders named Rar, TextFiles, and Debs. I'm looking for speed here so the faster the script the better. I want it to be a function that I pass 1 argument... (4 Replies)
Discussion started by: DC Slick
4 Replies

5. Shell Programming and Scripting

Change all filenames under different folders...

Hi, all: I'd love to use shell script to change all filenames under different folders once for all: I've got over 100 folders, in each of them, there is a file named "a.ppm". I wanna change all these "a.ppm" to "b.ppm", and still . Visually, the directory structure looks like: and hope... (1 Reply)
Discussion started by: jiapei100
1 Replies

6. Shell Programming and Scripting

How best to remove certain characters from filenames and folders recursively

hello, I'm trying to figure out which tool is best for recursively renaming and files or folders using the characters \/*?”<>| in their name. I've tried many examples that use Bash, Python and Perl, but I'm not much of a programmer I seem to have hit a roadblock. Does anyone have any... (15 Replies)
Discussion started by: prometheon123
15 Replies

7. Shell Programming and Scripting

Linux Script to compare two folders and copy missing files

Hi, I need help in shell scripting. If someone can help me, that would be great! Problem. I want Linux Script to compare two folders and copy missing files. Description. I have two directories /dir1 /dir2 I need to copy all distinct/new/unique/missing files from /dir1 and that... (1 Reply)
Discussion started by: S.Praveen Kumar
1 Replies

8. Shell Programming and Scripting

How to copy files with the same filenames as those in another folder to that same folder?

Hello All A similar question like this was asked before but I need to change part of the question. I've two folders, Folder A contains some image files in 150 subfolders; Folder B contains text files in 350 subfolders. All image files in Folder A have the same filename as the text... (5 Replies)
Discussion started by: chlade
5 Replies

9. Shell Programming and Scripting

How do you compare two folders and copy the difference to a third folder in a remote server?

How do you compare one local folder and a remote folder and copy the difference to a third folder in a remote folder.e.g. Folder A -- Is in a remote server and it has the following files TEST1.OUT TEST2.OUT TEST3.OUT Folder B --Is in a local server and it has the following files ... (5 Replies)
Discussion started by: cumeh1624
5 Replies

10. Shell Programming and Scripting

Script to compare partial filenames in two folders and delete duplicates

Background: I use a TV tuner card to capture OTA video files (.mpeg) and then my Plex Media Server automatically optimizes the files (transcodes for better playback) and places them in a new directory. I have another Plex Library pointing to the new location for the optimized .mp4 files. This... (2 Replies)
Discussion started by: shaky
2 Replies

LEARN ABOUT DEBIAN

parallel-rsync

PARALLEL-RSYNC(1)														 PARALLEL-RSYNC(1)

NAME

       parallel-rsync - deploy files to listed hosts

SYNOPSIS

       parallel-rsync [OPTIONS] -h hosts.txt local remote

DESCRIPTION

       pssh provides a number of commands for executing against a group of computers, using SSH. It's most useful for operating on clusters of
       homogenously-configured hosts.

       parallel-rsync deploy files files to all hosts you listed.

OPTIONS

       -r --recursive
	   recusively copy directories (OPTIONAL)

       -a ----archive
	   use rsync -a (archive mode) (OPTIONAL)

       -z --compress
	   use rsync compression (OPTIONAL)

       -h --hosts
	   hosts file (each line "host[:port] [user]")

       -l --user
	   username (OPTIONAL)

       -p --par
	   max number of parallel threads (OPTIONAL)

       -o --outdir
	   output directory for stdout files (OPTIONAL)

       -e --errdir
	   output directory for stderr files (OPTIONAL)

       -t --timeout
	   timeout (secs) (-1 = no timeout) per host (OPTIONAL)

       -O --options
	   SSH options (OPTIONAL)

       -v --verbose
	   turn on warning and diagnostic messages (OPTIONAL)

EXAMPLE

	   # parallel-rsync -r -h hosts.txt -l irb2 foo /home/irb2/foo

ENVIRONMENT

       All four programs take similar sets of options. All of these options can be set using the following environment variables:

       o   PSSH_HOSTS
       o   PSSH_USER
       o   PSSH_PAR
       o   PSSH_OUTDIR
       o   PSSH_VERBOSE
       o   PSSH_OPTIONS

SEE ALSO

       parallel-ssh(1), parallel-scp(1), parallel-slurp(1), parallel-nuke(1), ssh(1), rsync(1)

AUTHOR

       Brent N. Chun <bnc@theether.org>

COPYING

       Copyright: 2003, 2004, 2005, 2006, 2007 Brent N. Chun

NOTES

	1. bnc@theether.org
	   mailto:bnc@theether.org

								    03/30/2009							 PARALLEL-RSYNC(1)
All times are GMT -4. The time now is 08:57 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy