Sponsored Content
Top Forums Shell Programming and Scripting How do you compare two folders and copy the difference to a third folder in a remote server? Post 302982351 by rovf on Wednesday 28th of September 2016 09:38:48 AM
Old 09-28-2016
Quote:
Originally Posted by cumeh1624
I know how to use rsync to sync up remote destination folder but my requirement is to compare files in local directory to files in remote directory and copy the difference to another directory in same remote server named difference.
You mean: You have a local directory A, and remote directories B and C, and want to copy those files in A to be copied to C, which are different from B? Sorry to misunderstand you.

In this case you indeed need to rule out your own algorithm, but unless I overlook something, you haven't answered yet my earlier question, how you exactly define "difference". Content? Last modification time?

You still can use rsync for getting the difference. Please look at the --dry-run switch of rsync. The idea is to first do a dry run of rsync of A vs. B, collect from it the names of the files which would be copied, and then use this list to actually copy the files to B, using for instance scp or rcp.
 

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

How to compare the difference between a file and a folder??

Hi, I have a .txt file which has to be compared with a folder and print the difference to some other .txt file. I did try with the diff command..i mean diff /tmp/aaa/bbb.txt /space/aaa/bbb/ /***bbb.txt contains all the files names which may or may not exist in the folder bbb..so i need... (2 Replies)
Discussion started by: kumarsaravana_s
2 Replies

2. Shell Programming and Scripting

Remote compare of folders

Hi, Is there a way (either commands/tools/scripts/logic) to compare two given folders on different unix boxes. I want to compare folder a in Unix box 'A' with folder 'b' in Unix box 'B'. I can run the script in Unix box 'A'. I am looking. for following results: files/sub folders only in a... (1 Reply)
Discussion started by: sunilav
1 Replies

3. Shell Programming and Scripting

copy some files from users home folders to my folder

i have users home directories in /home all the users have some files starting with character e and i want to copy all these files in a folder in my (root) home using a script i tried the script for i in m5 do cd m5 cp e1* /home/pc/exam cd .. done but get these... (3 Replies)
Discussion started by: pcrana
3 Replies

4. UNIX for Advanced & Expert Users

How to copy folder from one server to remote one using SSH

ok guys. I have a Centos server running a large website I own. Its is severely struggling under the load and I am having to move to a new load balanced setup. The servers are both remote (to me) and although I can do most things admin wise in nix, I am struggling a bit to come up with a way... (1 Reply)
Discussion started by: anderow
1 Replies

5. Shell Programming and Scripting

Find files of type within folder copy multiple results to different folders

Ok question number two: I'd like to search a directory for multiple file types (rar, txt, deb) and depending on what's found, copy those files to folders named Rar, TextFiles, and Debs. I'm looking for speed here so the faster the script the better. I want it to be a function that I pass 1 argument... (4 Replies)
Discussion started by: DC Slick
4 Replies

6. Shell Programming and Scripting

HELP! Need to compare 2 folders on 2 different systems, and copy unmatched filenames to other folder

This has been tearing my hair out. I need to: 1: compare server1:/data/archive/ to server2:/data/archive/ (through rsync, ssh, etc) 2: filenames that don't match, get copied (scp) to server2:/data/ server1 and server2 have ssh, scp, rsync access between eachother. Is there any option in... (3 Replies)
Discussion started by: damang111
3 Replies

7. Shell Programming and Scripting

Copy folder and files from unix server to linux server

We would be migrating unix solaries to Linux redhat. Basically source is unix and target is linux. i would like to copy entire file system unix/source/* to target linux/souce/* but target linux has only folder setup so what ever files copied need to be placed in the linux server with same... (8 Replies)
Discussion started by: balajikalai
8 Replies

8. Shell Programming and Scripting

Linux Script to compare two folders and copy missing files

Hi, I need help in shell scripting. If someone can help me, that would be great! Problem. I want Linux Script to compare two folders and copy missing files. Description. I have two directories /dir1 /dir2 I need to copy all distinct/new/unique/missing files from /dir1 and that... (1 Reply)
Discussion started by: S.Praveen Kumar
1 Replies

9. Shell Programming and Scripting

Compare Directories and folders on two remote lpar

Hello, i'm trying to script to compare the same directory on the lpar. The problem is i'm using the command : find /etc -type d -ls but i cannot reach the remote lpar. I have another solution to put that command in a text file and compare with Excel. but i could be fine to have the... (7 Replies)
Discussion started by: steiner
7 Replies

10. Shell Programming and Scripting

Copy local files to single remote host but multiple folders using rsync

I'm trying to copy a file myfile.scr from my local Linux server to multiple folders on remote AiX server using single rsync command. Below command helps me copy the file "myfile.scr" from my localhost to a remote host folder "/app/deployment/tmpfiles" rsync --delay-updates -F --compress... (1 Reply)
Discussion started by: mohtashims
1 Replies
RDUP-BACKUPS(7) 						       rdup							   RDUP-BACKUPS(7)

NAME
rdup-backups - introduction into making backups with rdup INTRODUCTION
rdup is a simple program that prints out a list of files and directories that are changed changed on a filesystem. It is more sophisticated than for instance find, because rdup will find files that are removed or directories that are renamed. A long time ago rdup included a bunch of shell and Perl scripts that implemented a backup policy. These could be used in a pipeline to per- form a backup. Currently rdup consists out of three basic utilities: rdup With rdup you create the file list on which later programs in the pipeline can work. The default output format also includes the files' content. rdup can be seen as a tar replacement in this respect, but rdup also allows for all kinds of transformations of the content (encryption, compression, reversal), see the -P switch in rdup(1) for more information. rdup-tr With rdup-tr you can transform the files rdup delivers to you. You can create tar, cpio or pax files. You can encrypt pathnames. rdup-tr is filter that reads from standard input and writes to standard output. See rdup-tr(1) for more information. With rdup and rdup-tr you can create an encrypted archive which is put in a directory structure that is also encrypted. rdup-up With rdup-up you can update an existing directory structure with the updates as described by rdup. rdup-up reads rdup input and will create the files, symbolic links, hard links and directories (and sockets, pipes and devices) in the file system. See rdup-up(1) for more information. So the general backup pipeline for rdup will look something like this: create filelist | transform | update filesystem ( rdup | rdup-tr | rdup-up ) Note 1: The same sequence is used for restoring. In both cases you want to move files from location A to B. The only difference is that the transformation is reversed when you restore. Note 2: The use of rdup-tr is optional. BACKUPS AND RESTORES
For rdup there is no difference between backups and restores. If you think about this for a minute you understand why. Making a backup means copying a list of files somewhere else. Restoring files is copying a list of files back to the place they came from. Same difference. So rdup can be used for both, if you did any transformation with rdup during the backup you just need to reverse those operations during the restore. BACKUPS
It is always best to backup to another medium, be it a different local harddisk or a NFS/CIFS mounted filesystem. You can also use ssh to store file on a remote server, ala rsync (although not as network efficient). If you backup to a local disk you can just as well use rsync or plain old tar, but if you store your files at somebody else's disk you will need encryption. This is where you go beyond rsync and rdup comes in. Rsync cannot do per-file encryption, sure you can encrypt the network traffic with ssh, but at the remote side your files are kept in plain view. If you implement remote backups, the easy route is to allow root access on the backup medium. If the backup runs without root access the created files will not have their original ownership. For NFS this can be achieved by using no_root_squash, for ssh you could enable PermitRootLogin. Note that this may be a security risk. SNAPSHOT BACKUPS
We need a little help here in the form of the rdup-simple script. Keep in mind that the following scripts can also be run remotely with the help of ssh. The following script implements the algorithm of rdup-simple. #!/bin/bash # some tmp files are saved in ~/.rdup. This directory must exist DIR=/home # what to backup BACKUP=/vol/backup TODAY=$(date +%Y%m/%d) LIST=~/.rdup/list-$HOSTNAME STAMP=~/.rdup/timestamp-$HOSTNAME # for remote backup, this has to run on the remote host! BUGBUG RET=$? case $RET in 2|*) echo Error >&2 exit 1 ;; 1) # full dump, remove file-list and time-stamp file rm $LIST $STAMP ;; 0) # inc dump # do nothing here ;; esac # this is the place where you want to modify the command line # right now, nothing is translated we just use 'cat' rdup -N $STAMP -Pcat $LIST $DIR | rdup-up $BACKUP/$HOSTNAME/$TODAY # or do a remote backup #rdup -N $STAMP -Pcat $LIST $DIR | ssh root@remotehost # rdup-up $BACKUP/$HOSTNAME/$TODAY LOCAL BACKUPS
With rdup-simple you can easily create backups. Backing up my home directory to a backup directory: rdup-simple ~ /vol/backup/$HOSTNAME This will create a backup in /vol/backup/$HOSTNAME/200705/15. So each day will have its own directory. Multiple sources are allowed, so: rdup-simple ~ /etc/ /var/lib /vol/backup/$HOSTNAME Will backup your home directory, /etc and /var/lib to the backup location. Also if you need to compress your backup, simple add a '-z' switch: rdup-simple -z ~ /etc/ /var/lib /vol/backup/$HOSTNAME REMOTE BACKUPS
For a remote backup to work, both the sending machine and the receiving machine must have rdup installed. The currently implemented proto- col is ssh. Dumping my homedir to the remote server: rdup-simple ~ ssh://miekg@remote/vol/backup/$HOSTNAME The syntax is almost identical, only the destination starts with the magic string 'ssh://'. Compression and encryption are just as easily enabled as with a local backup, just add '-z' and/or a '-k keyfile' argument: rdup-simple -z -k 'secret-file' ~ ssh://miekg@remote/vol/backup/$HOSTNAME Remember though, that because of these advanced features (compression, encryption, etc, ...) the network transfer can never be as efficient as rsync. ALSO SEE
rdup(1), rdup-tr(1), rdup-up(1) and http://www.miek.nl/projects/rdup/ 1.1.x 15 Dec 2008 RDUP-BACKUPS(7)
All times are GMT -4. The time now is 02:03 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy