Help needed with Content synchronization script using rsync


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Help needed with Content synchronization script using rsync
# 1  
Old 02-19-2014
Help needed with Content synchronization script using rsync

Hi All,

I need a .sh script which serves the below purpose. It would be of great help.
I am unable to perform the second need of mapping texts within the file and updating only those contents.
For Eg: There is a html file containing link abc.com. This should be replaced with xyz.com link in the destination folder.

##########################################
1)content sync from source to target machine, based on a extensive list of directories and hostnam/ip

2)search and replace of the mapped texts from source to target

3)replacement should be only in changed/updated files and to a configurable set of file extensions
#######################################

Below is my script workthrough as of now:
Code:
#############################################
#!/bin/sh
#!/usr/bin/perl

##################################################################################
#
#
# This script is used to sync the contents from source directory of local #machine
# to destination directory of target machine.
# 
##################################################################################

################################################################################
# Logging directory
################################################################################
LOGDIR="Path to logging directory"
################################################################################
# RSync binary location
################################################################################
rsync="/opt/mw/rsync-2.6.6/bin/rsync"

################################################################################
# RSync binary location on remote server
# this is only used when source and target directories are on different servers
# otherwise it should be same rsync value
################################################################################
rsync_path="/opt/mw/rsync-2.6.6/bin/rsync"

################################################################################
# For Content mapping below mentioned directory paths and hostnames are required
################################################################################
SOURCEDIR="Path to Source Directory on Local Machine"
TARGETDIR="Path to Target Directory on Remote Machine"
BACKUPDIR="Path to Backup Directory"
TARGETSERVER="Hostname"
TargetUser="Username"
LOGFILE="Path to Log file"
Rsync_Options = "--recursive --times --archive --verbose --update --links --checksum --progress --delete"
#-v, --verbose increase verbosity 
#-a, --archive archive mode
#-r, --recursive recurse into directories 
#-u, --update skip files that are newer on the receiver 
#-t, --times preserve modification times 
#-l, --links copy symlinks as symlinks
#-c, --checksum skip based on checksum, not mod-time & size 
#--dry-run
#rsync --recursive --times --archive --verbose --update --links --checksum --progress --delete


################################################################################
# RSync filters
# file/folders mentioned in the filter will be skipped for synchronization
# for more details on rsync filter read rsync documentation
################################################################################
rsync_filter="- .backup/"
#################################################################################
# Check if the log file exists
################################################################################
if [ ! -e $LOGFILE ]; then
     touch $LOGFILE
fi
if [ ! -d $TARGETDIR ]; then
     echo "Log destination directory doesn't seem to exist. Please investigate"
     exit 2
fi

echo $'\n\n' >> $LOGFILE

#####################################################################
#
# Gather a list of the files from source directory and the remote directory.
# It finds the file differences and copy the files to the destination directory.
# Only missing files are copied.
#
#####################################################################

#for i in `find $SOURCEDIR -type f`
#do
rsync $Rsync_Options -e ssh $SOURCEDIR/ $TargetUser@$TARGETSERVER :$TARGETDIR/ 
#done
echo "Sync Completed at:`/bin/date`" >> $LOGFILE


Last edited by zaxxon; 02-19-2014 at 08:42 AM..
# 2  
Old 02-19-2014
If you have some disk space available you could:

1- do a rsync to a temp dir.
2- Change all your files/links
3- rsync the temp dir to target server

Benefits would be that you don't mod the source files at the price of performance (since you do two rsync instead of one)
# 3  
Old 02-20-2014
Hi Maverick,

Thanks for the reply. As i am not having much disk space, i want to include any command within the script which does a mapping of text.
# 4  
Old 02-21-2014
Well you do not have much options then.

Best thing would be to do your rsync and change the files on the remote server:

- Rsync on remote server
- Find/edit/replace your files

You still don't touch your source files and you do not need double disk space on the other end since you will be editing the files directly.

Since you should probably have ssh keys on source and remove servers you could triger your script to ssh <command> to edit your files.

What are the conditions/changes/results your are targeting at the end?
Login or Register to Ask a Question

Previous Thread | Next Thread

9 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Help Needed for creating the folder by checking today's date and, take backup using rsync command

How to create a shell script to create a folder by using the today's date to take backup using rsync command on every evening around 7 pm. Kindly help. Thanks. To be more precise, I want to create a script which matches the today's date with server's date format, if matches then creates the... (2 Replies)
Discussion started by: bakula10
2 Replies

2. Shell Programming and Scripting

Rsync Error: rsync: link_stat failed: No such file or directory (2)

I wish to copy all the files & folder under /web/Transfer_Files/data/ on mymac1 (Linux) to remote server mybank.intra.com (Solaris 10) /tmp/ location I am using Ansible tool synchronize module which triggers the unix rsync command as below:rsync --delay-updates -F --compress --archive --rsh=ssh... (2 Replies)
Discussion started by: mohtashims
2 Replies

3. Shell Programming and Scripting

Help needed in sending file content with colors and borders

HI i am running a shell script in cron and storing the output of that script in a file say test.then i am copying the content of test to test1 and i will send the output of test to some email ids using mutt. Next time when the script executes i am comparing the contents of test and test1 and... (3 Replies)
Discussion started by: venkitesh
3 Replies

4. Shell Programming and Scripting

Help Needed on RSYNC

My Script as below ####### #!/usr/bin/ksh DIR=$1 MN=`echo $1 | awk -F'/' '{print $NF}'` LF=./rsynclog RSYNC=/usr/local/bin/rsync mv $LF.$MN $LF.$MN.prev compress $LF.$MN.prev $RSYNC -arz --rsync-path=$RSYNC --log-file=$LF.$MN SERVER1:$DIR/ $DIR While running it I am getting... (1 Reply)
Discussion started by: Atp3530
1 Replies

5. Solaris

I need to synchronization by rsync

Hi everyone! I wonder about the rsync. The purpose I just want: synchronization from server 1 to server 2 by rsync, example: Node 1: /app/logs/* Node 2: /app/logs/* in Node 1 when I used rsync -aru /app/logs/ node2:/app/logs/, then everything will be copied and update to node 2. But,... (1 Reply)
Discussion started by: trantuananh24hg
1 Replies

6. Shell Programming and Scripting

Synchronization, rsync option

Dear all, I am synchronizing some files on 2 hosts. host2 is a backup server of host1, but can have some more up to date files. I use on host2: rsync -auv usrer@host1:/Source_Dir Dest_DirOption -u (update) will copy and replace the newest files from host2 to host1. Before this is done I... (1 Reply)
Discussion started by: freddie50
1 Replies

7. Shell Programming and Scripting

help with sed needed to extract content from html tags

Hi I've searched for it for few hours now and i can't seem to find anything working like i want. I've got webpage, saved in file par with form like this: <html><body><form name='sendme' action='http://example.com/' method='POST'> <textarea name='1st'>abc123def678</textarea> <textarea... (9 Replies)
Discussion started by: seb001
9 Replies

8. Shell Programming and Scripting

Help with rsync script

Good day all, I need your help in editing a rsync script that I am trying to write and using ksh shell. I want to add multiple patterns --include and --exclude options with variable inputs called SID. I have created two files and named them N1 and N2 and created subfiles in each one In... (4 Replies)
Discussion started by: Noura-83
4 Replies

9. Shell Programming and Scripting

Help needed for rsync

How can i combine the below two rsync commands into one: rsync -auvL --delete /etc /usr/local/MediaServer /mnt/bak/livecore/filesystem.0 > /tmp/backup_log/rsync.log 2>&1 and rsync -auv --delete --exclude='live/public_html/video' --exclude='live/public_html.old' /var/www... (0 Replies)
Discussion started by: proactiveaditya
0 Replies
Login or Register to Ask a Question