Help with changing rsync script


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Help with changing rsync script
# 1  
Old 02-22-2011
Help with changing rsync script

Code:
#/bin/bash
#set -vx
DST_SRV=<destination_hostname>
MDATE=`date +%Y%m%d%H%M`
SRC_CONTENT="/home/prad/sourcecontent/"
DST_CONTENT="/tmp/prad/destinationfolder/"
DST_LOG="/tmp/prad/STATS"
CURRENT_LOG="/home/prad/STATS/rsync-current.log"
EMAIL="/home/prad/EMAIL/email.log"
LOG="/home/prad/STATS/rsync.log"
ELOG="/home/prad/STATS/rsync-error.log"
RCMD="/usr/bin/rsync -avr --stats --progress --delete --timeout=30"
MAILCMD="mail -s"
EMAILGR="user@domain.com"

> $EMAIL
> $CURRENT_LOG

if [ `ps -elf |grep -v grep |grep /usr/bin/rsync | awk '{ print $4 }'` ]

then
        echo "-----  Process is running  ----`date`----" >> $ELOG 2>&1

else
        echo "----  Starting rsync  -----" >> $CURRENT_LOG 2>&1
        SDATE=`date`
        $RCMD $SRC_CONTENT $DST_SRV:$DST_CONTENT >> $CURRENT_LOG 2>&1

        if [ $? -eq "0" ]
        then
                echo "----  Rsync started at $SDATE  ---"  >> $CURRENT_LOG 2>&1
                echo "----  Rsync completed at `date`  ----" >> $CURRENT_LOG 2>&1
                grep "Rsync started at" $CURRENT_LOG >> $EMAIL
                grep "Number of files:" $CURRENT_LOG >> $EMAIL
                grep "Number of files transferred:" $CURRENT_LOG >> $EMAIL
                grep "Total file size:" $CURRENT_LOG >> $EMAIL
                grep "Total transferred file size:" $CURRENT_LOG >> $EMAIL
                grep "Total bytes sent:" $CURRENT_LOG >> $EMAIL
                grep "Total bytes received:" $CURRENT_LOG >> $EMAIL
                grep "Rsync completed at" $CURRENT_LOG >> $EMAIL
                cat $CURRENT_LOG >> $LOG
                cat $EMAIL | $MAILCMD "Prad Test script - RSYNC of source content completed successfully" $EMAILGR
        else
                echo "----  Rsync Error at `date` ----" >> $CURRENT_LOG 2>&1
                echo "----  Rsync Error at `date` ----" >> $EMAIL
                echo "Please check the error log" >> $EMAIL
                cat $CURRENT_LOG >> $ELOG
                cat $EMAIL | $MAILCMD "Prad test script- RSYNC of source content failed" $EMAILGR
        fi
fi
scp $LOG $DST_SRV:$DST_LOG
echo "=======Log File uploaded to destination server-<destination_hostname> ==`date`====" >> $LOG 2>&1

This script is pretty simple and self explanatory. All it does is that it copies the content from source server to destination server. The cron is set up in such a way that this rsync script runs very frequently! So result is that you get hundreds of emails stating "Prad Test script - RSYNC of source content completed successfully".

Here is what I am planning to do:
1. I should get only 1 email at end of day instead of my mail box getting flooded with rsync success or failure messages.
2. In that email report, I should be able to see the number of times rsync of source content was "success" and also the number of times rsync of source content "failed"


My approach:
1. I am trying to send the results to a file instead of email
2. Count the keywords "completed" and "failed" from that file.

Drawback:
The file will just grow in size and I need to come up with a way to remove the contents of the file after certain days.
Note: I am yet to research this step!

Please let me know if my approach makes sense?? If not, please suggest me some alternatives!

Last edited by pludi; 02-22-2011 at 09:25 PM..
# 2  
Old 02-22-2011
I think that you should send rsync output into a temporary file ,and cron another script to check the temporary file , mail the result to your box,then empty the file
# 3  
Old 02-23-2011
Here is what I tried

I sent the success and failure messages to 2 different files- rsync-success-outputfile and rsync-failure-outputfile.

Wrote an other script called count.sh
Code:
#!/bin/ksh
success=0
failure=0
for i in  $(cat /tmp/OUTPUT/rsync-success-outputfile)
do
if [ $i == RSYNC of content store completed successfully ]; #name which you want to find from the given file
then
success=`expr $success + 1`
fi
done
for i in  $(cat /tmp/OUTPUT/rsync-failure-outputfile)
do
if [ $i == RSYNC of content store failed ]; #name which you want to find from the given file
then
failure=`expr $failure + 1`
fi
done
echo $success
echo $failure


Other than counting, I want to display a report of some sort stating rsync failed 4 times and they were at 1PM, 2PM, 3PM, 4PM ....whatever.

The sole aim is to know at what time rsync failed. I really do not bother about the time frame where rsync was a success. All, I need is to know how many times rsync error occured and at what time specifically?

Any suggestion?

BTW, thanks for your response. It helped me a lot.

Last edited by Franklin52; 02-24-2011 at 03:31 AM.. Reason: Please use code tags
# 4  
Old 02-27-2011
i think that you should log the time when your rsync command run into your log file ,like

time:
your rsync output
....
time:
your rsync output

====

that will be easy to analysis (use sed tool or something you like)
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Rsync Error: rsync: link_stat failed: No such file or directory (2)

I wish to copy all the files & folder under /web/Transfer_Files/data/ on mymac1 (Linux) to remote server mybank.intra.com (Solaris 10) /tmp/ location I am using Ansible tool synchronize module which triggers the unix rsync command as below:rsync --delay-updates -F --compress --archive --rsh=ssh... (2 Replies)
Discussion started by: mohtashims
2 Replies

2. Shell Programming and Scripting

Help with rsync file restore script

Hello Friends, I am trying to create an rsync file restore script. I will post want I have below and explain the problem I am having. #!/bin/bash # # Partial Rsync File Restore Script # # clear echo # Shell loads into /raid0/data/backup/bin/ cd .. # I cd to the backup... (1 Reply)
Discussion started by: jdavis_33
1 Replies

3. Shell Programming and Scripting

Cp script like rsync

Hi Folks, I have a NAS running busybox with a ARM processor. Several PC's backup full weekly and several daily incrementals to the NAS. The NAS has two USB attached disks, which are swapped off site weekly. The problem is the built in rsync only transfers at 2Mb/s to the external USB... (2 Replies)
Discussion started by: bigal.nz
2 Replies

4. Shell Programming and Scripting

Rsync script and backup problems.

Hello everybody I'm triing since few days to do this. So sorry if my question looks stupide, but i've tried. I have to get picture from a folder (who is updated automaticly and with subfolder) with theirs extensions (i'm ok on that) and this files have to me copied in a folder where a website... (7 Replies)
Discussion started by: boytruc
7 Replies

5. Shell Programming and Scripting

Help with rsync script

Good day all, I need your help in editing a rsync script that I am trying to write and using ksh shell. I want to add multiple patterns --include and --exclude options with variable inputs called SID. I have created two files and named them N1 and N2 and created subfiles in each one In... (4 Replies)
Discussion started by: Noura-83
4 Replies

6. Shell Programming and Scripting

Help with rsync script

Good day all, I need your help in editing a rsync script that I am trying to write and using ksh shell. I want to add multiple patterns --include and --exclude options with variable inputs called SID. I have created two files and named them N1 and N2 and created subfiles in each one In N1... (1 Reply)
Discussion started by: Noura-83
1 Replies

7. UNIX for Advanced & Expert Users

Rsync Script Modification

Hi I have a rsync that runs from a remote location to our central data store. The sync generates a log locally not in the central location. I would like to modify the rsync to add the following The log, instead of filing locally it gets sent to a area on the central data store once the syncs... (0 Replies)
Discussion started by: treds
0 Replies

8. OS X (Apple)

rsync is changing permissions

I have the following command. This is meant to download all files from my server to the Downloads folder of my startup drive: /usr/local/bin/rsync -avve ssh --numeric-ids --delete --ignore-errors -R grndlvl@myserver.com:/usr/home/./grndlvl grndlvl@myserver.com:/usr/home/./grndlvl/mail_boxes/... (8 Replies)
Discussion started by: rlinsurf
8 Replies

9. Shell Programming and Scripting

rsync script ...

Hi all, I am a newbie in unix and i want a script which should sync files between two servers like rsync. The condition is that. after rsync command is completed it should also list me how many files has been changed during the rsync execution. if the rsync execution has failed , then... (2 Replies)
Discussion started by: abalakrishnan
2 Replies

10. Shell Programming and Scripting

Rsync script in cron from stepping on itself

I have the following rsync script that I use for syncing MySQL files from one server to another. I run the script at 20 minutes past every second hour in cron. I want to make sure that the script completes in it's entirety before it is set to kick off again. For example, when the script starts at... (3 Replies)
Discussion started by: sunsysadm2003
3 Replies
Login or Register to Ask a Question