rsync a file as it gets created and after 3 minutes old
- run a backup job
- The jobs creates partial files one after the other, about 2 minutes interval.
What i want to do is that while the job is still running or while a file was last modified or created 3 minutes ago, the
file should be rsync to a remote server untill the last file has been created and 3 minutes old.
I am learning how to script.
I have tried different options but no success
The closest i got was that the files were copied up to the current last file created and the script exited without waiting for the subsequent files to be created.
In other words i could not get the script to loop until it picked up all the files before exiting.
In my test i used ' echo " nohup rsync -a --inplace /mara/backup/mala$filename mara@tronic9922:/mara/backup/mara/ $ " >> thejobs ' instead of doing rsync dynamically.
But what i really need is to use rsync, to i can send each file to the remove server as it is created and its is 3 minutes old.
Hopefully someone can fix this for me or come up with a new script all together.
- see bottom for the files that get created.
Thanks
Code:
#!/bin/ksh
for filename in `find . -mmin -3 -type f|grep bkup`;
do
BKPPROCCNT=`ps -ef |grep "/royal/current/bin/_muller runbkup"|grep -v grep |wc -l` echo $filename
if [[ $BKPPROCCNT -eq 1 ]] || [[ -s $filename ]];
then
nohup rsync -a --inplace /home/backup/mala/$filename mala@tronic9922:/home/backup/mala/ $
else
fi
sleep 3
done
Code:
#!/bin/ksh
for filename in `find . -mmin +2 -type f|grep bkup`;
do
BKPPROCCNT=`ps -ef |grep "/royal/current/bin/_muller runbkup"|grep -v grep |wc -l`
echo $filename
if [[ $BKPPROCCNT -eq 1 ]] && [[ -s $filename ]];
then
nohup rsync -a --inplace /home/backup/mala$filename mala@tronic9922:/home/backup/mala/ $
else
echo "backup ended or no file found"
fi
sleep
done
TRYING TO USE A WHILE LOOP AS THE BACKUP JOB IS RUNNING - this i could not figure it out.
Code:
#!/bin/ksh
set BKPID = `ps -ef |grep "/royal/current/bin/_muller runbkup"|awk '{print $2}'|head -1`
while ("$BKID" != "")
do
if [ test `find /vpas/backup/mala -name bkup* -mmin +5` ] then
then
---
nohup rsync -a --inplace /home/backup/mala$filename mala@tronic9922:/home/backup/mala/ $
else
echo "backup ended or no file found"
fi
sleep 3
done
cd /home/backup/mala
ls -lrt
-rw-rw-r-- 1 mala dev 7516081152 Apr 11 19:50 bkup1
-rw-rw-r-- 1 mala dev 7516081152 Apr 11 19:51 bkup2
-rw-rw-r-- 1 mala dev 7516081152 Apr 11 19:52 bkup3
-rw-rw-r-- 1 mala dev 7516081152 Apr 11 19:53 bkup4
-rw-rw-r-- 1 mala dev 7516081152 Apr 11 19:55 bkup5
-rw-rw-r-- 1 mala dev 7516081152 Apr 11 19:56 bkup6
-rw-rw-r-- 1 mala dev 7516081152 Apr 11 19:57 bkup7
-rw-rw-r-- 1 mala dev 7516081152 Apr 11 19:57 bkup8
-rw-rw-r-- 1 mala dev 7516081152 Apr 11 19:58 bkup9
-rw-rw-r-- 1 mala dev 7516081152 Apr 11 19:59 bkup10
-rw-rw-r-- 1 mala dev 3943402496 Apr 11 19:59 bkup11
Moderator's Comments:
Please do wrap your codes or samples in CODE TAGS as per forum rules.
Last edited by RavinderSingh13; 04-12-2019 at 02:02 AM..
Do you know when the "last file" has been created? Are you able to identify it?
You seem to know the creating program / script. Are you able to modify it?
Do you REALLY have to rsyncbefore the last files has arrived? Can't you wait until they're complete? The process doesn't seem to take that long...
Do you know when the "last file" has been created? Are you able to identify it? NO
You seem to know the creating program / script. Are you able to modify it? If you can modify it to work that will be great.
Quote:
Do you REALLY have to rsync before the last files has arrived? Can't you wait until they're complete? The process doesn't seem to take that long.
So the sample i showed you is just for splitting a 73 G file. In actual fact the file i have to split is 3 TB. It will take over 5 hours just to split all the files. It will take another 15 hours just to rsync to complete sending all the files to a remote location which about 1200 miles away. If I ship the 3TB+ as a whole it will take about 2 days to rsync the bulk file over 1200 miles between hostA and hostB.
The reason behind not waiting for all the files to complete the splitting, is because i want to save time.
Once i receive all the files on the remote server , i will have to worry about total time to restore. If i go through the old school method of shipping 3TB at once to the remote site and restoring at the remote site it takes me about 3 days depending the distance between server A and server B.
If i have to wait until all the files are split before i rsync each file , then I have to endure the time it takes for all the files to split, which depending on the size of the file could be humongous.
The smart thing, is to split, dont wait for the next file, rsync the first spit file , wait till the nest split file to complete and rynsc and so forth until the last file is split. Hopefull by ussing a "nohup" , i can rsynch each file using different sessions.
Last edited by RavinderSingh13; 04-12-2019 at 02:37 PM..
I'd be suprised if sending the split files in parallel would improve the time consumed, as the bandwidth will be saturated anyhow. What be the available bandwidth, btw? Do you have independent routes from A to B?
RudiC, I don't think he's trying to parallelize the upload, just send the first tenth without waiting for the computer to finish processing the last tenth.
Step back a little. Why are you using rsync to send 10 files? What exact problem is being solved by splitting and transferring the file this way? Is your connection unreliable?
I don't think you can "trick" rsync into using incomplete files, you may need to arrange something else. It may be possible to avoid splitting the file at all!
What kind of connection do you have between yourself and the destination? 2 tb in 2 days sounds pretty fast. Are you on the same subnet, even virtually?
What's your operating system? On Linux, I think you can use losetup to create a device that's a "view" into a file on disk. So instead of creating a 200 gigabyte file, you'd just create a device which maps to the first 200 gigabytes with no copying or fuss. Whether rsync will swallow loopback files remains to be seen.
If you don't need to use rsync to transfer, you can just grab sections of file by any means you like and transfer them without writing the split file to disk at all.
Last edited by Corona688; 04-12-2019 at 04:38 PM..
A process xyz is running and creating file1, file2, file3, .... filen. how do i know if the process has stopped and createtime of the last file (filen) is older than 5 minutes?
OS is AIX (3 Replies)
Hello all,
Info:
System RedHat 7.5
I need to create a script that based on the creation time,
if the file is older then 5 minutes then execute some stuff, if not exit.
I thought to get the creation time and minutes like this.
CreationTime=$(stat -c %y /tmp/test.log | awk -F" " '{ print... (3 Replies)
Hi all,
System Ubuntu 16.04.3 LTS
i have the following log
INFO 2019-02-07 15:13:31,099 module.py:700] default: "POST /join/8550614e-3e94-4fa5-9ab2-135eefa69c1b HTTP/1.0" 500 2042
INFO 2019-02-07 15:13:31,569 module.py:700] default: "POST /join/6cb9c452-dcb1-45f3-bcca-e33f5d450105... (15 Replies)
Hi, please help with below time conversion to minutes.
one column values:
2 minutes 16 seconds 420 msec
43 seconds 750 msec
0 days 3 hours 29 minutes 58 seconds 480 msec
11 seconds 150 msec
I need output in minutes(total elapsed time in minutes) (2 Replies)
I wish to copy all the files & folder under /web/Transfer_Files/data/ on mymac1 (Linux) to remote server mybank.intra.com (Solaris 10) /tmp/ location
I am using Ansible tool synchronize module which triggers the unix rsync command as below:rsync --delay-updates -F --compress --archive --rsh=ssh... (2 Replies)
I wanted to send an email to the client whenever there is failed record created in a /feed/HR-76/failed folder after processing of feed file.
I can find out with the help of below script that what is the new file created but that file didn't make just 15 minutes before.
... (1 Reply)
Hi ,
I need help in getting how many minutes ago the last file, matching some pattern in file name, was created in a folder.
Thanks in advance. (8 Replies)
Hi
Can some one please help me
How can i search a file which has been created or modified in last five minutes
I have used the command
find . -mmin -5
and it does not work
i get an error -mmin is bad option
Please help
Much regards
Tarun (2 Replies)
04-12-2019
