rsync a file as it gets created and after 3 minutes old Post: 303033858

Sponsored Content

Top Forums UNIX for Beginners Questions & Answers rsync a file as it gets created and after 3 minutes old Post 303033858 by Corona688 on Friday 12th of April 2019 03:17:16 PM

04-12-2019

Registered User

RudiC, I don't think he's trying to parallelize the upload, just send the first tenth without waiting for the computer to finish processing the last tenth.

Step back a little. Why are you using rsync to send 10 files? What exact problem is being solved by splitting and transferring the file this way? Is your connection unreliable?

I don't think you can "trick" rsync into using incomplete files, you may need to arrange something else. It may be possible to avoid splitting the file at all!

What kind of connection do you have between yourself and the destination? 2 tb in 2 days sounds pretty fast. Are you on the same subnet, even virtually?

What's your operating system? On Linux, I think you can use losetup to create a device that's a "view" into a file on disk. So instead of creating a 200 gigabyte file, you'd just create a device which maps to the first 200 gigabytes with no copying or fuss. Whether rsync will swallow loopback files remains to be seen.

If you don't need to use rsync to transfer, you can just grab sections of file by any means you like and transfer them without writing the split file to disk at all.

Last edited by Corona688; 04-12-2019 at 04:38 PM..

This User Gave Thanks to Corona688 For This Post:

Corona688

View Public Profile for Corona688

Visit Corona688's homepage!

Find all posts by Corona688

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

How can i search a file which has been created or modified in last five minutes

Hi Can some one please help me How can i search a file which has been created or modified in last five minutes I have used the command find . -mmin -5 and it does not work i get an error -mmin is bad option Please help Much regards Tarun

2. Shell Programming and Scripting

find files created within 30 minutes

find . -name *.txt -mmin -30 This is working in Redhat but not in Solaris.. What is the equivalent option in Solaris?

3. Shell Programming and Scripting

Ftp get files created in last 30 minutes

Is it possible in an ftp script to get remote files based on whether they have been created in the last 30 minutes?

4. Shell Programming and Scripting

Check how many minutes ago the last file created

Hi , I need help in getting how many minutes ago the last file, matching some pattern in file name, was created in a folder. Thanks in advance.

5. Shell Programming and Scripting

How to send a file in UNIX through email which is created only 15 minutes before the current time?

I wanted to send an email to the client whenever there is failed record created in a /feed/HR-76/failed folder after processing of feed file. I can find out with the help of below script that what is the new file created but that file didn't make just 15 minutes before. ...

6. Shell Programming and Scripting

Rsync Error: rsync: link_stat failed: No such file or directory (2)

I wish to copy all the files & folder under /web/Transfer_Files/data/ on mymac1 (Linux) to remote server mybank.intra.com (Solaris 10) /tmp/ location I am using Ansible tool synchronize module which triggers the unix rsync command as below:rsync --delay-updates -F --compress --archive --rsh=ssh...

7. UNIX for Beginners Questions & Answers

How to convert days hours minutes seconds to minutes?

Hi, please help with below time conversion to minutes. one column values: 2 minutes 16 seconds 420 msec 43 seconds 750 msec 0 days 3 hours 29 minutes 58 seconds 480 msec 11 seconds 150 msec I need output in minutes(total elapsed time in minutes)

8. Shell Programming and Scripting

Grep a log file for the last 5 minutes of contents every 5 minutes

Hi all, System Ubuntu 16.04.3 LTS i have the following log INFO 2019-02-07 15:13:31,099 module.py:700] default: "POST /join/8550614e-3e94-4fa5-9ab2-135eefa69c1b HTTP/1.0" 500 2042 INFO 2019-02-07 15:13:31,569 module.py:700] default: "POST /join/6cb9c452-dcb1-45f3-bcca-e33f5d450105...

9. Shell Programming and Scripting

Check file creation Time minutes and if file older then 5 minutes execute some stuff

Hello all, Info: System RedHat 7.5 I need to create a script that based on the creation time, if the file is older then 5 minutes then execute some stuff, if not exit. I thought to get the creation time and minutes like this. CreationTime=$(stat -c %y /tmp/test.log | awk -F" " '{ print...

10. UNIX for Beginners Questions & Answers

Find if create time of last created file in a directory is older than 5 minutes

A process xyz is running and creating file1, file2, file3, .... filen. how do i know if the process has stopped and createtime of the last file (filen) is older than 5 minutes? OS is AIX

LEARN ABOUT DEBIAN

ocf_heartbeat_rsyncd

OCF_HEARTBEAT_RSYNCD(7) 					OCF resource agents					   OCF_HEARTBEAT_RSYNCD(7)

NAME

       ocf_heartbeat_rsyncd - Manages an rsync daemon

SYNOPSIS

       rsyncd [start | stop | monitor | meta-data | validate-all]

DESCRIPTION

       This script manages rsync daemon

SUPPORTED PARAMETERS

       binpath
	   The rsync binary path. For example, "/usr/bin/rsync" (optional, string, default rsync)

       conffile
	   The rsync daemon configuration file name with full path. For example, "/etc/rsyncd.conf" (optional, string, default /etc/rsyncd.conf)

       bwlimit
	   This option allows you to specify a maximum transfer rate in kilobytes per second. This option is most effective when using rsync with
	   large files (several megabytes and up). Due to the nature of rsync transfers, blocks of data are sent, then if rsync determines the
	   transfer was too fast, it will wait before sending the next data block. The result is an average transfer rate equaling the specified
	   limit. A value of zero specifies no limit.  (optional, string, no default)

SUPPORTED ACTIONS

       This resource agent supports the following actions (operations):

       start
	   Starts the resource. Suggested minimum timeout: 20s.

       stop
	   Stops the resource. Suggested minimum timeout: 20s.

       monitor
	   Performs a detailed status check. Suggested minimum timeout: 20s. Suggested interval: 60s.

       validate-all
	   Performs a validation of the resource configuration. Suggested minimum timeout: 20s.

       meta-data
	   Retrieves resource agent metadata (internal use only). Suggested minimum timeout: 5s.

EXAMPLE

       The following is an example configuration for a rsyncd resource using the crm(8) shell:

	   primitive p_rsyncd ocf:heartbeat:rsyncd 
	     op monitor depth="0" timeout="20s" interval="60s"

SEE ALSO

       http://www.linux-ha.org/wiki/rsyncd_(resource_agent)

AUTHOR

       Linux-HA contributors (see the resource agent source for information about individual authors)

resource-agents UNKNOWN 					    03/09/2014						   OCF_HEARTBEAT_RSYNCD(7)