04-12-2019
RudiC, I don't think he's trying to parallelize the upload, just send the first tenth without waiting for the computer to finish processing the last tenth.
Step back a little. Why are you using rsync to send 10 files? What exact problem is being solved by splitting and transferring the file this way? Is your connection unreliable?
I don't think you can "trick" rsync into using incomplete files, you may need to arrange something else. It may be possible to avoid splitting the file at all!
What kind of connection do you have between yourself and the destination? 2 tb in 2 days sounds pretty fast. Are you on the same subnet, even virtually?
What's your operating system? On Linux, I think you can use losetup to create a device that's a "view" into a file on disk. So instead of creating a 200 gigabyte file, you'd just create a device which maps to the first 200 gigabytes with no copying or fuss. Whether rsync will swallow loopback files remains to be seen.
If you don't need to use rsync to transfer, you can just grab sections of file by any means you like and transfer them without writing the split file to disk at all.
Last edited by Corona688; 04-12-2019 at 04:38 PM..
This User Gave Thanks to Corona688 For This Post:
10 More Discussions You Might Find Interesting
1. Shell Programming and Scripting
Hi
Can some one please help me
How can i search a file which has been created or modified in last five minutes
I have used the command
find . -mmin -5
and it does not work
i get an error -mmin is bad option
Please help
Much regards
Tarun (2 Replies)
Discussion started by: tarundeepdhawan
2 Replies
2. Shell Programming and Scripting
find . -name *.txt -mmin -30
This is working in Redhat but not in Solaris..
What is the equivalent option in Solaris? (1 Reply)
Discussion started by: tene
1 Replies
3. Shell Programming and Scripting
Is it possible in an ftp script to get remote files based on whether they have been created in the last 30 minutes? (5 Replies)
Discussion started by: gefa
5 Replies
4. Shell Programming and Scripting
Hi ,
I need help in getting how many minutes ago the last file, matching some pattern in file name, was created in a folder.
Thanks in advance. (8 Replies)
Discussion started by: Anupam_Halder
8 Replies
5. Shell Programming and Scripting
I wanted to send an email to the client whenever there is failed record created in a /feed/HR-76/failed folder after processing of feed file.
I can find out with the help of below script that what is the new file created but that file didn't make just 15 minutes before.
... (1 Reply)
Discussion started by: puneetkhullar
1 Replies
6. Shell Programming and Scripting
I wish to copy all the files & folder under /web/Transfer_Files/data/ on mymac1 (Linux) to remote server mybank.intra.com (Solaris 10) /tmp/ location
I am using Ansible tool synchronize module which triggers the unix rsync command as below:rsync --delay-updates -F --compress --archive --rsh=ssh... (2 Replies)
Discussion started by: mohtashims
2 Replies
7. UNIX for Beginners Questions & Answers
Hi, please help with below time conversion to minutes.
one column values:
2 minutes 16 seconds 420 msec
43 seconds 750 msec
0 days 3 hours 29 minutes 58 seconds 480 msec
11 seconds 150 msec
I need output in minutes(total elapsed time in minutes) (2 Replies)
Discussion started by: ramu.badugula
2 Replies
8. Shell Programming and Scripting
Hi all,
System Ubuntu 16.04.3 LTS
i have the following log
INFO 2019-02-07 15:13:31,099 module.py:700] default: "POST /join/8550614e-3e94-4fa5-9ab2-135eefa69c1b HTTP/1.0" 500 2042
INFO 2019-02-07 15:13:31,569 module.py:700] default: "POST /join/6cb9c452-dcb1-45f3-bcca-e33f5d450105... (15 Replies)
Discussion started by: charli1
15 Replies
9. Shell Programming and Scripting
Hello all,
Info:
System RedHat 7.5
I need to create a script that based on the creation time,
if the file is older then 5 minutes then execute some stuff, if not exit.
I thought to get the creation time and minutes like this.
CreationTime=$(stat -c %y /tmp/test.log | awk -F" " '{ print... (3 Replies)
Discussion started by: charli1
3 Replies
10. UNIX for Beginners Questions & Answers
A process xyz is running and creating file1, file2, file3, .... filen. how do i know if the process has stopped and createtime of the last file (filen) is older than 5 minutes?
OS is AIX (3 Replies)
Discussion started by: malaika
3 Replies
LEARN ABOUT SUNOS
queuedefs
queuedefs(4) File Formats queuedefs(4)
NAME
queuedefs - queue description file for at, batch, and cron
SYNOPSIS
/etc/cron.d/queuedefs
DESCRIPTION
The queuedefs file describes the characteristics of the queues managed by cron(1M). Each non-comment line in this file describes one queue.
The format of the lines are as follows:
q.[njobj][nicen][nwaitw]
The fields in this line are:
q The name of the queue. a is the default queue for jobs started by at(1); b is the default queue for jobs started by batch (see
at(1)); c is the default queue for jobs run from a crontab(1) file.
njob The maximum number of jobs that can be run simultaneously in that queue; if more than njob jobs are ready to run, only the first
njob jobs will be run, and the others will be run as jobs that are currently running terminate. The default value is 100.
nice The nice(1) value to give to all jobs in that queue that are not run with a user ID of super-user. The default value is 2.
nwait The number of seconds to wait before rescheduling a job that was deferred because more than njob jobs were running in that job's
queue, or because the system-wide limit of jobs executing has been reached. The default value is 60.
Lines beginning with # are comments, and are ignored.
EXAMPLES
Example 1: A sample file.
#
#
a.4j1n
b.2j2n90w
This file specifies that the a queue, for at jobs, can have up to 4 jobs running simultaneously; those jobs will be run with a nice value
of 1. As no nwait value was given, if a job cannot be run because too many other jobs are running cron will wait 60 seconds before trying
again to run it.
The b queue, for batch(1) jobs, can have up to 2 jobs running simultaneously; those jobs will be run with a nice(1) value of 2. If a job
cannot be run because too many other jobs are running, cron(1M) will wait 90 seconds before trying again to run it. All other queues can
have up to 100 jobs running simultaneously; they will be run with a nice value of 2, and if a job cannot be run because too many other jobs
are running cron will wait 60 seconds before trying again to run it.
FILES
/etc/cron.d/queuedefs queue description file for at, batch, and cron.
SEE ALSO
at(1), crontab(1), nice(1), cron(1M)
SunOS 5.10 1 Mar 1994 queuedefs(4)