Sponsored Content
Top Forums UNIX for Advanced & Expert Users Limiting size of rsync batch output Post 302641871 by Corona688 on Wednesday 16th of May 2012 03:19:10 PM
Old 05-16-2012
sneakernet is a bit of a kludge in the first place if you have to do it in sections. Personally, I'd just use the mechanical drive every trip instead of trying to be 'clever' about it. Ideally you could just run gigabit ethernet and be done with it...

Here are two matching scripts which are a bit of a kludge themselves, creating 50 gigabyte portions of a tar file individually and extracting them individually, using pipes and fifos so it doesn't need to create more than one at once. Technically the data being fed into tar never breaks at the end of a file -- just stalls until the next file is read. It sees EOF by when the input file's actually missing.

Code:
#!/bin/sh
# megasend.sh -- for the sending system.

SOURCE="./source"
OUTPUT="output"

# 50 binary gigabytes.  Guessing low since overfilling the drive
# would mean data loss.
MAXBLOCKS="$((1024*1024*50))"

trap "rm -f /tmp/$$-fifo" EXIT

mkfifo /tmp/$$-fifo
tar -cf /tmp/$$-fifo "$SOURCE" &

PART=1

while true
do
        # Check for EOF by reading one sector
        dd of=/tmp/$$ bs=1024 count=1 2>/dev/null
        [ -s "/tmp/$$" ] || break # Quit if /tmp/$$ is an empty file

        echo "Hit enter when drive is ready to store part $PART"
        read REPLY </dev/tty

        # Don't let the drive run out of space here.  Guess a little low.
        ( cat /tmp/$$ ; dd bs=1024 count=$MAXBLOCKS ) >"$OUTPUT" 2>/dev/null

        ls -lh output
        echo "Part $PART is now ready"
        ((PART++))
done < /tmp/$$-fifo

Code:
#!/bin/bash
# megaget.sh -- for the receiving system.

function die
{
        echo "$@" >&2
        exit 1
}

SOURCE="./output"
DEST="dest/"

[ -d "$DEST" ] || die "No such folder $DEST"

PART=1
while true
do
        echo "Hit enter when drive containing part $PART is ready" >&2
        read REPLY </dev/tty

        # Check for EOF by looking for blank file
        [ -e output ] || break

        ls -lh "$SOURCE" >&2
        cat "$SOURCE"

        echo "Part $PART restored" >&2

        ((PART++))
done | tar -C "$DEST" -xf -

 

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

tar file size (volume) limiting

Hi I am trying to create tar files of a whole bunch of files and want to limit them to 50Mb each. I have tried using the -k option but cannot get it to work. Has anyone out there had success creating these? Cheers Ian (1 Reply)
Discussion started by: bigjeff
1 Replies

2. UNIX for Dummies Questions & Answers

limiting the size of syslog log files

I would like to limit the size of syslog log files. Is there a setting I can enter in syslog.conf that does this for me. Ideally I would like something along the lines of a circular buffer of N bytes. P.S. I'm a new user, and this site is awesome. I wish I found it earlier. Thanks, David (1 Reply)
Discussion started by: dmirza
1 Replies

3. UNIX for Dummies Questions & Answers

limiting home directory size for a group

Is there a way to set the size of the home directory for every single user in a specific group, in more details: I have a group & i will have to add about 20 users to it to be their home directories. i want each of the home directories for this group to be limited to 50 MB Help? (11 Replies)
Discussion started by: Katkota
11 Replies

4. HP-UX

Limiting SFTP Users While Not Limiting Regular Users?

Hi, I have searched the web and have come back with nothing that is satisfactory for what I require. SFTP is my corporations new file transfer standard. What I require is a method to lock down SFTP users to their directory (they may go to sub directories) while not restricting regular users. ... (2 Replies)
Discussion started by: Emancipator
2 Replies

5. Solaris

Rsync Size ????

Hi I have been using rsync for the past few days and would vouch for it anytime.However i am unable to find the total size of files being transferred. The output of rsync looks something like this: sent 2.92M bytes received 90.75K bytes 6.78K bytes/sec total size is 6.27G speedup... (2 Replies)
Discussion started by: Hari_Ganesh
2 Replies

6. UNIX for Dummies Questions & Answers

Limiting User mailbox size in /var/spool

How can one limit the size of user mailboxes in /var/spool/mail? (0 Replies)
Discussion started by: proactiveaditya
0 Replies

7. Windows & DOS: Issues & Discussions

Check the file size using Batch script in windows

Hi, I need to check the file size using a batch script. Pls advise. (0 Replies)
Discussion started by: krackjack
0 Replies

8. Shell Programming and Scripting

Limiting output file size

Hi guys, I want to know if there is a way to check the current size of the file that I output "stuff" to. For example, if I run a command that outputs data (like another shell script or C program) and i do something like `./a.out &> tempfile.txt` within the script, I want to be constantly... (2 Replies)
Discussion started by: solaris7
2 Replies

9. UNIX for Dummies Questions & Answers

Rsync size descrepancies

I'm using rsync to transfer data from one system (nfs01) to another (nfs02) but I'm seeing 28GB more data on the target than what's on the source. The source and target filesystems are both 138 GB. The source shows 100GB used and after running rsync the target shows 128 GB used. Shouldn't they be... (5 Replies)
Discussion started by: westmoreland
5 Replies

10. UNIX for Beginners Questions & Answers

Limiting Bash array single line output

#!/bin/bash PH=(AD QD QC 5H 6C 8C 7D JH 3H 3S) echo ${PH} In the above array, how can I print to screen just the first 8 elements of ${PH} and have the last 2 elements print just below the first line starting underneath AD? I need to do this in order to save terminal window spacing... (5 Replies)
Discussion started by: cogiz
5 Replies
rsync_selinux(8)					rsync Selinux Policy documentation					  rsync_selinux(8)

NAME
rsync_selinux - Security Enhanced Linux Policy for the rsync daemon DESCRIPTION
Security-Enhanced Linux secures the rsync server via flexible mandatory access control. FILE_CONTEXTS SELinux requires files to have an extended attribute to define the file type. Policy governs the access daemons have to these files. If you want to share files using the rsync daemon, you must label the files and directories public_content_t. So if you created a special directory /var/rsync, you would need to label the directory with the chcon tool. chcon -t public_content_t /var/rsync To make this change permanent (survive a relabel), use the semanage command to add the change to file context configuration: semanage fcontext -a -t public_content_t "/var/rsync(/.*)?" This command adds the following entry to /etc/selinux/POLICYTYPE/contexts/files/file_contexts.local: /var/rsync(/.*)? system_u:object_r:publix_content_t:s0 Run the restorecon command to apply the changes: restorecon -R -v /var/rsync/ SHARING FILES
If you want to share files with multiple domains (Apache, FTP, rsync, Samba), you can set a file context of public_content_t and pub- lic_content_rw_t. These context allow any of the above domains to read the content. If you want a particular domain to write to the pub- lic_content_rw_t domain, you must set the appropriate boolean. allow_DOMAIN_anon_write. So for rsync you would execute: setsebool -P allow_rsync_anon_write=1 BOOLEANS
system-config-selinux is a GUI tool available to customize SELinux policy settings. AUTHOR
This manual page was written by Dan Walsh <dwalsh@redhat.com>. SEE ALSO
selinux(8), rsync(1), chcon(1), setsebool(8), semanage(8) dwalsh@redhat.com 17 Jan 2005 rsync_selinux(8)
All times are GMT -4. The time now is 08:54 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy