Limiting size of rsync batch output


 
Thread Tools Search this Thread
Top Forums UNIX for Advanced & Expert Users Limiting size of rsync batch output
# 1  
Old 05-16-2012
Limiting size of rsync batch output

Anyone know if there's a way to limit the size of rsync batch output blob? I need each batch to fix on a 64GB USB key.

Using syntax like:

rsync -av --only-write-batch=/Volumes/usb/batch --stats /Users/dfbadmin/sandbox/ /Users/dfbadmin/archives/
# 2  
Old 05-16-2012
rsync doesn't sound like the right tool... How can it synchronize to something that can't hold the folder's entire contents?

That's the kind of thing I'd use an archiver like tar for, and split the archive across different sizes.

How to do that can vary.

What's your system? What's your shell?
# 3  
Old 05-16-2012
I'm syncing some very large filesystems across the net with rsync, but essentially using a sneakernet to move the bandwidth-intensive files. I've been using a hard drive, but thought it'd be nice to just use a large USB thumb drive instead.

Right now, I just run as-is on the USB drive and if it errors out, I pull out a large mechanical drive and use that.

I'm running this on OSX in the default bash shell.

My example would have looked better like this:

rsync -av --only-write-batch=/Volumes/usb/batch --stats /Users/dfbadmin/sandbox/ remote.server.com:/Users/dfbadmin/archives/
# 4  
Old 05-16-2012
sneakernet is a bit of a kludge in the first place if you have to do it in sections. Personally, I'd just use the mechanical drive every trip instead of trying to be 'clever' about it. Ideally you could just run gigabit ethernet and be done with it...

Here are two matching scripts which are a bit of a kludge themselves, creating 50 gigabyte portions of a tar file individually and extracting them individually, using pipes and fifos so it doesn't need to create more than one at once. Technically the data being fed into tar never breaks at the end of a file -- just stalls until the next file is read. It sees EOF by when the input file's actually missing.

Code:
#!/bin/sh
# megasend.sh -- for the sending system.

SOURCE="./source"
OUTPUT="output"

# 50 binary gigabytes.  Guessing low since overfilling the drive
# would mean data loss.
MAXBLOCKS="$((1024*1024*50))"

trap "rm -f /tmp/$$-fifo" EXIT

mkfifo /tmp/$$-fifo
tar -cf /tmp/$$-fifo "$SOURCE" &

PART=1

while true
do
        # Check for EOF by reading one sector
        dd of=/tmp/$$ bs=1024 count=1 2>/dev/null
        [ -s "/tmp/$$" ] || break # Quit if /tmp/$$ is an empty file

        echo "Hit enter when drive is ready to store part $PART"
        read REPLY </dev/tty

        # Don't let the drive run out of space here.  Guess a little low.
        ( cat /tmp/$$ ; dd bs=1024 count=$MAXBLOCKS ) >"$OUTPUT" 2>/dev/null

        ls -lh output
        echo "Part $PART is now ready"
        ((PART++))
done < /tmp/$$-fifo

Code:
#!/bin/bash
# megaget.sh -- for the receiving system.

function die
{
        echo "$@" >&2
        exit 1
}

SOURCE="./output"
DEST="dest/"

[ -d "$DEST" ] || die "No such folder $DEST"

PART=1
while true
do
        echo "Hit enter when drive containing part $PART is ready" >&2
        read REPLY </dev/tty

        # Check for EOF by looking for blank file
        [ -e output ] || break

        ls -lh "$SOURCE" >&2
        cat "$SOURCE"

        echo "Part $PART restored" >&2

        ((PART++))
done | tar -C "$DEST" -xf -

# 5  
Old 05-16-2012
That's pretty great! I have another project that this will be perfect for.

I'm using currently using rsync because it allows me to compare the filesystems, compute the differences and then copy the missing files to my transfer drive. If I were really fancy, I'd check the total, computed transfer size and make a judgment as to whether the files could realistically be transferred over the net or if they should be moved to the local drive for manual shuttling.
# 6  
Old 05-16-2012
You could use rsync to generate a list of files to tar up, I suppose.
# 7  
Old 05-17-2012
If I have time tomorrow, maybe I'll glue the two approaches together.
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

Limiting Bash array single line output

#!/bin/bash PH=(AD QD QC 5H 6C 8C 7D JH 3H 3S) echo ${PH} In the above array, how can I print to screen just the first 8 elements of ${PH} and have the last 2 elements print just below the first line starting underneath AD? I need to do this in order to save terminal window spacing... (5 Replies)
Discussion started by: cogiz
5 Replies

2. UNIX for Dummies Questions & Answers

Rsync size descrepancies

I'm using rsync to transfer data from one system (nfs01) to another (nfs02) but I'm seeing 28GB more data on the target than what's on the source. The source and target filesystems are both 138 GB. The source shows 100GB used and after running rsync the target shows 128 GB used. Shouldn't they be... (5 Replies)
Discussion started by: westmoreland
5 Replies

3. Shell Programming and Scripting

Limiting output file size

Hi guys, I want to know if there is a way to check the current size of the file that I output "stuff" to. For example, if I run a command that outputs data (like another shell script or C program) and i do something like `./a.out &> tempfile.txt` within the script, I want to be constantly... (2 Replies)
Discussion started by: solaris7
2 Replies

4. Windows & DOS: Issues & Discussions

Check the file size using Batch script in windows

Hi, I need to check the file size using a batch script. Pls advise. (0 Replies)
Discussion started by: krackjack
0 Replies

5. UNIX for Dummies Questions & Answers

Limiting User mailbox size in /var/spool

How can one limit the size of user mailboxes in /var/spool/mail? (0 Replies)
Discussion started by: proactiveaditya
0 Replies

6. Solaris

Rsync Size ????

Hi I have been using rsync for the past few days and would vouch for it anytime.However i am unable to find the total size of files being transferred. The output of rsync looks something like this: sent 2.92M bytes received 90.75K bytes 6.78K bytes/sec total size is 6.27G speedup... (2 Replies)
Discussion started by: Hari_Ganesh
2 Replies

7. HP-UX

Limiting SFTP Users While Not Limiting Regular Users?

Hi, I have searched the web and have come back with nothing that is satisfactory for what I require. SFTP is my corporations new file transfer standard. What I require is a method to lock down SFTP users to their directory (they may go to sub directories) while not restricting regular users. ... (2 Replies)
Discussion started by: Emancipator
2 Replies

8. UNIX for Dummies Questions & Answers

limiting home directory size for a group

Is there a way to set the size of the home directory for every single user in a specific group, in more details: I have a group & i will have to add about 20 users to it to be their home directories. i want each of the home directories for this group to be limited to 50 MB Help? (11 Replies)
Discussion started by: Katkota
11 Replies

9. UNIX for Dummies Questions & Answers

limiting the size of syslog log files

I would like to limit the size of syslog log files. Is there a setting I can enter in syslog.conf that does this for me. Ideally I would like something along the lines of a circular buffer of N bytes. P.S. I'm a new user, and this site is awesome. I wish I found it earlier. Thanks, David (1 Reply)
Discussion started by: dmirza
1 Replies

10. UNIX for Dummies Questions & Answers

tar file size (volume) limiting

Hi I am trying to create tar files of a whole bunch of files and want to limit them to 50Mb each. I have tried using the -k option but cannot get it to work. Has anyone out there had success creating these? Cheers Ian (1 Reply)
Discussion started by: bigjeff
1 Replies
Login or Register to Ask a Question