Login or Register to Ask a Question and Join Our Community

Sponsored Content
Full Discussion: Limiting size of rsync batch output
Top Forums UNIX for Advanced & Expert Users Limiting size of rsync batch output Post 302641871 by Corona688 on Wednesday 16th of May 2012 03:19:10 PM
Old 05-16-2012
sneakernet is a bit of a kludge in the first place if you have to do it in sections. Personally, I'd just use the mechanical drive every trip instead of trying to be 'clever' about it. Ideally you could just run gigabit ethernet and be done with it...

Here are two matching scripts which are a bit of a kludge themselves, creating 50 gigabyte portions of a tar file individually and extracting them individually, using pipes and fifos so it doesn't need to create more than one at once. Technically the data being fed into tar never breaks at the end of a file -- just stalls until the next file is read. It sees EOF by when the input file's actually missing.

Code:
#!/bin/sh
# megasend.sh -- for the sending system.

SOURCE="./source"
OUTPUT="output"

# 50 binary gigabytes.  Guessing low since overfilling the drive
# would mean data loss.
MAXBLOCKS="$((1024*1024*50))"

trap "rm -f /tmp/$$-fifo" EXIT

mkfifo /tmp/$$-fifo
tar -cf /tmp/$$-fifo "$SOURCE" &

PART=1

while true
do
        # Check for EOF by reading one sector
        dd of=/tmp/$$ bs=1024 count=1 2>/dev/null
        [ -s "/tmp/$$" ] || break # Quit if /tmp/$$ is an empty file

        echo "Hit enter when drive is ready to store part $PART"
        read REPLY </dev/tty

        # Don't let the drive run out of space here.  Guess a little low.
        ( cat /tmp/$$ ; dd bs=1024 count=$MAXBLOCKS ) >"$OUTPUT" 2>/dev/null

        ls -lh output
        echo "Part $PART is now ready"
        ((PART++))
done < /tmp/$$-fifo

Code:
#!/bin/bash
# megaget.sh -- for the receiving system.

function die
{
        echo "$@" >&2
        exit 1
}

SOURCE="./output"
DEST="dest/"

[ -d "$DEST" ] || die "No such folder $DEST"

PART=1
while true
do
        echo "Hit enter when drive containing part $PART is ready" >&2
        read REPLY </dev/tty

        # Check for EOF by looking for blank file
        [ -e output ] || break

        ls -lh "$SOURCE" >&2
        cat "$SOURCE"

        echo "Part $PART restored" >&2

        ((PART++))
done | tar -C "$DEST" -xf -

 

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

tar file size (volume) limiting

Hi I am trying to create tar files of a whole bunch of files and want to limit them to 50Mb each. I have tried using the -k option but cannot get it to work. Has anyone out there had success creating these? Cheers Ian (1 Reply)
Discussion started by: bigjeff
1 Replies

2. UNIX for Dummies Questions & Answers

limiting the size of syslog log files

I would like to limit the size of syslog log files. Is there a setting I can enter in syslog.conf that does this for me. Ideally I would like something along the lines of a circular buffer of N bytes. P.S. I'm a new user, and this site is awesome. I wish I found it earlier. Thanks, David (1 Reply)
Discussion started by: dmirza
1 Replies

3. UNIX for Dummies Questions & Answers

limiting home directory size for a group

Is there a way to set the size of the home directory for every single user in a specific group, in more details: I have a group & i will have to add about 20 users to it to be their home directories. i want each of the home directories for this group to be limited to 50 MB Help? (11 Replies)
Discussion started by: Katkota
11 Replies

4. HP-UX

Limiting SFTP Users While Not Limiting Regular Users?

Hi, I have searched the web and have come back with nothing that is satisfactory for what I require. SFTP is my corporations new file transfer standard. What I require is a method to lock down SFTP users to their directory (they may go to sub directories) while not restricting regular users. ... (2 Replies)
Discussion started by: Emancipator
2 Replies

5. Solaris

Rsync Size ????

Hi I have been using rsync for the past few days and would vouch for it anytime.However i am unable to find the total size of files being transferred. The output of rsync looks something like this: sent 2.92M bytes received 90.75K bytes 6.78K bytes/sec total size is 6.27G speedup... (2 Replies)
Discussion started by: Hari_Ganesh
2 Replies

6. UNIX for Dummies Questions & Answers

Limiting User mailbox size in /var/spool

How can one limit the size of user mailboxes in /var/spool/mail? (0 Replies)
Discussion started by: proactiveaditya
0 Replies

7. Windows & DOS: Issues & Discussions

Check the file size using Batch script in windows

Hi, I need to check the file size using a batch script. Pls advise. (0 Replies)
Discussion started by: krackjack
0 Replies

8. Shell Programming and Scripting

Limiting output file size

Hi guys, I want to know if there is a way to check the current size of the file that I output "stuff" to. For example, if I run a command that outputs data (like another shell script or C program) and i do something like `./a.out &> tempfile.txt` within the script, I want to be constantly... (2 Replies)
Discussion started by: solaris7
2 Replies

9. UNIX for Dummies Questions & Answers

Rsync size descrepancies

I'm using rsync to transfer data from one system (nfs01) to another (nfs02) but I'm seeing 28GB more data on the target than what's on the source. The source and target filesystems are both 138 GB. The source shows 100GB used and after running rsync the target shows 128 GB used. Shouldn't they be... (5 Replies)
Discussion started by: westmoreland
5 Replies

10. UNIX for Beginners Questions & Answers

Limiting Bash array single line output

#!/bin/bash PH=(AD QD QC 5H 6C 8C 7D JH 3H 3S) echo ${PH} In the above array, how can I print to screen just the first 8 elements of ${PH} and have the last 2 elements print just below the first line starting underneath AD? I need to do this in order to save terminal window spacing... (5 Replies)
Discussion started by: cogiz
5 Replies

LEARN ABOUT SUSE

xfs_estimate

xfs_estimate(8) 					      System Manager's Manual						   xfs_estimate(8)

NAME

       xfs_estimate - estimate the space that an XFS filesystem will take

SYNOPSIS

       xfs_estimate [ -h? ] [ -b blocksize ] [ -i logsize ]
		    [ -e logsize ] [ -v ] directory ...

DESCRIPTION

       For  each directory argument, xfs_estimate estimates the space that directory would take if it were copied to an XFS filesystem.  xfs_esti-
       mate does not cross mount points.  The following definitions are used:
	      KB = *1024
	      MB = *1024*1024
	      GB = *1024*1024*1024

       The xfs_estimate options are:

       -b blocksize
	      Use blocksize instead of the default blocksize of 4096 bytes.  The modifier k can be used after the number to  indicate  multiplica-
	      tion by 1024.  For example,

		   xfs_estimate -b 64k /

	      requests an estimate of the space required by the directory / on an XFS filesystem using a blocksize of 64K (65536) bytes.

       -v     Display more information, formatted.

       -h     Display usage message.

       -?     Display usage message.

       -i, -e logsize
	      Use  logsize instead of the default log size of 1000 blocks.  -i refers to an internal log, while -e refers to an external log.  The
	      modifiers k or m can be used after the number to indicate multiplication by 1024 or 1048576, respectively.

	      For example,

		   xfs_estimate -i 1m /

	      requests an estimate of the space required by the directory / on an XFS filesystem using an internal log of 1 megabyte.

EXAMPLES

       % xfs_estimate -e 10m /var/tmp
       /var/tmp will take about 4.2 megabytes
	       with the external log using 2560 blocks or about 10.0 megabytes

       % xfs_estimate -v -e 10m /var/tmp
       directory		     bsize   blocks    megabytes    logsize
       /var/tmp 		      4096	792	   4.0MB   10485760

       % xfs_estimate -v /var/tmp
       directory		     bsize   blocks    megabytes    logsize
       /var/tmp 		      4096     3352	  14.0MB   10485760

       % xfs_estimate /var/tmp
       /var/tmp will take about 14.0 megabytes

																   xfs_estimate(8)
All times are GMT -4. The time now is 09:54 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy