UNIX for Dummies Questions & Answers

This forum is closed for new posts. Please post beginner questions to learn unix and learn linux in this forum UNIX for Beginners Questions & Answers

Rsync whole folder and head of each file


 

    #1  
Old 11-08-2013
yifangt yifangt is offline VIP Member  
UNIX.COM VIP Member
 
Join Date: Sep 2009
Last Activity: 16 July 2018, 4:28 PM EDT
Location: Saskatchewan, Canada
Posts: 534
Thanks: 352
Thanked 10 Times in 9 Posts
Rsync whole folder and head of each file

Hello,
I am trying to backup the whole folder which contains years of data and huge size, some files are > 10GB. I want keep the exact organization of the folder, except that for bigger file only part of it (say head -50, or simply only the name of the file) will be kept. Then the structure of the folder will be copied exactly as "original" but not a real copy.
I have tried:
Code:
rsync -avzh /storage/path/yifangt --exclude=backups --max-size='1024k' /home/yifangt/BACKUP

How to modify my script to get that? Thanks a lot!

yifangt

Last edited by yifangt; 11-08-2013 at 08:36 PM.. Reason: unclear description
Sponsored Links
    #2  
Old 11-29-2013
DGPickett DGPickett is offline Forum Advisor  
Registered User
 
Join Date: Oct 2010
Last Activity: 1 February 2016, 3:35 PM EST
Location: Southern NJ, USA (Nord)
Posts: 4,673
Thanks: 8
Thanked 588 Times in 561 Posts
rsync is not interested in your short or empty files, but you may be able to ask it to not sync large files, let's look: https://www.unix.com/man-page/linux/1/rsync/ Yes:
--max-size=SIZE don't transfer any file larger than SIZE
Now, for the big files, you need to script up a find and process to make the exerp or empty files under the right relative paths.

If you make a local clone tree of the original directory tree, modified with exerpts, you can rsync that all, so your script does not have to ssh for every file it creates. A clone tree is a dir on the same device where you make all the same sub-directories and then hard-link all the files of the original tree (and any other inodes including any relative withing subtree symbolic links, but not absolute or relative outside subtree symolic links). When you want a file to be different, you must delete the hard link and replace it with a new, real file. (Only such files, with one link, should be written, else you destroy the original.)
Sponsored Links
    #3  
Old 11-29-2013
yifangt yifangt is offline VIP Member  
UNIX.COM VIP Member
 
Join Date: Sep 2009
Last Activity: 16 July 2018, 4:28 PM EDT
Location: Saskatchewan, Canada
Posts: 534
Thanks: 352
Thanked 10 Times in 9 Posts
Thanks, but not really catch all of your ideas. Will reads more of the manpage as suggested. Have a great weekend!
Sponsored Links
 

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

More UNIX and Linux Forum Topics You Might Find Helpful
Thread Thread Starter Forum Replies Last Post
rsync delete specific files - from different target folder MVEERA Shell Programming and Scripting 0 11-22-2011 08:49 AM
rsync delete single file from the target folder MVEERA Shell Programming and Scripting 3 11-21-2011 05:56 PM
File Management: How do I move all JPGS in a folder structure to a single folder? guptaxpn Shell Programming and Scripting 4 06-11-2009 02:41 AM
Parse the .txt file for folder name and FTP to the corrsponding folder. MeganP Shell Programming and Scripting 3 07-03-2007 01:54 PM



All times are GMT -4. The time now is 05:02 PM.

Unix & Linux Forums Content Copyrightę1993-2018. All Rights Reserved.
×
UNIX.COM Login
Username:
Password:  
Show Password





Not a Forum Member?
Forgot Password?