Unix/Linux Go Back    


UNIX for Dummies Questions & Answers This forum is closed for new posts. Please post beginner questions to learn unix and learn linux in this forum UNIX for Beginners Questions & Answers

Rsync whole folder and head of each file

UNIX for Dummies Questions & Answers


Tags
backup, linux, rsync

 
 
Thread Tools Search this Thread Display Modes
    #1  
Old Unix and Linux 11-08-2013   -   Original Discussion by yifangt
yifangt's Unix or Linux Image
yifangt yifangt is offline VIP Member  
UNIX.COM VIP Member
 
Join Date: Sep 2009
Last Activity: 19 February 2018, 4:39 PM EST
Location: Saskatchewan, Canada
Posts: 522
Thanks: 338
Thanked 8 Times in 8 Posts
Rsync whole folder and head of each file

Hello,
I am trying to backup the whole folder which contains years of data and huge size, some files are > 10GB. I want keep the exact organization of the folder, except that for bigger file only part of it (say head -50, or simply only the name of the file) will be kept. Then the structure of the folder will be copied exactly as "original" but not a real copy.
I have tried:


Code:
rsync -avzh /storage/path/yifangt --exclude=backups --max-size='1024k' /home/yifangt/BACKUP

How to modify my script to get that? Thanks a lot!

yifangt

Last edited by yifangt; 11-08-2013 at 09:36 PM.. Reason: unclear description
Sponsored Links
    #2  
Old Unix and Linux 11-29-2013   -   Original Discussion by yifangt
DGPickett's Unix or Linux Image
DGPickett DGPickett is offline Forum Advisor  
Registered User
 
Join Date: Oct 2010
Last Activity: 1 February 2016, 3:35 PM EST
Location: Southern NJ, USA (Nord)
Posts: 4,673
Thanks: 8
Thanked 587 Times in 560 Posts
rsync is not interested in your short or empty files, but you may be able to ask it to not sync large files, let's look: https://www.unix.com/man-page/linux/1/rsync/ Yes:
--max-size=SIZE don't transfer any file larger than SIZE
Now, for the big files, you need to script up a find and process to make the exerp or empty files under the right relative paths.

If you make a local clone tree of the original directory tree, modified with exerpts, you can rsync that all, so your script does not have to ssh for every file it creates. A clone tree is a dir on the same device where you make all the same sub-directories and then hard-link all the files of the original tree (and any other inodes including any relative withing subtree symbolic links, but not absolute or relative outside subtree symolic links). When you want a file to be different, you must delete the hard link and replace it with a new, real file. (Only such files, with one link, should be written, else you destroy the original.)
Sponsored Links
    #3  
Old Unix and Linux 11-29-2013   -   Original Discussion by yifangt
yifangt's Unix or Linux Image
yifangt yifangt is offline VIP Member  
UNIX.COM VIP Member
 
Join Date: Sep 2009
Last Activity: 19 February 2018, 4:39 PM EST
Location: Saskatchewan, Canada
Posts: 522
Thanks: 338
Thanked 8 Times in 8 Posts
Thanks, but not really catch all of your ideas. Will reads more of the manpage as suggested. Have a great weekend!
Sponsored Links
 

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Linux More UNIX and Linux Forum Topics You Might Find Helpful
Thread Thread Starter Forum Replies Last Post
rsync delete specific files - from different target folder MVEERA Shell Programming and Scripting 0 11-22-2011 09:49 AM
rsync delete single file from the target folder MVEERA Shell Programming and Scripting 3 11-21-2011 06:56 PM
File Management: How do I move all JPGS in a folder structure to a single folder? guptaxpn Shell Programming and Scripting 4 06-11-2009 03:41 AM
Parse the .txt file for folder name and FTP to the corrsponding folder. MeganP Shell Programming and Scripting 3 07-03-2007 02:54 PM



All times are GMT -4. The time now is 04:27 PM.