Rsync whole folder and head of each file


 
Thread Tools Search this Thread
Top Forums UNIX for Dummies Questions & Answers Rsync whole folder and head of each file
# 1  
Old 11-08-2013
Rsync whole folder and head of each file

Hello,
I am trying to backup the whole folder which contains years of data and huge size, some files are > 10GB. I want keep the exact organization of the folder, except that for bigger file only part of it (say head -50, or simply only the name of the file) will be kept. Then the structure of the folder will be copied exactly as "original" but not a real copy.
I have tried:
Code:
rsync -avzh /storage/path/yifangt --exclude=backups --max-size='1024k' /home/yifangt/BACKUP

How to modify my script to get that? Thanks a lot!

yifangt

Last edited by yifangt; 11-08-2013 at 10:36 PM.. Reason: unclear description
# 2  
Old 11-29-2013
rsync is not interested in your short or empty files, but you may be able to ask it to not sync large files, let's look: https://www.unix.com/man-page/linux/1/rsync/ Yes:
--max-size=SIZE don't transfer any file larger than SIZE
Now, for the big files, you need to script up a find and process to make the exerp or empty files under the right relative paths.

If you make a local clone tree of the original directory tree, modified with exerpts, you can rsync that all, so your script does not have to ssh for every file it creates. A clone tree is a dir on the same device where you make all the same sub-directories and then hard-link all the files of the original tree (and any other inodes including any relative withing subtree symbolic links, but not absolute or relative outside subtree symolic links). When you want a file to be different, you must delete the hard link and replace it with a new, real file. (Only such files, with one link, should be written, else you destroy the original.)
# 3  
Old 11-29-2013
Thanks, but not really catch all of your ideas. Will reads more of the manpage as suggested. Have a great weekend!
 
Login or Register to Ask a Question

Previous Thread | Next Thread

9 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Help Needed for creating the folder by checking today's date and, take backup using rsync command

How to create a shell script to create a folder by using the today's date to take backup using rsync command on every evening around 7 pm. Kindly help. Thanks. To be more precise, I want to create a script which matches the today's date with server's date format, if matches then creates the... (2 Replies)
Discussion started by: bakula10
2 Replies

2. Shell Programming and Scripting

Rsync Error: rsync: link_stat failed: No such file or directory (2)

I wish to copy all the files & folder under /web/Transfer_Files/data/ on mymac1 (Linux) to remote server mybank.intra.com (Solaris 10) /tmp/ location I am using Ansible tool synchronize module which triggers the unix rsync command as below:rsync --delay-updates -F --compress --archive --rsh=ssh... (2 Replies)
Discussion started by: mohtashims
2 Replies

3. Shell Programming and Scripting

Rsync - how to copy hidden folder or hidden files when using full path

Hello. I use this command : rsync -av --include=".*" --dry-run "$A_FULL_PATH_S" "$A_FULL_PATH_D"The data comes from the output of a find command. And no full source directories are in use, only some files. Source example... (2 Replies)
Discussion started by: jcdole
2 Replies

4. Shell Programming and Scripting

Script to tar/rsync/rm multiple folder names

hi all, i attach a link with what im trying to do automatically via script but i have some questions i need answering please, bear in mind i am really new to bash scripting, the only thing i know how to do is commands in scripts like cd rm tar rsync cp stuff like that i have mutiple project... (48 Replies)
Discussion started by: robertkwild
48 Replies

5. Shell Programming and Scripting

rsync delete specific files - from different target folder

Hi, I need to use rsync to delete multiple files(only specified files not all) using --delete option, these files are located in different target folders. Instead of running rsync command multiple times for each file, can we achieve this with one time execution? your help is much... (0 Replies)
Discussion started by: MVEERA
0 Replies

6. Shell Programming and Scripting

rsync delete single file from the target folder

Hi We want to delete a single file from the destiantion directory using rsync. Is it possible to do this ? or Do we have any alternate approaches in rsync( for ex applying some filters ..etc) For ex: ----------------------------------------------- Source (Folder) ... (3 Replies)
Discussion started by: MVEERA
3 Replies

7. Shell Programming and Scripting

How can i get the head of file while using awk?

Hi All, I'm a newbie here, I'm just wondering how can i get the head of my file while using awk? input data: nik1,nik2,nik3 nik2,nik3,nik4 nik3,nik4,nik5 expected output is: nik1 because it is in the top and it is in the first delimeted. i tried awk -F "," '{print $1}' but i... (2 Replies)
Discussion started by: nikki1200
2 Replies

8. UNIX for Advanced & Expert Users

does head create new file

hi, Here my doubt is can we create a file using head/tail command in unix. for example: my file consists 2000 lines , i want to cut it into 2. does this work please give correct command line. head -1000 abc.txt > acd.txt tail -1000 abc.txt > bdc.txt hi, Here my doubt is can we... (5 Replies)
Discussion started by: chittisri
5 Replies

9. Shell Programming and Scripting

head command with more than one file

Hi, I have the following problem. I have files with one column of data (let's say file1.dat, file2.dat...file6.dat), and I would like to record the first value of the column of each file into another file (let's name it fileall.dat), which would have the the six values, one in each column. I use to... (4 Replies)
Discussion started by: josegr
4 Replies
Login or Register to Ask a Question

Featured Tech Videos