Sponsored Content
Top Forums UNIX for Advanced & Expert Users How to rsync or tar directory trees, with hidden directory, but without files? Post 302454642 by drl on Sunday 19th of September 2010 12:54:42 PM
Old 09-19-2010
Hi.

I found it easier to think of replicating the directory tree, omitting the non-directory items, and then tarring up that tree:
Code:
#!/usr/bin/env bash

# @(#) s1	Demonstrate replication of directory structure.

# Utility functions: print-as-echo, print-line-with-visual-space.
pe() { for i;do printf "%s" "$i";done; printf "\n"; }
pl() { pe;pe "-----" ;pe "$*"; }

pl " Original tree \"a\" with files:"
tree -a -F a

pl " Original tree \"a\", listing directories only:"
tree -d -a -F a

rm -rf skeleton skeleton-t1
find . -type d ! -name '*skeleton*' > skeleton-t1
# cat skeleton-t1
sed 's|^[.]|skeleton|' skeleton-t1 > skeleton-t2
# cat skeleton-t2

IFS=$'\012'
while read dir
do
  mkdir "$dir"
done < skeleton-t2

pl " New upper tree named \"skeleton\", same structure, expecting no files:"
tree -a -F skeleton

pl  " Final tar operation:"
tar cvf skeleton.tar -C skeleton .

pl " Listing of tar file:"
tar xvf skeleton.tar

exit 0

producing on an existing example tree "a":
Code:
% ./s1

-----
 Original tree "a" with files:
a
|-- .a4
|-- a-3
|-- a1
|-- a2
|-- b/
|   |-- b1
|   |-- b2
|   |-- b3 with spaces
|   `-- d/
|       |-- d1
|       `-- d2
`-- c/
    |-- c1
    |-- c2
    `-- e-dir with spaces/
        |-- .f-hidden/
        |   |-- f1
        |   `-- f2
        |-- e1
        `-- e2

5 directories, 15 files

-----
 Original tree "a", listing directories only:
a
|-- b
|   `-- d
`-- c
    `-- e-dir with spaces
        `-- .f-hidden

5 directories

-----
 New upper tree named "skeleton", same structure, expecting no files:
skeleton
`-- a/
    |-- b/
    |   `-- d/
    `-- c/
        `-- e-dir with spaces/
            `-- .f-hidden/

6 directories, 0 files

-----
 Final tar operation:
./
./a/
./a/b/
./a/b/d/
./a/c/
./a/c/e-dir with spaces/
./a/c/e-dir with spaces/.f-hidden/

-----
 Listing of tar file:
./
./a/
./a/b/
./a/b/d/
./a/c/
./a/c/e-dir with spaces/
./a/c/e-dir with spaces/.f-hidden/

I ran this on a tree that had 60 MB in 280 directories, and it went too quickly for me to see anything except the last part of the list.

There may be other shorter methods as well ... cheers, drl
 

8 More Discussions You Might Find Interesting

1. UNIX for Advanced & Expert Users

rsync: taking advantage of files in different directory other than destination

Dear Folks, I have to backup pgsql database dump everynight on a routine. The database dump actually contains sql(text) statements. The actual size of the database dump is aroung 800 MB. Between two days backup, only few lines of statements are modified/added/deleted. I dont want to do... (1 Reply)
Discussion started by: rssrik
1 Replies

2. Shell Programming and Scripting

Best way to diff two huge directory trees

Hi I have a job that will be running nightly incremental backsup of a large directory tree. I did the initial backup, now I want to write a script to verify that all the files were transferred correctly. I did something like this which works in principle on small trees: diff -r -q... (6 Replies)
Discussion started by: same1290
6 Replies

3. UNIX for Dummies Questions & Answers

tar files in directory

can someone give me a script to tar files that is older than 5 days in a directory that is not something like this: fileArray=($(find -mtime +5 asdfasdf)) tar -cvf asfadfasdfa ${fileArray} as the Unix I'm using has some problem with ($( )), I need another way to tar files in the folder.... (1 Reply)
Discussion started by: s3270226
1 Replies

4. Shell Programming and Scripting

How to copy very large directory trees

I have constant trouble with XCOPY/s for multi-gigabyte transfers. I need a utility like XCOPY/S that remembers where it left off if I reboot. Is there such a utility? How about a free utility (free as in free beer)? How about an md5sum sanity check too? I posted the above query in another... (3 Replies)
Discussion started by: siegfried
3 Replies

5. UNIX for Advanced & Expert Users

Find all files in the current directory excluding hidden files and directories

Find all files in the current directory only excluding hidden directories and files. For the below command, though it's not deleting hidden files.. it is traversing through the hidden directories and listing normal which should be avoided. `find . \( ! -name ".*" -prune \) -mtime +${n_days}... (7 Replies)
Discussion started by: ksailesh1
7 Replies

6. Shell Programming and Scripting

Rsync to copy specific subfolders and files to new directory

RootFolderI: RootFolderI/FolderA/Subfolder1/Subsub1/JPG1.jpg -> want this jpg RootFolderI/FolderA/Subfolder2/Subsub1/JPG2.jpg -> want this jpg RootFolderI/FolderA/Subfolder2/Subsub2/JPG3.jpg . . . RootFolderI/FolderB/Subfolder1/Subsub1/JPG4.jpg -> want this jpg ... (1 Reply)
Discussion started by: blocnt
1 Replies

7. Shell Programming and Scripting

ksh - Checking directory trees containing wild cards

Hi Can somebody please show me how to check from within a KSH script if a directory exists on that same host when parts of the directory tree are unknown? If these wildcard dirs were the only dirs at that level then ... RETCODE=$(ls -l /u01/app/oracle/local/*/* | grep target_dir) ... will... (4 Replies)
Discussion started by: user052009
4 Replies

8. Shell Programming and Scripting

Rsync - how to copy hidden folder or hidden files when using full path

Hello. I use this command : rsync -av --include=".*" --dry-run "$A_FULL_PATH_S" "$A_FULL_PATH_D"The data comes from the output of a find command. And no full source directories are in use, only some files. Source example... (2 Replies)
Discussion started by: jcdole
2 Replies
RDUP-BACKUPS(7) 						       rdup							   RDUP-BACKUPS(7)

NAME
rdup-backups - introduction into making backups with rdup INTRODUCTION
rdup is a simple program that prints out a list of files and directories that are changed changed on a filesystem. It is more sophisticated than for instance find, because rdup will find files that are removed or directories that are renamed. A long time ago rdup included a bunch of shell and Perl scripts that implemented a backup policy. These could be used in a pipeline to per- form a backup. Currently rdup consists out of three basic utilities: rdup With rdup you create the file list on which later programs in the pipeline can work. The default output format also includes the files' content. rdup can be seen as a tar replacement in this respect, but rdup also allows for all kinds of transformations of the content (encryption, compression, reversal), see the -P switch in rdup(1) for more information. rdup-tr With rdup-tr you can transform the files rdup delivers to you. You can create tar, cpio or pax files. You can encrypt pathnames. rdup-tr is filter that reads from standard input and writes to standard output. See rdup-tr(1) for more information. With rdup and rdup-tr you can create an encrypted archive which is put in a directory structure that is also encrypted. rdup-up With rdup-up you can update an existing directory structure with the updates as described by rdup. rdup-up reads rdup input and will create the files, symbolic links, hard links and directories (and sockets, pipes and devices) in the file system. See rdup-up(1) for more information. So the general backup pipeline for rdup will look something like this: create filelist | transform | update filesystem ( rdup | rdup-tr | rdup-up ) Note 1: The same sequence is used for restoring. In both cases you want to move files from location A to B. The only difference is that the transformation is reversed when you restore. Note 2: The use of rdup-tr is optional. BACKUPS AND RESTORES
For rdup there is no difference between backups and restores. If you think about this for a minute you understand why. Making a backup means copying a list of files somewhere else. Restoring files is copying a list of files back to the place they came from. Same difference. So rdup can be used for both, if you did any transformation with rdup during the backup you just need to reverse those operations during the restore. BACKUPS
It is always best to backup to another medium, be it a different local harddisk or a NFS/CIFS mounted filesystem. You can also use ssh to store file on a remote server, ala rsync (although not as network efficient). If you backup to a local disk you can just as well use rsync or plain old tar, but if you store your files at somebody else's disk you will need encryption. This is where you go beyond rsync and rdup comes in. Rsync cannot do per-file encryption, sure you can encrypt the network traffic with ssh, but at the remote side your files are kept in plain view. If you implement remote backups, the easy route is to allow root access on the backup medium. If the backup runs without root access the created files will not have their original ownership. For NFS this can be achieved by using no_root_squash, for ssh you could enable PermitRootLogin. Note that this may be a security risk. SNAPSHOT BACKUPS
We need a little help here in the form of the rdup-simple script. Keep in mind that the following scripts can also be run remotely with the help of ssh. The following script implements the algorithm of rdup-simple. #!/bin/bash # some tmp files are saved in ~/.rdup. This directory must exist DIR=/home # what to backup BACKUP=/vol/backup TODAY=$(date +%Y%m/%d) LIST=~/.rdup/list-$HOSTNAME STAMP=~/.rdup/timestamp-$HOSTNAME # for remote backup, this has to run on the remote host! BUGBUG RET=$? case $RET in 2|*) echo Error >&2 exit 1 ;; 1) # full dump, remove file-list and time-stamp file rm $LIST $STAMP ;; 0) # inc dump # do nothing here ;; esac # this is the place where you want to modify the command line # right now, nothing is translated we just use 'cat' rdup -N $STAMP -Pcat $LIST $DIR | rdup-up $BACKUP/$HOSTNAME/$TODAY # or do a remote backup #rdup -N $STAMP -Pcat $LIST $DIR | ssh root@remotehost # rdup-up $BACKUP/$HOSTNAME/$TODAY LOCAL BACKUPS
With rdup-simple you can easily create backups. Backing up my home directory to a backup directory: rdup-simple ~ /vol/backup/$HOSTNAME This will create a backup in /vol/backup/$HOSTNAME/200705/15. So each day will have its own directory. Multiple sources are allowed, so: rdup-simple ~ /etc/ /var/lib /vol/backup/$HOSTNAME Will backup your home directory, /etc and /var/lib to the backup location. Also if you need to compress your backup, simple add a '-z' switch: rdup-simple -z ~ /etc/ /var/lib /vol/backup/$HOSTNAME REMOTE BACKUPS
For a remote backup to work, both the sending machine and the receiving machine must have rdup installed. The currently implemented proto- col is ssh. Dumping my homedir to the remote server: rdup-simple ~ ssh://miekg@remote/vol/backup/$HOSTNAME The syntax is almost identical, only the destination starts with the magic string 'ssh://'. Compression and encryption are just as easily enabled as with a local backup, just add '-z' and/or a '-k keyfile' argument: rdup-simple -z -k 'secret-file' ~ ssh://miekg@remote/vol/backup/$HOSTNAME Remember though, that because of these advanced features (compression, encryption, etc, ...) the network transfer can never be as efficient as rsync. ALSO SEE
rdup(1), rdup-tr(1), rdup-up(1) and http://www.miek.nl/projects/rdup/ 1.1.x 15 Dec 2008 RDUP-BACKUPS(7)
All times are GMT -4. The time now is 01:57 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy