Sponsored Content
Top Forums Shell Programming and Scripting Copy Directory Tree from one Server to Another Post 302937595 by drl on Friday 6th of March 2015 12:09:31 PM
Old 03-06-2015
Hi.

I may be missing some important point, but I see:

1)
Quote:
Originally Posted by mohtashims
... it takes more than an hour for the find command as it is affected by the thousands of files which are present in the folder ...
I tried the find in the following situations:
Code:
Starting directory "/usr"
 Found directories: 9536
 Found files      : 125144
 Mean files/dir   : 13.12
 Size (bytes)     : 3.0G

real	0m2.607s
user	0m0.472s
sys	0m2.064s

and
Code:
Starting directory "VirtualBox VMs/" (different computer)
 Found directories: 56
 Found files      : 163
 Mean files/dir   : 2.91
 Size (bytes)     : 277G

real	0m0.062s
user	0m0.008s
sys	0m0.016s

and on the very slow ZFS fielsystem (Linux SuSE):
Code:
Starting directory "/usr" (in a VM)
 Found directories: 8987
 Found files      : 119022
 Mean files/dir   : 13.24
 Size (bytes)     : 3.8G

real	1m19.483s
user	0m1.855s
sys	0m25.096s

Still, the file and directory names (not size) were listed for ZFS in:
Code:
real	0m7.468s
user	0m1.000s
sys	0m4.339s

and to write the directory names to a file 3.143s

While I have not created 40K files in the 277G directory, it still seems odd that the operation would an take an hour.

2) Although individual directories can be made with mkdir, the command also allows multiple parameters, so
Code:
mkdir path1 path2 path3 ...

could be used.

3) If the parents options is used, then the creation order is irrelevant (as far as I can tell):
Code:
#!/usr/bin/env bash

# @(#) s1	Demonstrate mkdir "parents" option.

pe() { for _i;do printf "%s" "$_i";done; printf "\n"; }
pl() { pe;pe "-----" ;pe "$*"; }
db() { ( printf " db, ";for _i;do printf "%s" "$_i";done;printf "\n" ) >&2 ; }
db() { : ; }
C=$HOME/bin/context && [ -f $C ] && $C mkdir tree

# Remove debris from previous runs.
rm -rf a

# Create a tree.
pl " Create a tree with "parents" mode:"
mkdir -p a/b/c/d/e
tree -n -d a

# Create a sub-tree.
pl " Create a sub-tree with "parents" mode -- expecting no change:"
mkdir -p a/b/c
tree -n -d a

exit

producing:
Code:
$ ./s1

Environment: LC_ALL = , LANG = en_US.UTF-8
(Versions displayed with local utility "version")
OS, ker|rel, machine: Linux, 2.6.26-2-amd64, x86_64
Distribution        : Debian 5.0.8 (lenny, workstation) 
bash GNU bash 3.2.39
mkdir (GNU coreutils) 6.10
tree v1.5.2 (c) 1996 - 2008 by Steve Baker, Thomas Moore, Francesc Rocher, Kyosuke Tokoro 

-----
 Create a tree with parents mode:
a
`-- b
    `-- c
        `-- d
            `-- e

4 directories

-----
 Create a sub-tree with parents mode -- expecting no change:
a
`-- b
    `-- c
        `-- d
            `-- e

4 directories

Best wishes ... cheers, drl
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Read from fileList.txt, copy files from directory tree

Hi, hopefully this is a fairly simple Q&A. I have a clean file list of approximately 180 filenames with no directory or slashes in front of the filename nor any extension or dot ".". I would like to read from this list, find these files recursively down through directory trees, copy the files... (1 Reply)
Discussion started by: fxvisions
1 Replies

2. Shell Programming and Scripting

Recursively copy only specific files from a directory tree

Hi I am a shell-script newbie and am looking to synchronize certain files in two directory structures. Both these directory-trees are in CVS and so I dont want the CVS directory to be copied over. I want only .sh and .pl files in each subdirectory under these directory trees to be... (3 Replies)
Discussion started by: sharpsharkrocks
3 Replies

3. UNIX for Dummies Questions & Answers

Move all files in a directory tree to a signal directory?

Is this possible? Let me know If I need specify further on what I am trying to do- I just want to spare you the boring details of my personal file management. Thanks in advance- Brian- (2 Replies)
Discussion started by: briandanielz
2 Replies

4. Shell Programming and Scripting

copy the latest file in the remote server's directory

Hi Expert Team, I performed the below piece of code to copy the latest file in the remote server's directory to the same server's other directory. But it is not working properly. How can i handle this? Can you please help me..? ssh ${REMOTE_USERID}@${REMOTE_HOSTNAME} "cp -p `ssh... (3 Replies)
Discussion started by: spkandy
3 Replies

5. Shell Programming and Scripting

Copy a directory from a server (UNIX) to a PC (Windows)

Hello, I already do some question about this topic. I already establish the conection from the server (Centus) whit the windows PC using SSH. Now I have two problems: 1- The server always ask me about a password to copy the files to the windows pc, I don't know how to avoid this, if someone can... (2 Replies)
Discussion started by: yeestrada
2 Replies

6. UNIX for Dummies Questions & Answers

How to copy a tree of directory

Mi question is how can you copy only de three of directory and not the files in it. Only a need the three of directorys not the files (6 Replies)
Discussion started by: enkei17
6 Replies

7. UNIX for Dummies Questions & Answers

Copy directory tree with files

Iam in the process of copying a directory with thousands of directories and files into a new directory. I need to preserve permissions, owner, group, date and timestamps, everything. Iam using AIX and would need help of writing the command whether it is cp-RP or cpio. Apprecaite your... (3 Replies)
Discussion started by: baanprog
3 Replies

8. Shell Programming and Scripting

Need to copy a directory structure from one server to another

Hello All, I have got a dev server and a production server will call it as D server and P server. I have a dir structure in my D server and i need to create the same dir structure in P server also using a shell script. If i do a find . in my D server, i am getting an o/p like :- . ./vio... (9 Replies)
Discussion started by: Renjesh
9 Replies

9. Shell Programming and Scripting

Specific directory parsing in a directory tree

Hi friends, Hello again :) i got stuck in problem. Is there any way to get a special directory from directory tree? Here is my problm.." Suppose i have one fix directory structure "/abc/xyz/pqr/"(this will be fix).Under this directory structure i have some other directory and... (6 Replies)
Discussion started by: harpal singh
6 Replies

10. Shell Programming and Scripting

Copy real file from a shortcut preserving the time stamps and directory tree

I have directory with shorcuts of files. for example: gara@yn\short\name1 ( shortcut to gara@yn\FOLDER\OPT\GARA\1.jpg ) gara@yn\short\name2 ( shortcut to gara@yn\FOLDER\OPT\GARA\11.jpg ) gara@yn\short\name3 ( shortcut to gara@yn\MARA\URSA\2.jpg ) gara@yn\short\name4 ( shortcut to... (6 Replies)
Discussion started by: gogok_bg
6 Replies
All times are GMT -4. The time now is 09:50 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy