I have a serverA with Directory "/opt/app/myfiles" which is 8.5GB with more than 40000 files and 45 folders in it.
I wish to transfer the folder "/opt/app/myfiles" tree structure with all the sub-directories and skip all files to another serverB.
I would appreciate a command that is Quick and I would not mind if the directory structure is copied locally so I manually transfer and untar it at the destination.
Note: In any case my source serverA Directory "/opt/app/myfiles" should not be harmed in any way.
Last edited by jim mcnamara; 03-05-2015 at 10:07 AM..
echo 'mkdir /opt/app/myfiles' > mkdirs.shl
find /opt/app/myfiles -type d |
while read rec
do
echo "mkdir $rec"
done >> mkdirs.shl
chmod +mkdirs.shl
scp mkdirs.shl serverB:
ssh serverB './mkdirs.shl'
You see this works but it takes more than an hour for the find command as it is affected by the thousands of files which are present in the folder. Any better fix ?
I might suggest a sort after the find then pipe to xargs -n1 mkdir to ensure that directories are created in ascending order so that there is no risk of trying to create /a/b/c/d/e before you create /a/b/c I admit it is unlikely, but possible.
if you are using servers that use backup & restore, fsdump & fsrestore or ufsdump and ufsrestore then you can take a dump and I think there is a way using the interactive restore to generate all the directories ready for data restore, at which point you just cancel. It's been a long time since I used them though. They are on AIX, Solaris etc. I'm not sure if there is a Linux variety.
Location: Saint Paul, MN USA / BSD, CentOS, Debian, OS X, Solaris
Posts: 2,288
Thanks Given: 430
Thanked 480 Times in 395 Posts
Hi.
I may be missing some important point, but I see:
1)
Quote:
Originally Posted by mohtashims
... it takes more than an hour for the find command as it is affected by the thousands of files which are present in the folder ...
I tried the find in the following situations:
Code:
Starting directory "/usr"
Found directories: 9536
Found files : 125144
Mean files/dir : 13.12
Size (bytes) : 3.0G
real 0m2.607s
user 0m0.472s
sys 0m2.064s
and
Code:
Starting directory "VirtualBox VMs/" (different computer)
Found directories: 56
Found files : 163
Mean files/dir : 2.91
Size (bytes) : 277G
real 0m0.062s
user 0m0.008s
sys 0m0.016s
and on the very slow ZFS fielsystem (Linux SuSE):
Code:
Starting directory "/usr" (in a VM)
Found directories: 8987
Found files : 119022
Mean files/dir : 13.24
Size (bytes) : 3.8G
real 1m19.483s
user 0m1.855s
sys 0m25.096s
Still, the file and directory names (not size) were listed for ZFS in:
Code:
real 0m7.468s
user 0m1.000s
sys 0m4.339s
and to write the directory names to a file 3.143s
While I have not created 40K files in the 277G directory, it still seems odd that the operation would an take an hour.
2) Although individual directories can be made with mkdir, the command also allows multiple parameters, so
Code:
mkdir path1 path2 path3 ...
could be used.
3) If the parents options is used, then the creation order is irrelevant (as far as I can tell):
Code:
#!/usr/bin/env bash
# @(#) s1 Demonstrate mkdir "parents" option.
pe() { for _i;do printf "%s" "$_i";done; printf "\n"; }
pl() { pe;pe "-----" ;pe "$*"; }
db() { ( printf " db, ";for _i;do printf "%s" "$_i";done;printf "\n" ) >&2 ; }
db() { : ; }
C=$HOME/bin/context && [ -f $C ] && $C mkdir tree
# Remove debris from previous runs.
rm -rf a
# Create a tree.
pl " Create a tree with "parents" mode:"
mkdir -p a/b/c/d/e
tree -n -d a
# Create a sub-tree.
pl " Create a sub-tree with "parents" mode -- expecting no change:"
mkdir -p a/b/c
tree -n -d a
exit
producing:
Code:
$ ./s1
Environment: LC_ALL = , LANG = en_US.UTF-8
(Versions displayed with local utility "version")
OS, ker|rel, machine: Linux, 2.6.26-2-amd64, x86_64
Distribution : Debian 5.0.8 (lenny, workstation)
bash GNU bash 3.2.39
mkdir (GNU coreutils) 6.10
tree v1.5.2 (c) 1996 - 2008 by Steve Baker, Thomas Moore, Francesc Rocher, Kyosuke Tokoro
-----
Create a tree with parents mode:
a
`-- b
`-- c
`-- d
`-- e
4 directories
-----
Create a sub-tree with parents mode -- expecting no change:
a
`-- b
`-- c
`-- d
`-- e
4 directories
I have directory with shorcuts of files.
for example:
gara@yn\short\name1 ( shortcut to gara@yn\FOLDER\OPT\GARA\1.jpg )
gara@yn\short\name2 ( shortcut to gara@yn\FOLDER\OPT\GARA\11.jpg )
gara@yn\short\name3 ( shortcut to gara@yn\MARA\URSA\2.jpg )
gara@yn\short\name4 ( shortcut to... (6 Replies)
Hi friends,
Hello again :)
i got stuck in problem. Is there any way to get a special directory from directory tree?
Here is my problm.." Suppose i have one fix directory structure "/abc/xyz/pqr/"(this will be fix).Under this directory structure i have some other directory and... (6 Replies)
Hello All,
I have got a dev server and a production server will call it as D server and P server.
I have a dir structure in my D server and i need to create the same dir structure in P server also using a shell script.
If i do a find . in my D server, i am getting an o/p like :-
.
./vio... (9 Replies)
Iam in the process of copying a directory with thousands of directories and files into a new directory.
I need to preserve permissions, owner, group, date and timestamps, everything.
Iam using AIX and would need help of writing the command whether it is cp-RP or cpio.
Apprecaite your... (3 Replies)
Hello, I already do some question about this topic. I already establish the conection from the server (Centus) whit the windows PC using SSH. Now I have two problems:
1- The server always ask me about a password to copy the files to the windows pc, I don't know how to avoid this, if someone can... (2 Replies)
Hi Expert Team,
I performed the below piece of code to copy the latest file in the remote server's directory to the same server's other directory.
But it is not working properly.
How can i handle this? Can you please help me..?
ssh ${REMOTE_USERID}@${REMOTE_HOSTNAME} "cp -p `ssh... (3 Replies)
Is this possible? Let me know If I need specify further on what I am trying to do- I just want to spare you the boring details of my personal file management.
Thanks in advance-
Brian- (2 Replies)
Hi
I am a shell-script newbie and am looking to synchronize certain files in two directory structures.
Both these directory-trees are in CVS and so I dont want the CVS directory to be copied over.
I want only .sh and .pl files in each subdirectory under these directory trees to be... (3 Replies)
Hi, hopefully this is a fairly simple Q&A.
I have a clean file list of approximately 180 filenames with no directory or slashes in front of the filename nor any extension or dot ".". I would like to read from this list, find these files recursively down through directory trees, copy the files... (1 Reply)
03-05-2015
Copy Directory Tree from one Server to Another
