Move directory recursive and leave symlinks at source


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Move directory recursive and leave symlinks at source
# 1  
Old 12-14-2017
Move directory recursive and leave symlinks at source

Looking for a script or command to -

Move a very large directory with tens of thousands of files and sub-directories recursively (filenames can include spaces) and replace with symlinks pointing to the new location at the same time so there is no downtime

Looking for speed + safety Smilie
# 2  
Old 12-14-2017
In other words, there will be activity going on in that directory while hoping to move...

Well unless your OS has special filesystem facilities theres very little chance to achieve what you want, the only thing I can think of is snapshots...
Then in command line or scripting the fact filenames can include spaces makes thing more hazardous, not impossible but that means needs monitoring to see if all is going well, so the task is not that easy if you hope to do it online
Offline the best tool to use in such case is cpio
Good luck
# 3  
Old 12-14-2017
Only read activity - no changes or writes being made to files, or new files / directories being added - it's like an archive
# 4  
Old 12-14-2017
Can you tell us a bit more about why you ned to do this? Is it a space issue or are you moving from using one SAN to another perhaps? That might colour our thinking rather than us guessing and giving (perhaps wildly inaccurate) guesses about what is best.

If it is read-only, then you would be better to copy the data to the new location. Then at an agreed time you can rename the top level directory and create the symbolic link. There will be the time between the rename and the creation of the link that things might not work but that should be a pretty small interval.

Perhaps an option (as root):
Code:
cd /original
tar -cvf - . | (cd /target ; tar -xvpf - )                 # Will run for quite a while

# When you are ready to switch
mv /original /original.old ; ln -s /target /original       # Rename the original and almost immediately create the link

When you are happy, remove /original.old but have a search through to make sure that there are no symbolic links inside it that a find ..... or rm -rf might follow. Take extra care at this point.



If you are moving SAN and assuming you have a volume manager, I would recommend that you attach the new LUNs and add them in to the old volume group. Then you can use the volume manager to firstly mirror the logical volumes then remove the old ones. You could use a volume manager to move the logical volume but if they are large I would always recommend a mirror and delete. It takes a little longer but you have less to worry about if there is an interruption. At worst you might have to drop the partial new copy and start again.


Like I suggested, very different options depending on what you are trying to achieve. Can you also confirm your OS and version with the output from uname -a (pasted in CODE tags) so we use the correct tools.

You also might have to consider SE-Linux, other security packages and Access Control Lists. Do you know if you have any of those?



I hope that this helps or we can continue the discussion.


Robin
# 5  
Old 12-14-2017
Thanks - that was exactly what i was after - especially the tar move, makes it very efficient
# 6  
Old 12-14-2017
Always check the exit status of a cd command!
Code:
cd /original &&
tar -cvf - . | (cd /target && tar -xvpf - )

An generic version is
Code:
(cd /original && tar -cvf - .) | (cd /target && tar -xvpf - )

where both the sending and the receiving tar work in certain directories in a subshell, while the current directory in the main shell is unchanged.
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

Move directory with rsync and delete from source

I need a rsync command which will exclude certain files and directories from source and copy the rest. I got this command working, sudo rsync -avzh --exclude 'bin' --exclude 'braintree' --exclude 'colinmollenhour' --exclude 'composer' --exclude 'doctrine' --exclude 'fabpot' --exclude... (2 Replies)
Discussion started by: Siddheshk
2 Replies

2. Shell Programming and Scripting

Shell script cannot create directory and move the file to that directory

I have a script, which is checking if file exists and move it to another directory if then mkdir -p ${LOCL_FILES_DIR}/cool_${Today}/monthly mv report_manual_alloc_rpt_A_I_ASSIGNMENT.${Today}*.csv ${LOCL_FILES_DIR}/cool_${Today}/monthly ... (9 Replies)
Discussion started by: digioleg54
9 Replies

3. UNIX for Dummies Questions & Answers

How to move gz files from one source directory to destination directory?

Hi All, Daily i am doing the house keeping in one of my server and manually moving the files which were older than 90 days and moving to destination folder. using the find command . Could you please assist me how to put the automation using the shell script . ... (11 Replies)
Discussion started by: venkat918
11 Replies

4. Shell Programming and Scripting

List files with date, create directory, move to the created directory

Hi all, i have a folder, with tons of files containing as following, on /my/folder/jobs/ some_name_2016-01-17-22-38-58_some name_0_0.zip.done some_name_2016-01-17-22-40-30_some name_0_0.zip.done some_name_2016-01-17-22-48-50_some name_0_0.zip.done and these can be lots of similar files,... (6 Replies)
Discussion started by: charli1
6 Replies

5. Shell Programming and Scripting

Rsync move with symlinks

Hi, I use rsync to move from source to target, but there are cases that I need to exclude: Suppose in my file system, I have a soft link ~/data -> /media/volgrp/data. Under data folder, there is a file hello.txt. After moving command "rsync --remove-source-files -aH --force ~/data/... (3 Replies)
Discussion started by: huangyingw
3 Replies

6. Shell Programming and Scripting

Move all files from source to destination directory based on the filename

Move all files starting with a specific name to different directory. This shell script program should have three parameters File Name Source Directory Destination Directory User should be able to enter ‘AB_CD*' in file name parameter. In this case all the files starting with AB_CD will... (1 Reply)
Discussion started by: chetancrsp18
1 Replies

7. UNIX for Dummies Questions & Answers

recursive wc on a directory?

Hi all, I need to count the number of lines in all the files under a directory (several levels deep). I am feeling extremely dumb, but I don't know how to do that. Needless to say, I am not a shell script wiz... Any advice? thanks in advance! (13 Replies)
Discussion started by: bimba17
13 Replies

8. Programming

Recursive remove directory.

What is the best way to completely remove dir with it's content ??? rmdir deletes only EMPTY dirs as i know. The man page of remove function says "remove() deletes a name from the file system." Can it remove any dir recursively ??? :rolleyes: (7 Replies)
Discussion started by: Trump
7 Replies

9. Programming

recursive copy of the directory

I want to copy a directory recursively ( it again has directories) and the directory is on windows and is nfsmounted in vxWorks, i am using unix to develop the code for this, can any one suggest me how to copy the directories recursively. (7 Replies)
Discussion started by: deepthi.s
7 Replies

10. Shell Programming and Scripting

non recursive search in the current directory only

Hi, Am trying for a script which should delete more than 15 days older files in my current directory.Am using the below piece of code: "find /tmp -type f -name "pattern" -mtime +15 -exec /usr/bin/ls -altr {} \;" "find /tmp -type f -name "pattern" -mtime +15 -exec /usr/bin/rm -f {} \;" ... (9 Replies)
Discussion started by: puppala
9 Replies
Login or Register to Ask a Question