Unix/Linux Go Back    


Filesystems, Disks and Memory Discuss NAS, SAN, RAID, Robotic Libraries, backup devices, RAM, DRAM, SCSI, IDE, EIDE topics here.

Faster way: SAN hd to SAN hd copying

Filesystems, Disks and Memory


Tags
san

Reply    
 
Thread Tools Search this Thread Display Modes
    #1  
Old Unix and Linux 3 Weeks Ago   -   Original Discussion by rino19ny
rino19ny's Unix or Linux Image
rino19ny rino19ny is offline
Registered User
 
Join Date: Aug 2017
Last Activity: 27 November 2017, 3:08 AM EST
Posts: 3
Thanks: 0
Thanked 0 Times in 0 Posts
Faster way: SAN hd to SAN hd copying

hi! i got a rhel 6.3 host that already have an xfs filesystem mounted from a SAN (let's call it SAN-1) whose size is 9TB.

i will be receiving another SAN (let's call it SAN-2) storage of 15TB size. this new addition is physically on another SAN storage. SAN-1 is on a Pillar storage while the new one is on a Dell SAN storage.

i supposed brand does not matter here. my initial reaction to move the directories and files from SAN-1 to SAN-2 is either using "tar" or "dd". in both cases, i need to do it online as there is continuous data being duplicated from SAN-1 to tape storage.

is using "tar" or "dd" the only online option? are there better ways?
Sponsored Links
    #2  
Old Unix and Linux 3 Weeks Ago   -   Original Discussion by rino19ny
Don Cragun's Unix or Linux Image
Don Cragun Don Cragun is offline Forum Staff  
Administrator
 
Join Date: Jul 2012
Last Activity: 13 December 2017, 10:10 PM EST
Location: San Jose, CA, USA
Posts: 10,774
Thanks: 590
Thanked 3,771 Times in 3,219 Posts
Why not cp (if you're trying to make copies of the files on the old SAN so they also appear on the new SAN) or mv (if you're trying to move files from one SAN to the other SAN)?
Sponsored Links
    #3  
Old Unix and Linux 3 Weeks Ago   -   Original Discussion by rino19ny
rino19ny's Unix or Linux Image
rino19ny rino19ny is offline
Registered User
 
Join Date: Aug 2017
Last Activity: 27 November 2017, 3:08 AM EST
Posts: 3
Thanks: 0
Thanked 0 Times in 0 Posts
coz i want to keep the links, permissions, owners, etc.

old school way. so i was wondering if nowadays there are better ways specially if a SAN storage is involved (as opposed to physical hd to hd).
    #4  
Old Unix and Linux 3 Weeks Ago   -   Original Discussion by rino19ny
Don Cragun's Unix or Linux Image
Don Cragun Don Cragun is offline Forum Staff  
Administrator
 
Join Date: Jul 2012
Last Activity: 13 December 2017, 10:10 PM EST
Location: San Jose, CA, USA
Posts: 10,774
Thanks: 590
Thanked 3,771 Times in 3,219 Posts
Don't you think it would have been a good idea to tell us
Note that cp -Rp source destination can be used to copy filesystems maintaining dates, permissions, and ownership (but may lose links) as long as you're running with the appropriate privileges.

You can use cpio, pax, or tar to maintain links, owners, and permissions as long as you're running with the appropriate privileges.

If you use dd to copy individual files, you will lose links, owners, and permissions.

If you use dd to copy an entire filesystem, you can't change the filesystem type or size with the copy and you will destroy any files that were on the SAN you are overwriting with the dd. And, the filesystem cannot be active (i.e., should not be mounted (or, if it is mounted, mount it read-only)) during the dd. After you copy the filesystem, depending on the filesystem type, you may be able to grow the filesystem in the new SAN to occupy the additional space available there depending on the tools available on the new SAN to do such things. Note that some SAN vendors use proprietary filesystem and/or RAID designs that might not work on another SAN vendor's hardware. Be very sure that you know what you're doing before trying to use dd to replace one SAN filesystem with another SAN filesystem.
Sponsored Links
    #5  
Old Unix and Linux 3 Weeks Ago   -   Original Discussion by rino19ny
DukeNuke2's Unix or Linux Image
DukeNuke2 DukeNuke2 is offline Forum Staff  
Soulman
 
Join Date: Jul 2006
Last Activity: 13 December 2017, 11:36 AM EST
Location: Berlin, Germany
Posts: 5,701
Thanks: 73
Thanked 302 Times in 288 Posts
I don't know if the cpio command is available on Linux but that is what I use in the Solaris world... AFAIK the faster'ish way to copy whole file systems. Read a bit here:

Copying Directories Between File Systems (cpio Command) - Oracle Solaris Administration: Devices and File Systems
Sponsored Links
    #6  
Old Unix and Linux 3 Weeks Ago   -   Original Discussion by rino19ny
rbatte1's Unix or Linux Image
rbatte1 rbatte1 is offline Forum Staff  
Root armed
 
Join Date: Jun 2007
Last Activity: 13 December 2017, 7:43 AM EST
Location: Lancashire, UK
Posts: 3,407
Thanks: 1,477
Thanked 669 Times in 602 Posts
When you say 'a faster way', are you trying to move the data from one SAN to another, so that you can detach the original and you need to make the offline time as short as possible?

If your disk is LVM, then you might be better to proceed like this:-
  • Get the new LUN assigned and re-scan to pick it up if necessary
  • Make the whole new LUN into an LVM Physical Volume with pvcreate
  • Add the PV into the volume group containing the data you want to move
  • Alter the logical volume holding your filesystem/data to have two copies, specifying the new PV as the target. This will take a long time, but the applications remain available because the data is still accessible.
  • Verify that the logical volume has no errors reported.
  • Alter the logical volume to have just a single copy, specifying the old SAN device to be removed
  • Repeat the mirror/drop for as many logical volumes as you need to move to completely free the old PV
  • Remove the old PV from the volume group
  • Detach the old LUN from the OS
  • Detach the old LUN from the SAN side

Using the tools in the OS rather than trying to do the work yourself should be a lot neater. The logical volume is moved, but the filesystem has no idea about it, so all the files remain exactly as they were and your applications, backups or whatever can happily carry on.

I prefer to add a mirror to the new device rather than just execute a move for large logical volumes just in case there are any issues in the process. You don't destroy the original so you can always drop the new copy and start again.


Have I missed the point and waffled on about something irrelevant? My apologies if I have, but I hope that this helps. If you are looking to make another copy to be used separately, then we might be able to use LVM to do that too, but that would be a copy at a fixed point in time that would then go out of date with respect to the original as either is changed (you suggest that there are many concurrent processes working on it)


Anyway, I hope that this helps. Please feel free to correct me if I'm way off target.


Kind regards,
Robin
Sponsored Links
    #7  
Old Unix and Linux 3 Weeks Ago   -   Original Discussion by rino19ny
Corona688's Unix or Linux Image
Corona688 Corona688 is offline Forum Staff  
Mead Rotor
 
Join Date: Aug 2005
Last Activity: 13 December 2017, 10:16 AM EST
Location: Saskatchewan
Posts: 22,546
Thanks: 1,159
Thanked 4,286 Times in 3,955 Posts
There's nothing particularly slow about tar, either.

In the end it's 15 tb of data, transferring it will take time.
Sponsored Links
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Linux More UNIX and Linux Forum Topics You Might Find Helpful
Thread Thread Starter Forum Replies Last Post
Which system is faster? SkySmart UNIX for Dummies Questions & Answers 6 04-14-2013 01:58 PM
Which command will be faster? y? karthi_g UNIX for Dummies Questions & Answers 4 07-30-2009 07:31 PM
Which is faster AWK or CUT dopple Shell Programming and Scripting 10 08-26-2008 04:42 AM
Faster then cp ? yoavbe Shell Programming and Scripting 6 07-10-2006 03:14 PM
faster way to loop? tads98 UNIX for Advanced & Expert Users 9 05-10-2006 05:07 PM



All times are GMT -4. The time now is 12:13 AM.