Unix/Linux Go Back    


Filesystems, Disks and Memory Discuss NAS, SAN, RAID, Robotic Libraries, backup devices, RAM, DRAM, SCSI, IDE, EIDE topics here.

Help finding a Unix friendly RAID 1 backup

Filesystems, Disks and Memory


Closed    
 
Thread Tools Search this Thread Display Modes
    #8  
Old Unix and Linux 03-23-2011   -   Original Discussion by c.wakeman
Corona688's Unix or Linux Image
Corona688 Corona688 is offline Forum Staff  
Mead Rotor
 
Join Date: Aug 2005
Last Activity: 14 December 2017, 4:58 PM EST
Location: Saskatchewan
Posts: 22,547
Thanks: 1,159
Thanked 4,287 Times in 3,955 Posts
Quote:
Originally Posted by c.wakeman View Post
Quote:
The server I want to backup is an NTFS file system.
Okay so here is the setup as best I know it. We have a unix terminal running Linux (I logged in and used the uname command to check) that is used as a data server. Users in the office, mostly on pcs but some on macs, can access the drive (through windows) by mapping a network drive under Tools in My Computer
At the level they're using it they say "OK server, give me <filename>" and the server goes "okay, here is <filename>". They don't use it at the filesystem level, and only care what it is because FAT won't let them create files >4GB while NTFS will.

It's almost certainly not actually NTFS.
Quote:
By doing, do you mean, what is it used for? If so, the department uses it to store research data.
I mean in more detail. What daemons are running to do what, using what files?
Quote:
To be quite honest I don't know much more beyond that, nor do I think does anyone else at this point.
Someone had to set this up at some point in time.
Quote:
So that would be simply plugging the drive into the server via USB, mounting (with code) and creating the tarball (more code)? How can I tell if my Linux server can understand XFS or not?

Code:
$ cat /proc/filesystems
nodev   sysfs
nodev   rootfs
nodev   bdev
nodev   proc
nodev   cgroup
nodev   cpuset
nodev   tmpfs
nodev   binfmt_misc
nodev   debugfs
nodev   sockfs
nodev   usbfs
nodev   pipefs
nodev   anon_inodefs
nodev   rpc_pipefs
nodev   devpts
        ext3
        ext2
        ext4
nodev   ramfs
nodev   hugetlbfs
        vfat
        msdos
        iso9660
nodev   nfs
nodev   nfs4
nodev   autofs
        xfs
nodev   mqueue
nodev   selinuxfs
        ntfs
$

(don't get excited over the NTFS entry. Linux's built-in NTFS support is still poor and, by necessity, read-only. There's a better driver under development but it's external to the kernel so quite cumbersome to use right now.)
Quote:
The uname -a command gave more info, would that help?
Probably.
Quote:
How long approximately would that take?
I have no idea. How long it takes depends on how large it is and how fast your computer is and how fast your drive is.
Quote:
and vs. say doing it through the udpcast as you suggested above?
Direct connection would probably be faster.
Quote:
When I do the backup do I need to prevent other activity on the server?
Depends what you're backing up, and what activity you're preventing.

You need to learn more about your system. I don't even know your distro at this point so I can't begin to help you find out what's even running on it at this point.

A good start would be all the output of
Code:
df -h
fdisk -l


Last edited by Corona688; 03-23-2011 at 12:52 PM..
Sponsored Links
    #9  
Old Unix and Linux 03-23-2011   -   Original Discussion by c.wakeman
c.wakeman's Unix or Linux Image
c.wakeman c.wakeman is offline
Registered User
 
Join Date: Mar 2011
Last Activity: 27 June 2011, 2:25 PM EDT
Posts: 16
Thanks: 3
Thanked 0 Times in 0 Posts
Quote:
I mean in more detail. What daemons are running to do what, using what files?
I don't know. How would I determine that? I tried a ps -aux but it didn't seem to tell me anything that made sense to me; only one file said daemon.

Quote:
Someone had to set this up at some point in time.
Yes, unfortunately, that individual is no longer with the company, I contacted him and he was either unwilling or unable to help me further with what he had done. (Though, it would be amusing to think of a Linux server spontaneously setting itself up somewhere; divine code-eption?. OK, bad joke, I'll stop now.)


Code:
uname -a command return:
Linux servername.location 2.6.26-2-686 #1 SMP Wed May 12 21:56:10 UTC 2010 i686 GNU/Linux

Quote:
I have no idea. How long it takes depends on how large it is and how fast your computer is and how fast your drive is. Quote:
and vs. say doing it through the udpcast as you suggested above?
Direct connection would probably be faster.
Fair enough. You answered my poorly written question despite its ambiguity. I understand it is dependent on computer speed, disk size etc.

Quote:
A good start would be all the output of
Code:
df -h
fdisk -l
Okay, here it is:

Code:
df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/sda5              37G  948M   34G   3% /
tmpfs                1014M     0 1014M   0% /lib/init/rw
udev                   10M  672K  9.4M   7% /dev
tmpfs                1014M     0 1014M   0% /dev/shm
/dev/sda1             464M   19M  421M   5% /boot
/dev/sda6             419G  232G  165G  59% /home


Code:
fdisk -l
Disk /dev/sda: 499.9 GB, 499930628096 bytes
255 heads, 63 sectors/track, 60779 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk identifier: 0x000da2f6

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1               1          61      489951   83  Linux
/dev/sda2              62       60779   487717335    5  Extended
/dev/sda5              62        4924    39062016   83  Linux
/dev/sda6            4925       60363   445313736   83  Linux
/dev/sda7           60364       60779     3341488+  82  Linux swap / Solaris

I hope that helps clarify things. Again, thank you for your continuing help.
Sponsored Links
    #10  
Old Unix and Linux 03-23-2011   -   Original Discussion by c.wakeman
Corona688's Unix or Linux Image
Corona688 Corona688 is offline Forum Staff  
Mead Rotor
 
Join Date: Aug 2005
Last Activity: 14 December 2017, 4:58 PM EST
Location: Saskatchewan
Posts: 22,547
Thanks: 1,159
Thanked 4,287 Times in 3,955 Posts
Did you get the output of cat /proc/filesystems ?

Writing a longer reply, but wanted to catch you whiel you were still online

---------- Post updated at 03:18 PM ---------- Previous update was at 02:34 PM ----------

Those help a lot. You've got a single-disk IBM-PC compatible system. There's no worrying about which disk to back up, and the backup you purchased is definitely larger than the disk the server uses. And all the files you care about, your company's data-files, are probably all in /home/, stored in /dev/sda6, nicely separated from the rest.

You've got two options:

A) Offline backup. Turn your server off, boot some backup software, run the backup, reboot, done. It won't be able to serve files while it's backing up.

500 gigs might take 15 hours on a 100baseT network, or even twice that if half-duplex, so I'm not sure a network backup is practical. Too bad, since a udpcast boot CD would make it much easier for you.

Instead I'd suggest booting a gentoo minimal liveCD. The amd64 disk is probably better if your system can boot 64-bit at all (whether your server's OS is 64-bit isn't relevant for an offline backup, just your CPU). You can make your server reboot cleanly by running /sbin/reboot, it may take several minutes as it will try to shut down things in an orderly fashion. When it reboots, make it boot the Gentoo CD, it should boot you to a raw Linux root prompt. From there you can do this:
  1. fdisk -l to see what your main hard drive is. It'll almost certainly be /dev/sda but it's good to be sure! (Ignore sda1, sda2, etc -- those are partitions. We want the whole disk.)
  2. chmod 400 /dev/sda* Prevent yourself from writing to your company drive and any partitions on it. Just insurance. The setting doesn't exist outside of the livecd's tiny mind, so it'll forget this next reboot.
  3. Plug in your Buffalo disk with USB, wait 15 seconds, then fdisk -l again. It will probably show up as /dev/sdb. Press enter to get a prompt back if kernel debug messages print garbage over it.
  4. dd if=/dev/sda of=/dev/sdb bs=16777216 && /sbin/poweroff or, "copy the raw contents of /dev/sda into /dev/sdb, then turn off the server". It will be stonily silent while doing this, but your hard drive activity light will suddenly be mostly solid-on. (The bs=16777216 is just for efficiency -- copy 16 megs at a time instead of 512 bytes at a time.)
  5. Once it finishes, it will turn off(literally, power itself completely off) to let you know the backup is complete and it's now safe to remove the USB drive, CD, and let the server boot back up normally. If an error happens, it won't turn off, just go back to a prompt and wait.

This will take roughly four to eight hours, I think. In the end you'll have a raw, bare-metal backup; Windows won't be able to use it, but Linux can, and if your server's hard drive dies, you could crack the drive out of your Buffalo's case and expect it to boot normally inside your server. (Assuming, of course, it has the proper connectors.)

There might even be ways to keep the /home partition on it fresh once you make it since your Linux server will still be able to talk to this disk.

B) Online backup. It won't back up the whole server, just your company's datafiles. You may be able to do this with minimal interruption to the server's clients, but, files in use may not be backed up properly.
  • Plug the USB drive into the server, wait 15 seconds, fdisk -l. See if it lists the partitions on your USB disk. There may be several but the largest one should be the data one.
  • Try to mount the disk. mkdir /mnt/backup then mount /dev/sdb1 /mnt/backup. May not be sdb1 of course!
  • tar -vcpf /mnt/backup/filename-date.tar /home/ Will create one giant tarball under filename-date.tar. Filenames will pour across the screen as it adds them.
  • Once it's done, umount /mnt/backup then sync. Once the sync command finishes you know it will be safe to power down and unplug your USB drive.

Last edited by Corona688; 03-23-2011 at 06:47 PM..
    #11  
Old Unix and Linux 03-24-2011   -   Original Discussion by c.wakeman
c.wakeman's Unix or Linux Image
c.wakeman c.wakeman is offline
Registered User
 
Join Date: Mar 2011
Last Activity: 27 June 2011, 2:25 PM EDT
Posts: 16
Thanks: 3
Thanked 0 Times in 0 Posts
Quote:
Did you get the output of cat /proc/filesystems ?
Sorry, didn't realize in your earlier post you were asking for this as well.


Code:
$ cat /proc/filesystems
nodev   sysfs
nodev   rootfs
nodev   bdev
nodev   proc
nodev   cgroup
nodev   cpuset
nodev   debugfs
nodev   securityfs
nodev   sockfs
nodev   pipefs
nodev   anon_inodefs
nodev   tmpfs
nodev   inotifyfs
nodev   devpts
nodev   ramfs
nodev   hugetlbfs
nodev   mqueue
nodev   usbfs
        ext3
nodev   rpc_pipefs
nodev   nfsd
$

I'm not seeing XFS or NTFS...

Quote:
A) Offline backup. Turn your server off, boot some backup software, run the backup, reboot, done. It won't be able to serve files while it's backing up.

Instead I'd suggest booting a gentoo minimal liveCD. The amd64 disk is probably better if your system can boot 64-bit at all (whether your server's OS is 64-bit isn't relevant for an offline backup, just your CPU).

This will take roughly four to eight hours, I think. In the end you'll have a raw, bare-metal backup; Windows won't be able to use it, but Linux can, and if your server's hard drive dies, you could crack the drive out of your Buffalo's case and expect it to boot normally inside your server. (Assuming, of course, it has the proper connectors.)

There might even be ways to keep the /home partition on it fresh once you make it since your Linux server will still be able to talk to this disk.

B) Online backup. It won't back up the whole server, just your company's datafiles. You may be able to do this with minimal interruption to the server's clients, but, files in use may not be backed up properly.
It seems option B might be easier but (from what I can gather) has the following limitation: it only copies the datafiles, not the entire hard drive. So, with the first option, if the server hard drive fails, I could conceivably turn the external hard drive into the new server hard drive (assuming, as you say, the connections match). With the second option, if the server hard drive fails, I would need to purchase a new server hard drive, program it accordingly, and upload/unpack the .tar file onto the new hard drive. Is this correct? Do you have any preferences/suggestions for what you would do?

Thanks again!
Sponsored Links
    #12  
Old Unix and Linux 03-24-2011   -   Original Discussion by c.wakeman
Corona688's Unix or Linux Image
Corona688 Corona688 is offline Forum Staff  
Mead Rotor
 
Join Date: Aug 2005
Last Activity: 14 December 2017, 4:58 PM EST
Location: Saskatchewan
Posts: 22,547
Thanks: 1,159
Thanked 4,287 Times in 3,955 Posts
Quote:
Originally Posted by c.wakeman View Post
It seems option B might be easier but (from what I can gather) has the following limitation: it only copies the datafiles, not the entire hard drive.
Exactly.
Quote:
So, with the first option, if the server hard drive fails, I could conceivably turn the external hard drive into the new server hard drive (assuming, as you say, the connections match).
Yep.
Quote:
With the second option, if the server hard drive fails, I would need to purchase a new server hard drive, program it accordingly, and upload/unpack the .tar file onto the new hard drive. Is this correct?
You're on a roll, that's exactly what would happen.

The main advantage of the online backup, besides that it's easy, would be that you could conceivably access the files in a pure Windows system if you really, really needed to get at them. You'd need to install something like 7zip to extract the tarball, and it'd take a long time, but you could do it.

Except it doesn't look quite so easy now since your system can't understand XFS. You'd have to reformat the drive as something else, or install XFS drivers. It's possible you already have them, just haven't loaded them yet -- try modprobe xfs. (The offline backup doesn't care what's on the USB drive -- it overwrites it all raw.)
Quote:
Do you have any preferences/suggestions for what you would do?
I like bare-metal backups. Having an entire working installation to throw in when things go seriously pear-shaped has let me stumble through a few awful mistakes learning experiences mostly unscathed. It's neither quick, pretty, nor elegant, but it's powerful. Harder to do for a complicated system, but yours only has one disk.

You could even keep the backup "fresh" in a similar way to the online backup, once you have it, since almost nothing but user files are going to change.

Last edited by Corona688; 03-24-2011 at 05:11 PM..
Sponsored Links
    #13  
Old Unix and Linux 03-25-2011   -   Original Discussion by c.wakeman
c.wakeman's Unix or Linux Image
c.wakeman c.wakeman is offline
Registered User
 
Join Date: Mar 2011
Last Activity: 27 June 2011, 2:25 PM EDT
Posts: 16
Thanks: 3
Thanked 0 Times in 0 Posts
Quote:
You'd need to install something like 7zip to extract the tarball, and it'd take a long time, but you could do it.
How long are we talking? For ~200+ GBs, hours, days? My window machine is a core2 duo e8600 running at 3.33 GHz with 3.25 GB RAM.

Quote:
Except it doesn't look quite so easy now since your system can't understand XFS. You'd have to reformat the drive as something else,
I think the drive comes XFS base, reformatting to another format shouldn't be too difficult. What would you suggest, NTFS? Based upon the output of the $ cat /proc/filesystems command, the system doesn't support NTFS either. Would NTFS still allow the tarball to be read by windows in a pinch?

Quote:
or install XFS drivers. It's possible you already have them, just haven't loaded them yet -- try modprobe xfs.
Does this command just see if they are there or load them if they are? Is this safe, stability wise? I don't want to make changes that jeopardize the safety of the data prior to having it backed up...

Quote:
(The offline backup doesn't care what's on the USB drive -- it overwrites it all raw.)
By that do you mean it doesn't care what files are on the drive or what format it is, or both?

Quote:
You could even keep the backup "fresh" in a similar way to the online backup, once you have it, since almost nothing but user files are going to change.
So what you're saying is, I would make a mirror of the entire system hard drive, and then weekly, could do backups, more similar to the online style, where I just upload file changes?

This would be opposed to downloading a tarball onto the drive, and then moving forward downloading new tarfiles on a weekly basis?

Choosing between these options should all be kept in the context that, at some point, I am going to ideally be installing the RAID 1 backup to do nightly backups. At that point, the external drive will be either kept in a separate location in the office or as you have suggested kept off-site (if possible) and brought in to do weekly backups.

When I do install the RAID system, is it better to have them simply backup data files or for them to act as a mirrored system? Planning ahead could help determine what should be done now.

Finally, if I really wanted to, could I do both? Would there be any advantage? disadvantage?

Thanks!
Sponsored Links
    #14  
Old Unix and Linux 03-25-2011   -   Original Discussion by c.wakeman
Corona688's Unix or Linux Image
Corona688 Corona688 is offline Forum Staff  
Mead Rotor
 
Join Date: Aug 2005
Last Activity: 14 December 2017, 4:58 PM EST
Location: Saskatchewan
Posts: 22,547
Thanks: 1,159
Thanked 4,287 Times in 3,955 Posts
Quote:
Originally Posted by c.wakeman View Post
How long are we talking? For ~200+ GBs, hours, days? My window machine is a core2 duo e8600 running at 3.33 GHz with 3.25 GB RAM.
The limit is mostly disk speed. Since you're reading and writing to the same disk, the speed is reduced. The time might approach double-digit hours.
Quote:
I think the drive comes XFS base, reformatting to another format shouldn't be too difficult. What would you suggest, NTFS?
Definitely not NTFS. I already noted that the "easy" ntfs driver is read-only. There's a better one available, but even the "good" one is still incomplete -- and intentionally difficult to get. They had to do that for their own sanity. People were ignoring an amazing quantity of red flashing "USE AT YOUR OWN RISK" warnings and getting all mad when important data was lost.
Quote:
Does this command just see if they are there or load them if they are? Is this safe, stability wise? I don't want to make changes that jeopardize the safety of the data prior to having it backed up...
All it does is load the ability to understand XFS filesystems, if available. It's safe.
Quote:
By that do you mean it doesn't care what files are on the drive or what format it is, or both?
All dd does is this:
  • Read sector 0 from /dev/sda
  • Write sector 0 to /dev/sdb
  • Read sector 1 from /dev/sda
  • Write sector 1 to /dev/sdb
  • ...
  • Read sector N from /dev/sda
  • Write sector N to /dev/sdb
  • End of file. Quit.
It dumps the contents of sda into sdb with complete ignorance of the meaning of data in sda and complete disregard for any current contents of sdb. (getting them backwards would be very bad. Hence the chmod step to prevent disaster.)

The way disks and partitions work means it doesn't have to understand what's in sda is to make a perfect copy. Dumping it raw replicates everything with perfection, including boot sectors, boot loaders, partition layouts, partition types, hidden recovery partitions, empty space, remnants of deleted files sitting in empty space, etc, including any things you didn't know about or didn't care about. The only caveat is the destination disk must be of equal or larger size than the source.

This single-minded blindness means it can even clone a system alien to it. Contents that seem like nothing but garbage will be replicated faithfully. I regularly do replications of Windows machines from Linux disks when installing a larger hard drive -- clone the old disk onto the new one and enlarge the partition later. I've done replications with macs once or twice.

Quote:
So what you're saying is, I would make a mirror of the entire system hard drive, and then weekly, could do backups, more similar to the online style, where I just upload file changes?
Yeah.
Quote:
Choosing between these options should all be kept in the context that, at some point, I am going to ideally be installing the RAID 1 backup to do nightly backups. At that point, the external drive will be either kept in a separate location in the office or as you have suggested kept off-site (if possible) and brought in to do weekly backups.
An up-to-date clone of the server would be a nice thing to have handy if you're trying to set up a RAID. If the RAID controller does anything unexpected like blank your drives when you configure an array, you can dub your backup back onto it to get everything back.
Quote:
When I do install the RAID system, is it better to have them simply backup data files or for them to act as a mirrored system? Planning ahead could help determine what should be done now.
It'd hardly be a RAID if you didn't mirror or strip or extend the disks in some way. It'd just be an extra hard drive.

If you want to keep it simple, a hardware RAID mirror would be simple. It'd act like the system you already have, a one-disk system, and all the advice I've given you so far would still apply to it. There's more complicated, fault-tolerant setups like raid5, but this takes a lot more disks and hardware, money that's probably better spent on things like a good uninterruptible power supply and a fire safe for your backups.
Quote:
Finally, if I really wanted to, could I do both?
Absolutely. A hardware mirror can swallow a single-disk failure and keep going, plus weekly backups to your external drive can save you from more drastic things.

Last edited by Corona688; 03-25-2011 at 02:44 PM..
Sponsored Links
Closed

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Linux More UNIX and Linux Forum Topics You Might Find Helpful
Thread Thread Starter Forum Replies Last Post
Converting MSVC /showInclude to unix friendly path krad Shell Programming and Scripting 7 02-17-2011 08:46 PM
User friendly Unix ksh prompt to type directories/files abdurrouf UNIX for Advanced & Expert Users 3 02-15-2008 06:07 PM
RAID 0 on Unix zmachine04 HP-UX 5 12-22-2005 10:37 PM
Unix Sco 5.0.7 Raid Problems josramon UNIX for Advanced & Expert Users 0 12-21-2004 01:10 PM
RAID Unix command zuinc UNIX for Dummies Questions & Answers 2 05-07-2002 11:19 AM



All times are GMT -4. The time now is 08:34 PM.