Filesystems, Disks and Memory

At the level they're using it they say "OK server, give me <filename>" and the server goes "okay, here is <filename>". They don't use it at the filesystem level, and only care what it is because FAT won't let them create files >4GB while NTFS will.

It's almost certainly not actually NTFS. I mean in more detail. What daemons are running to do what, using what files? Someone had to set this up at some point in time.

Code:

$ cat /proc/filesystems
nodev   sysfs
nodev   rootfs
nodev   bdev
nodev   proc
nodev   cgroup
nodev   cpuset
nodev   tmpfs
nodev   binfmt_misc
nodev   debugfs
nodev   sockfs
nodev   usbfs
nodev   pipefs
nodev   anon_inodefs
nodev   rpc_pipefs
nodev   devpts
        ext3
        ext2
        ext4
nodev   ramfs
nodev   hugetlbfs
        vfat
        msdos
        iso9660
nodev   nfs
nodev   nfs4
nodev   autofs
        xfs
nodev   mqueue
nodev   selinuxfs
        ntfs
$

(don't get excited over the NTFS entry. Linux's built-in NTFS support is still poor and, by necessity, read-only. There's a better driver under development but it's external to the kernel so quite cumbersome to use right now.) Probably. I have no idea. How long it takes depends on how large it is and how fast your computer is and how fast your drive is. Direct connection would probably be faster.
Depends what you're backing up, and what activity you're preventing.

You need to learn more about your system. I don't even know your distro at this point so I can't begin to help you find out what's even running on it at this point.

A good start would be all the output of

Code:

df -h
fdisk -l

I don't know. How would I determine that? I tried a ps -aux but it didn't seem to tell me anything that made sense to me; only one file said daemon.

Yes, unfortunately, that individual is no longer with the company, I contacted him and he was either unwilling or unable to help me further with what he had done. (Though, it would be amusing to think of a Linux server spontaneously setting itself up somewhere; divine code-eption?. OK, bad joke, I'll stop now.)

Code:

uname -a command return:
Linux servername.location 2.6.26-2-686 #1 SMP Wed May 12 21:56:10 UTC 2010 i686 GNU/Linux

Fair enough. You answered my poorly written question despite its ambiguity. I understand it is dependent on computer speed, disk size etc.

Okay, here it is:

Code:

df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/sda5              37G  948M   34G   3% /
tmpfs                1014M     0 1014M   0% /lib/init/rw
udev                   10M  672K  9.4M   7% /dev
tmpfs                1014M     0 1014M   0% /dev/shm
/dev/sda1             464M   19M  421M   5% /boot
/dev/sda6             419G  232G  165G  59% /home

Code:

fdisk -l
Disk /dev/sda: 499.9 GB, 499930628096 bytes
255 heads, 63 sectors/track, 60779 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk identifier: 0x000da2f6

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1               1          61      489951   83  Linux
/dev/sda2              62       60779   487717335    5  Extended
/dev/sda5              62        4924    39062016   83  Linux
/dev/sda6            4925       60363   445313736   83  Linux
/dev/sda7           60364       60779     3341488+  82  Linux swap / Solaris

I hope that helps clarify things. Again, thank you for your continuing help.

Did you get the output of cat /proc/filesystems ?

Writing a longer reply, but wanted to catch you whiel you were still online

---------- Post updated at 03:18 PM ---------- Previous update was at 02:34 PM ----------

Those help a lot. You've got a single-disk IBM-PC compatible system. There's no worrying about which disk to back up, and the backup you purchased is definitely larger than the disk the server uses. And all the files you care about, your company's data-files, are probably all in /home/, stored in /dev/sda6, nicely separated from the rest.

You've got two options:

A) Offline backup. Turn your server off, boot some backup software, run the backup, reboot, done. It won't be able to serve files while it's backing up.

500 gigs might take 15 hours on a 100baseT network, or even twice that if half-duplex, so I'm not sure a network backup is practical. Too bad, since a udpcast boot CD would make it much easier for you.

Instead I'd suggest booting a gentoo minimal liveCD. The amd64 disk is probably better if your system can boot 64-bit at all (whether your server's OS is 64-bit isn't relevant for an offline backup, just your CPU). You can make your server reboot cleanly by running /sbin/reboot, it may take several minutes as it will try to shut down things in an orderly fashion. When it reboots, make it boot the Gentoo CD, it should boot you to a raw Linux root prompt. From there you can do this:

This will take roughly four to eight hours, I think. In the end you'll have a raw, bare-metal backup; Windows won't be able to use it, but Linux can, and if your server's hard drive dies, you could crack the drive out of your Buffalo's case and expect it to boot normally inside your server. (Assuming, of course, it has the proper connectors.)

There might even be ways to keep the /home partition on it fresh once you make it since your Linux server will still be able to talk to this disk.

B) Online backup. It won't back up the whole server, just your company's datafiles. You may be able to do this with minimal interruption to the server's clients, but, files in use may not be backed up properly.

Sorry, didn't realize in your earlier post you were asking for this as well.

Code:

$ cat /proc/filesystems
nodev   sysfs
nodev   rootfs
nodev   bdev
nodev   proc
nodev   cgroup
nodev   cpuset
nodev   debugfs
nodev   securityfs
nodev   sockfs
nodev   pipefs
nodev   anon_inodefs
nodev   tmpfs
nodev   inotifyfs
nodev   devpts
nodev   ramfs
nodev   hugetlbfs
nodev   mqueue
nodev   usbfs
        ext3
nodev   rpc_pipefs
nodev   nfsd
$

I'm not seeing XFS or NTFS...

It seems option B might be easier but (from what I can gather) has the following limitation: it only copies the datafiles, not the entire hard drive. So, with the first option, if the server hard drive fails, I could conceivably turn the external hard drive into the new server hard drive (assuming, as you say, the connections match). With the second option, if the server hard drive fails, I would need to purchase a new server hard drive, program it accordingly, and upload/unpack the .tar file onto the new hard drive. Is this correct? Do you have any preferences/suggestions for what you would do?

Thanks again!

Exactly. Yep. You're on a roll, that's exactly what would happen.

The main advantage of the online backup, besides that it's easy, would be that you could conceivably access the files in a pure Windows system if you really, really needed to get at them. You'd need to install something like 7zip to extract the tarball, and it'd take a long time, but you could do it.

Except it doesn't look quite so easy now since your system can't understand XFS. You'd have to reformat the drive as something else, or install XFS drivers. It's possible you already have them, just haven't loaded them yet -- try modprobe xfs. (The offline backup doesn't care what's on the USB drive -- it overwrites it all raw.) I like bare-metal backups. Having an entire working installation to throw in when things go seriously pear-shaped has let me stumble through a few awful mistakes learning experiences mostly unscathed. It's neither quick, pretty, nor elegant, but it's powerful. Harder to do for a complicated system, but yours only has one disk.

You could even keep the backup "fresh" in a similar way to the online backup, once you have it, since almost nothing but user files are going to change.

How long are we talking? For ~200+ GBs, hours, days? My window machine is a core2 duo e8600 running at 3.33 GHz with 3.25 GB RAM.

I think the drive comes XFS base, reformatting to another format shouldn't be too difficult. What would you suggest, NTFS? Based upon the output of the $ cat /proc/filesystems command, the system doesn't support NTFS either. Would NTFS still allow the tarball to be read by windows in a pinch?

Does this command just see if they are there or load them if they are? Is this safe, stability wise? I don't want to make changes that jeopardize the safety of the data prior to having it backed up...

By that do you mean it doesn't care what files are on the drive or what format it is, or both?

So what you're saying is, I would make a mirror of the entire system hard drive, and then weekly, could do backups, more similar to the online style, where I just upload file changes?

This would be opposed to downloading a tarball onto the drive, and then moving forward downloading new tarfiles on a weekly basis?

Choosing between these options should all be kept in the context that, at some point, I am going to ideally be installing the RAID 1 backup to do nightly backups. At that point, the external drive will be either kept in a separate location in the office or as you have suggested kept off-site (if possible) and brought in to do weekly backups.

When I do install the RAID system, is it better to have them simply backup data files or for them to act as a mirrored system? Planning ahead could help determine what should be done now.

Finally, if I really wanted to, could I do both? Would there be any advantage? disadvantage?

Thanks!

The limit is mostly disk speed. Since you're reading and writing to the same disk, the speed is reduced. The time might approach double-digit hours. Definitely not NTFS. I already noted that the "easy" ntfs driver is read-only. There's a better one available, but even the "good" one is still incomplete -- and intentionally difficult to get. They had to do that for their own sanity. People were ignoring an amazing quantity of red flashing "USE AT YOUR OWN RISK" warnings and getting all mad when important data was lost. All it does is load the ability to understand XFS filesystems, if available. It's safe. All dd does is this:It dumps the contents of sda into sdb with complete ignorance of the meaning of data in sda and complete disregard for any current contents of sdb. (getting them backwards would be very bad. Hence the chmod step to prevent disaster.)

The way disks and partitions work means it doesn't have to understand what's in sda is to make a perfect copy. Dumping it raw replicates everything with perfection, including boot sectors, boot loaders, partition layouts, partition types, hidden recovery partitions, empty space, remnants of deleted files sitting in empty space, etc, including any things you didn't know about or didn't care about. The only caveat is the destination disk must be of equal or larger size than the source.

This single-minded blindness means it can even clone a system alien to it. Contents that seem like nothing but garbage will be replicated faithfully. I regularly do replications of Windows machines from Linux disks when installing a larger hard drive -- clone the old disk onto the new one and enlarge the partition later. I've done replications with macs once or twice.

Yeah. An up-to-date clone of the server would be a nice thing to have handy if you're trying to set up a RAID. If the RAID controller does anything unexpected like blank your drives when you configure an array, you can dub your backup back onto it to get everything back. It'd hardly be a RAID if you didn't mirror or strip or extend the disks in some way. It'd just be an extra hard drive.

If you want to keep it simple, a hardware RAID mirror would be simple. It'd act like the system you already have, a one-disk system, and all the advice I've given you so far would still apply to it. There's more complicated, fault-tolerant setups like raid5, but this takes a lot more disks and hardware, money that's probably better spent on things like a good uninterruptible power supply and a fire safe for your backups. Absolutely. A hardware mirror can swallow a single-disk failure and keep going, plus weekly backups to your external drive can save you from more drastic things.