12-22-2010
4,673,
588
Join Date: Oct 2010
Last Activity: 1 February 2016, 3:35 PM EST
Location: Southern NJ, USA (Nord)
Posts: 4,673
Thanks Given: 8
Thanked 588 Times in 561 Posts
I always wondered if some compressed file system could help here, maybe hierarchical so files not recently references or not recently changed go to compressed storage only, or if mirrored, compressed on one side. Solaris had a ClientFS that was an NFS FS with local cached copies on disk and of course RAM. It seemed very interesting, as you could configure most local disk for striped no parity one logical volume, so speed was optimized, since the data was all backed up on the NFS master copy. The remote NFS could be a linux system with a compressed file system, even the NTFS with compression turned on. Speed on the compressed volume is not an issue, because local disk take care of high bandwidth needs. Mirroring between two independent servers, perhaps on indpendent GB Ethernet NICs, would give great access rates.
It does not hurt if the RDBMS is run in append only table mode, where you keep 100% history with insert only churn and use a subquery or materialized view to find the status (latest) or active (latest=active) row set. This way, RDBMS supporting raw files goe quiescent once full, and fill is 100%. It saves design time, too, as there is no "What to save" discussion -- disk is cheaper than talk, and if a new app has a boo-boo, due to new code or new users, nothing is lost in an update or delete. Read about Plan9 OS at Bell Labs -- everything just a local copy of the optical. I did find that one needed an archival administrative delete and update, but that can be done with 100% safety using clone tables with archive event keys, so you can undo an archival delete or update, and undo any undo.
Even NFS might be OK if yoou have enough RAM! Maybe you could trade some expensive disks for ram and cheap disk, a do-it-yourself SAN. A GB Ethernet card is like $30, when I checked a few years ago, and you can buy or make a back-to-back cables so it serves one host full duplex -- take that, fiber kings! Low latency with no collisions or switch cut through delay might make NFS a lot faster, more so than bandwidth alone.
