distributed filesystem over internet/VPN


Login or Register to Reply

 
Thread Tools Search this Thread
# 1  
Old 09-04-2012
distributed filesystem over internet/VPN

On this forum was already posted similar question, but it was 4 years ago and didn't give me answers.

I have two groups of engineers that works in far locations connected via VPN. Physically, the connection is a DSL. Currently we have a linux server in one location that provide files over SMB/CIFS. Thus live of people in the second location is a nightmare.

I can change a lot on the server, but one thing must remain. Workstation need to access files using standard windows mechanisms (SMB/CIFS). No extra software (drivers, clients etc).

To solve it, I want to install second server in the second location and use a distributed filessystem that mainly works in replication mode. Files on both servers will be redistributed by samba in the local network.

There is a great list of distributed filesystems on wikipedia.

The most advanced seams to be lustre. But I have some doubts if it will work for me. Lustre is dedicated to a true cluster with a high speed connection (10GigE or special hardware like RDMA). It also stripe space from all nodes, while I rather want to have replication/mirroring.

XtreemFS seams to be more adequate to me. At least from functional site. But it seams that it is JAVA based solution... I'm very afraid for the performance. There is also some limitation for read/write replication.

GlusterFS: according to some blog this filesystem is not good choice for high-latency connection like DSL. (sorry, forum disallow me to publish valid link, add http prefix: joejulian.name/blog/glusterfs-replication-dos-and-donts/)

What is your recommendation?
Had anyone similar problem?
# 2  
Old 09-04-2012
We did have a problem, we used completely local filesystems instead. Then set up several dynamic rsync connections to keep files synced across machines. Our files on the servers are never more than about 1 minute behind time. We use inotify: checksum and then copy files to a special directory on close if there is a change. rsync is called to update the remote file.

None of our files is huge, all less than 1MB. The remote boxes are in remote locations with only DSL or satellite available.
# 3  
Old 09-05-2012
Quote:
Originally Posted by jim mcnamara
Our files on the servers are never more than about 1 minute behind time.
Quote:
Originally Posted by jim mcnamara
None of our files is huge, all less than 1MB.
Private I agree with you, 1 min is acceptable from technical point of view. But I know human's nature. When something get wrong, people will blame that solution (and me personally). I need something that guarantee consistence by design.

Additional, our files usually are rather big, usually over 10MB, some are 100MB.
# 5  
Old 09-05-2012
Quote:
Originally Posted by Corona688
You want instant concurrency, with 100MB files?
As I said, from technical point of view rsync is enough good. Problem is in humans nature. They blame everything else than they own faults.
1min uncertainty is a nice opportunity to "hide" humans fault and blame the tool (or what worse me).

Quote:
Originally Posted by Corona688
How fast are your network links?
In fact 512kbit/s. Exactly it is ADSL, so the real throughput is limited by upstreams of both sites.
Login or Register to Reply

|
Thread Tools Search this Thread
Search this Thread:
Advanced Search

More UNIX and Linux Forum Topics You Might Find Helpful
Internet connection single interface through vpn tschmi IP Networking 0 12-23-2015 06:01 PM
Mount Filesystem in AIX Unable to read /etc/filesystem m_raheelahmed AIX 2 09-19-2011 05:21 AM
Solaris Filesystem vs. Windows FileSystem arah Solaris 4 09-09-2011 02:24 AM
hwo to find shared filesystem and local filesystem in AIX kamranjalal UNIX for Dummies Questions & Answers 2 04-18-2011 01:27 PM
SFTP an internet address from a system behind an internet proxy toobrown1 Linux 4 10-09-2009 09:44 AM
Filesystem - error when extend the filesystem SmartAntz Solaris 11 08-13-2009 02:52 PM
Wanted: Geographically distributed filesystem solution otheus Filesystems, Disks and Memory 6 10-19-2008 10:42 AM
Distributed veitcha UNIX for Dummies Questions & Answers 3 11-06-2000 08:58 AM