Sync(3pm)						User Contributed Perl Documentation						 Sync(3pm)

NAME

	 MLDBM::Sync - safe concurrent access to MLDBM databases

SYNOPSIS

	 use MLDBM::Sync;			# this gets the default, SDBM_File
	 use MLDBM qw(DB_File Storable);	# use Storable for serializing
	 use MLDBM qw(MLDBM::Sync::SDBM_File);	# use extended SDBM_File, handles values > 1024 bytes
	 use Fcntl qw(:DEFAULT);		# import symbols O_CREAT & O_RDWR for use with DBMs

	 # NORMAL PROTECTED read/write with implicit locks per i/o request
	 my $sync_dbm_obj = tie %cache, 'MLDBM::Sync' [..other DBM args..] or die $!;
	 $cache{"AAAA"} = "BBBB";
	 my $value = $cache{"AAAA"};

	 # SERIALIZED PROTECTED read/write with explicit lock for both i/o requests
	 my $sync_dbm_obj = tie %cache, 'MLDBM::Sync', '/tmp/syncdbm', O_CREAT|O_RDWR, 0640;
	 $sync_dbm_obj->Lock;
	 $cache{"AAAA"} = "BBBB";
	 my $value = $cache{"AAAA"};
	 $sync_dbm_obj->UnLock;

	 # SERIALIZED PROTECTED READ access with explicit read lock for both reads
	 $sync_dbm_obj->ReadLock;
	 my @keys = keys %cache;
	 my $value = $cache{'AAAA'};
	 $sync_dbm_obj->UnLock;

	 # MEMORY CACHE LAYER with Tie::Cache
	 $sync_dbm_obj->SyncCacheSize('100K');

	 # KEY CHECKSUMS, for lookups on MD5 checksums on large keys
	 my $sync_dbm_obj = tie %cache, 'MLDBM::Sync', '/tmp/syncdbm', O_CREAT|O_RDWR, 0640;
	 $sync_dbm_obj->SyncKeysChecksum(1);
	 my $large_key = "KEY" x 10000;
	 $sync{$large_key} = "LARGE";
	 my $value = $sync{$large_key};

DESCRIPTION

       This module wraps around the MLDBM interface, by handling concurrent access to MLDBM databases with file locking, and flushes i/o explicity
       per lock/unlock.  The new [Read]Lock()/UnLock() API can be used to serialize requests logically and improve performance for bundled reads &
       writes.

	 my $sync_dbm_obj = tie %cache, 'MLDBM::Sync', '/tmp/syncdbm', O_CREAT|O_RDWR, 0640;

	 # Write locked critical section
	 $sync_dbm_obj->Lock;
	   ... all accesses to DBM LOCK_EX protected, and go to same tied file handles
	   $cache{'KEY'} = 'VALUE';
	 $sync_dbm_obj->UnLock;

	 # Read locked critical section
	 $sync_dbm_obj->ReadLock;
	   ... all read accesses to DBM LOCK_SH protected, and go to same tied files
	   ... WARNING, cannot write to DBM in ReadLock() section, will die()
	   ... WARNING, my $v = $cache{'KEY'}{'SUBKEY'} will trigger a write so not safe
	   ...	 to use in ReadLock() section
	   my $value = $cache{'KEY'};
	 $sync_dbm_obj->UnLock;

	 # Normal access OK too, without explicity locking
	 $cache{'KEY'} = 'VALUE';
	 my $value = $cache{'KEY'};

       MLDBM continues to serve as the underlying OO layer that serializes complex data structures to be stored in the databases.  See the MLDBM
       BUGS section for important limitations.

       MLDBM::Sync also provides built in RAM caching with Tie::Cache md5 key checksum functionality.

INSTALL

       Like any other CPAN module, either use CPAN.pm, or perl -MCPAN "-e" shell, or get the file MLDBM-Sync-x.xx.tar.gz, unzip, untar and:

	 perl Makefile.PL
	 make
	 make test
	 make install

LOCKING

       The MLDBM::Sync wrapper protects MLDBM databases by locking and unlocking around read and write requests to the databases.  Also necessary
       is for each new lock to tie() to the database internally, untie()ing when unlocking.  This flushes any i/o for the dbm to the operating
       system, and allows for concurrent read/write access to the databases.

       Without any extra effort from the developer, an existing MLDBM database will benefit from MLDBM::sync.

	 my $dbm_obj = tie %dbm, ...;
	 $dbm{"key"} = "value";

       As a write or STORE operation, the above will automatically cause the following:

	 $dbm_obj->Lock; # also ties
	 $dbm{"key"} = "value";
	 $dbm_obj->UnLock; # also unties

       Just so, a read or FETCH operation like:

	 my $value = $dbm{"key"};

       will really trigger:

	 $dbm_obj->ReadLock; # also ties
	 my $value = $dbm{"key"};
	 $dbm_obj->Lock; # also unties

       However, these lock operations are expensive because of the underlying tie()/untie() that occurs for i/o flushing, so when bundling reads &
       writes, a developer may explicitly use this API for greater performance:

	 # tie once to database, write 100 times
	 $dbm_obj->Lock;
	 for (1..100) {
	   $dbm{$_} = $_ * 100;
	   ...
	 }
	 $dbm_obj->UnLock;

	 # only tie once to database, and read 100 times
	 $dbm_obj->ReadLock;
	 for(1..100) {
	   my $value = $dbm{$_};
	   ...
	 }
	 $dbm_obj->UnLock;

CACHING

       I built MLDBM::Sync to serve as a fast and robust caching layer for use in multi-process environments like mod_perl.  In order to provide
       an additional speed boost when caching static data, I have added an RAM caching layer with Tie::Cache, which regulates the size of the
       memory used with its MaxBytes setting.

       To activate this caching, just:

	 my $dbm = tie %cache, 'MLDBM::Sync', '/tmp/syncdbm', O_CREAT|O_RDWR, 0640;
	 $dbm->SyncCacheSize(100000);  # 100000 bytes max memory used
	 $dbm->SyncCacheSize('100K');  # 100 Kbytes max memory used
	 $dbm->SyncCacheSize('1M');    # 1 Megabyte max memory used

       The ./bench/bench_sync.pl, run like "bench_sync.pl "-c"" will run the tests with caching turned on creating a benchmark with 50% cache
       hits.

       One run without caching was:

	=== INSERT OF 50 BYTE RECORDS ===
	 Time for 100 writes + 100 reads for  SDBM_File 		 0.16 seconds	  12288 bytes
	 Time for 100 writes + 100 reads for  MLDBM::Sync::SDBM_File	 0.17 seconds	  12288 bytes
	 Time for 100 writes + 100 reads for  GDBM_File 		 3.37 seconds	  17980 bytes
	 Time for 100 writes + 100 reads for  DB_File			 4.45 seconds	  20480 bytes

       And with caching, with 50% cache hits:

	=== INSERT OF 50 BYTE RECORDS ===
	 Time for 100 writes + 100 reads for  SDBM_File 		 0.11 seconds	  12288 bytes
	 Time for 100 writes + 100 reads for  MLDBM::Sync::SDBM_File	 0.11 seconds	  12288 bytes
	 Time for 100 writes + 100 reads for  GDBM_File 		 2.49 seconds	  17980 bytes
	 Time for 100 writes + 100 reads for  DB_File			 2.55 seconds	  20480 bytes

       Even for SDBM_File, this speedup is near 33%.

KEYS CHECKSUM

       A common operation on database lookups is checksumming the key, prior to the lookup, because the key could be very large, and all one
       really wants is the data it maps too.  To enable this functionality automatically with MLDBM::Sync, just:

	 my $sync_dbm_obj = tie %cache, 'MLDBM::Sync', '/tmp/syncdbm', O_CREAT|O_RDWR, 0640;
	 $sync_dbm_obj->SyncKeysChecksum(1);

	!! WARNING: keys() & each() do not work on these databases
	!! as of v.03, so the developer will not be fooled into thinking
	!! the stored key values are meaningful to the calling application
	!! and will die() if called.
	!!
	!! This behavior could be relaxed in the future.

       An example of this might be to cache a XSLT conversion, which are typically very expensive.  You have the XML data and the XSLT data, so
       all you do is:

	 # $xml_data, $xsl_data are strings
	 my $xslt_output;
	 unless ($xslt_output = $cache{$xml_data.'&&&&'.$xsl_data}) {
	   ... do XSLT conversion here for $xslt_output ...
	   $cache{$xml_data.'&&&&'.xsl_data} = $xslt_output;
	 }

       What you save by doing this is having to create HUGE keys to lookup on, which no DBM is likely to do efficiently.  This is the same method
       that File::Cache uses internally to hash its file lookups in its directories.

New MLDBM::Sync::SDBM_File
       SDBM_File, the default used for MLDBM and therefore MLDBM::Sync has a limit of 1024 bytes for the size of a record.

       SDBM_File is also an order of magnitude faster for small records to use with MLDBM::Sync, than DB_File or GDBM_File, because the
       tie()/untie() to the dbm is much faster.  Therefore, bundled with MLDBM::Sync release is a MLDBM::Sync::SDBM_File layer which works around
       this 1024 byte limit.  To use, just:

	 use MLDBM qw(MLDBM::Sync::SDBM_File);

       It works by breaking up up the STORE() values into small 128 byte segments, and spreading those segments across many records, creating a
       virtual record layer.  It also uses Compress::Zlib to compress STORED data, reducing the number of these 128 byte records. In benchmarks,
       128 byte record segments seemed to be a sweet spot for space/time efficiency, as SDBM_File created very bloated *.pag files for 128+ byte
       records.

BENCHMARKS

       In the distribution ./bench directory is a bench_sync.pl script that can benchmark using the various DBMs with MLDBM::Sync.

       The MLDBM::Sync::SDBM_File DBM is special because is uses SDBM_File for fast small inserts, but slows down linearly with the size of the
       data being inserted and read.

       The results for a dual PIII-450 linux 2.4.7, with a ext3 file system blocksize 4096 mounted async on a RAID-1 2xIDE 7200 RPM disk were as
       follows:

	=== INSERT OF 50 BYTE RECORDS ===
	 Time for 100 writes + 100 reads for  SDBM_File 		 0.16 seconds	  12288 bytes
	 Time for 100 writes + 100 reads for  MLDBM::Sync::SDBM_File	 0.19 seconds	  12288 bytes
	 Time for 100 writes + 100 reads for  GDBM_File 		 1.09 seconds	  18066 bytes
	 Time for 100 writes + 100 reads for  DB_File			 0.67 seconds	  12288 bytes
	 Time for 100 writes + 100 reads for  Tie::TextDir .04		 0.31 seconds	  13192 bytes

	=== INSERT OF 500 BYTE RECORDS ===
	(skipping test for SDBM_File 100 byte limit)
	 Time for 100 writes + 100 reads for  MLDBM::Sync::SDBM_File	 0.52 seconds	 110592 bytes
	 Time for 100 writes + 100 reads for  GDBM_File 		 1.20 seconds	  63472 bytes
	 Time for 100 writes + 100 reads for  DB_File			 0.66 seconds	  86016 bytes
	 Time for 100 writes + 100 reads for  Tie::TextDir .04		 0.32 seconds	  58192 bytes

	=== INSERT OF 5000 BYTE RECORDS ===
	(skipping test for SDBM_File 100 byte limit)
	 Time for 100 writes + 100 reads for  MLDBM::Sync::SDBM_File	 1.41 seconds	1163264 bytes
	 Time for 100 writes + 100 reads for  GDBM_File 		 1.38 seconds	 832400 bytes
	 Time for 100 writes + 100 reads for  DB_File			 1.21 seconds	 831488 bytes
	 Time for 100 writes + 100 reads for  Tie::TextDir .04		 0.58 seconds	 508192 bytes

	=== INSERT OF 20000 BYTE RECORDS ===
	(skipping test for SDBM_File 100 byte limit)
	(skipping test for MLDBM::Sync db size > 1M)
	 Time for 100 writes + 100 reads for  GDBM_File 		 2.23 seconds	2063912 bytes
	 Time for 100 writes + 100 reads for  DB_File			 1.89 seconds	2060288 bytes
	 Time for 100 writes + 100 reads for  Tie::TextDir .04		 1.26 seconds	2008192 bytes

	=== INSERT OF 50000 BYTE RECORDS ===
	(skipping test for SDBM_File 100 byte limit)
	(skipping test for MLDBM::Sync db size > 1M)
	 Time for 100 writes + 100 reads for  GDBM_File 		 3.66 seconds	5337944 bytes
	 Time for 100 writes + 100 reads for  DB_File			 3.64 seconds	5337088 bytes
	 Time for 100 writes + 100 reads for  Tie::TextDir .04		 2.80 seconds	5008192 bytes

AUTHORS

       Copyright (c) 2001-2002 Joshua Chamas, Chamas Enterprises Inc.  All rights reserved.  Sponsored by development on NodeWorks
       http://www.nodeworks.com and Apache::ASP http://www.apache-asp.org

       This program is free software; you can redistribute it and/or modify it under the same terms as Perl itself.

SEE ALSO

	MLDBM(3), SDBM_File(3), DB_File(3), GDBM_File(3)

perl v5.10.0							    2002-07-03								 Sync(3pm)