ovdb(5) InterNetNews Documentation ovdb(5)
ovdb - Overview storage method for INN
Ovdb is a storage method that uses the BerkeleyDB library to store overview data. It
requires version 2.6.x or later of the BerkeleyDB library; it is known to work well with
version 2.7.7. There is also a 3.0.55, but it is not very stable (though ovdb will com-
pile correctly with it). Check the web page http://www.avalon.net/~hakehoe/inn/ for the
latest on BerkeleyDB compatibility.
Ovdb makes use of the full transaction/logging/locking functionality of the BerkeleyDB
environment. BerkeleyDB may be downloaded from http://www.sleepycat.com .
The BerkeleyDB distribution is (by default) installed into /usr/local/BerkeleyDB, with
subdirectories lib/, include/, and bin/ for the library, headers, and support tools,
respectively. Version 3.x uses /usr/local/BerkeleyDB.3.x instead. Ovdb uses some of the
support tools in the bin/ subdirectory; namely db_archive, db_checkpoint, and db_deadlock.
To build ovdb support into INN, specify the option `--with-berkeleydb' when running the
configure script. By default, configure will search for a BerkeleyDB tree in several
likely locations, and choose the highest version (based on the name of the directory,
e.g., BerkeleyDB.3.0) that it finds. There will be a message in the configure output
indicating the chosen pathname.
You can override this pathname by adding a path to the option, e.g., `--with-berke-
leydb=/usr/BerkeleyDB.3.1'. This directory is expected to have subdirectories include,
lib, and bin containing db.h, the library itself, and the db_* utilities, respectively.
The ovdb database will take up more disk space for a given spool than the other overview
methods. Plan on needing at least 1.1 KB for every article in your spool (not counting
crossposts). So, if you have 5 million articles, you'll need at least 5.5 GB of disk
space for ovdb. With BerkeleyDB 2.x, the db files are 'grow only'; the library will not
shrink them, even if data is removed. So, reserving extra space above the estimate is a
good idea. Plus, you'll need additional space for transaction logs: at least 40 MB,
preferably 100 MB or more. (The logs may be on a different filesystem -- see the DB_CON-
To enable ovdb, set the ovmethod parameter in inn.conf to `ovdb'. The ovdb database is
stored in the directory specified by the pathoverview paramter in inn.conf. This is the
"DB_HOME" directory. To start out, this directory should be empty (other than an optional
DB_CONFIG file, see the DB_CONFIG section for details) and innd (or makehistory) will cre-
ate the files as necessary in that directory.
Other parameters for configuring ovdb are in the optional ovdb.conf configuration file.
See also the sample ovdb.conf.
Overview data is split between this many files. Currently, innd will keep all of the
files open, so don't set this too high or innd may run out of file descriptors. The
nnrpds only open one at a time, regardless. May be set to one, or just a few, but
only do that if your OS supports large (>2G) files. Warning: do NOT change this on an
already established database. Default is 32.
If txn_nosync is set to false, BerkeleyDB flushes the log after every transaction.
This minimizes the number of transactions that may be lost in the event of a crash,
but results in significantly degraded performance. Default is true.
Size of the memory pool cache, in Kilobytes. The cache will have a backing store file
in the DB directory which will be at least as big. In general, the bigger the cache,
the better. Use `db_stat -m' to see cache hit percentages. If they're less than 80%,
try increasing the cache size. To make a change of this parameter take effect with an
existing database, remove the __db* files while there are no database processes run-
ning. Default is 8000 K, which may be adequate for small to medium-sized servers.
Full-feed servers will probably need at least 14000.
Sets the page size for the DB files (in bytes). Must be a power of 2. Best choices
are 4096, 8192, or 16384. The default is 8192. You can not change the pagesize of an
existing DB, it must be recreated.
Sets the minimum number of keys per page. See the BerkeleyDB documentation for more
info. Default is based on page size:
default_minkey = pagesize / 2048
You can not change the minkey of an existing DB, it must be recreated.
A file called DB_CONFIG may be placed in the database directory to customize where the
various database files and transaction logs are written. By default, all of the files are
written in the "DB_HOME" directory. One way to improve performance is to put the transac-
tion logs on a different disk. To do this, put:
in the DB_CONFIG file. If the pathname you give starts with a /, it is treated as an
absolute path; otherwise, it is relative to the "DB_HOME" directory. Make sure that any
directories you specify exist and have proper ownership/mode before starting INN, because
they won't be created automatically. Also, don't change the DB_CONFIG file while anything
that uses ovdb is running.
The DB_CONFIG functionality is part of BerkeleyDB itself.
You don't need to do anything special when starting or stopping INN when using ovdb, as
long as you use rc.news. When starting, rc.news:
o Runs `ovdb_recover'. This will perform any needed recovery on the database; e.g., if
there was a crash that may have left the database in an inconsistent state. This is
done before innd is started. If the innd.pid file is present but innd is not running
(i.e., unclean shutdown), rc.news will supply the `-f' argument to `ovdb_recover'
o Runs `dbprocs start'. This starts the DB housekeeping processes.
And when stopping INN, rc.news calls `dbprocs stop' after the other INN processes have
been shut down.
Problems relating to ovdb are logged to news.err with "OVDB" in the error message.
If a program accessing the database crashes, or otherwise exits uncleanly, it might leave
a stale lock in the database. This lock could cause other processes to deadlock on that
stale lock. To fix this, shut down all news processes (using `kill -9' if necessary) then
run `ovdb_recover -f' as the news user, e.g.: `su news -c "/PATHBIN/ovdb_recover -f"' This
will remove all locks plus repair any damage caused by killing deadlocked processes.
The ovmethod and pathoverview parameters are relevant to ovdb.
Optional configuration file for tuning. See CONFIGURATION section above.
Directory where the database goes. BerkeleyDB calls it the 'DB_HOME' directory.
Optional file to configure the layout of the database files.
I've modified innshellvars and innshellvars.pl to add the BerkeleyDB bin dir to PATH and
set DB_HOME to innconf->pathoverview. Since I don't know TCL, someone needs to modify
innshellvars.tcl in the same way.
Implement a way to limit how many databases can be open at once (to reduce file descriptor
usage); maybe using something similar to the cache code in ov3.c
Written by Heath Kehoe <email@example.com> for InterNetNews
rc.news(8), inn.conf(5), dbprocs(8), ovdb_recover(8), ovdb_upgrade(8)
BerkeleyDB documentation, in the docs directory of the BerkeleyDB source distribution, or
on the Sleepycat web page: http://www.sleepycat.com/
3rd Berkeley Distribution INN 2.3 ovdb(5)