LADDER(1) User Contributed Perl Documentation LADDER(1)
NAME
Ladder - creates migration repositories for software release sets
Description
Ladder creates a SecureApt repository to migrate production devices from one release milestone to the next. The repository contains all
binary packages which would be installed to upgrade the target package of the specified release, including base packages. Source packages
are not included as this would make the final tarball much larger than necessary. Sources should remain available via the main
repositories.
For the purposes of "ladder", the bare installation / rootfs should be considered to always precede the first software release. Subsequent
steps can then be based on the tarball of the previous milestone.
Note that if using "multistrap" or a foreign architecture "debootstrap", ensure that the rootfs inside the tarball is configured and
repacked before being used with "ladder". i.e. use the production tarball rather than the build system tarball.
Ladder checks the installed package list from the production tarball for that release, calculates the packages needed to migrate to the
specified milestone and prepares a repository containing those packages, including all dependencies.
If the specified package list and the specified milestone are NOT contiguous, errors can result if some of the contained packages need to
migrate between data formats. For most cases, create a ladder step for each software release and upgrade devices in the same sequence.
"ladder" steps can be chained by modifying the update scripts.
Config files
Ladder configuration files live in /etc/ladder.d/ and need to be named after the release described. e.g. /etc/ladder.d/internal.conf.
A minimal file to upgrade to Debian sid could look like:
[sid]
suite=unstable
location=http://ftp.uk.debian.org/debian
targetpackage=apt
A more comprehensive config file could look like:
[internal]
suite=interim
codename=milestone
branch=software_release_4
key=0xDEADBEEF
keyringdir=keys
location=copy:///srv/repo
rootpackage=libfoo3
targetpackage=metapackage
extrapackages=bar baz other
updatescript=true
mountpoint=/media/ladder
(It is possible to list more than one package, as a space separated list. Commas or other markers will not be parsed by apt.)
The section name (e.g. internal in the example above) is used as the milestone name, which can differ from the suite name and the branch
name.
For more information on the key and keyringdir options, see the section on SecureApt below.
Requirements
The rootfs is expected to carry some existing apt sources, the location specified in the config file should be the one additional source
which provides the updated packages and the expectation is that this will have a different suite name to the suite configured in the
rootfs. If the location and suite are the same, "apt" will print messages about duplicate source lists but these messages can be ignored.
In order for apt to calculate the packages needed for the update, all repositories which are enabled in the rootfs tarball including the
location specified in the config file must be accessible on the machine running "ladder".
Deployment of ladder tarballs
The final tarball contains an example apt source showing the syntax which would be suitable for use with the packaged repository. The full
path will need to be specified in the final sources list file. e.g.
deb copy:///milestone suite main
May need to be modified to:
deb copy:///media/usb0/milestone suite main
The example source is packaged as ladder.list in the tarball.
The key should normally already be part of a keyring package and installed on the devices. If not, an exported copy of the public key is
also included in the tarball which can be included into the device keyring using "apt-key" (which needs to be run as root):
apt-key add /path/milestone/ladder.gpg
Some scripting / programming support will be needed to make this process seamless on-device, in particular to provide the knowledge of the
actual sequence of milestone names, but this is beyond the scope of "ladder", if only because the ladder tarball needs to be unpacked
first.
If the system is set with some standard apt sources already, the upgrade will need to only allow "apt-get" to see the ladder repository
(because the normal network connection isn't available, so the update would fail). To do this, use apt command line options to reset the
location of the SourceList and SourceParts:
apt-get -o Dir::Etc::SourceList=ladder.list -o Dir::Etc::SourceParts=./dir/ update
(./dir/ should be an empty directory - or a directory containing empty .list files and nothing else.)
The only requirements to use the ladder tarball are to create the relevant source list file, ensure the key is available and then call apt-
get update; apt-get upgrade. There is no need for perl, reprepro or anything else used by "ladder" itself.
Example update script
If the configuration file includes the "updatescript" option an example script will be included, listing the value of the "rootpackage"
option to be removed. If the "mountpoint" option is set, the DIR variable will be set in the example script as well. (You may need to
invest time in a "udev" rule as part of your rootfs to get a known mount point but such rules are beyond the scope of this documentation.)
#!/bin/sh
set -e
DIR=/media/ladder
ROOTPKG=
CONFIG=-y -o Dir::Etc::SourceList=${DIR}/ladder.list -o Dir::Etc::SourceParts=${DIR}/list.d/
apt-key add ${DIR}/pubkey.asc
apt-get ${CONFIG} update
if [ -n "$ROOTPKG" ]; then
apt-get ${CONFIG} --purge autoremove $ROOTPKG
fi
apt-get ${CONFIG} dist-upgrade
apt-get ${CONFIG} autoclean
SecureApt
Signing a ladder step repository requires that the secret key is usable without a passphrase and that the secret key is accessible to the
root user, either directly or via sudo.
As with anything related to GnuPG, protecting the secret key is the sole responsibility of the key owner. It is recommended that ladder
steps are only created in a secure environment comparable with that used to generate the keys. The same requirements apply to the machines
which use the secret key to sign the internal milestone repositories, so it may be appropriate to create ladder steps on those machines.
Specifying a keyring directory and key ID
If "keyringdir" is used, the specified directory must contain the public and secret keyrings which contain the specified "key".
"ladder" will then make both the secret key and public key accessible to the root user using a temporary keyring in
/var/lib/ladder/keys. Only the key available in /var/lib/ladder/keys will be available to the repository signing process. "ladder" only
needs to be able to read the secret and public keyrings of the keyringdir specified. Ensure that the secret key is available - without
a passphrase - or the repository will not be signed.
Using just a key ID
If "keyringdir" is not used, the user must ensure that the key is available to the root user as ladder requires sudo/root to be able to
use apt. Ensure that the specified secret key is available - without a passphrase - to the root user or the repository will not be
signed.
sudo gpg --list-secret-key KEYID
Using "keyringdir" is generally the easiest option.
If the key is not available, the repository simply won't be signed and devices would need to pass the AllowUnauthenticated option to
"apt-get" when using the ladder repository. ladder does not add the unauthenticated option to generated upgrade scripts! You can tell a
SecureApt repository by the presence of the Release.gpg file.
It is possible to auto-generate GnuPG keys but "ladder" does not support this currently. The main problem is entropy - generating a new
GnuPG (or SSH) key requires a lot of entropy, especially as default key lengths increase. It is a lot easier to ensure high entropy when
the key generation process is interactive.
Keyring packages are recommended
With careful planning, the security of the step upgrades can be much improved by modifying the update scripts to not add the signing key
using "apt-key add" but instead to provide a keyring package in the rootfs itself which contains the public key which will be used to sign
the next milestone. This is how Debian arranges keys - the release of milestone A is not made until the key which will be used to sign
milestone B has the corresponding public key already included in the keyring package in milestone A.
Such keyring packages themselves need to be in the milestone repository because then the keyring package itself is protected by SecureApt.
Note that keyring packages will make it harder to use the downgrade solution explained below, hence the need for planning.
Key expiries
Key expiry dates will complicate "ladder" usage, especially if downgrades are to be available. If a device was released on milestone D and
needs to be downgraded to milestone B, you will have problems if the key used to sign milestone B has since expired. Equally, repairing or
servicing a device running milestone B becomes problematic if the key for milestone B has expired whilst the device was in use.
Avoid using expiry dates on keys unless you are very, very confident that a particular milestone will not be in use after a certain date.
Key management
Keys can be revoked but this relies on the devices which need to verify that key being able to download the revocation certificate and then
to still have a usable key available for the upgrade. Consider revoking the key for milestone B in the version of the keyring package
released with milestone D (milestone C still needs it to be able to upgrade). This allows keys to be revoked on-device but still be usable
should it become necessary to repair, service or downgrade.
If a key is compromised, then unless the keyring package in any one milestone still includes a usable key, there may be no way of securely
upgrading devices without manually adding a replacement key. Take care of your secret keys.
Steps and milestones
Ladder - as with Debian - only works forwards. Downgrades are not supported. If the rootfs tarball contains an existing apt source which
contains packages NEWER than the requested milestone, then the packages downloaded will be for the existing apt source, not the milestone.
Check the output with the "-n|--dry-run" option.
However, judicious use of the "rootpackage" option can assist with limited downgrades - especially when the software being downgraded is
under your own control. The generated updatescript can use "apt-get --purge autoremove" on the root package. Specifying a core library or
special platform dependency package here can allow the rootfs to be returned to a pristine state. The required milestone can then be
installed as if from a clean base. This is not quite the same as an explicit downgrade but is a much more reliable mechanism as it provides
the equivalent rootfs to when the original milestone was created.
For these reasons, always keep a copy of the original clean rootfs which has no complicating apt sources.
If your root package is a shared library, you can specify multiple root packages in the config file so that all released SONAME versions
are removed. Use only spaces to separate packages in the config file.
If you are using keyring packages, ensure that a suitable keyring package is available to the ladder step which purges the root package.
To be able to upgrade to the end milestone from a purged rootfs, the keyring package first needs to be upgraded to include the key used to
sign the end milestone (although the upgraded keyring package is free to include revoked copies of intermediary keys, if appropriate).
Output
Ladder works in the /var/lib/ladder directory, unpacking the tarball into ./rootfs and creating the repository in a directory named after
the milestone.
Results will be /var/lib/ladder/ladder-$name.tgz
Support
"ladder" was written with a specific purpose in mind but is available in Debian in the hope it will be useful for other situations as well.
If there are specific situations where "ladder" could be extended to be more useful for others, let me know using the Debian bug tracking
system: bugs.debian.org/ladder.
Note that "reprepro" already has snapshot support which is not the same as a "ladder" of milestones. Snapshots include full sources and
ancillary packages which are not needed on-device and are intended for build systems and developer use - ladder milestones are intended to
provide a small repository which can be used on machines after production.
perl v5.14.2 2012-03-12 LADDER(1)