Emdebian and Debian

Debian is a fantastic resource with its 20,000 packages and 12 architectures, but it's too fat for a lot of embedded and small-system use. Below we describe the emdebian process for producing a version of Debian that is smaller, but retains as much that is good about Debian as possible. The packaging system, the licensing, the distributed development, the vendor neutrality, the multi-architecture support, the guaranteed freedom, the availability of source, the build system, the consistency between packages.

Unfortunately we can't just change Debian policy to mandate smaller packages, or allow the removal of important documentation, as that wouldn't suit most Debian users. Yet simply taking a snapshot of Debian and modifying the packages will mean our packages are different from those in Debian and we will get left behind as Debian continues to develop. One solution to this is support for "dpkg variants" where changes needed to derive an Emdebian package can be fully integrated into the Debian package.

We need a scheme that allows Emdebian to stay in sync with Debian as much as possible, whilst having fine control over package generation. To do this we need to keep Emdebian package modifications in each package, maintained by package maintainers as much as possible. The initial modifications will be done by the Emdebian team in the form of new rules and control files in emdebian patch files. These modifications are passed back to maintainers or hopefully can be persuaded to keep them up to date for their packages.

The current composite method combines the best of previous methods by using patches stored in subversion. For details of the latest version of the composite method and details of previous methods, see the MetaData Wiki page.

This is a huge undertaking but fortunately we don't need to do this for every package in Debian, only the relatively small subset that most embedded/small systems need. Much of the process can be automated, at least to a first approximation, and once the principle is proved we can hopefully get the maintenance of the emdebian changes mandated as part of Debian policy. However that's well in the future. Right now we need to set up the system, make patches, persuade maintainers and persuade ourselves that we have a practical and useful system.

The Composite Plan

This scheme has been worked out over a succession of meetings during late 2006 by a number of project members, ably assisted by interested Debian developers. It is loosely based around a combination of the $DEBIAN_DIR and udebs methods. It has also been greatly improved by discussion on the mailing list. Ultimately Wookey and Neil Williams are to blame for any misunderstandings in committing it to text. So if you think it's rubbish you know who to blame. Criticism is welcome and changes may well need to be made as we try things out.

Initially, we will work just with a subset of packages often used in small systems - both Debian 'base' and such things as Busybox and uclibc. This is in order to get something that is genuinely useful whilst keeping the number of packages and maintainers we are dealing with down to a reasonable number. Once the principle is established we'll include more packages.

Each package will have a set of patches in emdebian SVN that implement the changes necessary to make an emdebian archive. In general, changes will be generated by scripts in the emdebian-tools package with as little manual editing as possible.

These new files will be used by the normal Debian build infrastructure with assistance from wrapper scripts from emdebian-tools. Tools will separate out translation files into generated packages - providing users with the mechanism to only install translation files for the languages in use. Tools will also remove documentation and (where possible) package optional components separately.

For some packages it will be best to change options to reduce the number of dependencies and/or be better suited to small systems.All packages will be built with -Os. Some secondary binary packages probably need not be built at all.

All this means that emdebian will have a different dependency tree to Debian, and auto-building Debian is already tricky due to many circular dependencies. Splitting out the documentation will remove many of these, but managing dependencies such that Emdebian remains buildable at all times will require care. A preliminary build of 550 packages with glibc has shown that the problem is manageable. Ensuring that build-depends are honoured for migration of a source package into Emdebian (unlike Debian proper's migration policy from unstable to testing) should help; so will keeping the total number of packages much smaller than Debian's. Advice from release gurus is welcome.

We need to

• Extend the build system for Crush (buildds)
• Identify and maintain the subset of Debian packages suitable for Emdebian Crush and for Emdebian Grip.
• Make and update Emdebian patches for each package
• Persuade maintainers to incorporate suitable Emdebian changes
• Build away and see how well it works.
• Document the process of creating an emdebian patch set for each package
• Document the build system so that people can monitor it and reproduce it
• Develop emdebian package policy to ensure consistency and minimalism

Volunteers are needed for all these tasks, otherwise this scheme will not get off the ground. Roll up, roll up.

Native compilation vs. cross-compilation

Debian is currently all natively compiled because some packages are difficult to cross-compile. This is less true than it was, and most of the things needed for Emdebian should cross-compile properly. For many embedded developers cross-compilation is normal, and with underpowered target platforms is often essential to get things compiled in a reasonable time.

People using binaries pre-compiled by Emdebian don't care how they got built so long as they work, but for those who need to compile their own packages cross-compiling is usually necessary. Native compilation works better for some architectures than others - e.g. fast new ARM machines are available which make native ARM compilation relatively quick. Native compilation means maintaining more build machines and some of them will be slow, but it removes one source of potential problems, and is compatible with the rest of Debian.

Recognising that most embedded developers need to be able to cross-compile, the standard emdebian build system will use the stag dpkg-based cross-compile environment. Thus a side-effect of the project will be making much of Debian cross-compile properly. For packages where this is not yet practical we will use scratchbox, which lets compilation happen on fast machines with configure scripts and other compile-time executables run natively.

Quality Checks

Cross building introduces a few, specialised, problems to do with keeping the prebuilt packages installable and tracking issues in the Debian to Emdebian progression of packages. The problems and issues are described in the Quality Assurance documentation for Emdebian.

Documentation

Most of the activity in the emdebian distribution is on the mailing list and archives and the Wiki.

manpages for emdebian-tools are available online.

A detailed description of what developers need to do is one of the tasks.

Emdebian is Debian, just smaller.

So, what is the relationship between Emdebian and Debian?

1. Emdebian is the distribution created by the Embedded Debian Project, an official sub-project of Debian.
2. Emdebian and Embedded Debian is explicitly supported by core parts of Debian, from ftp-master and release teams to gcc and glibc maintainers. Debian has chosen to support making changes that assist embedded development, via the Embedded Debian Project and has consistently done so since 2000 (around the time of Potato, if not before).
3. Emdebian uses Debian resources and systems - mailing list, IRC, bug tracking, packages, release schedules, suite codenames, migration methods and requirements.
4. Emdebian can be completely regenerated using nothing but Debian (except possibly during a release freeze in Debian where interim releases exist outside Debian for the length of the freeze). In fact, Emdebian relies on Debian to be able to build Emdebian itself - building Emdebian on Debian derivatives or other GNU/Linux operating systems is explicitly unsupported.
5. Emdebian uses Debian installation systems - debian-installer, debootstap etc.
6. Emdebian consists mostly of Debian Developers and actively promotes new members of Emdebian to become Debian Developers themselves.
7. Embedded Debian is determined to hold to all the Policies and practices of Debian that are compatible with an embedded target and to include such changes into Debian that allow for the number of actual differences to be as small as possible.