Emdebian tdeb support

Translation packages (tdebs)

Emdebian uses a technique from OpenEmbedded to handle translations which is becoming known within Debian as "Tdeb" - translation debian packages. TDebs are a method for splitting translation files out of packages and into a separate repository, organised by language and program, allowing individual users to only install translations files for the locales supported on their own machine.

Emdebian TDebs are different to the Debian TDeb proposal but development will progress in both types, improvements in Debian support being implemented in the Emdebian processes.

Differences with Debian

There are more Emdebian TDebs than the equivalent Debian TDebs and, predictably, each is smaller. For ~240 source packages, Emdebian Crush currently creates ~1,800 TDebs compared to ~700 that would be created under the draft specification for Debian. Further, Debian retains that 700 for all architectures when Emdebian Crush will add another 1,800 for each architecure supported.
TDebs are already implemented in Emdebian but Debian needs more work to resolve the scalability issues described above.
Emdebian will use the same TDebs for Emdebian Grip and Emdebian Crush until TDebs are supported in Debian (some time before the release of Debian 6.0 "Squeeze").
Once TDebs are supported in Debian, Emdebian Grip will switch to a derivative of Debian TDebs in order to gain from the improved tool support that will be implemented in Debian once TDebs are supported. The switch will add a few related locales to Grip installations but the change will otherwise be invisible.
Emdebian TDebs are currently organised one file per source package, per locale, per architecture. Debian TDebs will be organised per source package and contain per locale root sections within a single file. Emdebian Grip will use Debian TDebs to create TDebs as one file per source package per locale root.
Debian TDebs might or might not end up containing the untranslated documentation as well as translated ones. Emdebian TDebs will not contain any documentation.

Prior to Debian Squeeze, Emdebian Grip will use the same TDebs as Emdebian Crush. The following table shows the status of TDebs as expected to exist at the time of the Debian Squeeze release.

	Emdebian Crush (from Lenny)	Emdebian Grip (Squeeze)	Debian Squeeze (tool support only)
Organisation	one TDeb for one locale, per source, per architecture	one TDeb for one locale root, per source	one TDeb for one source package (with one extra, only in special cases).
filenames	$source-locale-$lang_$version_$arch.tdeb where $lang is pt_BR, distinct from pt	$source-tdeb-$root_$version_all.tdeb where root includes pt_BR with pt, distinct from fr.	$source-tdeb_$version_all.tdeb where the locale roots are separate data members within one archive.
Architecture	any	all	all
endianness	Architecture-dependent	use wrapper	use wrapper
Content	Only .mo files	Only .mo files	Translated manpages, images, documentation and .mo files together with changelog, copyright and other package docs.

langupdate is a frontend for the Emdebian implementation of tdebs to handle the selection and installation of TDebs to match supported locales and the current set of installed packages.

Disc space is not cheap

Current Debian packages collate all available translations into a single package which can easily lead to > 250Mb of unused translation files on a typical installation. i.e. the translations alone could take up four times the amount of space intended for the entire operating system. Emdebian generates a single package for every translation of each Emdebian package, leading to a 70% reduction in installation size but a tenfold increase in the number of binary packages built from each source package. To solve this scalability problem, langupdate supports a secondary sources list and secondary apt cache so that the main apt cache can be kept as small as possible.

In Debian 6.0 "Squeeze", Debian TDebs will be supported (actual TDebs will be in the subsequent release) but these will also contain all translations in a single file. The Debian TDeb will be organised to separate translation content for each locale root into dedicated sections within the TDeb file format. Emdebian Grip will use this functionality to separate those data components into separate TDebs. In due course, Debian is likely to deploy similar support on the client side to strip unwanted locale roots from downloaded TDebs. Emdebian Grip transfers that burden to the repository, to allow for smaller downloads to Grip devices.

langupdate queries the secondary apt cache to get a list of all translation packages (tdebs) that match the list of locales supported on the device. This list is then compared with the list of installed binaries. The translation packages that match the installed package are then installed with apt-get. langupdate will be adapted to work with both Emdebian Grip and Emdebian Crush.

Why C and C++?

The langupdate code is a mix of C and C++ because there is no direct hook into apt-get upgrade and no easy way of reading the apt-cache from C without forking a call to apt-cache itself. The apt bindings already require a fork to call apt-get. So langupdate uses libapt-pkg which is a C++ library. The rest of the code is C because it was the easiest way to do it at the time. If anyone fancies porting langupdate to just C++ for use on devices that do not have room for libglib2.0-0, patches are welcome.

langupdate uses a temporary file for the sources.list and updates it from the list of supported locales each time langupdate is run, i.e. it combines the effects of 'apt-get update' with 'apt-get upgrade'. tdeb packages have no dependencies so 'dist-upgrade' is not required.

Source

http://buildd.emdebian.org/svn/browser/current/host/trunk/langupdate/trunk/.

Development documentation

Code documentation is available via doxygen.

Package

langupdate is a native Emdebian package - it may appear in Debian at a later date when Debian supports tdebs.

This means that the default version of langupdate is cross built using emdebian-tools.

http://www.emdebian.org/packages/search.php?package=langupdate.

Repository support

em_installtdeb is a new script in emdebian-tools based on debhelper, to create translation packages (tdebs) for Emdebian. em_installtdeb is intended to separate out the individual translation files from the current Debian packages into packages without any translation files and a series of tdeb locale packages, one per translation. Generated packages use the syntax: $package-locale-$language_code_$version_all.deb

Once a package uses em_installtdeb, translation files should be removed from all packages in the normal build. em_installtdeb runs as a second build (very small, very fast) that simply converts and packages the *.po files into tdebs. Tdeb packages have no dependencies and no packages may depend upon them.

Each package now built for Emdebian includes tdeb support so that if the package contains gettext-compliant translations, tdeb packages are created. Emdebian also supports a locale repository with search support. Currently, certain compromises are made to allow tdebs to work without changes to packages like dpkg or reprepro - including the need to make two uploads, one for the main package (without translations) and one for the generated tdeb .changes file and all of the tdeb packages. For more information on em_installtdeb, see the em_installtdeb manpage.

Other resources on tdebs

Original Emdebian tdeb plans (Neil Williams).
Draft TDeb specification for Debian.

Emdebian needs tdebs urgently so the decision was taken to proceed on an experimental basis and then feed back to the Debian tdeb implementation and fold the Emdebian code into the Debian tdeb support.

See the emdebian contact page for information on contacting us.