start

Translate Toolkit Developers Guide

The goal of the translate toolkit is to simplify and unify the process of translation.

History

The initial toolkit was designed to convert Mozilla .dtd and .properties files into Gettext PO format. The logic was not that PO was in any way superior but that by simplifying the translations process i.e. allowing a translator to use one format and one tool that we could get more people involved and more translators.

The tools have now evolved to include other formats such as OpenOffice.org and the goal is still to migrate various formats to a common format, PO and in the future XLIFF as more tools become available for that format.

These tools we group as converters. Along the way we developed other tools that allowed us to manipulate PO files and check them for consistency. As we evolved the converter tools we have also improved and abstracted the classes that read the various file types. In the future we hope to define these better so that we have a more or less stable API for converters.

Resources

Subversion access

We moved to the SourceForge Subversion repository in August 2006.

svn co https://translate.svn.sourceforge.net/svnroot/translate/src/trunk translate

Bugzilla

Communication

Source code map

The tools are stored in SVN https://translate.svn.sourceforge.net/viewvc/translate/src/trunk/ (previously in CVS). This rough map will allow you to navigate SVN:

Current source state

Note that this documentation is almost always for trunk.

Development trunk
Current stable branch Pootle-toolkit-1.2

Setup

The toolkit is installed by running:

./setup.py install

As root

The various setup options are yours to explore

To run / test things without installing, see the instructions for Pootle. Those instructions sets your environment for both Pootle and the toolkit

General overview of the programs

Each tool in the toolkit has both a core program and a command line wrapper. For example the oo2po converter:

This is done so that the tools can be used from within the Pootle server thus reusing the toolkit easily.

Command line options

Getting lost with the command line options? Well you might want to volunteer to move some of them into configuration files. But in terms of programming you might be confused as to where they are located. Many of the command line options are implemented in each tool. Things such as --progress and --errorlevel are used in each program. Thus these are abstracted in misc/optrecurse.py. While each tools unique command line options are implemented in xxx.py.

Converters

The converters each have a class that handles the conversion from one format to another. This class has one important method convertfile which handles the actual conversion.

A function convertXXX manages the conversion for the command line equivalent and essentially has at least 3 parameters: inputfile, outputfile and templatefile. It itself will call the conversion class to handle conversion of individual files. Recursing through multiple files is handled by the optrecurse.py logic.

The converters main function handles any unique command line options.

Where we are headed is to get to a level where the storage formats themselves are more aware of themselves and their abilities. Thus the converter could end up as one program that accepts storage format plugins to convert from anything to almost anything else. Although our target localisation formats are PO and XLIFF only.

If you want to create a new converter it is best to look at a simple instance such as csv2tbx or txt2po and their associated storage classes. The storage base class documentation will give you the information you need for the storage class implementation.

Tools

The tools in some way copy the logic of the converters. We have a class so that we can reuse a lot of the functionality in Pootle. We have a core function that take: input, output and templates. And we have a main function to handle the command line version.

pocount should be converted to this but does not follow this conventions. In fact pocount should move the counting to the storage formats to allow any format to return its own word count.

Checks

There's really only one, pofilter. But there are lots of helper functions for pofilter. pofilters main task is to check for errors in PO or XLIFF files. Here are the helper file and their descriptions.

pofilter is now relatively mature. The best areas for contributions are:

The API documentation is a good start if you want to add a new tests. To add a new language have a look at a language you understand amongst those already implemented.

Storage

These are the heart of the converters. Each destination storage format is implemented in its own file. Up until toolkit version 0.8, there was no formally defined API (the tools have been evolving and only recently stabilised), but they generally followed this structure. These classes are defined:

In the XML based formats e.g. TMX, XLIFF and HTML there is usually just an extended parser to manage the file creation.

Within each storage format there are many helper functions for escaping and managing the unique features of the actual format.

You can help by:

Base Classes

From toolkit 0.9 onwards, we are moving towards basing all storage formats on a set of base classes, in the move to a universal API. We're also fixing things so that escaping is much more sane and handled within the class itself not by the converters.

In base classes we have different terminology

We have also tried to unify terminology but this has been filtered into the old classes as far as possible.