Tagging, Building, and Source Tarballs
In preparation for starting automation, we recently started to use a script,
release_sanity.py, that was originally written by a Release Engineering summer intern. This Python script assists a release engineer with double-checking that all configurations for a release match what is checked into our tools and configuration repositories. It also checks what is in the specified release code revisions for
mozilla-release and all the (human) languages for this release, which will be what the builds and language repacks are generated from.
Figure 3: Automated tagging.
The script accepts the buildbot config files for any release configurations that will be used (such as desktop or mobile), the branch to look at (
mozilla-release), the build and version number, and the names of the products that are to be built (such as "fennec" or "firefox"). It will fail if the release repositories do not match what's in the configurations, if locale repository changesets don't match our shipping locales and localization changeset files, or if the release version and build number don't match what has been given to our build tools with the tag generated using the product, version, and build number. If all the tests in the script pass, it will reconfigure the buildbot master where the script is being run and where release builders will be triggered, and then generate the "send change" that starts the automated release process.
After a release engineer starts up builders, the first automated step in the Firefox release process is tagging all the related source code repositories to record which revision of the source, language repositories, and related tools are being used for this version and build number of a release candidate. These tags allow us to keep a history of Firefox and Fennec (mobile Firefox) releases' version and build numbers in our release repositories. For Firefox releases, one example tag set is
FIREFOX_10_0_RELEASE FIREFOX_10_0_BUILD1 FENNEC_10_0_RELEASE FENNEC_10_0_BUILD1.
A single Firefox release uses code from about 85 version control repositories that host things such as the product code, localization strings, release automation code, and helper utilities. Tagging all these repositories is critical to ensure that future steps of the release automation are all executed using the same set of revisions. It also has several other benefits: Linux distributions and other contributors can reproduce builds with exactly the same code and tools that go into the official builds, and it also records the revisions of source and tools used on a per-release basis for future comparison of what changed between releases.
Once all the repositories are branched and tagged, a series of dependent builders automatically start up: one builder for each release platform plus a source bundle that includes all source used in the release. The source bundle and built installers are all uploaded to the release directory as they become available. This allows anyone to see exactly what code is in a release, and gives a snapshot that would allow us to re-create the builds if we ever needed to (for example, if our VCS failed somehow). For the Firefox build's source, sometimes we need to import code from an earlier repository. For example, with a beta release this means pulling in the signed-off revision from Mozilla-Aurora (our more-stable-than-nightly repository) for Firefox 10.0b1. For a release, it means pulling in the approved changes from Mozilla-Beta (typically the same code used for 10.0b6) to the Mozilla-Release repository. This release branch is then created as a named branch whose parent changeset is the signed-off revision from the "go to build" provided by the Release Coordinator.
The release branch can be used to make release-specific modifications to the source code, such as bumping version numbers or finalizing the set of locales that will be built. If a critical security vulnerability is discovered in the future that requires an immediate fix a chemspill a minimal set of changes to address the vulnerability will be landed on this relbranch and a new version of Firefox generated and released from it. When we have to do another round of builds for a particular release,
buildN, we use these relbranches to grab the same code that was signed off on for "go to build," which is where any changes to that release code will have been landed. The automation starts again and bumps the tagging to the new changeset on that relbranch.
Our tagging process does a lot of operations with local and remote Mercurial repositories. To streamline some of the most common operations we've written a few tools to assist us. retry.py is a simple wrapper that can take a given command and run it, retrying several times if it fails. It can also watch for exceptional output conditions and retry or report failure in those cases. We've found it useful to wrap retry.py around most of the commands which can fail due to external dependencies. For tagging, the Mercurial operations could fail due to temporary network outages, Web server issues, or the backend Mercurial server being temporarily overloaded. Being able to automatically retry these operations and continue saves a lot of our time, since we don't have to manually recover, clean up any fallout and then get the release automation running again. hgtool.py is a utility that encapsulates several common Mercurial operations, like cloning, pulling, and updating with a single invocation. It also adds support for Mercurial's share extension, which we use extensively to avoid having several full clones of repositories in different directories on the same machine. Adding support for shared local repositories significantly sped up our tagging process, since most full clones of the product and locale repositories could be avoided. An important motivation for developing tools like these is making our automation as testable as possible. Because tools like hgtool.py are small, single-purpose utilities built on top of reusable libraries, they're much easier to test in isolation.
Today our tagging is done in two parallel jobs: one for desktop Firefox which takes around 20 minutes to complete as it includes tagging 80+ locale repositories, and another for mobile Firefox which takes around 10 minutes to complete, because we have fewer locales currently available for our mobile releases. In the future, we would like to streamline our release automation process so that we tag all the various repositories in parallel. The initial builds can be started as soon as the product code and tools requirement repository is tagged, without having to wait for all the locale repositories to be tagged. By the time these builds are finished, the rest of the repositories will have been tagged so that localization repackages and future steps can be completed. We estimate this can reduce the total time to have builds ready by 15 minutes.
Localization Repacks and Partner Repacks
Once the desktop builds are generated and uploaded to ftp.mozilla.org, our automation triggers the localization repackaging jobs. A "localization repack" takes the original build (which contains the
en-US locale), unpacks it, replaces the en-US strings with the strings for another locale that we are shipping in this release, then repackages all the files back up again (this is why we call them repacks). We repeat this for each locale shipping in the release. Originally, we did all repacks serially. However, as we added more locales, this took a long time to complete, and we had to restart from the beginning if anything failed out mid-way through.
Figure 4: Repacking Firefox for each localization.
Now, instead, we split the entire set of repacks into six jobs, each processed concurrently on six different machines. This approach completes the work in approximately one-sixth of the time. This also allows us to redo a subset of repacks if an individual repack fails, without having to redo all repacks. (We could split the repacks into even more, smaller, concurrent jobs, but we found it took away too many machines from the pool, which affected other unrelated jobs triggered by developers on our continuous integration system.)
The process for mobile (on Android) is slightly different, as we produce only two installers: an English version and a multi-locale version with more than a dozen languages built into the installer instead of a separate build per locale. The size of this multi-locale version is an issue, especially with slow download speeds onto small mobile devices. One proposal for the future is to have other languages be requested on demand as add-ons from http://addons.mozilla.org.
In Figure 4, you can see that we currently rely on three different sources for our locale information:
l10n-changesets_mobile-release.json. (There is a plan to move all three into a unified JSON file.) These files contain information about the different localizations we have, and certain platform exceptions. Specifically, for a given localization we need to know which revision of the repository to use for a given release and we need to know if the localization can build on all of our supported platforms (for instance, Japanese for Mac comes from a different repository all together). Two of these files are used for the Desktop releases and one for the Mobile release (this JSON file contains both the list of platforms and the changesets).