Facebook's engineering team has confirmed that the organization is open sourcing Warp, a preprocessor for the C and C++ languages.
Authored by Walter Bright in a joint project association with Facebook itself, Warp draws upon the social giant's sophisticated build system that uses distributed compilation in conjunction with several cache layers backed up by memcache.
This sophisticated build is part of what makes Warp special to development shops that work with large C++ code bases, who always need to pay close attention to build times.
Facebook research scientist Andrei Alexandrescu says that while conventional wisdom has it that preprocessing time is a "negligible part" of building a C++ binary, C++ is in fact "notoriously difficult" to parse, which means that C++ parsers tend to be "large and slow" compared to their counterparts for other languages.
Alexandrescu points out that code generation is also quite the time sink, especially if a developer is using optimizations — and that linking is a large serial step at the end of each build.
"Therefore, it would seem that the lowly task of going through the included files and expanding macros would take a comparatively short time and be essentially impossible to improve upon. Not so! Replacing gcc's preprocessor with Warp has led to significant improvements of our end-to-end build times (including linking)," he said.
The team has measured debug build speed improvements ranging from 10% all the way to 40%, all of which were in complex projects with massive codebases and many dependencies.
Alexandrescu hosted a Q&A with Walter Bright himself to ask him how he feels about Warp today. When questioned on whether he was surprised that solid overall build speed improvements could be achieved by improving C++ preprocessing speed alone, Bright said not at all.
"But to really figure why… one would need some historical perspective. When C was created, RAM was a scarce commodity; so many programs wouldn't fit in memory. Therefore, a common approach back then was to do multi-pass processing — a complex program would be divided into multiple, simpler batch stages, each going over the output of the previous stage and producing a file for the next stage. The specification of the C preprocessor follows that mold. For example, the very first step is to blindly replace trigraphs. The second removes each backslash followed by newline. And so on and so forth. This design allowed each of these steps to be carried by a separate program, but is poorly suited to today's compiler with large memories and parallelism-hungry CPUs."
Explaining his approach to open source, Bright said that he would have approached Warp very differently 10 years ago.
"I would have probably used C-style hand-written code for everything, and I am not above using goto (laughs). But I am always on the lookout for improving the 'sound' of my coding, as I think any programmer should be. The reality is that a C preprocessor is fundamentally a filter program, so range-based algorithms and pipe composition fit it best. For Warp, then, I hoped to get a lot of mileage from composing with range-based algorithms everywhere, and that is indeed what happened."
