Channels ▼
RSS

Global Developer

Facebook's C/C++ Warp Preprocessor Open Sourced


Facebook's engineering team has confirmed that the organization is open sourcing Warp, a preprocessor for the C and C++ languages.

Authored by Walter Bright in a joint project association with Facebook itself, Warp draws upon the social giant's sophisticated build system that uses distributed compilation in conjunction with several cache layers backed up by memcache.

This sophisticated build is part of what makes Warp special to development shops that work with large C++ code bases, who always need to pay close attention to build times.

Facebook research scientist Andrei Alexandrescu says that while conventional wisdom has it that preprocessing time is a "negligible part" of building a C++ binary, C++ is in fact "notoriously difficult" to parse, which means that C++ parsers tend to be "large and slow" compared to their counterparts for other languages.

Alexandrescu points out that code generation is also quite the time sink, especially if a developer is using optimizations — and that linking is a large serial step at the end of each build.

"Therefore, it would seem that the lowly task of going through the included files and expanding macros would take a comparatively short time and be essentially impossible to improve upon. Not so! Replacing gcc's preprocessor with Warp has led to significant improvements of our end-to-end build times (including linking)," he said.

The team has measured debug build speed improvements ranging from 10% all the way to 40%, all of which were in complex projects with massive codebases and many dependencies.

Alexandrescu hosted a Q&A with Walter Bright himself to ask him how he feels about Warp today. When questioned on whether he was surprised that solid overall build speed improvements could be achieved by improving C++ preprocessing speed alone, Bright said not at all.

"But to really figure why… one would need some historical perspective. When C was created, RAM was a scarce commodity; so many programs wouldn't fit in memory. Therefore, a common approach back then was to do multi-pass processing — a complex program would be divided into multiple, simpler batch stages, each going over the output of the previous stage and producing a file for the next stage. The specification of the C preprocessor follows that mold. For example, the very first step is to blindly replace trigraphs. The second removes each backslash followed by newline. And so on and so forth. This design allowed each of these steps to be carried by a separate program, but is poorly suited to today's compiler with large memories and parallelism-hungry CPUs."

Explaining his approach to open source, Bright said that he would have approached Warp very differently 10 years ago.

"I would have probably used C-style hand-written code for everything, and I am not above using goto (laughs). But I am always on the lookout for improving the 'sound' of my coding, as I think any programmer should be. The reality is that a C preprocessor is fundamentally a filter program, so range-based algorithms and pipe composition fit it best. For Warp, then, I hoped to get a lot of mileage from composing with range-based algorithms everywhere, and that is indeed what happened."


Related Reading


More Insights






Currently we allow the following HTML tags in comments:

Single tags

These tags can be used alone and don't need an ending tag.

<br> Defines a single line break

<hr> Defines a horizontal line

Matching tags

These require an ending tag - e.g. <i>italic text</i>

<a> Defines an anchor

<b> Defines bold text

<big> Defines big text

<blockquote> Defines a long quotation

<caption> Defines a table caption

<cite> Defines a citation

<code> Defines computer code text

<em> Defines emphasized text

<fieldset> Defines a border around elements in a form

<h1> This is heading 1

<h2> This is heading 2

<h3> This is heading 3

<h4> This is heading 4

<h5> This is heading 5

<h6> This is heading 6

<i> Defines italic text

<p> Defines a paragraph

<pre> Defines preformatted text

<q> Defines a short quotation

<samp> Defines sample computer code text

<small> Defines small text

<span> Defines a section in a document

<s> Defines strikethrough text

<strike> Defines strikethrough text

<strong> Defines strong text

<sub> Defines subscripted text

<sup> Defines superscripted text

<u> Defines underlined text

Dr. Dobb's encourages readers to engage in spirited, healthy debate, including taking us to task. However, Dr. Dobb's moderates all comments posted to our site, and reserves the right to modify or remove any content that it determines to be derogatory, offensive, inflammatory, vulgar, irrelevant/off-topic, racist or obvious marketing or spam. Dr. Dobb's further reserves the right to disable the profile of any commenter participating in said activities.

 
Disqus Tips To upload an avatar photo, first complete your Disqus profile. | View the list of supported HTML tags you can use to style comments. | Please read our commenting policy.
 

Video