Channels ▼


Deja Parallel All Over Again

Uh oh--looks like the industry is about to rediscover parallel computing. I first got into it in the mid-1980s, when the Connection Machine (in the U.S.) and the transputer (in the UK) seemed to signal that we would soon all be writing code to run on hundreds or thousands of processors at once.

The excitement died pretty quickly, as did most of the companies trying to build a business around parallel computing: Of the more than two dozen firms we surveyed in 1991 , only a handful were still in business five years later, all of which had either moved into some other area, or set parallel computing aside.

Parallelism had a brief renaissance in the mid '90s, when Linux clusters first appeared. They have steadily been gaining ground as both servers and supercomputers, but up until now, most programmers have been able to ignore them. Judging from the recent flurry of discussion in the blogosphere, that may be about to change, thanks primarily to the appearance of multicore CPUs (which put several independent processors on a single chip). As Tim Bray recently pointed out , Sun, Intel, AMD, and IBM are all pursuing multicore architectures as a way of putting more CPU cycles in programmers' hands. Within two or three years, most of us could well be faced with hardware that only reaches its potential if multithreading and other parallel tricks are used.

Bray then went on to discuss some of the obstacles that ubiquitous parallelism faces. Number five on his list (after legacy apps, observability, Java mutxes, and LAMP, and just ahead of the non-problem "how many is enough?") is "testing and debugging". As far as I'm concerned, this is the only problem worth mentioning. Most scientific programmers turn their backs on massively-parallel machines because they simply can't get the damn things to work. Task farms? No problem. Simple regular grids? Piece of cake. But anything else--any program that actually had lots of different things going on at once--requires heroic effort to debug and tune.

The underlying problem is that thinking concurrently is a jillion times harder than thinking about one thing at a time. Adding non-determinism (which is a feature of almost all parallel programs), makes it a jillion times harder again, and today's tools simply aren't powerful enough to help us. (Yes, I've used Totalview. Yes, it's better than the competition. No, it doesn't help enough--sorry.)

Enter the notion of omniscient, or replay, debuggers. These capture information about programs as they execute, so that developers can step backward in time and see what happened just before a breakpoint or a crash. Bil Lewis described one in the June 2005 issue of Dr. Dobb's Journal, but the idea has been around a long time--a colleague of mine, Irving Reid, remembers using one to debug microcontroller code in the 1980s.

Replay debuggers are wonderful even when you're dealing with sequential code, but they're practically indispensable when you're faced with interrupts, context switches, and messages. The problem is, logging enough information to be useful slows programs down considerably. This distorts the sequence of events, which can in turn move or hide the bug you're chasing.

In that light, I'd like to know whether anyone at Sun or elsewhere is planning to use one of the cores in a multicore to log the activities of the others, or (more radically) to architect one of the cores purely for logging purposes. This wouldn't just be for developers' sakes: As a sometime system administrator, the thought of having applications log what they're doing, 24/7, so that I can send crash trace to vendors when their software fails on me, is very attractive.

Devoting that much silicon to logging and debugging might seem extravagant, but there's no point producing hardware whose full power can't ever be realized. I'm still fascinated by the potential of parallel computing; maybe this time, we'll finally see that potential become a reality.

Gregory Wilson is a DDJ contributing editor. He can be contacted at


Related Reading

More Insights

Currently we allow the following HTML tags in comments:

Single tags

These tags can be used alone and don't need an ending tag.

<br> Defines a single line break

<hr> Defines a horizontal line

Matching tags

These require an ending tag - e.g. <i>italic text</i>

<a> Defines an anchor

<b> Defines bold text

<big> Defines big text

<blockquote> Defines a long quotation

<caption> Defines a table caption

<cite> Defines a citation

<code> Defines computer code text

<em> Defines emphasized text

<fieldset> Defines a border around elements in a form

<h1> This is heading 1

<h2> This is heading 2

<h3> This is heading 3

<h4> This is heading 4

<h5> This is heading 5

<h6> This is heading 6

<i> Defines italic text

<p> Defines a paragraph

<pre> Defines preformatted text

<q> Defines a short quotation

<samp> Defines sample computer code text

<small> Defines small text

<span> Defines a section in a document

<s> Defines strikethrough text

<strike> Defines strikethrough text

<strong> Defines strong text

<sub> Defines subscripted text

<sup> Defines superscripted text

<u> Defines underlined text

Dr. Dobb's encourages readers to engage in spirited, healthy debate, including taking us to task. However, Dr. Dobb's moderates all comments posted to our site, and reserves the right to modify or remove any content that it determines to be derogatory, offensive, inflammatory, vulgar, irrelevant/off-topic, racist or obvious marketing or spam. Dr. Dobb's further reserves the right to disable the profile of any commenter participating in said activities.

Disqus Tips To upload an avatar photo, first complete your Disqus profile. | View the list of supported HTML tags you can use to style comments. | Please read our commenting policy.