Dr. Dobb's contributing editor Mike Riley recently had an opportunity to talk with Joe Duffy, lead developer and architect for Microsoft's Parallel Extensions to .NET, and author of Concurrent Programming on Windows.
Q: Joe, what are the biggest challenges facing Windows developers seeking to upgrade their legacy code to multicore environments?
A: Dealing with side-effects.
Most programs on Windows are imperative + shared-memory, which means that data structures are typically mutable. This is the legacy we inherit from our C-programming heritage and has only been encouraged by the gradual shift to OOP which encapsulates and hides that mutability. Finding compute-bound regions of code that would benefit from parallelism is also quite challenging. A simple for-each loop might be doing a lot of work, and thus is a strong candidate for parallelism. But it may also have subtle effects on its environment that make parallelism impossible (e.g., loop-carried dependencies) or very tricky (e.g. thread-unsafe updates to objects accessible within the loop). It's not always obvious which is the case.
This is a tangled web. Imagine you've found a costly loop you want to parallelize, but can't because it contains side-effects. Papering over the effects with locks can, with patience and great care, lead to safety, but even when done right comes at a performance penalty. The cost and risk are high, and the benefit is marginalized. Writing efficient parallel code really does demand an alternative style of programming that mixes functional and imperative programming in a way that our languages currently don't provide in a first-class way. This is why a 900-page book on the subject is necessary. Those who figure it out can be rewarded handsomely -- with 4x, 8x, 16x, performance improvements -- but the task must be undertaken with great care.
Q: How long do you expect the programming technology to be as simple as a check box in a language's compiler options (i.e., check a "Optimize for multi-core systems" similar to the way "Compile for 64-bit systems" works today)?
A: We're a long way away from that. The key, I think, is abstraction. Somebody in the stack will always have to worry about the hard problems I mention above. Over time, however, languages and frameworks will help higher level developers write "the right" code by default. Having people write more declarative programs will enable smarter compilers and runtimes to parallelize, provided the declarative syntax side-steps the side-effects problem. For example, PLINQ in .NET 4.0 is highly parallel because developers express problems in terms of bulk-data operations. MATLAB and logic solvers can parallelize because programs are declarative and not operational. The trend towards DSLs [Domain-Specific Languages] will broaden the impact of declarative programming.
Q: Which of the 16 chapters in your book was the most challenging to write? Why?
A: Each was challenging for its own reasons. Chapter 13, "Data and Task Parallelism," was hard because it's the money chapter. Ultimately, that's the one that helps people to write parallel code on today's Windows OS, with today's C++ or .NET Framework, on today's generation of multiprocessors. The rest of the book was a collection of many useful facts. But Chapter 13 tries to steer people towards a set of best practices for writing parallel code.
Q: What is the most exciting real-world parallel programming application you have seen using the Parallel Extension to .NET? Do you expect these extensions to simply be part of the main .NET Framework in future .NET releases?
A: An MSR team internally uses PLINQ as part of a more holistic distributed parallel programming model. They use very similar techniques to PLINQ internally to distribute LINQ queries to nodes in a cluster, and then once on a node in the cluster they parallelize using PLINQ (and indirectly TPL). They call their project DryadLINQ, and have a public website where you can learn more.. I'm very excited about this project, because it shows that the same programming model (LINQ) can scale horizontally + vertically at once. You get interactive access to terabytes and terabytes of data, which only distributed + local parallelism can give you. This is a glimpse of the future of commoditized parallelism.
There have also been several simulations with impressive visualizations (e.g., graphics, stock market modeling, health-care analysis). These are great because they demonstrate why we still need a desktop PC and can't just rely on the cloud for everything.
As to whether PLINQ, TPL, and the other components in Parallel Extensions to the .NET Framework will show up in .NET sometime, you bet. They are shipping in Visual Studio 10, a.k.a. .NET Framework 4.0. We also have a set of new C++ components too, including some more extensibility points for custom schedulers than you'll find in .NET. Check out http://msdn.microsoft.com/en-us/concurrency/ for more information, including links to our team blogs.
Q: Where can interested readers go on the web to learn more about your book, Concurrent Programming on Windows, as well as your thoughts on parallel application development in general?
A: They can start with my blog. From there, they can find a sample chapter from my book and random musings on the state of concurrency. I also recorded a Channel9 interview recently where I discuss the future of programming languages in lieu of the concurrency problem.