Is the Future Fine-Grained Or Coarse-Grained Parallelism?
In his current editorial, our esteemed Dr. Dobb's Editor-in-Chief Andrew Binstock discusses whether or not parallel programming will ever be taken up by the masses. He comes to the conclusion that (*spoiler alert*) the multiple cores in his devices will be more likely be used to run multiple and separate processes rather than a single, multithreaded process.
I can't really disagree with his conclusion. When I first encountered parallel programming, it was in the context of scientific and technical computing. These areas of endeavor will always need more and more computational power and parallelism provides the excellent means to provide an ever-increasing number of relatively cheap processing cycles.
As multicore processors began to be announced, I had to wonder if there would really be much need for them. My test for the usefulness of just about any new-fangled technology is whether or not my 75-year-old mother would be able to get any benefit from it. I had a hard time coming up with a reason for her to invest in a multicore processor. She sends and reads e-mail, surfs the Interwebs, and keeps up with her grandkids via Facebook. Not much need for multiple cores within that scenario. Even so, I could easily see where multiple processes (coarse-grained parallelism), each running on separate cores, could give her a throughput benefit, as Andrew points out.
I know that my mother isn't the typical user or target market of new platforms and she isn't likely to be an avid user of those applications that can and do make use of parallelism on a multicore processor. These consumer apps tend to be specialized (i.e., games, image processing) and are few and far between. No "killer app" has come forward for multicore processors, so why do we have multicore processors? To answer that, we have to go back to rocket exhaust.
In order to justify the advent of multicore processors, chip manufacturers forecast that keeping on the path charted by Moore's Law, with respect to processor clock speed, would lead to heat generation by chips rivaling the temperatures seen at the exhaust nozzle of a firing rocket. Convincing the public to buy a laptop that could heat a 50-room mansion (not to mention what it would do to their lap) was deemed to present, at best, a tough marketing challenge. To keep on a path of increasing computational power without increasing the clock speed, multiple cores were built into current and future chips. The clock speed was also lowered on multicore chips. Thus, to get more performance in an application than the app had on the previous generation of processors, some form of parallelism would be needed.
One of the comments on Andrew's post states that parallel programming is so difficult that it takes a Nobel Laureate to do it well. (I'm still waiting for my Nobel Prize and have an open spot on the mantle reserved for it.) While that is a bit of an exaggeration, it's not too far off, at least in reference back to when multiple cores were first showing up. So, if almost no one was doing any parallel programming 6-7 years ago, besides avoiding the technological problem of heating and cooling, what were chip manufacturers thinking when they foisted multicore processors on an unprepared crowd of software developers? I believe they were hoping for something akin to the catch phrase from the movie Field of Dreams: "If you build it, they will come."
I can attest that Intel was aware of this seeming "cart before the horse" paradox and had been involved with the training of software engineers both internally and externally in how to do parallel programming (fine-grain parallelism). We also actively reached out to university and college faculty to convince them that the future of programming was to not make a distinction between (serial) programming and parallel programming; the latter would simply be "programming" from then on. One measure of the success of this effort is the recognition of parallelism as a necessary part of any computationally based degree program and the inclusion of parallel programming topics in the forthcoming ACM Computer Science Curriculum Guidelines.
Another success I see is one that Andrew cites himself: the creation of programming models that are aimed at making parallel programming easy. OpenMP, Intel Cilk Plus, Microsoft Task Parallel Library, and languages like Go with parallelism built in as a "first-class citizen" are all examples of the strides that have been made to facilitate parallelism in code.
There have been some inhibitors to parallel programming due to the changes in technology. The mobile computing space (smartphones, tablets, and such) wasn't as big then as it is today. As this technology evolves, the low-power, single-core processors that had been used up to the present are being replaced by low-power, multicore processors. Even though this is the popular area for software developers trying to make a name or an extra buck or to exercise some creativity and have some fun, it is an industry that is just getting started and there hasn't been any need or much support for parallelism. But now that the chips going into your tablet and phone are multicore, I think the parallel programming will soon follow.
I believe that we're still in the infancy of multicore processors and widespread parallel programming models. Looking back at the history of programming language development, I like to bring up how long something like the
for-loop took to be developed into its modern syntax. It was many years going from coding with machine code or assembly language that required a manual test of the counter and a GOTO that loops back over the body to equivalent higher-level language instructions to the syntactic sugar of the
for-loop where all of the niggly little details are taken care of in the background. If we consider MPI and threads as the assembly languages of parallelism, then OpenMP and TPL are the first passes of implementing parallelism in a much easier, higher-level way. Having parallelism as an integral part of the definition of a programming language is the next step in the evolution that will make things easier.
Will fine-grained parallel programming ever be embraced? I have confidence that the answer is "yes." It won't be tomorrow or the day after or even next year, but I do foresee a day when "parallel programming" is known simply as programming.