Maestro: A Special-Purpose Language for Parallel Programming

Joining us is Niklas Gustafsson, a software architect and member of Microsoft's Parallel Computing Platform team.

Editor's Note: Subsequent to this article being published, the 'Maestro' name has been changed to 'Axum'.

Q: Niklas, what is Maestro?

A: First of all, Maestro is an incubation project within Microsoft's Developer Division. That means that we're somewhere between research and product development, trying to take some advanced ideas and put them to practical use, but not quite sure what the timeframe for release will be.

Maestro is what I call a "special-purpose language" (SPL) meant to address some of the complexities of authoring parallel applications. By "special-purpose" I mean that I fully expect that Maestro would not be the only language you use to write an application, it is designed to be used with an object-oriented language like C# or VB. Unlike some domain-specific languages, Maestro has enough general-purpose features so that you don't have to jump out into C# or VB just because you want to add two numbers or scan through a list, but its focus is on orchestrating concurrent component interactions, not implementing algorithms or authoring type libraries.

The foundation for Maestro is .NET, but it is not just another OO language. While it can access all of .NET's class libraries and capabilities, you wouldn't use Maestro to define new classes and types. Instead, the foundational idea behind Maestro is the need to isolate parallel components and provide a much simplified model for concurrency within each isolated component. Maestro is an actor-oriented language relying on a simplified model declarative coordination within a component, and message-passing between components that are freely concurrent.

The same model is achieved by using a separate OS process for each component, but that has a very high overhead on all current operating systems. Maestro relies on the compiler and the supporting runtime to provide the constraints necessary to realize the benefits of the isolated model.

Q: What problems does Maestro solve/address?

A: The basic principle of Maestro is to reduce the complexity of a parallel application to the point where most programmers should be able to write a parallel application without even thinking about it as being parallel. In a model based solely on shared memory, such as we have with all object-oriented languages, you have enormous problems with reference aliasing. This means that any two given reference variables of the same type may or may not point at the same object. If they don't, concurrent access is safe, if they do, it's typically not; a lot of programmer productivity is lost trying to rein in this complexity.

The alternatives to this model are starkly different; for decades, there's been tension between those who believe that a shared memory model is necessary, and those who think that a pure message-passing model is required. It's like the war between the big-endian and little-endian camps in Gulliver's Travels.

With .NET, relying on a pure message-passing model is not realistic, as you would have to forgo the rich assets that have been built up in the base class library as well as components such as WCF, WF, WPF, etc. They are all based on object-oriented principles. At the same time, the emergence of cloud-based services in the mainstream forces us to deal with message-passing as a first-class concept in our programming tool box. Modeling message-passing as method calls is doomed to relegate it to second-class status and will not allow us to handle the complexities associated with handling a multitude of messages with a reasonable effort.

Maestro is neither a pure message-passing language, nor does it rely only on shared memory. Instead, it defines domains of shared memory and provides a simple declarative model for coordinating access to that shared state. Then, it requires asynchronous message-passing for interactions between domains and prevents reference aliasing. This way, you can still use all of .NET and have a well-defined concurrency model.

Q: It's similiarity to Erlang suggests it is best suited to parallel programming. Is this correct?

A: Maestro is best suited to partitioning an application into domains of state and active agents (actors) that read and manipulate that state. What you wind up with is a parallel application, but my hope is that you shouldn't ever really have to think of it as one. Just like most Web developers don't think of their application(s) as parallel, which they are, Maestro developers shouldn't, either.

In fact, the programming model of the Web is one of the inspirations behind Maestro -- here we have this massively parallel application running on hundreds of millions of processors and being programmed by tens of thousands of developers. Recipe for disaster, right? Wrong -- it works wonderfully: scales well, is remarkably robust and resilient to partial failures.

The similarities between Maestro and Erlang are fairly superficial. Both are actor-oriented, but Erlang achieves its isolation by being purely functional, which means having to use a radically different approach to programming throughout. It requires different idioms, different tools, a different debugger experience, etc., to do it justice. Maestro is a .NET language and adds only a couple of new concepts that the developer has to learn.

Q: If readers want to learn more about Maestro, where can they go?

A: Right now, there's not a lot published on Maestro, but we have some exciting things that we hope to be able to bring to the public in the spring. For now, I would recommend the Channel 9 interview Charles Torre did with the Maestro team. We've also started discussing it a bit on the PFX team blog, and I will start talking more about it on my relatively sparse blog.