Dryad: Tools for Concurrency

In Greek mythology, dryads were tree nymphs -- female deities that hung out in the woods. Since then, dryads have popped up in literature (C.S. Lewis's "The Chronicles of Narnia"), computer games (World of Warcraft), and even TV shows (Xena: Warrior Princess and Scooby-Do). And now their latest incarnation is as a development environment for concurrent programming.

More specifically, Dryad is an infrastructure out of Microsoft Research designed to help programmers access the resources of thousands of machines without knowing anything about concurrent programming.

The goal of the Dryad project, according to senior researcher Michael Isard, "is to abstract away a lot of the practical details of the cluster -- the data placement, the network hierarchy, the strategy for fault tolerance -- so that the programmer can work at a higher level and concentrate more on the structure of the computation and rely on Dryad to do the scheduling, the fault tolerance, and those kinds of things."

In Dryad: Distributed Data-Parallel Programs from Sequential Building Blocks, a paper written by Isard, Mihai Budiu, Yuan Yu, Andrew Birrell, and Dennis Fetterly, the researchers explain:

The Dryad execution engine handles all the difficult problems of creating a large, distributed, concurrent application: scheduling the use of computers and their CPUs, recovering from communication or computer failures, and transporting data between [computation] vertices.

Moreover, say the authors, Dryad is designed to scale effectively, from a single, multicore computer to small clusters of computers to datacenters with thousands of computers.

One thing that's particularly interesting is the team has leveraged non-research oriented Microsoft technologies to extend Dryad. For instance, the researches have rolled out DryadLINQ which combines Dryad and .NET's LINQ. Dryad takes care of the distributed for parallel applications, while LINQ lets you write and debug applications in a SQL-like query language in a familiar environment like Visual Studio.

"Dryad takes away the programmer’s need to understand low-level concurrency, but it still relies on the programmer to think at some level of abstraction about how the job could be divided up," explains Isard. "Programmers don’t have to worry about low-level synchronization primitives, but they do still have to understand something about the structure of what needs to be done and what depends on what else."

All in all, Dryad sound interesting and important, especially as the need for tools that enable concurrency accelerates.

-- Jonathan Erickson
jerickson@ddj.com

Posted by Jon Erickson at 09:14 AM  Permalink