Channels ▼

Christopher Diggins

Dr. Dobb's Bloggers

Parallel Map and Reduce in Heron

February 03, 2010

I've been rather quiet lately, while I continue to labor away on Heron in the background. Rather than freezing the feature set and moving as fast as I can towards an official v1.0 release of Heron, I've decided to make it a bit more sexy by parallelizing the reduce and map operators.

Heron has several operators for list processing:

 

  • map - Transforms a list into a new list, by applying a unary function to each item in the original list to get the values in the new list 
  • accumulate - Iterates over items in a list, combining each item with an accumulator value using a binary function. 
  • select  - Creates a list by selecting items from another list that satisfy a unary predicate. 
  • reduce - Combines items in a list by applying a binary function to two items in a list in a nondeterministic order.

 

The current implementation of Heron runs these operations sequentially. However, of these operations the "map", "select", and "reduce" operator all can be parallelized (read as: run on multiple cores).

Releasing a language with the potential to be interpreted in parallel, without actually doing so in my default implementation, would be kind of anticlimactic so I decided that I really should try to get at least the map and reduce operators parallel.

If "map and reduce" sounds familiar, but you aren't sure where, MapReduce  is the name of a framework for performing distributed computing on large data sets.  

So why do am I doing all of this? Well because I am too darn lazy to learn how to do concurrent programming properly using threads. I figured it would be easier to make my language manage concurrency automatically for me. The irony is I still have to learn to use threads properly to make the implementation multithreaded. Oh well, at least I'm having fun, I think. 

Related Reading


More Insights






Currently we allow the following HTML tags in comments:

Single tags

These tags can be used alone and don't need an ending tag.

<br> Defines a single line break

<hr> Defines a horizontal line

Matching tags

These require an ending tag - e.g. <i>italic text</i>

<a> Defines an anchor

<b> Defines bold text

<big> Defines big text

<blockquote> Defines a long quotation

<caption> Defines a table caption

<cite> Defines a citation

<code> Defines computer code text

<em> Defines emphasized text

<fieldset> Defines a border around elements in a form

<h1> This is heading 1

<h2> This is heading 2

<h3> This is heading 3

<h4> This is heading 4

<h5> This is heading 5

<h6> This is heading 6

<i> Defines italic text

<p> Defines a paragraph

<pre> Defines preformatted text

<q> Defines a short quotation

<samp> Defines sample computer code text

<small> Defines small text

<span> Defines a section in a document

<s> Defines strikethrough text

<strike> Defines strikethrough text

<strong> Defines strong text

<sub> Defines subscripted text

<sup> Defines superscripted text

<u> Defines underlined text

Dr. Dobb's encourages readers to engage in spirited, healthy debate, including taking us to task. However, Dr. Dobb's moderates all comments posted to our site, and reserves the right to modify or remove any content that it determines to be derogatory, offensive, inflammatory, vulgar, irrelevant/off-topic, racist or obvious marketing or spam. Dr. Dobb's further reserves the right to disable the profile of any commenter participating in said activities.

 
Disqus Tips To upload an avatar photo, first complete your Disqus profile. | View the list of supported HTML tags you can use to style comments. | Please read our commenting policy.
 


Video