Channels ▼

Web Development

Open Source Developers Build on Amazon Web Services

Clustered grid computing is increasingly available to the masses. Thanks to developer Doug Cutting, a Yahoo employee and prominent open source developer, aspiring programmers have access to the Lucene search engine library, an open source Web search crawler called Nutch, and Hadoop, an open source implementation of Google's MapReduce algorithm for processing large data sets.

U.K.-based software developer Tom White calls Nutch "Google in a jar."

Companies such as Krugle, Powerset, Wikipedia, and Zimbra have reached into the Google jar and are putting this open source code to use.

Used in conjunction with Amazon Web Services, Hadoop, Lucene, and Nutch promise anyone the fuel to fire up their own Google using Amazon S3 for storage and Amazon EC2 for processing. (PhDs, Google's ad sales system, a well-stocked kitchen, and the Google T-Rex are not included.)

Powerset, a natural language search startup, is pressing ahead with its own search engine based on Cutting's open source code. It will rely on Amazon's EC2 service for processing power.

Would-be search barons still have some work to do connecting the open source software and Amazon's hardware, but the open source community is rapidly doing just that: Developers like Cutting and White have been working to implement the Hadoop file system on S3, Amazon's storage service.

"I wanted to run some large natural language processing jobs on Hadoop but couldn't since I only had a handful of machines at my disposal," White said in an e-mail.

With the debut of Amazon's EC2 service last August, metered processing and storage gave developers the tools for ad hoc grid computation.

"The vision is that I write a MapReduce job, put my data on S3, run a simple script to run the job on a cluster of EC2 machines, and out pops the result on S3 for me to pick up," White said.

That vision is not far off.

Related Reading

More Insights

Currently we allow the following HTML tags in comments:

Single tags

These tags can be used alone and don't need an ending tag.

<br> Defines a single line break

<hr> Defines a horizontal line

Matching tags

These require an ending tag - e.g. <i>italic text</i>

<a> Defines an anchor

<b> Defines bold text

<big> Defines big text

<blockquote> Defines a long quotation

<caption> Defines a table caption

<cite> Defines a citation

<code> Defines computer code text

<em> Defines emphasized text

<fieldset> Defines a border around elements in a form

<h1> This is heading 1

<h2> This is heading 2

<h3> This is heading 3

<h4> This is heading 4

<h5> This is heading 5

<h6> This is heading 6

<i> Defines italic text

<p> Defines a paragraph

<pre> Defines preformatted text

<q> Defines a short quotation

<samp> Defines sample computer code text

<small> Defines small text

<span> Defines a section in a document

<s> Defines strikethrough text

<strike> Defines strikethrough text

<strong> Defines strong text

<sub> Defines subscripted text

<sup> Defines superscripted text

<u> Defines underlined text

Dr. Dobb's encourages readers to engage in spirited, healthy debate, including taking us to task. However, Dr. Dobb's moderates all comments posted to our site, and reserves the right to modify or remove any content that it determines to be derogatory, offensive, inflammatory, vulgar, irrelevant/off-topic, racist or obvious marketing or spam. Dr. Dobb's further reserves the right to disable the profile of any commenter participating in said activities.

Disqus Tips To upload an avatar photo, first complete your Disqus profile. | View the list of supported HTML tags you can use to style comments. | Please read our commenting policy.