Channels ▼


How Do We Loop-The-Data-Loop With Hadoop?

Pervasive Software has this month released Data Integrator v10 - Hadoop Edition to the Apache Hadoop user community with a view to helping the flow of data both to and from Hadoop-based big data stores.

The argument here is that developers are now tasked with integrating big data sets into and out of environments where devices (and the applications they are running) may be working with what can only logically be called "little data" by comparison.

Pervasive CTO Mike Hoskins argues that programmers will need to find the agility to combine and process data from all their operations within the new highly scalable data stores of Hadoop.

"The combination of our high performance HDFS and HBase connectors and Pervasive Data Integrator visual ETL tooling eradicates the need for custom MapReduce code for executing data import-export operations," said Hoskins.

NOTE: Hadoop Distributed File System (HDFS) is the primary storage system used by Hadoop applications. HDFS creates multiple replicas of data blocks and distributes them on compute nodes throughout a cluster to enable reliable, extremely rapid computations. HBase is the Hadoop database.

"I'm particularly jazzed about our high-performance HBase loading. For the first time, users can (with a single click) move data from traditional data stores including DB2, MySQL, Netezza, PostgreSQL, SQLServer, Oracle, Teradata, and Vertica directly into HBase, the dominant NoSQL database provided free with all Hadoop distributions," Hoskins added.

Industry comments suggest that Hadoop may now need visual data integration tooling of this kind — especially given the need to execute increasingly complex workloads against massive amounts of data at high speed. Demand for powerful big data analytic platforms appears to be coming at us quickly right now.

Pervasive says it is helping to bridge non-Hadoop data more easily into Hadoop with no MapReduce code — and this is the loop-the-loop balancing act that now needs to be pulled off.

Related Reading

More Insights

Currently we allow the following HTML tags in comments:

Single tags

These tags can be used alone and don't need an ending tag.

<br> Defines a single line break

<hr> Defines a horizontal line

Matching tags

These require an ending tag - e.g. <i>italic text</i>

<a> Defines an anchor

<b> Defines bold text

<big> Defines big text

<blockquote> Defines a long quotation

<caption> Defines a table caption

<cite> Defines a citation

<code> Defines computer code text

<em> Defines emphasized text

<fieldset> Defines a border around elements in a form

<h1> This is heading 1

<h2> This is heading 2

<h3> This is heading 3

<h4> This is heading 4

<h5> This is heading 5

<h6> This is heading 6

<i> Defines italic text

<p> Defines a paragraph

<pre> Defines preformatted text

<q> Defines a short quotation

<samp> Defines sample computer code text

<small> Defines small text

<span> Defines a section in a document

<s> Defines strikethrough text

<strike> Defines strikethrough text

<strong> Defines strong text

<sub> Defines subscripted text

<sup> Defines superscripted text

<u> Defines underlined text

Dr. Dobb's encourages readers to engage in spirited, healthy debate, including taking us to task. However, Dr. Dobb's moderates all comments posted to our site, and reserves the right to modify or remove any content that it determines to be derogatory, offensive, inflammatory, vulgar, irrelevant/off-topic, racist or obvious marketing or spam. Dr. Dobb's further reserves the right to disable the profile of any commenter participating in said activities.

Disqus Tips To upload an avatar photo, first complete your Disqus profile. | View the list of supported HTML tags you can use to style comments. | Please read our commenting policy.