Dr. Dobb's is part of the Informa Tech Division of Informa PLC

This site is operated by a business or businesses owned by Informa PLC and all copyright resides with them. Informa PLC's registered office is 5 Howick Place, London SW1P 1WG. Registered in England and Wales. Number 8860726.

Channels ▼

Ken North

Dr. Dobb's Bloggers

100 GB to 30,000 GB: Size, Speed, and Benchmarks

June 21, 2012

Anyone who's spent much time around software designers and developers knows they often compare the merits of various database systems. Those discussions usually focus on differentiators such as performance and ease of programming. This was true as long ago as the famous database debate between two ACM Turing Award winners, Charles Bachman and Edgar F. Codd.

Performance, data integrity, and ease of use are key characteristics to look for when comparing database systems. (In this context, ease of use is often a function of the tools available for modeling, design, and programming.)

When comparing database systems, we often make subjective judgments. At the same time, we yearn for an objective yardstick for measuring a database management system (DBMS) and database applications that use it.

Many in the database community have a strong interest in instrumentation and measurement of database performance. For developers and DBMS providers, performance is a characteristic we want to measure and quantify in terms that permit comparisons.

System architects are worth their weight in gold when they can identify and eliminate performance penalties. Numerous factors can hinder performance, including less than optimal hardware, shoddy database design, network latency, inferior database drivers, poor application coding, and queries that require optimization.

Benchmarks have proven to be a valuable tool for studying performance under different database workloads. They can provide information that's useful for identifying and isolating problems. They can be used in combination with regression testing as a quality assurance tool for new software releases. But the most high-profile use of database benchmarks is providing a quantitative comparison of differences between hardware and software used for database work.

Database Benchmarks

The computing industry has funded the Transaction Processing Council (TPC) to produce foremost benchmarks for database performance measurement, but there are others.

The Open Source Database Benchmark (OSDB) is a community-maintained benchmark that's modeled after AS3AP, the ANSI SQL Scalable and Portable Benchmark. There's been very little recent activity in the OSDB community since the 2010 announcement of version 0.90. PolePosition is a benchmark for comparing a combination of database engines with object-relational mapping technology.

Yahoo! Research created a benchmark that's well known in the NoSQL community. The Yahoo Cloud Serving Benchmark provides common workloads for comparing key-value stores and other cloud databases, such as Cassandra, HBase, and Sherpa. This benchmark is not designed for testing complex query operations, but it provides useful metrics for CRUD-type operations (read and write latency).

The gold standard of database benchmarks is the collection developed by the TPC. Computer manufacturers frequently cite the TPC results for publication when they've established a new price-performance record. The people who develop the TPC benchmarks include representatives of member organizations, such as Cisco, Dell, Fujitsu, HP, IBM, Intel, Microsoft, Oracle, Sybase, and VMware.

Related Reading

More Insights

Currently we allow the following HTML tags in comments:

Single tags

These tags can be used alone and don't need an ending tag.

<br> Defines a single line break

<hr> Defines a horizontal line

Matching tags

These require an ending tag - e.g. <i>italic text</i>

<a> Defines an anchor

<b> Defines bold text

<big> Defines big text

<blockquote> Defines a long quotation

<caption> Defines a table caption

<cite> Defines a citation

<code> Defines computer code text

<em> Defines emphasized text

<fieldset> Defines a border around elements in a form

<h1> This is heading 1

<h2> This is heading 2

<h3> This is heading 3

<h4> This is heading 4

<h5> This is heading 5

<h6> This is heading 6

<i> Defines italic text

<p> Defines a paragraph

<pre> Defines preformatted text

<q> Defines a short quotation

<samp> Defines sample computer code text

<small> Defines small text

<span> Defines a section in a document

<s> Defines strikethrough text

<strike> Defines strikethrough text

<strong> Defines strong text

<sub> Defines subscripted text

<sup> Defines superscripted text

<u> Defines underlined text

Dr. Dobb's encourages readers to engage in spirited, healthy debate, including taking us to task. However, Dr. Dobb's moderates all comments posted to our site, and reserves the right to modify or remove any content that it determines to be derogatory, offensive, inflammatory, vulgar, irrelevant/off-topic, racist or obvious marketing or spam. Dr. Dobb's further reserves the right to disable the profile of any commenter participating in said activities.

Disqus Tips To upload an avatar photo, first complete your Disqus profile. | View the list of supported HTML tags you can use to style comments. | Please read our commenting policy.