Channels ▼

Mark Nelson

Dr. Dobb's Bloggers

Pigz on the Wing

January 17, 2010

Mark Adler  just might have the distinction of having his code running on more computers than anyone else on earth. This is because Mark (in conjunction with Jean-loup Gailly and many more volunteers) wrote zlib - the free library that reads and writes streams compressed with the deflate algorithm.

Of course, this means that Mark's code is used just about everywhere to read and write files in Zip and gzip format - so it is deployed on virtually every desktop, running any O/S. It is also deployed on zillions of embedded systems running Linux and other operating systems.

It's nice to daydream about how rich Mark would be if he had a nickel for every deployment of zlib, but of course, if zlib cost a nickel, it probably wouldn't be so ubiquitous.  I think he will have to be content with eventual enshrinement in the FOSS Hall of Hame.

Next Up: Pigz

Apparently keeping his finger on the pulse of multicore hysteria, Mark recently took some time out to produce a nice little program called pigz, a parallel version of gzip. Pigz has been out for a couple of years now, and this month rolled up to version 2.1.6.

Pigz uses the pthreads library to parallelize compression of data, by grabbing big chunks and processing each one in its own thread. Decompression is not so easy to break apart. Pigz will still use multiple threads when decompressing, but it is basically just decomposing I/O and computation.

What the World Needs Now

There are two things we could do to make parallel decompression a reality.

First, Mark points out that specially constructed deflate streams could be used for parallel processing during decompression. Presumably we would need some modification to the deflate specification  to add labeling or marking of some sort for these special streams.

Second, there is no reason that most archivers can't start using parallel threads right now when extracting multiple files from archives. Programs like WinZip could be doing this today, but I don't believe it is implemented in many products.  

A Challenge

It will be tough to resolve this question, but can anyone make a case for some other library code that is more widely deployed than zlib? I can think of a few candidates, but nothing that clearly beats it. One possible winner would be some O/S code for a super-cheap 4- or 8-bit processor that has been produced in quantities exceeding a billion. Unfortunately I don't have the kind of research reports handy to figure that out.

Author's Note: I apologize for the title of the article, which deliberately mangles the preferred pronunciation of the product as pig-zee.

Related Reading

More Insights

Currently we allow the following HTML tags in comments:

Single tags

These tags can be used alone and don't need an ending tag.

<br> Defines a single line break

<hr> Defines a horizontal line

Matching tags

These require an ending tag - e.g. <i>italic text</i>

<a> Defines an anchor

<b> Defines bold text

<big> Defines big text

<blockquote> Defines a long quotation

<caption> Defines a table caption

<cite> Defines a citation

<code> Defines computer code text

<em> Defines emphasized text

<fieldset> Defines a border around elements in a form

<h1> This is heading 1

<h2> This is heading 2

<h3> This is heading 3

<h4> This is heading 4

<h5> This is heading 5

<h6> This is heading 6

<i> Defines italic text

<p> Defines a paragraph

<pre> Defines preformatted text

<q> Defines a short quotation

<samp> Defines sample computer code text

<small> Defines small text

<span> Defines a section in a document

<s> Defines strikethrough text

<strike> Defines strikethrough text

<strong> Defines strong text

<sub> Defines subscripted text

<sup> Defines superscripted text

<u> Defines underlined text

Dr. Dobb's encourages readers to engage in spirited, healthy debate, including taking us to task. However, Dr. Dobb's moderates all comments posted to our site, and reserves the right to modify or remove any content that it determines to be derogatory, offensive, inflammatory, vulgar, irrelevant/off-topic, racist or obvious marketing or spam. Dr. Dobb's further reserves the right to disable the profile of any commenter participating in said activities.

Disqus Tips To upload an avatar photo, first complete your Disqus profile. | View the list of supported HTML tags you can use to style comments. | Please read our commenting policy.