A few weeks back I was debating whether to use iterators or streaming operations for I/O on my data compression code. I bemoaned the fact that the C++ iterators that perform stream I/O use the insertion and extraction operators, making them unsuitable for binary data compression.

It looks like I spoke too soon.

The Problem

The canonical way to use iterators to go over an iostream is via istream_iterator or its partner, ostream_iterator. In this short sample, I create a binary file with values 0 to 256, then try to read back the data using this iterator type:

#include <iostream>
#include <fstream>
#include <iterator>
using namespace std;

int main(int argc, char* argv[])
{
    ofstream temp_out("temp.dat", std::ios_base::binary );
    for ( int i = 0 ; i < 256 ; i++ )
        temp_out.put( (char) i );
    temp_out.close();
    ifstream temp_in("temp.dat", std::ios_base::binary );
    char last = -1;
    int count = 0;
    istream_iterator<char> ii( temp_in );
    while ( ii != std::istream_iterator<char>() ) {
        char c = *ii++;
        count++;
        if ( c != char(last+1) )
            cout << "Error on character number: " << (int) last << endl;
        last = c;
    }
    cout << "Count: " << count << endl;
    return 0;
}

Running this program gives the following output:

Error on character number: 8
Error on character number: 31
Count: 250

Due to the fact that the extraction operator uses whitespace as a delimiter, we get errors trying to read tab, line feed, vertical tab, form feed, carriage return, and space — which means we simply don't see six characters in the input file. This is not too bad when parsing ascii text, but when trying to do compression on binary data, it just won't work.

I figured this was the end of it until reader Fred Jardon chimed in with:

Isn't istreambuf_iterator the iterator you're looking for?

http://cplusplus.com/reference/std/iterator/istreambuf_iterator/

It directly reads from the inner streambuf without using the extraction operator.

The opposite iterator exists: ostreambuf_iterator

Well, Fred is quite right, and it only serves to show my lack of depth when it comes to iostreams. The two classes, istreambuf_iterator and ostreambuf_iterator read directly from the underlying buffer, and don't use the extraction operator, which means they don't have the whitespace issues seen above. Changing just two lines fixes the problem:

    istreambuf_iterator<char> jj(temp_in.rdbuf());
    while ( ii != std::istreambuf_iterator<char>() ) {

Running that I get the pristine output I long for:

Count: 256

I'd like to tell you the ramifications of diving down into the object and working on the buffer, but I'm afraid the implementation details still elude me. There's a lot more going on in iostreams than in good old <stdio.h>, and half of what I learned about it was wiped out in the standardization process. So be it.

In any case, I think the proper use of istreambuf_iterator tips the scale in favor of iterators for me, which reverses my previous thinking.

And many thanks to Fred for straightening out the mess.