Last week, I described a bug in a multipass compiler in which:

• One of the compiler's passes would sometimes — but not always — complain about spurious characters in the input that it had obtained from the previous pass.
• It was possible to capture the file that the previous pass produced for this pass as input.
• When I did so, and ran the failing compiler pass on the captured file, it would always work.

I invited readers to speculate about what might cause such a state of affairs, and got a number of interesting comments. In fact, several of these comments are closely related to what turned out to be the actual problem, as we shall see.

Let me simplify the description of the situation by calling the two compiler passes A and B. What I had learned was:

• If I run A and send its output to B, it sometimes fails.
• If I run A, capture its output, and run B with the captured file as its input, B always succeeds.

One reader speculated that the problem must be in how the file is being passed from A to B, and that's what I thought, too. However, when I looked at what was happening, the framework around the compiler was doing exactly the same thing that I was doing: Run A with its output directed to a temporary file, then run B with its input coming from the temporary file.

After thinking about this situation for a long time, I realized that there was one crucial difference between how the compiler was normally run and how I ran it when I was trying to provoke a failure:

When the compiler ran normally, phase A ran afresh every time; when I captured the output and ran phase B with the captured file, I ran A only once and then ran B repeatedly.

What if I was looking in the wrong place? That is, what if the problem was not that B was behaving differently with the same input, but rather that A was sometimes producing different output? Such questions are easy to answer once asked: I wrote a shell script to run A repeatedly and compare its output each time with that from the previous run.

Sure enough, once in a while, compiler phase A would produce different output from what it normally produced. Moreover, that difference was that occasional characters would be replaced by zero characters. Not a byte with all bits zero, but the ASCII representation of the digit 0.

What I had just learned was that regardless of what might or might not be wrong with phase B, there was something wrong with phase A — and until I found that problem, there was no point in looking further at phase B. More generally, when a program occasionally produces different output from the same input, maybe you're mistaken about it being the same input.

Of course, I now had to figure out why phase A occasionally produced output with spurious characters in it. Next week, I'll reveal more details about what I learned.