People with little experience in the field wonder why designing secure systems is so hard. Here's a good example.Several months ago I bought some tickets to a concert. These tickets were additions to a subscription, and subscribers have the privelege of exchanging tickets for alternative dates until shortly before the performance. Accordingly I asked the box office to hold the tickets, and I would pick them up before the concert.

The box office told me that as a security precaution, I would be asked to show them the credit card that I had used to buy the tickets. This is a reasonable request, as from their viewpoint, anyone could show up claiming to be one of the people in my party and asking to pick up the tickets.

A while later I got a letter from my bank. Someone had stolen about a million credit-card numbers from one of their data centers, so as a security precaution they were going to send me a new credit card. My old card would automatically be cancelled the first time I used the new one. This is a reasonable precaution; even though they might be especially vigilant about anyone trying to use one of the stolen numbers, it is clearly better all around to issue new cards.

Now consider the combined effects of these two reasonable precautions: Someone has to create an unforeseen exception to the system in order for me to be able to pick up the tickets that I bought and paid for. Once I received my new credit card, I destroyed the old one, not realizing that I would need that credit card to pick up the tickets.

I think that security systems in general have two problems, neither of which is widely appreciated.

The first is the one I just described: Two security precautions, each of which is reasonable by itself, combine in unanticipated ways to cause annoying side effects.

The second is more subtle: Dealing with these side effects often requires creating ad hoc exceptions, and these exceptions can create their own vulnerabilities. In this case, for example, the theft of credit-card numbers made it possible, at least in principle, for someone to use tickets for which someone else has paid. All the interloper needs to do is convince the people at the box office that the legitimate credit-card number has been changed to whatever credit card the interloper happens to hold.

I think that an underappreciated difficulty in designing security systems is balancing the need to deal with unexpected problems with the need to repel clever opportunists.Dealing with these side effects often requires creating ad hoc exceptions, and these exceptions can create their own vulnerabilities.