Cameron and I have been talking about Q&A systems over the course of our blog. We have discussed the difficulty of NLP in answering a simple question like "Which is better Coke or Pepsi?" and how parallelism can be used to help Q&A systems. Q&A research tries to develop approaches, algorithms, techniques to search a collections of documents in order to retrieve answers. Q&A research has to deal with a number of challenges or issues such as question classification and processing, ambiguity, context, decomposition, domains of questions, answer extraction and formulation.We have blogged about some of these issues when dealing with the aforementioned question. This type of research has been apart of AI for decades in our attempt to get a computer to answer a question and give a human-like answer.

In the early 1960's Joseph Weizenbaum from MIT developed ELIZA, a computer program that was an early and primitive example of natural language processing. It answered questions according to a script giving very human-like responses. One example of such an interaction was DOCTOR which simulated a psychotherapist.

This technology led to a series of computer programs called "chatterbots" designed to simulate human-like conversation. I remember a similar system back in the 1990's called Dr. Sbaitso that accompanied our Sound Blaster sound card. We used Dr. Sbaitso as an example of the potential of computer interfaces in the future at a local science fair with our children and children in the neighborhood. An example interaction would be:

Responder: I don't like that girl. Dr. Sbaitso: Why do you say you don't like that girl? Responder: Because she is not nice to me. Dr. Sbaitso: Why is it that she is not nice to you? Responder: I don't know. Dr. Sbaitso: Why don't you know? Responder: I just don't. Dr. Sbaitso: Would you like to change the topic?

It is evident in such an exchange there is no understanding by the system. It is just cleverly manipulating the response into another question. Also the nature of such an exchange does not require an answer that contains any information. The purpose of chatterbots were not to retrieve information but to be an example of a Turing Test. The argument has been, although one may be fooled to believe they were chatting with a human and not a program, intelligence has to do with understanding.

Now Watson is a long way from Dr. Sbaitso. As a matter of fact Watson is not carrying on a conversation but nevertheless its performance was quite astounding. I have even heard he beat some congressman at Jeopardy (that's not surprising). Does Watson understand? Well, Cameron and I was interviewed by some local news journalist about Watson and its technology. He asked us would Watson win. "Of course", we responded. But also we knew Watson would give some wrong answers. We also knew that it would probably give a few ridiculous answers. Not because the answer was not stored or could be implied or learned from the millions of pages of documents stored. It would be because it did not understand the question. Understanding the question as far as context, able to resolve any ambiguity (issues of pragmatics), etc., understand the question as a whole.

The "big error" Watson made was in its response to the question about "U.S. Cities":

"Its largest airport is named for a World War II hero; its second largest, for a World War II battle"

Watson said "What is Toronto?" Well, it had a very low confidence in that answer but because it was the final Jeopardy question, it had to give an answer. Now when considering Watson's approach to answering questions, there is a number of explanations to why Watson gave that answer. Could it be that Watson just didn't understand?