Dr. Dobb's is part of the Informa Tech Division of Informa PLC

This site is operated by a business or businesses owned by Informa PLC and all copyright resides with them. Informa PLC's registered office is 5 Howick Place, London SW1P 1WG. Registered in England and Wales. Number 8860726.

Channels ▼

zz_unused_Marianne Kolbasuk zz_unused_McGee

Dr. Dobb's Bloggers

Watson is a long way from Dr. Sbaitso

March 03, 2011

Cameron and I have been talking about Q&A systems over the course of our blog. We have discussed the difficulty of NLP in answering a simple question like "Which is better Coke or Pepsi?" and how parallelism can be used to help Q&A systems. Q&A research tries to develop approaches, algorithms, techniques to search a collections of documents in order to retrieve answers. Q&A research has to deal with a number of challenges or issues such as question classification and processing, ambiguity, context, decomposition, domains of questions, answer extraction and formulation.We have blogged about some of these issues when dealing with the aforementioned question. This type of research has been apart of AI for decades in our attempt to get a computer to answer a question and give a human-like answer.

In the early 1960's Joseph Weizenbaum from MIT developed ELIZA, a computer program that was an early and primitive example of natural language processing. It answered questions according to a script giving very human-like responses. One example of such an interaction was DOCTOR which simulated a psychotherapist.

This technology led to a series of computer programs called "chatterbots" designed to simulate human-like conversation. I remember a similar system back in the 1990's called Dr. Sbaitso that accompanied our Sound Blaster sound card. We used Dr. Sbaitso as an example of the potential of computer interfaces in the future at a local science fair with our children and children in the neighborhood. An example interaction would be:

Responder: I don't like that girl. Dr. Sbaitso: Why do you say you don't like that girl? Responder: Because she is not nice to me. Dr. Sbaitso: Why is it that she is not nice to you? Responder: I don't know. Dr. Sbaitso: Why don't you know? Responder: I just don't. Dr. Sbaitso: Would you like to change the topic?

It is evident in such an exchange there is no understanding by the system. It is just cleverly manipulating the response into another question. Also the nature of such an exchange does not require an answer that contains any information. The purpose of chatterbots were not to retrieve information but to be an example of a Turing Test. The argument has been, although one may be fooled to believe they were chatting with a human and not a program, intelligence has to do with understanding.

Now Watson is a long way from Dr. Sbaitso. As a matter of fact Watson is not carrying on a conversation but nevertheless its performance was quite astounding. I have even heard he beat some congressman at Jeopardy (that's not surprising). Does Watson understand? Well, Cameron and I was interviewed by some local news journalist about Watson and its technology. He asked us would Watson win. "Of course", we responded. But also we knew Watson would give some wrong answers. We also knew that it would probably give a few ridiculous answers. Not because the answer was not stored or could be implied or learned from the millions of pages of documents stored. It would be because it did not understand the question. Understanding the question as far as context, able to resolve any ambiguity (issues of pragmatics), etc., understand the question as a whole.

The "big error" Watson made was in its response to the question about "U.S. Cities":

"Its largest airport is named for a World War II hero; its second largest, for a World War II battle"

Watson said "What is Toronto?" Well, it had a very low confidence in that answer but because it was the final Jeopardy question, it had to give an answer. Now when considering Watson's approach to answering questions, there is a number of explanations to why Watson gave that answer. Could it be that Watson just didn't understand?

Related Reading

More Insights

Currently we allow the following HTML tags in comments:

Single tags

These tags can be used alone and don't need an ending tag.

<br> Defines a single line break

<hr> Defines a horizontal line

Matching tags

These require an ending tag - e.g. <i>italic text</i>

<a> Defines an anchor

<b> Defines bold text

<big> Defines big text

<blockquote> Defines a long quotation

<caption> Defines a table caption

<cite> Defines a citation

<code> Defines computer code text

<em> Defines emphasized text

<fieldset> Defines a border around elements in a form

<h1> This is heading 1

<h2> This is heading 2

<h3> This is heading 3

<h4> This is heading 4

<h5> This is heading 5

<h6> This is heading 6

<i> Defines italic text

<p> Defines a paragraph

<pre> Defines preformatted text

<q> Defines a short quotation

<samp> Defines sample computer code text

<small> Defines small text

<span> Defines a section in a document

<s> Defines strikethrough text

<strike> Defines strikethrough text

<strong> Defines strong text

<sub> Defines subscripted text

<sup> Defines superscripted text

<u> Defines underlined text

Dr. Dobb's encourages readers to engage in spirited, healthy debate, including taking us to task. However, Dr. Dobb's moderates all comments posted to our site, and reserves the right to modify or remove any content that it determines to be derogatory, offensive, inflammatory, vulgar, irrelevant/off-topic, racist or obvious marketing or spam. Dr. Dobb's further reserves the right to disable the profile of any commenter participating in said activities.

Disqus Tips To upload an avatar photo, first complete your Disqus profile. | View the list of supported HTML tags you can use to style comments. | Please read our commenting policy.