User Testing: Homework or Fun?

Who says user testing is supposed to be fun? (DDJ blogger Michael Hunter notwithstanding, of course.) Who says, for that matter, that education is supposed to be fun? Well, researchers at the University of Southern California, that's who.

To prove their point that you can have your game and play it to, USC researchers are developing user-testing tools for games that capture and analyze play experience to automatically detect design weakness and flaws.

"Traditionally," says Tim Marsh, a post-doctoral researcher at USC's Viterbi School of Engineering, "game companies hire teenagers, and turn them loose trying to find flaws and gaps in the game," which they report either verbally or in writing, along with their impressions."

Marsh contends that this approach is neither systematic nor scientific. Instead, the approach he and his team, which includes Cyrus Shahabi, Kiyoung Yang, and Shamus Smith, are proposing analyzes "immersidata." Shahabi came up with the term when referring to a machine-readable record of commands sent to the computer by keyboards, joysticks, and other controls, collected in parallel with a videotape recording of the player at the game session.

The testing tool, called "ISIS" (short for "Immersidata AnalySIS')identifies data of interest and index events within the videotape. For the game development application, ISIS returns examples of six different kinds of occurrences, or "points" in the immersidata/video record:

• Activity completion points.
  
• Task completion points, which let researchers review the performance of a task.
  
• Break points, when nothing is happening.
  
• Wandering points, when user-player are moving, but not selecting objects .
  
• Critical events, actions leading up to accomplishment or non-accomplishment.
  
• Navigation errors. Collisions which indicate inadequate or poor design.

By backtracking from the points, researchers can see how the point developed.

According to Marsh, the system effectively finds problems in the areas it is set to look for. Improvements to ISIS include adding functionality for finding and identifying other potential problem areas--recognizing repetition patterns by players, and replacing and/or supplementing the video capture with a replay of the game from the player's point of view. Marsh is also working on ways to use immersidata to capture more aspects of the game experience, including emotional/empathetic elements.

Posted by Jon Erickson at 08:24 AM  Permalink