Dr. Dobb's is part of the Informa Tech Division of Informa PLC

This site is operated by a business or businesses owned by Informa PLC and all copyright resides with them. Informa PLC's registered office is 5 Howick Place, London SW1P 1WG. Registered in England and Wales. Number 8860726.

Channels ▼

Pablo Santos

Dr. Dobb's Bloggers

Don't Develop GUI Tests, Teach Your App To Test Itself!

June 16, 2014

While developing native GTK- and Cocoa-based versions of the Plastic GUI, my team and I faced the challenge of developing a new test suite.

But instead of adopting a couple of new tools, one for Linux and one for Mac, we decided to teach the GUIs to test themselves. Read on to find out how.

To GUI Test or To Unit Test

New code brings the opportunity to reduce the number of GUI tests by introducing more unit tests, which is always a good option since GUI tests tend to be slower and more fragile.

But, at the end of the day, we decided that having some GUI tests would prevent user interface issues, so the challenge was to reduce the development cost (considering one GUI would be GTK based and the other one Cocoa based).

Welcome to GUI Test Hell

We've used a commercial GUI test package in the past for most of our Windows-based GUI testing. I won't reveal the name here since basically I'm stating that we're not totally happy with it.

The main problems we faced were:

  • It was slow.
  • It failed to detect false positives.
  • It was hard to maintain since it was not written in a production language, but rather in some sort of scripting language.

Initially (years ago), the situation was even worse: We "recorded" the tests using one of the great "record and we'll auto generate the test code" features that every decent GUI testing tool includes.

Don't do that!

Auto-generated code is crap and you'll regret it each time you make a change.

I don't know about non-product-oriented code: I mean, you develop your project in 5 months, release it to the customer, and forget it (lucky you!). In that instance, auto-recording probably makes sense. For a long-running project like Plastic, it definitely doesn't. Clean, well-organized code is always a better choice.

The Fundamental Problem of GUI Testing Toolkits

Here's basic problem we always faced with the tools we used: Sometimes the testing framework won't be able to find a control, for some obscure reason, and the test will fail. You repeat the test and it works, but you'll have a false positive.

It also happens in a different form: Sometimes it takes seconds for a visual control to become "visible" for the testing toolkit, slowing down the entire test execution.

Initially, we thought it was a product issue, but then we tried the UI Automation APIs that were included in Windows long ago. We used them through White and we ended up with exactly the same issue: Sometimes a control was not "visible" for the test code, not even UI Spy, the native app, was able to detect it.

This was frustrating, so when we started the GTK/Cocoa GUI code, we wanted to try something different.

GUI Test Cheating: Do It Yourself

Basically, the premise for this development was:

  • What are the issues with GUI tests? Control detection and ease of development and maintenance.
  • How can we fix them? Well, let's avoid having to "find" the controls, and instead make them always available. Then, let's use our favorite language and environment to write the test code.

We also tried the following idea (which we still don't know whether it is 100% valid or not, but it is working so far): Load the test code "inside" the app so a different thread runs the test while the main GUI thread responds to events. This way, the "test thread" can access the "visible controls" using some predefined interfaces for each dialog, window, table (grid), and so on, do actual clicks sending events, and easily check the results of the actions being performed.

We created a few tests and it actually looked promising:

  • The test code was written in C#, our main production language, so every developer feels at home with it.
  • The environment to launch tests would be the same we use for Unit Tests and command-line automation tests: our own PNUnit, a "parallel" extension we contributed to NUnit eons ago and still use and evolve today. It allows us to distribute tests among different machines, coordinate tests involving several clients and servers on different platforms, and so on. While all this is supported by any modern commercial GUI test package, using our own system made us feel at home.
  • It runs fast, we didn't have to wait for controls to become visible, and we never missed a control.

Related Reading

More Insights

Currently we allow the following HTML tags in comments:

Single tags

These tags can be used alone and don't need an ending tag.

<br> Defines a single line break

<hr> Defines a horizontal line

Matching tags

These require an ending tag - e.g. <i>italic text</i>

<a> Defines an anchor

<b> Defines bold text

<big> Defines big text

<blockquote> Defines a long quotation

<caption> Defines a table caption

<cite> Defines a citation

<code> Defines computer code text

<em> Defines emphasized text

<fieldset> Defines a border around elements in a form

<h1> This is heading 1

<h2> This is heading 2

<h3> This is heading 3

<h4> This is heading 4

<h5> This is heading 5

<h6> This is heading 6

<i> Defines italic text

<p> Defines a paragraph

<pre> Defines preformatted text

<q> Defines a short quotation

<samp> Defines sample computer code text

<small> Defines small text

<span> Defines a section in a document

<s> Defines strikethrough text

<strike> Defines strikethrough text

<strong> Defines strong text

<sub> Defines subscripted text

<sup> Defines superscripted text

<u> Defines underlined text

Dr. Dobb's encourages readers to engage in spirited, healthy debate, including taking us to task. However, Dr. Dobb's moderates all comments posted to our site, and reserves the right to modify or remove any content that it determines to be derogatory, offensive, inflammatory, vulgar, irrelevant/off-topic, racist or obvious marketing or spam. Dr. Dobb's further reserves the right to disable the profile of any commenter participating in said activities.

Disqus Tips To upload an avatar photo, first complete your Disqus profile. | View the list of supported HTML tags you can use to style comments. | Please read our commenting policy.