Don't Develop GUI Tests, Teach Your App To Test Itself!
While developing native GTK- and Cocoa-based versions of the Plastic GUI, my team and I faced the challenge of developing a new test suite.
But instead of adopting a couple of new tools, one for Linux and one for Mac, we decided to teach the GUIs to test themselves. Read on to find out how.
To GUI Test or To Unit Test
New code brings the opportunity to reduce the number of GUI tests by introducing more unit tests, which is always a good option since GUI tests tend to be slower and more fragile.
But, at the end of the day, we decided that having some GUI tests would prevent user interface issues, so the challenge was to reduce the development cost (considering one GUI would be GTK based and the other one Cocoa based).
Welcome to GUI Test Hell
We've used a commercial GUI test package in the past for most of our Windows-based GUI testing. I won't reveal the name here since basically I'm stating that we're not totally happy with it.
The main problems we faced were:
- It was slow.
- It failed to detect false positives.
- It was hard to maintain since it was not written in a production language, but rather in some sort of scripting language.
Initially (years ago), the situation was even worse: We "recorded" the tests using one of the great "record and we'll auto generate the test code" features that every decent GUI testing tool includes.
Don't do that!
Auto-generated code is crap and you'll regret it each time you make a change.
I don't know about non-product-oriented code: I mean, you develop your project in 5 months, release it to the customer, and forget it (lucky you!). In that instance, auto-recording probably makes sense. For a long-running project like Plastic, it definitely doesn't. Clean, well-organized code is always a better choice.
The Fundamental Problem of GUI Testing Toolkits
Here's basic problem we always faced with the tools we used: Sometimes the testing framework won't be able to find a control, for some obscure reason, and the test will fail. You repeat the test and it works, but you'll have a false positive.
It also happens in a different form: Sometimes it takes seconds for a visual control to become "visible" for the testing toolkit, slowing down the entire test execution.
Initially, we thought it was a product issue, but then we tried the UI Automation APIs that were included in Windows long ago. We used them through White and we ended up with exactly the same issue: Sometimes a control was not "visible" for the test code, not even UI Spy, the native app, was able to detect it.
This was frustrating, so when we started the GTK/Cocoa GUI code, we wanted to try something different.
GUI Test Cheating: Do It Yourself
Basically, the premise for this development was:
- What are the issues with GUI tests? Control detection and ease of development and maintenance.
- How can we fix them? Well, let's avoid having to "find" the controls, and instead make them always available. Then, let's use our favorite language and environment to write the test code.
We also tried the following idea (which we still don't know whether it is 100% valid or not, but it is working so far): Load the test code "inside" the app so a different thread runs the test while the main GUI thread responds to events. This way, the "test thread" can access the "visible controls" using some predefined interfaces for each dialog, window, table (grid), and so on, do actual clicks sending events, and easily check the results of the actions being performed.
We created a few tests and it actually looked promising:
- The test code was written in C#, our main production language, so every developer feels at home with it.
- The environment to launch tests would be the same we use for Unit Tests and command-line automation tests: our own PNUnit, a "parallel" extension we contributed to NUnit eons ago and still use and evolve today. It allows us to distribute tests among different machines, coordinate tests involving several clients and servers on different platforms, and so on. While all this is supported by any modern commercial GUI test package, using our own system made us feel at home.
- It runs fast, we didn't have to wait for controls to become visible, and we never missed a control.