Channels ▼

Community Voices

Dr. Dobb's Bloggers

Five Questions With Michael Kelly

September 22, 2008

Michael Kelly is a tester's tester. He talks about testing on his blog as well as in magazines ranging from InformIT to Better Software. He talks about testing at the Indianapolis Workshops on Software Testing he co-founded as well as at conferences ranging from the Indianapolis Quality Assurance Association Quality Enrichment Conference to the upcoming Pacific Northwest Software Quality Conference. And he tests software for companies large and small as well as for the fun of it.

Regardless of how you come to know Michael, his passion for improving the craft of testing and for helping testers improve themselves comes shining through. Here is what Michael has to say:

DDJ: What was your first introduction to testing? What did that leave you thinking about the act and/or concept of testing?

MDK: I started my career as an intern doing GUI-level test automation in a Fortune 500 company. In a matter of months I had found some of the works on that topic by Kaner, Pettichord, Marick, and others and was applying everything I could to the problems we were working to solve. After less than a year, myself and other $12-an-hour interns were teaching the $150-an-hour consultants how to write more maintainable tests and how they could get better coverage by making simple changes. The experience left me with an immediate sense of skepticism on the industry. I wondered at how those well-paid, well-meaning, and otherwise intelligent people could do work so poorly.

Once I entered the workforce proper, it didn't take me long to figure out testing would be my permanent home. It also didn't take me long to figure out what the problem had been. I found that most popular testing literature trivialized the work that we do and that people actually believed what tool vendors said. It was amazing to me that a group of twenty-year-olds could see through the marketing hype and experienced professionals couldn't. So, I sought out the only group of people I saw questioning the status quo. It was at the Austin Workshop on Test Automation that I eventually found the community that I could call home. They questioned and challenged everything, asked for empirical evidence of claims, and viewed testing as a cognitively-complex and interdisciplinary profession.

DDJ: What has most surprised you as you have learned about testing/in your experiences with testing?

MDK: What has surprised me the most is how much I keep learning. I still feel like I did as an intern. There's too much to read, too much to try, not enough time to practice, and never enough time to collaborate with others. I know that I know more then I did ten years ago, but when I look at everything I still only have a passing understanding of, I feel overwhelmed! It's not simply a problem of keeping up with technology, it's also finding time to learn our craft's history and to explore what I can learn about testing from philosophy, art, mathematics, and business; a few of the places from where I've developed my understanding of what we do as testers.

DDJ: How would you describe your testing philosophy?

MDK: I identify myself as a member of the context-driven community, and find that much of my testing philosophy mirrors the work done by James Bach, Jon Bach, Michael Bolton, Jonathan Kohl, Cem Kaner, and others doing work in rapid testing and exploratory testing. I think most testers would be well served by developing a deeper understanding of scientific and systems thinking, heuristics and biases, and project and business management. I would also encourage them to explore areas of interest outside of software testing and to bring something back to the field from those interests. As I write this, we just finished the third conference for the Association for Software Testing and at the conference we had talks pulling lessons from music, magic, lacrosse, finance, improv comedy, statistics, and civil engineering.

DDJ: What is the most interesting bug you have seen?

MDK: I've written about this before, but I remember my first security bug. It was so simple, I stumbled over it accidentally. (Well, I told the very angry people who were upset with me that it was an accident.) The problem started with a developer who had left his or her user ID in a code comment on the login page for a production system. It looked something like this:

<TABLE cellSpacing = 0 cellPadding = 1 border = 0 width = "98%" align = "center">
 <TBODY>
 <tr>
  <td class = "wb">
  <table border = 0>
   <FORM method = "POST" name = "login" action = "/login" enctype = "application/x-www-form-urlencoded">
    <tr>
    <td class = sb>
     login
     <input type = "text" name = "login_id" size = "32" maxlength = "64"/>
    </td>
    <td class = small>
     enter your username
    </td>
    </tr>
    <tr>
    <td class = sb>
     password
     <input type = "password" name = "password" size = "16" maxlength = "32"/>
    </td>
    <td class = small>
     enter your password
    </td>
    </tr>
    <tr>
    <td class = sb align = right>
     <input type = "submit" name = "submit" value = "login"/>
     <!-- u2x34t - Oct 12, 2004: Removed link to defect tracking system-->
    </td>
    </tr>
   </form>
  </table>
  </td>
 </tr>
 </TBODY>
</TABLE>

Note the comment enclosed in the <!-- --> tags. Out of curiosity, I entered the user ID (u2x34t) from the source code into the username field and tried to guess the password. I was rewarded with this:

"The password you entered is incorrect. Please try again."

I say rewarded because the first user ID I tried gave me this error:

"The user id you entered is not recognized. Please try again."

The specificity of the error messages for the system indicated that I was on the right track. I wouldn't have known that if the system had consistently displayed an error message similar to this one:

"The user id and/or password you entered are incorrect, please try again."

At this point, I knew I had a valid user ID, but I still didn't have the password. I didn't want to simply guess because I didn't want to lock the ID (earlier tests had shown that to be a problem). Instead, I started to wonder about that comment the developer made in the source code:

"Removed link to defect tracking system"

I asked myself some questions: What tracking system? How did they remove it? And where did it go? I looked at the source for more clues, but none could be found, at least to my untrained eye. I needed more source code. I figured that if the developer left comments on the login page, there was a good chance he or she left them in other code as well. At the bottom of the login page was a link to a help page. ("Need help logging in? Forgot your ID?") I followed that link and looked at the source for the help screen, where I found something similar to this:

<TABLE cellSpacing = 0 cellPadding = 1 border = 0 width = "98%" align = "center">
 <TBODY>
 <tr>
  <td class = "wb">
  <table border = 0>
   <tr>
   <td class = sb align = right>
    <!--<a href="http://URLForDefectTrackingSystem/%userIdParameter%.aspx" mce_href="http://URLForDefectTrackingSystem/%userIdParameter%.aspx">
    Submit a ticket.</a>-->
   </td>
   </tr>
   <tr>
   <!--Help text was here...-->
   </tr>
  </table>
  </td>
 </tr>
 </TBODY>
</TABLE>

To remove the link to the defect tracking system, the developer had simply commented out the link. Not only that—the link included a parameter for the user currently logged in so it would know who submitted the ticket. At that point, I had a URL that required a user ID, and I had a user ID. I simply copied the URL, pasted it into the address bar, typed the user ID in the appropriate place, and hit Enter. The system displayed an error page for a defect tracking system, stating that the system was no longer in use. I initially thought I had hit a dead end, but then I saw a link at the bottom of the page: "Return to application." I clicked the link and was rewarded with the home page (in production) for the application I was attempting to access. No password required!

After that, I was hooked, and I had to learn more about security testing.

DDJ: Is there something which is typically emphasized as important regarding testing that you think can be ignored, is unimportant?

MDK: I think there is a large focus on counting things in our industry. Many people count tests, use cases, defects, and even testers. We're obsessed with counting. I hear people say things like, "We have 4,000 tests" or "We found 50 defects" without any indication about what's actually being covered by the tests and what risks are addressed or what severity the defects are. There seems to be an impression that everything is equivalent or that if you just communicate a number you've communicated some meaning.

I don't find counting to be useful outside of its use as a general indicator for size or volume within a specific project context. On past projects, I've seen people count tests like they are lines of code. I find that you fall into all the same problems programmers run into when they do that. It may be a useful metric in some instances, but rarely is it a meaningful metric.

 

 

[See my Table Of Contents post for more details about this interview series.]

Related Reading


More Insights






Currently we allow the following HTML tags in comments:

Single tags

These tags can be used alone and don't need an ending tag.

<br> Defines a single line break

<hr> Defines a horizontal line

Matching tags

These require an ending tag - e.g. <i>italic text</i>

<a> Defines an anchor

<b> Defines bold text

<big> Defines big text

<blockquote> Defines a long quotation

<caption> Defines a table caption

<cite> Defines a citation

<code> Defines computer code text

<em> Defines emphasized text

<fieldset> Defines a border around elements in a form

<h1> This is heading 1

<h2> This is heading 2

<h3> This is heading 3

<h4> This is heading 4

<h5> This is heading 5

<h6> This is heading 6

<i> Defines italic text

<p> Defines a paragraph

<pre> Defines preformatted text

<q> Defines a short quotation

<samp> Defines sample computer code text

<small> Defines small text

<span> Defines a section in a document

<s> Defines strikethrough text

<strike> Defines strikethrough text

<strong> Defines strong text

<sub> Defines subscripted text

<sup> Defines superscripted text

<u> Defines underlined text

Dr. Dobb's encourages readers to engage in spirited, healthy debate, including taking us to task. However, Dr. Dobb's moderates all comments posted to our site, and reserves the right to modify or remove any content that it determines to be derogatory, offensive, inflammatory, vulgar, irrelevant/off-topic, racist or obvious marketing or spam. Dr. Dobb's further reserves the right to disable the profile of any commenter participating in said activities.

 
Disqus Tips To upload an avatar photo, first complete your Disqus profile. | View the list of supported HTML tags you can use to style comments. | Please read our commenting policy.
 


Video