Channels ▼
RSS

Testing

Debugging Multithreaded Applications in Windows


Putting It All Together

Let's stop for a minute and take a look at applying the previously discussed principles to a simplified real-world example. Assume that you are writing a data-acquisition application. Your design calls for a producer thread that samples data from a device every second and stores the reading in a global variable for subsequent processing. A consumer thread periodically runs and processes the data from the producer. In order to prevent data corruption, the global variable shared by the producer and consumer is protected with a Critical Section. An example of a simple implementation of the producer and consumer threads is shown in Listing Two. (Note that error handling is omitted for readability.)

Listing Two: Simple data acquisition device.

  	static int m_global = 0;
	static CRITICAL_SECTION hLock; // protect m_global

	// Simple simulation of data acquisition
	void sample_data()
	{
	EnterCriticalSection(&hLock);
	m_global = rand();
	LeaveCriticalSection(&hLock);
	}

		// This function is an example
		// of what can be done to data
		// after collection
		// In this case, you update the display
		// in real time
	void process_data()
	{
		EnterCriticalSection(&hLock);
		printf("m_global = 0x%x\n", m_global);
		LeaveCriticalSection(&hLock);
	}

		// Producer thread to simulate real time
		// data acquisition. Collect 30 s
		// worth of data
		unsigned __stdcall Thread1(void *)
		{
			int count = 0;
			SetThreadName(-1, "Producer");
			while (1)
			{
				// update the data
				sample_data();

				Sleep(1000);
				count++;
				if (count > 30)
					break;
			}
			return 0;
		}

		// Consumer thread
		// Collect data when scheduled and
		// process it. Read 30 s worth of data
		unsigned __stdcall Thread2(void *)
		{
			int count = 0;
			SetThreadName(-1, "Consumer");
			while (1)
			{
				process_data();

				Sleep(1000);
				count++;
				if (count > 30)
					break;
			}
			return 0;
		}

The producer samples data on line 34 and the consumer processes the data in line 53. Given this relatively simple situation, it is easy to verify that the program is correct and free of race conditions and deadlocks. Now assume that the programmer wants to take advantage of an error-detection mechanism on the data acquisition device that indicates to the user that the data sample collected has a problem. The changes made to the producer thread by the programmer are shown in Listing Three.

Listing Three: Sampling data with error checking.

        void sample_data()
        {
             EnterCriticalSection(&hLock);
             m_global =   rand();
             if   ((m_global % 0xC5F) == 0)
             {
                  //   handle error
                  return;
             }
             LeaveCriticalSection(&hLock);
        }

After making these changes and rebuilding, the application becomes unstable. In most instances, the application runs without any problems. However, in certain circumstances, the application stops printing data. How do you determine what's going on?

The key to isolating the problem is capturing a trace of the sequence of events that occurred prior to the system hanging. This can be done with a custom trace buffer manager or with tracepoints. This example uses the trace buffer implemented in Listing One.

Now armed with a logging mechanism, you are ready to run the program until the error case is triggered. Once the system fails, you can stop the debugger and examine the state of the system. To do this, run the application until the point of failure. Then, using the debugger, stop the program from executing. At this point, you'll be able bring up the Threads window to see the state information for each thread, such as the one shown in Figure 1.

multithreaded debugging
Figure 1: Examining thread state information using Visual Studio.

When you examine the state of the application, you can see that the consumer thread is blocked, waiting for the process_data() call to return. To see what occurred prior to this failure, access the trace buffer. With the application stopped, call the PrintTraceBuffer() method directly from Visual Studio's debugger. The output of this call in this sample run is shown in Figure 2.

     Thread ID| Timestamp|Msg
     ---------|----------|------------------------------------
     0x0000728|1137395188|Producer: sampled data value: 0x29
     0x00005a8|1137395188|Consumer: processed data value: 0x29
     0x0000728|1137395189|Producer: sampled data value: 0x78
     0x00005a8|1137395189|Consumer: processed data value: 0x78
     0x0000728|1137395190|Producer: sampled data value: 0x18BE
     0x0000728|1137395190|Producer: sampled data value: 0x6784
     0x0000728|1137395190|Producer: sampled data value: 0x4AE1
     0x0000728|1137395191|Producer: sampled data value: 0x3D6C

Figure 2: Output from trace buffer after error condition occurs.

Examination of the trace buffer log shows that the producer thread is still making forward progress. However, no data values after the first two make it to the consumer. This coupled with the fact that the thread state for the consumer thread indicates that the thread is stuck, points to an error where the critical section is not properly released. Upon closer inspection, it appears that the data value in line 7 of the trace buffer log is an error value. This leads up back to your new handling code, which handles the error but forgets to release the mutex. This causes the consumer thread to be blocked indefinitely, which leads to the consumer thread being starved. (Technically, this isn't a deadlock situation, as the producer thread is not waiting on a resource that the consumer thread holds.)

What this example shows is that by combining several features — the ability to name threads, tracepoints, and logging — it becomes possible to zero in on the problem without having to try to do all the diagnostic work after the fact. Developing other specialized activities that are tied to tracepoints can provide further assistance in bug hunting. Good luck!


Shameem Akhter is a platform architect at Intel, focusing on single socket multi-core architecture and performance analysis. Jason Roberts is a senior software engineer at Intel Corporation. This article is adapted from in the book Multi-Core Programming: Increasing Performance through Software Multi-threading, published by Intel Press, Copyright 2011 Intel Corporation.


Related Reading


More Insights






Currently we allow the following HTML tags in comments:

Single tags

These tags can be used alone and don't need an ending tag.

<br> Defines a single line break

<hr> Defines a horizontal line

Matching tags

These require an ending tag - e.g. <i>italic text</i>

<a> Defines an anchor

<b> Defines bold text

<big> Defines big text

<blockquote> Defines a long quotation

<caption> Defines a table caption

<cite> Defines a citation

<code> Defines computer code text

<em> Defines emphasized text

<fieldset> Defines a border around elements in a form

<h1> This is heading 1

<h2> This is heading 2

<h3> This is heading 3

<h4> This is heading 4

<h5> This is heading 5

<h6> This is heading 6

<i> Defines italic text

<p> Defines a paragraph

<pre> Defines preformatted text

<q> Defines a short quotation

<samp> Defines sample computer code text

<small> Defines small text

<span> Defines a section in a document

<s> Defines strikethrough text

<strike> Defines strikethrough text

<strong> Defines strong text

<sub> Defines subscripted text

<sup> Defines superscripted text

<u> Defines underlined text

Dr. Dobb's encourages readers to engage in spirited, healthy debate, including taking us to task. However, Dr. Dobb's moderates all comments posted to our site, and reserves the right to modify or remove any content that it determines to be derogatory, offensive, inflammatory, vulgar, irrelevant/off-topic, racist or obvious marketing or spam. Dr. Dobb's further reserves the right to disable the profile of any commenter participating in said activities.

 
Disqus Tips To upload an avatar photo, first complete your Disqus profile. | View the list of supported HTML tags you can use to style comments. | Please read our commenting policy.
 

Video