Dr. Dobb's is part of the Informa Tech Division of Informa PLC

This site is operated by a business or businesses owned by Informa PLC and all copyright resides with them. Informa PLC's registered office is 5 Howick Place, London SW1P 1WG. Registered in England and Wales. Number 8860726.

Channels ▼


Task-Based Programming in Windows

In 2005, Herb Sutter argued that the free lunch of Moore's Law was at an end, and that multicore systems demanded alternative approaches to coding. His point was that we could not simply rely on languages and compilers to shield us from rethinking the way we structure our code.

Several years on, and despite the near ubiquity of multicore systems, the hardware potential seems to run well ahead of general practice in software development. When developers do turn, perhaps in desperation, to parallelization, it's still not unusual for them to end up writing apps with a thousand threads and an architecture based on what is sometimes described as the "synchronize and suffer" model.

This article discusses a different, rather straightforward approach I have found useful in building a number of multithreaded applications (server and client) over the years using existing features of the Windows API.

For more on Windows programming, it is the focus of the March 2013 issue of Dr. Dobb's Journal: Understand the new Win8 socket API, prevent XSS attacks in ASP.NET Web apps, refresh your knowledge of atomic operations, and more! Download the PDF.


The design consists of the following sequence:

  1. From the main thread discrete pieces of work are handed off to an input queue. Each piece is discrete in that it is owned by only one thread at a time, which minimizes the amount of synchronization.
  2. The input queue (or more generically an I/O queue) is a wrapper around a Windows I/O completion port.
  3. A thread pool services the input queue. Each thread retrieves the next item from the queue and processes it.
  4. Active tasks can be returned to the input queue. That is, they can cooperatively yield or they can run until the current work is exhausted. The thread then returns to the input queue for its next piece of work. If the input queue doesn't have enough work to keep all the threads in the pool working, then threads are effectively parked until there is more work.
  5. Output is posted to an output queue (another I/O queue instance), which may be on the main thread or another dedicated, special-purpose thread that acts as a coordinator, or collator of the output. For example, sorting or reassembling data which may have been processed out of order in the thread pool.
  6. The collated output arrives at its final destination or sink (Figure 1).

Figure 1: An overview of the process.

There is nothing particularly new in this arrangement. It is a variation on a design previously presented in Dr. Dobb's. The goal rather is to show that the advantages in reliability and performance over "synchronize and suffer" can be obtained with only a modest effort.

Let's have a look at key pieces. The goal is for the application code to have minimal knowledge, if any, of its threading context. The threading should all be handled by the plumbing.

I/O Queues

Our input and output queues are instances of a generalized I/O queue, which is a fairly simple wrapper around a Windows I/O Completion Port (IOCP). IOCPs have been around since NT 4.0 and have been written about extensively.

Several Windows subsystems use IOCPs for asynchronous operations including Winsock, Named Pipes, and file I/O. In addition to supporting Windows subsystems, IOCPs enable programmer-developed asynchronous operations. This is the capability we're interested in here. Being able to seamlessly integrate OS-supplied asynchronous features with our own code in a single architecture goes a long way to reducing the overall complexity of our app. The basic queue is shown in Listing One.

Listing One: The queue.
class Queue
		    HANDLE _hiocp;
		    void Associate(HANDLE hAssoc);
		    void Run();
		    void Quit();
		    void Post(IAsyncState *pState);
		    void Fetch(IAsyncState **ppState, DWORD wait);

Work is posted to a queue on one thread and another thread can fetch the work item in FIFO order. When there is no work in the queue, the fetching thread simply waits, either indefinitely or for a chosen interval. The latter might be useful, for example, for raising a notification indicating that a thread is idle, which could be fed into a thread scaling mechanism.

Thread Pool

Generally, a thread pool with a modest number of workers ends up being the most efficient. See some of the earlier link references for why this is so. As a guideline, we usually don't want to stray too far from a thread count that matches the available number of cores. This is what the example thread pool does. However, the pool constructor allows the size to be set explicitly if you're inclined to experiment.

If we are expecting larger numbers of blocking operations (for example, writing the results to a database), we could consider some form of scaling for the thread pool. This might be necessary if you end up dealing with some synchronous library code.

The example app has been built, intentionally, to run on Windows XP. If you are targeting Windows 7 or later you could save some coding by using the built-in system thread pool, which is a considerable improvement over the version supplied in XP.

Once the pool is set up and fed, the worker thread function simply extracts items from the input queue and dispatches them via a generic mechanism, which I'll cover in a moment.

Thread:GetInput and Thread::GetOutput can be used anywhere to obtain pointers to these queues.

Windows supplies a top-level structured exception handler (SEH) for each thread. If the top-level handler is invoked, that usually ends the whole program. Ideally, any runtime errors should be captured and dealt with in the application code and an error escaping into the thread function should be non-continuable. For this example (Listing Two), I add just enough support to make an attempt to pass some diagnostic data back to the main thread, which responds by shutting down the pool.

Listing Two: Operations with exception handling.
static unsigned _stdcall Function(void* param)
	StructuredExceptionTranslator seh;

	try {
		Pool* pPool = reinterpret_cast<Pool*>(param);

 		IO::Queue *pInput = pPool->GetInput();
		IO::Queue *pOutput = pPool->GetOutput();

		for(;;) {
			IO::IAsyncState* pState = NULL;	
			pInput->Fetch(&pState, INFINITE);

			if(!pState) break;
				unsigned ret = pState->Dispatch();

			if(ret == IO::Queue::ERROR_YIELD) 
	} catch(StructuredException& e) {
		IO::Queue* pOutput = Thread::GetOutput();		
		std::string msg = "SEH Error";				
		AsyncError *pErr = new AsyncError(e, msg);

		//TODO:  copy exception record


	} catch(Win32Error& e) {

		IO::Queue* pOutput = Thread::GetOutput();
		std::string msg = e;
		AsyncError *pErr = new AsyncError(e, msg);

	} catch(std::exception& e) {

		IO::Queue* pOutput = Thread::GetOutput();
		std::string msg = e.what();
		AsyncError *pErr = new AsyncError(0xFFFFFFFF, msg);

	} catch(...) {

		IO::Queue* pOutput = Thread::GetOutput();
		std::string msg = "Unexpected error";
		AsyncError *pErr = new AsyncError(0xFFFFFFFF, msg);
	return 0;


Related Reading

More Insights

Currently we allow the following HTML tags in comments:

Single tags

These tags can be used alone and don't need an ending tag.

<br> Defines a single line break

<hr> Defines a horizontal line

Matching tags

These require an ending tag - e.g. <i>italic text</i>

<a> Defines an anchor

<b> Defines bold text

<big> Defines big text

<blockquote> Defines a long quotation

<caption> Defines a table caption

<cite> Defines a citation

<code> Defines computer code text

<em> Defines emphasized text

<fieldset> Defines a border around elements in a form

<h1> This is heading 1

<h2> This is heading 2

<h3> This is heading 3

<h4> This is heading 4

<h5> This is heading 5

<h6> This is heading 6

<i> Defines italic text

<p> Defines a paragraph

<pre> Defines preformatted text

<q> Defines a short quotation

<samp> Defines sample computer code text

<small> Defines small text

<span> Defines a section in a document

<s> Defines strikethrough text

<strike> Defines strikethrough text

<strong> Defines strong text

<sub> Defines subscripted text

<sup> Defines superscripted text

<u> Defines underlined text

Dr. Dobb's encourages readers to engage in spirited, healthy debate, including taking us to task. However, Dr. Dobb's moderates all comments posted to our site, and reserves the right to modify or remove any content that it determines to be derogatory, offensive, inflammatory, vulgar, irrelevant/off-topic, racist or obvious marketing or spam. Dr. Dobb's further reserves the right to disable the profile of any commenter participating in said activities.

Disqus Tips To upload an avatar photo, first complete your Disqus profile. | View the list of supported HTML tags you can use to style comments. | Please read our commenting policy.