Building a Testpoint Framework

In general, my design goals were:

• Testpoints had to be set to active/nonactive without changing the executable.

• When testpoints were not active, there should be no performance penalty.

• The framework should be robust enough that new testpoints could be easily added, and new functional requirements should be met with ease.

• Using/creating testpoints should be straightforward.

• The code should be portable to other operating systems. (I originally wrote the code for Windows using Visual C++ 6.0 and simple COM objects, but it easily ported to UNIX. The VideoClick server, for instance, runs on both Windows and UNIX. Luckily, there are implementations of the COM library for UNIX (such as that from Mainsoft), but we wanted to keep it simple, so I wrote a UNIX implementation of the COM library for INPROC -- an in-process COM server that is executed in the same process as the client.

Testpoints Framework Architecture

The testpoint framework I designed consists of three objects: the testpoint (TP), testpoint handler (TPH), and testpoint manager (TPM). When using the framework, you supply specific testpoints, while the TPH and TPM are unchanged. They are part of the TestPoints.DLL (available electronically).

The framework design is based on the Factory and Policy design patterns. The TPM is a factory: When a piece of code wants to set a TP, the TP object asks the TPM if this TP should be activated ("Should TP1 be activated?"). If the TPM's answer is "yes," the TPM supplies a new object (hence, the factory) that handles the data coming from this TP. This new object is the TPH.

From that point on, the TP works directly with the TPH. Data collected by the TP is passed to the TPH (using an interface agreed upon between the TP and the TPH), and processed by the TPH. Once a specific TPH is instantiated for a TP, the TP does not care about the internals of the handling -- it simply writes data into the interface of the TPH.

Although there is only one TPM in a system, there can be many TPHs -- each with an exposed interface to a TP (several TPHs can expose the same interface to the TP, thus allowing TPs to choose which TPH to use). There are any number of TPs, each with a unique ID. Each TP uses one of the TPHs.

Figure 3 is a UML diagram that shows the instantiation sequence of a TP. Once the TP object holds a TPH, the TPM is no longer needed and all communication is done directly to the TPH.

How does the TPM know which TPH is used for a given TP? Again, each TP has a unique ID. I use a Global Unique Identifier (GUID) to name each TP. The information for all testpoints is stored in a database accessed by the TPM. As usual with Windows development, this is the Registry (this does not contradict the portability required in the design goals; Registry implementations exist for other platforms as well). Figure 4 (from the Registry editor) shows that three testpoints are known in this particular machine (the testpoints in the actual system running may differ). Each testpoint has three values required by the TPM: Name, IsActive, and a CLSID (or ProgId) that identifies the TPH used for this handler. All other fields are used by the TPH. For example, the File value is used by the TPH (whose ProgId is Vsoft.DataSink.Handler) to know the filename where data shall be written.

Implementing the Framework

Figure 5 illustrates the COM objects the framework uses. The testpoint itself is not a COM object -- it is merely a client (although it can be a COM object).

The Testpoint Manager (TPM)

The TPM is the object consulted by a TP for the activity state and supplies the handler for this TP. This structure isolates the TP from where the configuration data for the TP is stored (for all the TP cares, it can be on another computer), and from which handler is instantiated.

For example, say you have a stream of data running in the system and passed from stage to stage using the interface shown as pseudocode in Listing One. You want to sample the data that passes from the data source to the data filter (as in Figure 2), so that, as Figure 6 illustrates, you implement a handler that exposes the IDataSink interface and connects it using a tee object (the tee object simply sends everything written to it to the two outputs).

The TP handler works great and writes all the data to the dump file. Now you want to see how big the buffers that are passed to the data filter are. All you have to do is implement a new handler (MyHistogramHandler, for example) that exposes the same IDataSink. You change the ProgId of the handler in the registry for this specific TP (other testpoints still want to use the old handler), and you are done. The application containing the testpoint need not be recompiled, and only the TPM (through the Registry) knows which handler is now in effect.

The Testpoint Handler (TPH)

The TPH acts upon the data supplied to it via the interface it exposes to the TP -- for instance, you can have a handler that exposes IDataSink and writes the data to disk (or network), one that exposes IDataSink and performs statistical calculations on the data and writes only the statistics, another that exposes IsomeInterface and performs a validity check on the interface semantics, and so on.

How does the TPH for a given TP know the value of parameters needed for its work? For example, the datasink handler needs a file name to dump the data. This is why every TPH exposes the IInitTestPoint interface having a single method: Set(char* registry_path_to_TP). When the TPM instantiates the TPH, it calls IInitTestPoint::Set() with the path to the TP's data. The TPH then reads any data it wishes.

The TPH should normally be implemented very efficiently; after all, you want the TP to have minimal impact when used.

The Testpoint (TP)

The testpoint itself is not necessarily a COM object -- it simply checks with TPM to see if it should be activated, and if so, connects to the supplied handler. From then on, the TPM is out of the picture and data is written directly to the handler.

Listing Two, sample pseudocode for the IDataSink in Figure 6, ensures that if the testpoint is not active or some error occurs during the initialization of the handler, the connection transparently proceeds directly to the next stage (the dataFilter). This underscores two important points: Testpoints should not disturb normal system work while not active; and, if a testpoint initialization fails, execution should proceed as usual.

Limitations of the Framework

Since the testpoints are used in a multithreaded environment, the TP framework must be thread safe. This is not a real limitation; just be careful what you do in the TPH and TPM.

Part of VideoClick is a server that should be up as much time as possible. Consequently, I want to activate/deactivate testpoints while the server is online and cause minimal disturbance. For example, the server has several concurrent video streams opened. If I turn on the TP for video-stream output, all streams from that point on are dumped into files -- even if I need to examine a single stream. Moreover, a TP activity is checked only when it is initialized (that is, when the connection between the data source and data filter is done). This implies that I cannot start dumping data from an already open object, nor stop dumping data before the object is destroyed. (Think of objects that live for days or weeks.) System administrators who wish to turn testpoints on/off should avoid altering the Registry. Even with the aforementioned limitations, the framework is currently deployed successfully in VideoClick.

The Revised Framework

Overcoming the limitations just described requires more cooperation on the TP's side and changes in the TPM. Consequently, I found it necessary to make changes to my original framework by adding features for:

• Identifying an instance of a unique TP (multithreading).
• Dynamic activation of a TP.

• Control of a TP on its activation.

• Control panel to see which testpoints exist in the system and which are active; controlling testpoint parameters and activation.

At first glance, it is enough to require the TP to expose an interface for turning it on/off, but then the TPM would have to hold a reference to the TP, which is strongly undesirable (even weak references won't help here). Also, how can the TPM know that the TP received the request to turn itself on? It is not enough that the TP sets a flag (remember the multithreading?).

The chosen solution involves polling. The TP polls the TPM whenever it wishes for the requested activity state and informs the TPM when the state is changed. Consequently, it is legal to have the TPM request from a TP to start dumping data, and for the TP to refuse to do so (since the containing object is in the middle of an urgent task); see Figure 7.

The Testpoint Control Panel

It is possible that users want to view the state of testpoints when no instances of the TP are created, so the TPM has to be available to the control panel at all times. It can still be an INPROC, but adding the next requirement leads to an OUTPROC server -- the control panel may connect to a TPM that resides on another machine. (Think how great it is that you can turn on/off testpoints on a machine at other sites.)

Converting the TPM to OUTPROC (either local or remote server) introduces new problems. Again, the TPM supplies an instance of the TPH to the TP. This TPH must be in the same process as the TP to keep performance high. If you use the standard mechanisms ("standard marshaling"), the TPH instance runs in the TPM's process, and every call from the TP to the TPH will have to be marshaled and cross between process boundaries -- something that's utterly unacceptable. This problem is solved using custom marshalling.

Conclusion

Testpoints were used in hardware systems long before computers existed. I found that applying the same principles to our software enhanced both testability and maintainability.

Acknowledgment

Thanks to Shahar Glixman for many ideas and help in the design.

References

Gamma, E. et al. Design Patterns. Addison-Wesley, 1995.

The COM Specification: http://www .microsoft.com/com/resources/comdocs.asp.

DDJ

Listing One

interface IDataSink
{
  Write(void* buff, UINT buffSize);
}

Back to Article

Listing Two

Connect_Source_And_Filter:

pDownStream = pDataFilter;
pTpm = GetTpm();
if(pTpm->IsActive( UUID_TP1))
{
    pHandler = pTpm->GetHandler(UUID_TP1);
    if(NULL != pHandler)
        pDownStream = new Tee( pDataFilter, pHandler);
}
pSource->Connect(pDownStream);

Back to Article