The Eclipse Test and Performance Tools Platform

Andy is a senior software engineer for Intel Corp. He can be reached at [email protected].

Software developers suffer from a peculiar delusion—we each think that our application is the center of the universe, that the only reason people buy computers is to run our applications. Oh sure, in the abstract, we know this isn't true, but when we're working on applications, well, that's the way we think.

As you might expect, solutions have been developed to accommodate this way of thinking. Operating systems let users run numerous applications side-by-side, while letting each one blissfully pretend that it is the only program running. But users know something the applications don't—several of these programs are solving related problems. Things such as integrated development environments (IDEs) simply put these related tools together in the same place.

But this doesn't really solve the problem—it only moves it. Now we have tools running together, but for the most part, unaware of one another. What programmers really want is for all of these tools to work together and leverage each other's results and capabilities.

So we've come to a fork in the road. Down one path, developers can lock into a single vendor solution, relying on proprietary Standards for interoperability. Down the other path, they can choose open Standards for interoperability and be able to pick and choose tools designed to work with that Standard. The Eclipse Test and Performance Tools Platform (TPTP) Project is an example of the second path. The advantages that open Standards and platforms provide include more options in selecting the best tool for every job and more possibilities as to how tools can interoperate. What may not be obvious is that this path also holds advantages for tool builders.

In this article, I examine the Eclipse Test and Performance Tools Platform Project (formerly known as "Hyades"), of which I'm a team member. I also explore ways in which tool developers can take advantage of the capabilities that TPTP provides to develop interoperable distributed test and performance tools.

What Is TPTP?

TPTP is an open platform that supplies frameworks, services, and data models to enable the development of integrated testing, tracing, profiling, and monitoring tools. Just as the Eclipse workbench provides a broad and extensible starting point for tool development in general, TPTP provides a starting point specifically for developing test and performance tools.

Although TPTP includes a number of fully functional exemplary tools that have been developed on top of this platform, the primary goal of the platform is to provide a common base upon which additional commercial, open-source, or in-house test and performance tools will be built.

The platform consists of a stack of components that includes data collectors, a communications framework, data models, analyzers, and viewers (see Figure 1). It is structured in such a way that proprietary tools can plug-in at any point and take advantage of the capabilities offered by the rest of the stack.

The idea behind the TPTP project is to move the common components of test and performance software into a shared, open-source code base. Consequently, tool developers can focus their attention on features, thereby producing better products, which (as an added benefit) plug-in to a common environment, paving the way for interoperability and symbiosis between tools.

Tool vendors get the benefit of not having to develop and maintain the code infrastructure that is necessary for their product but really secondary to its actual functionality. End users get the benefit of better tools that know how to work together. Everybody wins.

Sounds great, right? But how do we get there?

Naturally, a common user experience begins with a shared user interface. Just plugging-in to the Eclipse workbench is a start toward a common look-and-feel, as well as a set of components on which to build UIs. Beyond that, the TPTP project provides a set of viewers and editors specifically geared toward test and performance analysis.

But viewers aren't useful unless they have something to view. The heart of the TPTP user experience is a common set of data models. TPTP defines four EMF-based data models:

• Test model.
• Statistical model.
• Logging model.
• Trace model.

By feeding data into these models, tools open up possibilities for interaction with analysis engines and viewers that are designed around these models.

The data models are supported by a common framework for accessing data collection and test-execution agents. TPTP provides an extensible set of classes to discover and manage these agents and the data they provide. If an agent produces data in one of the formats TPTP recognizes, a standard data loader reads the data and populates the data model. Otherwise, the agent developer can provide a custom data loader to achieve the same end.

Finally, TPTP includes a flexible framework to connect agents with the workbench. In the rest of this article, I explore the agent framework, explaining the concepts it is based on and exploring how to develop agents to plug-in to the TPTP project.

Communications

Everyone who has developed a software tool with distributed execution has had to address the problem of how to connect the remote components with the local components. Companies with multiple tools or tools with long histories have probably solved this problem several times and are quite possibly considering doing it again. No one ever seems to be happy with their remoting solution.

But the absurdity is that this communications piece generally has nothing to do with the primary functionality that the tool is intended to provide. It's just something that has to be done along the way. It makes sense to adopt an open solution.

General solutions such as the Adaptive Communication Environment (ACE) tend to be heavier than you might like. The TPTP communication framework is specifically designed to support testing and the performance analysis engine, and provides a streamlined connection to the rest of TPTP.

Figure 2 shows the basic architecture of the TPTP communications framework. TPTP provides a client library that manages communications with an Agent Controller that can be running locally or on a remote system. This Agent Controller manages access to and control of agents running on the target system. In addition, TPTP provides template code for agent development to handle boiler-plate tasks such as registering with the Agent Controller and parsing incoming commands.

The actual communications between components are managed by dynamically loaded transport layers. TPTP provides transport layers to handle socket-based communications, and SSL and HTTP solutions are under development. Additional mechanisms can be supported through custom libraries written to the TPTP transport layer interface. The TPTP architecture is designed to isolate these layers from the rest of the system so that clients and agents can be written independent of the transport layer they use to communicate and transfer data.

The Agent Controller uses a standard but extensible protocol for sending asynchronous commands back and forth between agents and clients. TPTP defines this XML-based protocol with a command envelope and basic set of interfaces and commands, but agent developers can extend the protocol by defining their own interfaces and commands.

Agents and Interfaces

In the context of TPTP, an "agent" is defined as a logical object that exposes services through the TPTP Agent Controller. In TPTP, these agents run in a separate process from the Agent Controller (though multiple agents can exist within a single process).

The Agent Controller recognizes external components through the connections they establish. When something establishes a connection with the Agent Controller through a transport layer, the Agent Controller assigns it an ID and begins seeing it as a logical object. If that object then sends the Agent Controller a registerAgent command, it is an agent.

Data collectors are an important type of agent, but the agent concept is meant to be general within TPTP. Other agents can provide services such as file transfer or system information.

Clients can request access to a specific agent by name, but to support the goal of general interoperability, clients can also locate agents by querying based on the command interfaces they support.

Interfaces and commands are the basic building blocks of the protocol TPTP uses for communications between components. Again, the problem of common terms comes in. TPTP understands an interface as a contract regarding a set of commands. Although there is no programmatic enforcement (such as compilation or linking errors), TPTP relies on the assumption that if a component says it supports an interface, then it handles all the commands in that interface. These interfaces are simply an agreement between components.

Commands in TPTP are based on XML fragments. TPTP defines a command element with attributes such as source, destination, and context that it uses for routing these commands, but a subelement within the command fragment defines command-specific information that is only read by the component receiving the command.

TPTP defines a standard set of interfaces for general concepts such as agent, collector, and event provider, but the protocol is designed to be extended using custom interfaces.

Agent Development

How can you put TPTP into your application? Listing One is an elementary piece of client code that locates an agent and sends it a command. Listing Two is code for the agent invoked in Listing One.

If it seems like these two samples don't contain any meaningful code, that's by design. Remember that the goal of TPTP is to provide all of the infrastructure code, so that tool developers can focus on adding their particular functionality. Because this sample doesn't have any particular purpose other than demonstrating how to use TPTP to connect a client to an agent, all of the work is done in the libraries provided by TPTP.

The client code interacts with an Agent Controller to obtain an instance of the agent it wants to use. It uses the TPTP client library to connect to an Agent Controller that may be running on the same machine as the client or on a remote machine. From this point on, the client will not make any calls that are dependent on the location of the target system. That will be abstracted by TPTP.

Next the client creates an object to act as a delegate for the agent it wants to control and passes that object to the local delegate of the Agent Controller. Behind the scenes, the TPTP client library interacts with the real Agent Controller to request that it provide an instance of this agent. The Agent Controller checks to see if there is already an instance of this agent running and available. If not, it launches the agent and manages its subsequent lifecycle. In either case, the client is put in contact with a running agent. (TPTP also provides ways to handle special cases such as when an agent must be launched in conjunction with the application it is monitoring.)

Alternatively, if the client did not have a specific agent in mind, it could have queried the Agent Controller for a list of agents that supported a given interface, then obtained further metadata describing each of these agents before deciding which agent to request access to.

The agent itself runs as a standalone process. The main routine for the process simply creates an instance of the agent object, which derives from the BaseCollectorImpl class provided by TPTP, registers the agent, and waits for the agent to be terminated.

The registerAgent method manages all of the common work of locating the Agent Controller, establishing communications, starting a thread to listen for incoming messages, and sending the registration command. The base class implementation further handles the work of parsing incoming commands and calling the agent's implementation of these commands.

For standard interfaces, such as the collector and agent interfaces, the agent itself only needs to provide an implementation for those commands for which it wants to override the default implementation. If the agent wants to provide a custom interface, it also needs to override the processCommand method and interpret the command block for its custom commands.

In this example, I have presented both the client and the agent implemented in C++. Where does Eclipse comes into the picture? For the agent itself, Eclipse really stays out of the picture. The agent always communicates through the Agent Controller and never needs to be specifically aware that it is talking to an Eclipse-based client on the other end of the line. TPTP provides libraries to enable agent development in C, C++, and Java.

On the client side, TPTP provides a C++ version of the client library that isn't integrated with the Eclipse workbench to provide a low entry point for tools that are currently implemented in C++ (though it is our hope that such tools will eventually proceed to integration with the Eclipse workbench and the rest of the TPTP client environment). TPTP also offers a Java-based client library that is fully integrated with the Eclipse workbench and the TPTP client framework. The C++ and Java client libraries are very similar, and all of the concepts I've described here apply to both.

Conclusion

If, after you've evaluated TPTP, you think it's a good solution but doesn't quite do what you need, get in touch with us. We've designed TPTP to be flexible and extensible, but there's always room for improvement. TPTP is an open-source platform, and the TPTP project team is serious about listening to input, accepting new people into the conversation, and keeping our tools useful to the testing and performance tools community.

DDJ

Listing One

#include "tptp/client/INode.h"
#include "tptp/client/NodeFactory.h"
#include "tptp/client/Agent.h"
#include "tptp/client/Collector.h"
#include "tptp/client/IDataProcessor.h"

using namespace TPTP::Client;

class MyDataProcessor: public IDataProcessor
{
public:
   MyDataProcessor() {}
   ~MyDataProcessor() {}

   // Handle the data coming
   virtual void incomingData(char buffer[], int length, DIME_HEADER_PTR_T dimeHeader) 
   {
      // TODO: Do something with the data
   }

   virtual void invalidDataType(char data[], int length) {}
   virtual void waitingForData() {}
};

int main(int argc, char* argv[])
{
   // Create a Node that represents the Target Machine
   INode* SampleNode = NodeFactory::createNode("localhost");

   // Get the Agent Controller on the Node
   AgentController* agentCtrlr = SampleNode->connect(10002);

   // Request an instance of our agent
   Collector* myCollector = new Collector("com.ddj.tptp.sample.myCollector");
   agentCtrlr->getAgent(myCollector, TPTP_CONTROLLER_ACCESS);

   // Establish Data Path and Data Listener
   MyDataProcessor* dataProcessor = new MyDataProcessor();
   int dataConnectionID = myCollector->addDataListener(dataProcessor);

   // Starts the collector
   myCollector->run();
   Sleep( 5000 );

   // Stop the collector
   myCollector->stop();

   return 0;
}

#include "tptp/agents/BaseCollectorImpl.h"

class MyCollector : public BaseCollectorImpl
{
public:
   MyCollector() {}
   ~MyCollector() {}
virtual int run(CmdBlock* cmd)
   {
      char data[] = "<mySampleData>        \
                        <mood>happy</mood> \
                     </mySampleData>";

      // TODO: Do whatever it is that starts data collection

      // Send the replay command
      int destID = cmd->getDestID();
      int sourceID = cmd->getSourceID();
      int contextID = cmd->getContextID();

      char   commandFormat[] = "<Cmd src=\"%ld\" dest=\"%ld\ ""
        "" ctxt=\"%ld\"><agentRunning ""
        "" iid=\"org.eclipse.tptp.Collector\"></agentRunning></Cmd>";
      char   command[1024];
      sprintf( command, commandFormat, destID, sourceID, contextID );

      sendCommand( command );

      // Simulated activity for sample purposes only
      Sleep( 500 );

      sendData(sourceID, data, sizeof(data) );

      return 0;
   }

   virtual int stop(CmdBlock* cmd)
   {
      // TODO: Do whatever it is that starts data collection

      // Send the replay command
      int destID = cmd->getDestID();
      int sourceID = cmd->getSourceID();
      int contextID = cmd->getContextID();

      char   commandFormat[] = "<Cmd src=\"%ld\" dest=\"%ld\ ""
        "" ctxt=\"%ld\"><agentStopped ""
        "" iid=\"org.eclipse.tptp.Collector\"></agentStopped></Cmd>";
      char   command[1024];
      sprintf( command, commandFormat, destID, sourceID, contextID );

      sendCommand( command );

      return 0;
   }

   // This sample doesn't expect to receive data
   virtual int receiveData(int sourceID, char buffer[], int bytesRead, 
       DIME_HEADER_PTR_T dimeHeader) { return 0; }
};

int main(int argc, char* argv[])
{
   MyCollector* collector = new MyCollector();
   collector->registerAgent("com.ddj.tptp.sample.myCollector", "MyCollector");
   collector->waitForTermination();
   return 0;
}