How It Works
To adapt their analytics to run on a grid, the customer used the Digipede Framework SDK. They had already made the decision to use .NET for the analytic service, so they needed a solution that would work natively in that environment. Because the developers had already encapsulated their analytics in .NET classes for performing the calculations, the adaptation to a .NET-based system was quick. By encapsulating these classes in separate .NET assemblies, they were able to submit analytics requests for execution on the Digipede Network.
The service itself is built using ASP.NET and hosted in IIS 6, and manages session information in the system using a SQL Server database. When a new analytics request comes into the service, a token is generated and returned to the client. This token is the key to any analysis-specific information for the life of the computation.
Depending on the duration of the calculation necessary to fulfill the request and the parallelizability of the analysis, the service has two optionssend the analysis to the grid as a whole, or decompose it into separate, parallel pieces and send all of those to the grid.
Again, depending on the possible level of parallelization, the service instantiates one or more objects to perform the analysis. These objects act as data holders for initiating the computation remotely. Collectively, these objects make up a job on the grid. The service submits this job to the Digipede Server and receives back a job-specific ID uniquely identifying this job on the grid. That ID is stored in the session database indexed by the token for later reference.
Meanwhile, each of the analysis objects is serialized automatically and distributed to an appropriate machine on the grid. The Digipede Agent software, running on each of those machines, determines if the machine can work on the job (based on its hardware, software, and availability). If the objects require any assemblies that are not already on the computer (for example, if a DLL has been updated recently), the Agent retrieves the necessary files before working on the job. When ready, it deserializes an object, executes the analysis, then serializes it again (this time, with results in tow) and returns them to the Digipede Server. The server itself monitors the status and health of each agent and reassigns any work that does not complete successfully.
Meanwhile, the client can query the web service to check the status of the job. To accomplish this, a client calls the analysis service using the same token received in response to its initial call. The service uses that token to lookup the job ID in the session database, then queries the Digipede Server for the status and results. The status of the job and each individual task can be returned, along with the result objects. In the event that the job is not complete, the system can provide an estimate of completion time.
Conclusion
As our customer realized, scalability is an important consideration in designing and building a service-oriented architecture. Fortunately, they grasped this at the right timein the design phase. Trying to retrofit scalability onto an existing system can cost more in time and effort than the initial implementation. The risks are large: Exposing an SOA to your organization that will not be able to handle the load will endanger the success of the entire project.
To meet our customer's scalability needs, designing their SOA to run on a grid was the ideal solution. Their SOA has ample capacity and a simple plan for growththey can now handle nearly 50 times the usage that the single server could provide, and adding commodity hardware can further expand capacity. By utilizing parallelization, they were also able to dramatically speed up their longer transactions (analysis that formerly took over an hour now completes in just a few minutes). Best of all, this grid will underpin all of their SOA efforts going forward: By deploying a common infrastructure, they will save significant development time and IT effort.