Dr. Dobb's is part of the Informa Tech Division of Informa PLC

This site is operated by a business or businesses owned by Informa PLC and all copyright resides with them. Informa PLC's registered office is 5 Howick Place, London SW1P 1WG. Registered in England and Wales. Number 8860726.


Channels ▼
RSS

Web Development

Scaling SOA with Distributed Computing


How It Works

To adapt their analytics to run on a grid, the customer used the Digipede Framework SDK. They had already made the decision to use .NET for the analytic service, so they needed a solution that would work natively in that environment. Because the developers had already encapsulated their analytics in .NET classes for performing the calculations, the adaptation to a .NET-based system was quick. By encapsulating these classes in separate .NET assemblies, they were able to submit analytics requests for execution on the Digipede Network.

The service itself is built using ASP.NET and hosted in IIS 6, and manages session information in the system using a SQL Server database. When a new analytics request comes into the service, a token is generated and returned to the client. This token is the key to any analysis-specific information for the life of the computation.

Depending on the duration of the calculation necessary to fulfill the request and the parallelizability of the analysis, the service has two options—send the analysis to the grid as a whole, or decompose it into separate, parallel pieces and send all of those to the grid.

Again, depending on the possible level of parallelization, the service instantiates one or more objects to perform the analysis. These objects act as data holders for initiating the computation remotely. Collectively, these objects make up a job on the grid. The service submits this job to the Digipede Server and receives back a job-specific ID uniquely identifying this job on the grid. That ID is stored in the session database indexed by the token for later reference.

Meanwhile, each of the analysis objects is serialized automatically and distributed to an appropriate machine on the grid. The Digipede Agent software, running on each of those machines, determines if the machine can work on the job (based on its hardware, software, and availability). If the objects require any assemblies that are not already on the computer (for example, if a DLL has been updated recently), the Agent retrieves the necessary files before working on the job. When ready, it deserializes an object, executes the analysis, then serializes it again (this time, with results in tow) and returns them to the Digipede Server. The server itself monitors the status and health of each agent and reassigns any work that does not complete successfully.

Meanwhile, the client can query the web service to check the status of the job. To accomplish this, a client calls the analysis service using the same token received in response to its initial call. The service uses that token to lookup the job ID in the session database, then queries the Digipede Server for the status and results. The status of the job and each individual task can be returned, along with the result objects. In the event that the job is not complete, the system can provide an estimate of completion time.

Conclusion

As our customer realized, scalability is an important consideration in designing and building a service-oriented architecture. Fortunately, they grasped this at the right time—in the design phase. Trying to retrofit scalability onto an existing system can cost more in time and effort than the initial implementation. The risks are large: Exposing an SOA to your organization that will not be able to handle the load will endanger the success of the entire project.

To meet our customer's scalability needs, designing their SOA to run on a grid was the ideal solution. Their SOA has ample capacity and a simple plan for growth—they can now handle nearly 50 times the usage that the single server could provide, and adding commodity hardware can further expand capacity. By utilizing parallelization, they were also able to dramatically speed up their longer transactions (analysis that formerly took over an hour now completes in just a few minutes). Best of all, this grid will underpin all of their SOA efforts going forward: By deploying a common infrastructure, they will save significant development time and IT effort.


Related Reading


More Insights






Currently we allow the following HTML tags in comments:

Single tags

These tags can be used alone and don't need an ending tag.

<br> Defines a single line break

<hr> Defines a horizontal line

Matching tags

These require an ending tag - e.g. <i>italic text</i>

<a> Defines an anchor

<b> Defines bold text

<big> Defines big text

<blockquote> Defines a long quotation

<caption> Defines a table caption

<cite> Defines a citation

<code> Defines computer code text

<em> Defines emphasized text

<fieldset> Defines a border around elements in a form

<h1> This is heading 1

<h2> This is heading 2

<h3> This is heading 3

<h4> This is heading 4

<h5> This is heading 5

<h6> This is heading 6

<i> Defines italic text

<p> Defines a paragraph

<pre> Defines preformatted text

<q> Defines a short quotation

<samp> Defines sample computer code text

<small> Defines small text

<span> Defines a section in a document

<s> Defines strikethrough text

<strike> Defines strikethrough text

<strong> Defines strong text

<sub> Defines subscripted text

<sup> Defines superscripted text

<u> Defines underlined text

Dr. Dobb's encourages readers to engage in spirited, healthy debate, including taking us to task. However, Dr. Dobb's moderates all comments posted to our site, and reserves the right to modify or remove any content that it determines to be derogatory, offensive, inflammatory, vulgar, irrelevant/off-topic, racist or obvious marketing or spam. Dr. Dobb's further reserves the right to disable the profile of any commenter participating in said activities.

 
Disqus Tips To upload an avatar photo, first complete your Disqus profile. | View the list of supported HTML tags you can use to style comments. | Please read our commenting policy.