Channels ▼

Web Development

The Media Grid

November, 2005: The Media Grid

Aaron is Director of the open Standards group through which the Media Grid is designed and developed. He can be contacted at

The Media Grid is a digital media network infrastructure and software-development platform based on new and emerging distributed computational grid technology. The Media Grid ( is designed as an on-demand public computing utility that software programs and web sites can access for digital content delivery (graphics, video, animations, movies, music, games, and so forth), storage, and media processing services (such as data visualization and simulation, medical image sharpening and enhancement, motion picture scene rendering, special effects, media transformations and compositing, and other digital media manipulation capabilities). As an open platform that provides digital media delivery, storage, and processing services, the Media Grid's foundation rests on Internet, web, and grid standards. By combining relevant standards from these fields with new and unique capabilities, the Media Grid provides a novel software-development platform designed specifically for networked applications that produce and consume large quantities of digital media.

As an open and extensible platform, the Media Grid enables a wide range of applications not possible with the traditional Internet alone, including: on-demand digital cinema and interactive movies; distributed film and movie rendering; truly immersive multiplayer games and virtual reality; real-time visualization of complex data (weather, medical, engineering, and so forth); telepresence and telemedicine (remote surgery, medical imaging, drug design, and the like); telecommunications (such as video conferencing, voice calls, video phones, and shared collaborative environments); vehicle and aircraft design and simulation; computational science applications (computational biology, chemistry, physics, astronomy, mathematics, and so forth); biometric security such as real-time face, voice, and body recognition; and similar high-performance media applications. By giving software developers the ability to easily access a theoretically unlimited pool of computing resources optimized for digital media, we anticipate that the Media Grid will enable these types of applications almost immediately, while unlocking the potential for a new class of applications that we can't conceive of today.

The Media Grid has been under active development for several years and is now reaching critical mass thanks to partner organizations and foundation technologies such as the Globus Toolkit. In this article, I examine how some of the system's key capabilities are supported by the Globus Toolkit 4.0 (GT4). Along the way, I explore several significant GT4 features, including its support for standard web services.

Public Utility for Digital Media

At a conceptual level, the Media Grid is modeled after an improved national power grid, with added security and stability features that eliminate downtime and blackouts. As with the U.S. national power grid, which standardizes the production and consumption of power in the United States, the Media Grid is built with the intention of establishing a new generation of technology standards that enable computer applications to "plug-in" to digital media services over the public Internet. Applications that only need to consume media content, store or archive media files, or access media processing services can do so at a fair and standardized price (or for free in certain cases), which we anticipate will be greatly reduced compared to the cost of today's proprietary digital content delivery systems, while the owners of computers that host and deliver media or provide media processing services receive compensation for their contribution to the Media Grid.

Desktop computers, workstations, laptops, handhelds, PDAs, mobile phones, game consoles, and kiosks are just a few of the many types of computing devices that can tap into the Media Grid (see Figure 1). Devices that have enough computational power and fast enough network connections can become nodes on the Media Grid, meaning they can store, deliver, or process media for other users in exchange for credit; in this way, some users can earn enough credit to pay for all of the premium (for-fee) content and services they wish to consume. In contrast, less powerful devices may simply consume media and services provided by the Media Grid as shown in Figure 2. Regardless, any device that runs Media Grid software can be spontaneously networked together over the traditional Internet to form ad-hoc grids, or swarms, that exchange media and media processing services. Grids can also be assembled from specific devices and administrated much like a traditional managed network.

By providing a global computing fabric for digital media, the Media Grid provides a public utility infrastructure that should have great appeal to the industry. Rather than building and maintaining proprietary solutions, companies and individuals will be able to utilize the Media Grid at a fraction of the cost and with minimal effort as compared to custom in-house solutions. A company such as Apple Computer, for example, stands to significantly reduce the time, effort, and cost of hosting and managing its wildly popular iTunes music service by offloading some portion of that service to the Media Grid. Similarly, television and movie production companies might enjoy significant economic advantages by using the Media Grid as a massive on-demand rendering farm rather than continue to invest time and money in their own in-house rendering farms. Scientists and researchers, meanwhile, may find the Media Grid a fast and convenient alternative to more esoteric or complex data visualization and simulation systems. At the other end of the spectrum, individual software application developers and web developers will be able to use the Media Grid for their own work as well. Web developers, for instance, can increase the performance, reliability, and scalability of their web sites while simultaneously reducing hosting fees simply by hosting images, movies, and music files on the Media Grid instead of using traditional ISPs or web servers.

The Media Grid isn't intended to replace or circumvent existing grids, clusters, or rendering farms—it's designed to provide uniform and simplified access to a wide range of such systems. Like the Web before it, which shields users and developers from the complexity of the Internet, the Media Grid provides a unified view to an otherwise complex system. In the same way that the Web simplifies Internet development and provides a standard browser interface for text-oriented information and basic media content, the Media Grid aims to make it easy for developers to access computational resources provided by existing technology vendors such as Oracle, IBM, Sun Microsystems, Hewlett-Packard, Microsoft, and others as shown in Figure 1. By making digital media content and processing power available through unified APIs, grid services, and web services, the Media Grid provides a public computing infrastructure that both developers and owners of high-performance computer systems can benefit from.

Globus Toolkit 4

When I first hinted about the Media Grid to DDJ readers (see "Creating Java Grid Services," DDJ, September 2003) Version 3 of the Globus Toolkit had just been released. What a difference two years makes. Both the Globus Toolkit and the organization behind it have grown by leaps and bounds. Globus Toolkit 4.0 (GT4) is now available and features a range of new capabilities including support for established and emerging web-services standards, while the main organizational structure behind Globus has been formalized as the "Globus Alliance."

In the words of the alliance, the Globus Toolkit is:

A fundamental enabling technology for the "Grid," letting people share computing power, databases, and other tools securely online across corporate, institutional, and geographic boundaries without sacrificing local autonomy. The toolkit includes software services and libraries for resource monitoring, discovery, and management, plus security and file management. The toolkit includes software for security, information infrastructure, resource management, data management, communication, fault detection, and portability. It is packaged as a set of components that can be used either independently or together to develop applications. The Globus Toolkit was conceived to remove obstacles that prevent seamless collaboration. Its core services, interfaces and protocols allow users to access remote resources as if they were located within their own machine room while simultaneously preserving local control over who can use resources and when.

In short, the Globus Toolkit is open-source software that enables computational grids and distributed applications that run across grids. In terms of the Media Grid, GT4's most significant capability is its support for standard web services. GT4 consists of a number of software components that directly support web services and several original components that predate the system's general focus on web services (see Table 1). These so-called "nonWS" or "preWS" components may eventually be replaced with WS versions in the future, yet remain a vital part of the toolkit in any case.

GT4 supports a number of important web-services standards and technologies, as Table 2 illustrates. Significantly, GT4 also supports the Web Services Resource Framework (WSRF) specification from OASIS (, which defines an open standard for implementing stateful resources that are accessible to web services. The ability to provide access to resources that maintain state across service invocations gives GT4 the ability to support distributed computing capabilities via web services. In other words, WSRF is the infrastructure that makes stateful grid services possible in GT4 (as you may recall, a grid service is merely a special-purpose web service designed to operate in a grid environment).

Swarming Connections and Quality of Service

The Media Grid combines swarming network connections with Quality of Service (QoS) levels to deliver content over the public Internet at high speeds and without interruption. Preliminary research indicates that swarming alone can scale to deliver content under loads several orders of magnitude beyond what is possible with traditional client-server architectures while enabling servers to gracefully cope with flash crowds (aka the "Slashdot effect"). By combining swarming with QoS, the Media Grid overcomes the inherent limitations of client-server technology that are especially detrimental to servers running on low-end devices and over low-capacity network connections. Specifically, limited file storage space and sparse processing resources combined with network bottlenecks render low-end computers useless for delivering large quantities of content over the Internet. This is especially true for consumer devices in the home because many ISPs restrict upload bandwidth to prevent home systems from acting as servers. Asymmetrical network connections such as this are a technical barrier that client-server architectures appear unable to overcome.

By mobilizing devices into dynamic and spontaneously configured ad-hoc grids, however, swarming overcomes these limitations, as Figure 3 illustrates. Here, we see a small swarm of four consumer devices working together to deliver pieces of a DVD video in parallel to a single client over the public Internet. Media Grid software running on the client receives and assembles the pieces of the movie as they are delivered from the swarm so that the entire video can be played in real time as it is downloaded. QoS mechanisms ensure that if a device in the swarm slows down or disconnects from the network, a new device is dynamically added to the swarm so that transmission continues at full speed without interruption, while standby modes support instant failover and context switching. Using swarming alone, early tests of the Media Grid confirmed the system is capable of delivering full-length DVD movies (encoded in DivX format) to home users fast enough to allow such content to be viewed in real time within moments of initiating the download. QoS measures currently slated for implementation are expected to further enhance performance and increase the system's scalability.

GT4 provides a number of data-management tools that make it possible for the Media Grid to locate, transfer, and manage large quantities of distributed data. GridFTP, for example, is a reliable, secure, and high-performance transport mechanism that is, in essence, an enhanced FTP server optimized for both memory-to-memory and disk-to-disk data transfer between grid nodes. GT4's GridFTP server supports separate front-end (client) and back-end (data) processes and a "striped" configuration in which a single client can connect to multiple back-end data sources. Striped data transfers are distributed across all data nodes that, when combined with GridFTP's support for parallel and partial data transfers, facilitate Media Grid swarming.

Although GridFTP is not a web-services component itself, GT4 does provide a WS component called "Reliable File Transfer" (RFT) that can be used to manage multiple file transfers over GridFTP. In addition to exposing high-performance data-transfer capabilities through a web-services interface, RFT also makes data transfers more reliable. RFT, unlike GridFTP, does not require the client to maintain an open socket connection for the duration of the transfer (an especially important feature for occasionally connected client devices such as mobile and wireless devices). RFT also maintains the state of data transfers in storage so that client- or data-source failures are easier to recover from.

Location-Independent Content and Services

Applications that utilize the Media Grid can access content and services by formal name or metadata (keywords, descriptions, and so forth). By eliminating the need for location-based access mechanisms, such as the URL, the Media Grid is designed to eliminate "file not found" or "service unavailable" situations. Location-independent capabilities will be supported through an implementation of "Universal Cache" technology. Universal Cache is necessary because today's networked software applications typically maintain a private local cache that is not shared with other applications. This produces redundant storage and network transmission for each cached resource.

Universal Cache solves this problem by providing a method and apparatus for a shared local cache that enables applications to access resources regardless of where they are actually located.

GT4's Replica Location Service (RLS) provides a reliable and scalable foundation on top of which the Media Grid's Universal Cache functionality is implemented. RLS uses a distributed registry to store the location of file and dataset copies, or replicas, available across any number of grid nodes. When a user or software application places a digital media file on the Media Grid, for example, it can be automatically registered with RLS. Every time the file is downloaded by a user (that is, replicated or copied) RLS can be updated with that new location information. By tracking file ingress and data migrations in this way, the Media Grid hides low-level location details from software developers and instead provides them with higher level abstractions (such as names and metadata) to describe media files that are ultimately resolved by RLS.

Grid Gateways: Virtual Organizations

The Media Grid gateway system is similar, in concept, to the CGI mechanism introduced by the World Wide Web. Whereas the Web's CGI mechanism is a standard for interfacing external applications with information servers, such as web and HTTP servers, the Media Grid gateway mechanism defines a standard for interfacing Media Grid clients and middleware with back-end grids, clusters, render farms, scientific workstations, and similar high-performance computing systems. As Figure 4 illustrates, Media Grid gateways also enable resource sharing between grids residing across organizational boundaries.

By defining a uniform gateway interface between Media Grid software and back-end systems, the Media Grid can be extended to support any form of third-party grid, cluster, render farm, or other computational systems. This open and extensible architecture does not come without risks, however, which are expressed primarily in the form of security risks. Companies, universities, and even individuals are often loath to expose their computer systems to outsiders—it's hard enough to maintain the integrity of a high-performance computer system that is inside an organization, let alone one that is available to the outside world over the public Internet.

To address these understandable concerns, the Media Grid is being developed with high-grade security as a nonnegotiable system requirement. Indeed, the Media Grid is designed with mission-critical applications, such as medical and military applications, in mind. To this end, GT4 provides a secure infrastructure that is specifically designed to allow resource sharing between multiorganizational virtual organizations (VO). Resource sharing among VOs goes far beyond merely transferring files between entities, however, and includes direct access to computer resources such as processors, hard drives, software applications, datasets, and other resources that are typically off limits to the outside world. GT4 provides the tools necessary to establish and enforce the rules and policies involved with multiorganizational resource sharing between VOs, which in turn provides the substrate on top of which the Media Grid's gateway mechanism is built.

Tapping into the Media Grid

Despite numerous improvements that have been made to the general installation process and related documentation compared to the previous version, GT4 is by no means simple to install, configure, and use. Distributed computational grids are, by their very nature, complex systems that are typically difficult to build, manage, and maintain. In this respect, GT4-based grid development is no different. It took well over a month for one Media Grid team to build and deploy a modest 100+ node test bed grid network comprised of both Linux and Windows nodes running GT4.

Because the Media Grid is a public utility, however, developers do not have to install GT4 themselves and can instead access the network directly through Experts in grids, security, and digital media technology are encouraged to tap into the Media Grid today so that improvements can be made in anticipation of the formal public launch scheduled for 2006.


Related Reading

More Insights

Currently we allow the following HTML tags in comments:

Single tags

These tags can be used alone and don't need an ending tag.

<br> Defines a single line break

<hr> Defines a horizontal line

Matching tags

These require an ending tag - e.g. <i>italic text</i>

<a> Defines an anchor

<b> Defines bold text

<big> Defines big text

<blockquote> Defines a long quotation

<caption> Defines a table caption

<cite> Defines a citation

<code> Defines computer code text

<em> Defines emphasized text

<fieldset> Defines a border around elements in a form

<h1> This is heading 1

<h2> This is heading 2

<h3> This is heading 3

<h4> This is heading 4

<h5> This is heading 5

<h6> This is heading 6

<i> Defines italic text

<p> Defines a paragraph

<pre> Defines preformatted text

<q> Defines a short quotation

<samp> Defines sample computer code text

<small> Defines small text

<span> Defines a section in a document

<s> Defines strikethrough text

<strike> Defines strikethrough text

<strong> Defines strong text

<sub> Defines subscripted text

<sup> Defines superscripted text

<u> Defines underlined text

Dr. Dobb's encourages readers to engage in spirited, healthy debate, including taking us to task. However, Dr. Dobb's moderates all comments posted to our site, and reserves the right to modify or remove any content that it determines to be derogatory, offensive, inflammatory, vulgar, irrelevant/off-topic, racist or obvious marketing or spam. Dr. Dobb's further reserves the right to disable the profile of any commenter participating in said activities.

Disqus Tips To upload an avatar photo, first complete your Disqus profile. | View the list of supported HTML tags you can use to style comments. | Please read our commenting policy.
Dr. Dobb's TV