Channels ▼
RSS

JVM Languages

The MPEG-4 Java API & MPEGlets

Aaron E. Walsh

, April 01, 2002

Source Code Accompanies This Article. Download It Now.


Apr02: The MPEG-4 Java API & MPEGlets

Aaron E. Walsh is chairman of Mantis Development Corp., chair of the Web3D-MPEG working group, and an international bestselling technology author. He can be contacted at aaron@mantiscorp.com.


MPEG-4 is an ISO/IEC Standard designed for delivery of multimedia content to any platform over any network. MPEG-4 can be thought of as a toolkit for developing network-oriented multimedia applications based on any combination of still imagery, audio, video, 2D, and 3D content. Developed by the Moving Picture Experts Group (http://mpeg.telecomitalialab.com/), MPEG-4 builds on the past success of the Emmy Award winning MPEG-1 and MPEG-2 Standards (an earlier MPEG-3 effort was eventually rolled back into MPEG-2, leaving the rather curious gap in numbering).

After nearly a decade of development, MPEG-4 (http://mpeg.telecomitalialab.com/standards/mpeg-4/mpeg-4.htm) is ready for prime time. The first generation of MPEG-4 content servers, authoring tools, and players are available, while a variety of emerging products are poised to enter the fray. A number of companies are developing MPEG-4 products, including Apple (http://www.apple.com/quicktime/), blaxxun Interactive (http://www.blaxxun.com/), Envivio (http://www.envivio.com/), IBM (http://www.ibm.com/), and Sun Microsystems (http://www.sun.com/), among others.

Scenes Composed of Media Objects

MPEG-4 revolves around the concept of discrete media objects composed into scenes, and as such, builds on scene graph programming concepts popularized by the Virtual Reality Modeling Language (VRML; http://www.web3d.org/fs_specifications.htm) Standard. MPEG-4 introduces enhancements not supported by VRML, however, such as streaming, binary compression, content synchronization, intellectual property management and protection, and enhanced audio, video, and 2D capabilities.

MPEG-4 also supports a number of mechanisms that enable interactive content. Programmatic control over scenes using Java is one such mechanism, and the focus of this article. In particular, I'll introduce a suite of Java APIs standardized by MPEG-4 and show how these APIs are used to create MPEGlets for constructing highly dynamic and interactive content. In doing so, I'll focus on the Scene API in this article.

MPEG-J API and Application Engine

MPEG-Java (MPEG-J) offers programmatic control over MPEG-4 terminals (players) and content using Java programs. These programs can be stored locally on the terminal or they can be embedded in the content itself, in which case the programs are downloaded along with the content (although in a separate elementary stream). The MPEG-J portion of the MPEG-4 Standard defines the lifecycle, scope, delivery mechanism, and security model for MPEG-J programs.

In addition to providing access to the MPEG-4 terminal and content, MPEG-J also offers limited access to decoder resources as well as the network layer. These capabilities are encapsulated in a suite of MPEG-J APIs organized into six packages; see Table 1.

As Figure 1 illustrates, the MPEG-J Application Engine consists of the MPEG-J APIs and any other Java APIs supported by the run-time platform, as well as a Java Virtual Machine (JVM). Upon arriving at the terminal, an MPEG-J program is processed by the Application Engine, through which it gains access to terminal resources and data as well as fundamental Java packages such as java.lang, java.io, and java.util. The Application Engine is essentially an MPEG-J subsystem capable of controlling the MPEG-4 terminal's Presentation Engine.

Although MPEG-4 terminals are not required to support MPEG-J (it is an optional feature intended to enable sophisticated interactivity that may not be required by all forms of content), those that do implement MPEG-J typically follow the architecture in Figure 2. The top part of this block diagram illustrates the MPEG-J Application Engine, which sits atop the Presentation Engine illustrated in the lower half of the diagram. In this way, MPEG-J adds programmatic control to the otherwise parametric control offered by MPEG-4's Binary Format For Scenes (BIFS).

Potential Applications of MPEG-J

Because MPEG-J offers the highest possible level of MPEG-4 interactivity, a wide variety of applications created using this technology are possible. Such applications include (but aren't limited to) content that dynamically adapts itself to changing network conditions, interactive digital television, interactive product and data visualizations, distance learning, games, and multiuser 3D worlds.

Using a combination of MPEG-J resource and scene APIs, for example, content tradeoffs based on available resources can be made at run time. For instance, a low-priority video object (such as a picture-in-a-picture shown during a newscast) can be immediately replaced with a still-texture object when resources are low rather than degrading the entire scene. Likewise, a single piece of content such as a broadcast music video can be seamlessly scaled across a variety of playback devices: Low-power devices such as PDAs and outdated PCs might receive lower quality audio and video, while higher powered devices can receive higher quality content. By using MPEG-J's resource, decoder, scene, and network APIs, these decisions can even be made based on user preferences, thereby giving end users fine-grained control over their overall experience.

It's important to keep in mind, however, that most first-generation MPEG-4 players focus primarily on streaming audio and video, as well as more simple forms of interactivity offered by the Standard. As a result, many players and authoring tools don't yet support MPEG-J, opting instead to add this sophisticated capability in a subsequent release.

MPEGlets

MPEGlets are streamable MPEG-J programs that implement the MPEGlet interface defined by the MPEG-J specification. This interface is found in the org.iso.mpeg.mpegj package, which is imported into all MPEGlet programs. Listings One and Two, for example, import this entire package using the "wildcard" asterisk character:



import org.iso.mpeg.mpegj.*;</p>

This provides access to the various classes and interfaces in the base MPEG-J package, including the MPEGlet interface that defines the lifecycle methods that every MPEGlet must implement.

MPEGlet Lifecycle Methods

All MPEGlets share a common interface that lets them be initialized, run, stopped, and destroyed by their run-time environment. In this respect, MPEGlets can be compared to applets.

Although applets have the option of overriding any of the lifecycle methods defined in the java.applet.Applet class they extend, MPEGlets must override the four lifecycle methods defined in the MPEGlet interface. The method signatures for these required methods are:

  • public void init(), called by the MPEG-J player to inform the MPEGlet that it has been loaded into the system. Any operations necessary to perform before the MPEGlet runs go here. This method is always called before the MPEGlet is executed as a separate thread via the run() method.
  • public void run(), called by the MPEG-J player to inform the MPEGlet that it may now execute. Any operations necessary for the MPEGlet's main thread of execution go here. init() is always called before run().

  • public void stop(), called by the MPEG-J player to inform the MPEGlet that it should stop its execution. Any operations that are necessary to perform before the MPEGlet is stopped go here. This method is always called before destroy() and can also be called anytime execution of the MPEGlet must cease.

  • public void destroy(), called by the MPEG-J player to inform the MPEGlet that it is being reclaimed by the system and should, therefore, destroy any resources it has allocated that will not automatically be cleaned up via garbage collection. Any cleanup operations necessary to perform before the MPEGlet is destroyed go here. stop() is always called before destroy().

Listing One, which is skeleton code you can use as a boilerplate for your own MPEGlets, imports the base MPEG-J package and implements the MPEGlet interface and lifecycle methods defined therein. The run() method of this skeleton code contains a generic try-catch block to facilitate MPEGlet and run-time exception handling.

MPEGlet Security Restrictions

MPEGlets also resemble applets in terms of the security restrictions that are placed on them. Like applets, MPEGlets are not allowed to:

  • Load native libraries, or invoke or define native code.
  • Read, write, or otherwise access files on the terminal.

  • Execute programs on the terminal.

  • Access system properties other than those defined by the Capability API.

In addition, MPEGlets can only execute their own code and that of the MPEG-J API and other Java APIs supplied by the terminal (as well as classes streamed to the terminal over the MPEG-J stream). By design, the MPEGlets security model provides a robust yet secure system for delivery and execution of Java bytecode at the terminal.

MPEGlets in Action

Listing Two adds some meat to Listing One, using the Scene API defined in the org.iso.mpeg.mpegj.scene package to modify a node in the MPEG-4 scene graph. Specifically, this MPEGlet changes the color of a shape in the scene in response to user activity — the program inverts the color of a shape when it is clicked with the mouse.

Because this program has no special need for init(), stop(), or destroy(), it merely overrides these methods as required by the MPEGlet interface but does not provide implementation for them. It does, however, provide implementation for the run() method, within which the bulk of this MPEGlet's code appears.

After retrieving a reference to the scene, this MPEGlet uses that reference to dig into the scene and retrieve a reference to a TouchSensor node (whose identifier is "TOUCH"), after which it gets a reference to a Material2D node (named "MATERIAL"). These node references are retrieved by name because MPEG-J allows access only to nodes that have an associated identifier, meaning they must have been previously defined using the DEF construct.

Example 1 contains the code for a short but sweet VRML scene that includes two DEFed nodes. The TouchSensor and DirectionalLight nodes in this example have been defined as TOUCH and LIGHT, respectively. These node names are then used to route an event from within the scene definition (the light node is activated when the mouse moves over the cone shape), whereas MPEG-J gives us external programmatic control over scenes as illustrated by the MPEGlet in Listing Two.

In the case of the MPEGlet in Listing Two, a new color value for the shape's Material node is sent to the scene when the user clicks on that shape. Although simplistic in nature, Listing Two is a complete MPEGlet that illustrates how to access and change properties of the scene with which it is associated.

ClockLet.java (available electronically; see "Resource Center," page 5), on the other hand, is a more complex MPEGlet that heaps yet another layer of flesh on the skeleton shown in Listing One. In addition to using the MPEG-J MPEGlet and Scene APIs, ClockLet.java takes advantage of the Java APIs provided by the terminal. Specifically, this MPEGlet uses the Calendar class defined by the java.util package to dynamically update the static clock scene frozen in time in Figure 3.

Developed by IBM in conjunction with the MPEG-4 standardization effort, this "ClockLet" creates a Calendar object to keep track of time. It then continually updates the hour, minute, and second hands of the clock scene using the same node-identifier technique just described (ClockLet.scenegraph.txt, available electronically, is the corresponding scene graph code). ClockLet can also respond to start, stop, and pause events generated by corresponding buttons on IBM's MPEG-4 player (Figure 3), which is itself written in Java.

Similar to the example shown in Listing Two, the bulk of this MPEGlet's code appears in the run() method. However, ClockLet also contains code related to the listener it uses to trigger scene updates as well as code related to multithreading. Unlike applets, every MPEGlet is inherently multithreaded. The MPEGlet interface ultimately implements the Runnable interface, which is why run() is a required lifecycle method (as opposed to the start() method used by Applets, which must explicitly implement Runnable on a case-by-case basis).

These examples are straightforward illustrations of the MPEG-J API in action. In addition to the added interactivity that MPEG-J brings to the table, it also helps to enforce good programming practice by ensuring that a clean boundary exists between presentation (the scene) and logic (programs controlling the scene).

MPEG-J programs, such as the MPEGlets presented here, are written independent of the content that they access and manipulate. Furthermore, you benefit from the many advantages afforded by Java, such as its strong focus on object-oriented programming, built-in support for networking and security, portable bytecode that may be executed across any number of platforms, and the potential for an extensive class library (courtesy of the Java API) that may be provided at the MPEG-4 terminal.

Acknowledgments

Thanks to Viswanathan (Vishy) Swaminathan of Sun Microsystems who was instrumental in the design and development of MPEG-J within the MPEG-4 standards effort. Vishy, along with Alex MacAulay of Envivio, also wrote the Scene API example MPEGlet (Listing Two), code originally created for the chapter on MPEG-J in More MPEG-4 Jump-Start (Prentice Hall, 2002), which I coauthored with Mikael Bourges-Sevenier. Thanks also to Steve Wood, Michelle Kim, and Jeff Boston of IBM for providing me with a prerelease version of their MPEG-4 player in addition to the ClockLet code.

DDJ

Listing One

/** LifecycleSkeleton.java
  * Skeleton (boilerplate) code illustrating "lifecycle" methods 
  * common to all MPEGlets.
  * @author Aaron E. Walsh, Mantis Development Corp.
  */
import org.iso.mpeg.mpegj.*;   // import all classes in main MPEG-J package
public class LifecycleSkeleton implements MPEGlet {
 /** Any operations necessary to perform before the MPEGlet runs go here.
   * The init() lifecycle method is called by the MPEG-J player to inform the 
   * MPEGlet that it has been loaded into the system. init() is always called 
   * before the MPEGlet is executed as a separate thread via run() method.  
   */
  public void init()
  {
  }
 /** Any operations necessary for MPEGlet's main thread of execution go here. 
   * run() is called by the MPEG-J player to inform the MPEGlet that it may
   * now execute. init() is always called before run().  
   */
  public void run() 
  {
     try {
            // invoke methods that can throw exceptions related to MPEG-J and 
            // MPEGlets here, with corresponding catch clauses below. This 
            // boilerplate code catches and prints the stack trace for 
            // MPEGJException and InterruptedException objects, in addition 
            // to generic Exception objects, but doesn't do anything truly 
            // productive after they are caught:
         } catch (MPEGJException e) {
            e.printStackTrace();  
         } catch (InterruptedException e) {
            e.printStackTrace(); 
         } catch (Exception e) {
            e.printStackTrace();  
         }
  }
  /** Any operations necessary to perform before MPEGlet is stopped go here.
   * stop() is called by the MPEG-J player to inform the MPEGlet that it 
   * should stop its execution. stop() is always called before destroy() and
   * can also be called anytime execution of the MPEGlet must cease.
   */
  public void stop() 
  {
  }
  /** Cleanup operations necessary to perform before MPEGlet is destroyed go 
   * here. destroy() is called by MPEG-J player to inform MPEGlet it is being
   * reclaimed by system and should therefore destroy any resources it has 
   * allocated that won't automatically be cleaned up via garbage collection. 
   * stop() will always be called before destroy(). 
   */
  public void destroy()
  {
  }
}

Back to Article

Listing Two

/** SceneExampleMpeglet.java
  * Complete MPEGlet using the Scene API to invert the color of a shape when
  * users click on it. This software module is featured in "More MPEG-4 
  * Jump-Start" by Aaron E. Walsh and Mikael Bourges-Sevenier and appears here
  * courtesy of the publisher, Prentice Hall, & original authors of this code.
  * @author Vishy Swaminathan, Sun Microsystems, Inc.
  * @author Alex MacAulay, Envivio, Inc.
  */

import org.iso.mpeg.mpegj.*;
import org.iso.mpeg.mpegj.scene.*;
public class SceneExampleMpeglet implements
  MPEGlet, SceneListener, EventOutListener {
    Scene m_scene = null;
    boolean m_isTouched = false;
    public void run() {
        try {
            // Get the scene manager and register to be
            // notified when the scene is ready.
            MpegjTerminal terminal = new MpegjTerminal(this);
            SceneManager manager = terminal.getSceneManager();
            manager.addSceneListener(this);
            // Wait until the scene is ready.
            synchronized (this) {
                while (m_scene == null) {
                    wait();
                }
            }
            // Get the TouchSensor node.
            Node touch = m_scene.getNode("TOUCH");
            // Register to be notified when it's touched.
            touch.addEventOutListener(
              EventOut.TouchSensor.touchTime, this);
            // Wait until it's touched.
            synchronized (this) {
                while (!m_isTouched) {
                    wait();
                }
            }
            // Invert the color of a shape as feedback.
            Node material2d = m_scene.getNode("MATERIAL");
            FieldValue val = material2d.getEventOut(
              EventOut.Material2D.emissiveColor);
            final float[] color = ((SFColorFieldValue) val).getSFColorValue();
            color[0] = 1 - color[0];
            color[1] = 1 - color[1];
            color[2] = 1 - color[2];
            material2d.sendEventIn(
                EventIn.Material2D.emissiveColor,
                new SFColorFieldValue() {
                    public float[] getSFColorValue() {
                        return color;
                    }
                }
            );
        } catch (MPEGJException e) {
            e.printStackTrace();
        } catch (InterruptedException e) {
            return;
        }
    }
    // Called by the scene manager when the scene is ready.
    public synchronized void notify(int msg, Scene scene) {
        if (msg == SceneListener.Message.SCENE_READY) {
            m_scene = scene;
            notify();
        }
    }
    // Called when the touchTime eventOut is triggered.
    public synchronized void notify(int id, FieldValue v) {
        m_isTouched = true;
        notify();
    }
    // lifecycle methods that aren't implemented by this MPEGlet:
    public void init() {}
    public void stop() {}
    public void destroy() {}
}

Back to Article


Related Reading


More Insights






Currently we allow the following HTML tags in comments:

Single tags

These tags can be used alone and don't need an ending tag.

<br> Defines a single line break

<hr> Defines a horizontal line

Matching tags

These require an ending tag - e.g. <i>italic text</i>

<a> Defines an anchor

<b> Defines bold text

<big> Defines big text

<blockquote> Defines a long quotation

<caption> Defines a table caption

<cite> Defines a citation

<code> Defines computer code text

<em> Defines emphasized text

<fieldset> Defines a border around elements in a form

<h1> This is heading 1

<h2> This is heading 2

<h3> This is heading 3

<h4> This is heading 4

<h5> This is heading 5

<h6> This is heading 6

<i> Defines italic text

<p> Defines a paragraph

<pre> Defines preformatted text

<q> Defines a short quotation

<samp> Defines sample computer code text

<small> Defines small text

<span> Defines a section in a document

<s> Defines strikethrough text

<strike> Defines strikethrough text

<strong> Defines strong text

<sub> Defines subscripted text

<sup> Defines superscripted text

<u> Defines underlined text

Dr. Dobb's encourages readers to engage in spirited, healthy debate, including taking us to task. However, Dr. Dobb's moderates all comments posted to our site, and reserves the right to modify or remove any content that it determines to be derogatory, offensive, inflammatory, vulgar, irrelevant/off-topic, racist or obvious marketing or spam. Dr. Dobb's further reserves the right to disable the profile of any commenter participating in said activities.

 
Disqus Tips To upload an avatar photo, first complete your Disqus profile. | View the list of supported HTML tags you can use to style comments. | Please read our commenting policy.
 

Video