Silverlight Media Elements



May 19, 2008
URL:http://www.drdobbs.com/windows/silverlight-media-elements/207800861

David James Kelley is a senior software architect at IdentityMine and author of the upcoming book Hacking Silverlight 2 on which this article is based. Courtesy of Manning Publications. All rights reserved.


Silverlight is all about next-generation user interfaces on the web, and one of the coolest aspects of presenting next-generation UIs on the web is video and sound. When developing a next-generation user or Web 2.0 experience, it is the smooth, subtle animation in the background, the video playing in the text, and the sound reacting to a user's action that brings UIs alive. The media element in Silverlight is the way we add video and sound, from streaming video or scrubbing the timeline, to using audio and special effects with video brushes. Here we talk about all the ins-and-outs of the media element and all the things we can do with it. The Silverlight media experience even supports high-definition video streaming online. With all the cool things about using media elements in Silverlight, it is easy to over look the possible downside of it -- performance. So first let's talk a little about performance of media elements in Silverlight.

I recall sitting in the room and writing a class called MediaClip that wrapped all the functionality I wanted for videos -- manipulating video, starting video, stopping and pausing video. It even had embedded controls and did source swapping with different versions of a media source, so that different resolutions could be used, depending on what the current video size setting was for the object. The best part was that in its smallest mode, you could drag the video. For example, if in the application you had a line or group of videos, then you could click-and-drag the clips to the editor tool and make a play list of sorts. Each one of these little clips was an instance of my MediaClip class. Thinking this source swapping and drag-and-drop was cool, we went built out all functionality for this rich immersive media experience. And we were feeling pretty good about ourselves.

When we finally had something to play with and things were coming together, a few of the guys decided to spend time playing with the video, maybe with our little Vista CPU meter running and the memory usage thing going on down in the task bar. We played a few videos, dragged a few things around. Life was good.

Then I selected a different category and the application swapped out the current media group on the page. No worries right? No one really noticed much on the meters. Dragged some media, play a clip or two... no worries. Well, so the meter was slowing crawling up... hm... no worries right? (Say it with me... "no worries...") Pretty soon the memory meter was pegged and the drag operation was getting slow. Hmm.. no worries? Click on another category... a new set of videos animate out... and it started going sloooower and sloooower.

At this point a few red flags started coming up -- or maybe about 100 red flags, one for each media clip that was on or had been on the screen. This was not a good sign. Through a lot of jumping around, we found a number of key performance issues for Silverlight and the media element.

Rule #1: Don't have 500 media elements in the same application at the same time all bound to some media somewhere.

Rule #2: Do not leave all the media elements around if you're not using them. It is one thing to have 10 on the screen, but another to have 10 on the screen and 90 hiding somewhere at --9000 for Canvas.Left. (And yes I have actually seen that done.)

Rule #3: See Rule #2.

Rule #4: Encode your media for the size that it will be displayed. Do NOT stretch the video....

All this video is cool, but if you can't serve it up and make it fast, then there's no point. Let's get down to business and learn out to work with the media element and how to use it in such a way as to be reasonably fast.

What Is a Silverlight Media Element?

The "media element" is an XAML element that lets you embed media -- namely, video and sound -- into Silverlight applications, thereby exposing the functionality in Silverlight that lets you manipulate the media content.

In the example below, we have only added a few new properties. The first new property is the "source property." The source property for the media element is what tells the media element where to go to a particular file or bit of media. The isMuted property is set to True or False, which determines if the element has sound when it first plays. In other words, it turns on/off the mute setting of the media element.

The next property is the AutoPlay. AutoPlay determines if and when the video is first loaded, if the video should automatically start playing. You can see these in this example of a media element in XAML.

<MediaElement Name="MediaElementElement"
  Source="ImenSample.2.wmv"
    Canvas.Left="20"
    Canvas.Top="20"
    Width="100"
    Height="100"
  IsMuted="False"
  AutoPlay="True" />
Listing One: The Basic Media Element in XAML.

Let's take a look at the media element running in Figure 1. Note the size of the video and the left and top properties and how they place the element. The really bad thing about it is, that this particular video is not 100x100 so there are some performance issues with doing it this way. The problem with this is not a function of the fact that we are showing the image at 100x100 but that it was encoded for a different size then 100x100. When you show video at a different size then the size it was encoded for then there is a performance hit for having to transform the video in real time.

Figure 1: Simple media player running.

That is how we build a basic media element and set our source and properties. Silverlight also support streaming media which means that we can basically make a web request (that is, an HTTP request to some URL) for a some video and as we starting getting the packets, we can actually start doing something with the video without waiting for the video to fully download. How we determine when to start playing or how much we need to download is called "buffering". In Silverlight, we can use a number of events to make sure we play nice with our media and stream it if we supported by our server. From the standpoint of "Silverlight," we really don't have to do anything special (Silverlight deals with loading and buffering), but we can overwrite how much of a buffer Silverlight will wait for before firing the media loaded event and start playing. Next we start adding controls to make the video do things. In other words, we start to manipulate the video in our media element.

Media Element Controls: Play, Pause, Stop and Full-Screen

Let's address the basic operation of the media element. The key "methods" are straightforward -- Pause, Play, and Stop. Pause only works if the video is playing. If the video is playing, then when you call the Pause method on the media element the audio or video pauses in its current position. Unlike Pause, Stop actually stops the player and resets the player back to the start of the current media source. Play is the opposite of Pause. If the video is "paused" at some location in the current media source or "stopped" at the beginning -- which is the state it starts with assuming auto play is set to False -- then the play method starts the video or audio playing.

The next cool feature is the full-screen mode of the Silverlight control. Full-screen mode is not a function of the media player, but of the control. You could, of course, use the full-screen feature to do things other than media, but most of the time it wiill involve showing video on full screen. Since the media element doesn't change on its own, you would have to handle the UI change in your code from or/to full screen. Full-screen then is a property of the host Silverlight control and on full screen changed, an event on the top control, we can then change the UI to the state needed.

With these four simple UI elements that you can bind to events using the mouse down events, you can create the behavior needed to build a media player that looks like the simple media player in Figure 2.

Figure 2: Simple media player.

To make this example actually work, let's look at how we set it up, starting at Listing Two where the events are wired up to the button elements.

private SilverlightHost SLHost = new SilverlightHost();
public Page()
{
   InitializeComponent();
}
private void MediaElementElement_pause(object sender, MouseEventArgs e)
{
   MediaElementElement.Pause();
}
private void MediaElementElement_play(object sender, MouseEventArgs e)
{
   MediaElementElement.Play();
}
private void MediaElementElement_stop(object sender, MouseEventArgs e)
{
   MediaElementElement.Stop();
}
private void MediaElementElement_fullscreen(object sender,MouseButtonEventArgs e)
   {
      if (SLHost.Content.IsFullScreen)
   {
      SLHost.Content.IsFullScreen = false;
   }
   else
   {
      SLHost.Content.IsFullScreen = true;
   }
}
Listing Two: Media Control Button event handlers

Notice that in the constructor we programmatically bind an event onfullScreenChanged to a method to manipulate the UI between transitions if we want to. However, in this example we'll skip over that and focus on the methods we call on the media element.

The full-screen mode must be fired from a user event directly and we are not allowed to nest this or have it fired from a non-direct user event. This prevents bad people from taking over someone's screen without permission. Let's review the basic events we wired.

The play, pause, and stop events are a reference to the media element so they are straightforward. The method MediaElementElement_fullscreen is important for the full-screen feature to be used and wired in XAML. As noted earlier, they are user events and can launch the application into full-screen mode. This event actually handles both cases and changes the application state, depending on the current state when the event fires.

In more sophisticated applications, we might be arranging or exposing different controls. Here, we would do that in this method. With our basic controls wired up let us move onto a number of other events we might want to use against the media element to provide more complete interaction with the video.

Media Element Events

Now, let's look at a few of the other events that we can wire up off of the media element. The first four elements are downloadProgressChanged, mediaOpened, MediaEnded, and MouseLeftButtonDown. The first three are related to the media, and the last is a common event we used earlier. In this case MediaElement_MouseLeftbuttonDown is used to make the full-screen mode of the application resize back to normal. We can then reuse the event we bound to earlier to resize the application, without having to have two blocks of code that do the same thing. DownloadProgressChanged is just what it says: Anytime something changes about the download, we get this event and can do something like a download progress bar. MediaOpened is fired when the media is downloaded and ready to play. We might use this to have a preloader animation for our media element. MediaEnded is fired when the media is complete and/or when something goes wrong with the stream or download. We should be able to deal with a number of conditions here and can use this to do some error handling. All these events are wired in Listing Three.

<MediaElement Name="MediaElementElement" Canvas.Left="0"
  Canvas.Top="0" Width="464" Height="348" IsMuted="False" AutoPlay="True"
  DownloadProgressChanged="MediaElement_DownloadProgressChanged"
  MediaOpened="MediaPlayerElement_MediaOpened"
  MediaEnded="MediaElement_MediaEnded"
  MouseLeftButtonDown="MediaElement_MouseLeftButtonDown" />
Listing Three: Media Element with bound events

You can see from this code that it's straightforward to bind these events, and for the most part you will always use the first three; however, you would also need to give the media element a source. Now that you have a media element going, we can do even cooler stuff with a Video brush bound to our media element.

Video Brush and the Media Element

The "video brush" is a great way to create cool special effects. To make a video brush work you, have to have a named media element that you can bind to. That has a bound source. In Listing Four, we have a source. We have set the element to be muted but play and its opacity is 0 so no one sees it on screen. Then we add some text in a text block app and add a foreground element with a "video" brush. The video brush element is bound to the media element by using the media element name. Using a video brush you can basically make just about anything play video.

<MediaElement Name="MediaElementElement"
  Source="ImenSample.2.wmv"
  Canvas.Left="20" Canvas.Top="20"
  Width="100" Height="100"
  IsMuted="True" AutoPlay="True" Opacity="0" />
<TextBlock Name="Sample Text"
  FontSize="20" FontFamily="Verdana" FontWeight="Bold"
  Canvas.Left="10" Canvas.Top="50"
  Opacity="1" >PAINT WITH VIDEO
     <TextBlock.Foreground>
        <VideoBrush SourceName="MediaElementElement" />
     </TextBlock.Foreground>
  </TextBlock>
Listing Four: Media Element and Text Bock with a Video Brush

When this XAML is rendered, we get the text "Paint with video" that is playing video in the background of the text.

Now we can set properties, bind events, and use video elements in a video brush. With that basic understanding of media elements and being able to use a video brush, we can move on to more complicated composite tasks.

Hack: Video Scrubbing

A common feature of most video players is the ability to "scrub" video or "scrub" the timeline of a video. When we say "scrub," we basically mean being able to change the position of the video by clicking on a line or icon and dragging it along a line that represents the video and the video changes in real time. That leads us into how we make the timeline visually, then how to wire it up such that we can actually "scrub" it. The first thing we do is then is to create our XAML; see Listing Five.

<Rectangle x:Name="TBStatusBarBg" Opacity="0.5" Width="327.167"
Height="2" Fill="#FF8C8989" RadiusX="0" RadiusY="0" Canvas.Left="15"
Canvas.Top="7.667" />
   <Rectangle x:Name="TBStatusBarDownloadProg" Opacity="0.5"
Width="0" Height="2" Fill="#FF6C6B6B" RadiusX="0" RadiusY="0"
Canvas.Left="15.333" Canvas.Top="8" />
   <Rectangle x:Name="TBStatusBarPlayProgress" Opacity="0.5"
Width="0" Height="2" Fill="#FF353434" RadiusX="0" RadiusY="0"
Canvas.Left="15" Canvas.Top="8.333" />
   <Rectangle x:Name="ScrubIcon" Opacity="1" Width="3" Height="8"
Fill="#999999" RadiusX="5" RadiusY="5" Canvas.Left="15" Canvas.Top="6"
Cursor="Hand" />
   <Rectangle x:Name="ScrubMask" Width="327.167" Opacity="0"
Height="10" Fill="#FFF0E8E8" Canvas.Left="15" Canvas.Top="5"
Cursor="Hand" />
Listing Five: Video Scrubbing Rectangles

You can make your timeline any way you want. In our case, we use five simple rectangles. Using our timeline, we will show how much the current video is downloaded and the current position in the playback. So with that our first rectangle is the background. This is the core visual representation of the video length. It doesn't actually do anything and is static, regardless of the video source, but contrast of the progress and position indicator is relative to it on a percentage basis.

The next rectangle is used to show how much the video is downloaded. This rectangle is animated by a media element event.

The third rectangle is our position indicator that shows where the video is in playback and will be the key thing that changes when we actually "scrub" the video.

Following that we have a rectangle that we use as our indicator icon. The indicator icon is rendered perpendicular to the others and is generally what users click on to scrub the timeline.

The last rectangle is actually an invisible mask we place over the whole set. It actually is what we bind mouse events to so that things are not so hard to click on.

The next two XAML components are what we need to make for the timeline to work. They are a media element and storyboards for animation. There are two storyboards -- one is a timer and the other is the position play back animation to smooth out the position change. Together this smoothes out the changes in position when scrubbing and during playback as seen in Listing Six.

<Canvas.Resources>
<Storyboard x:Name="Timer"
Storyboard.TargetName="MediaPlayer" >
   <DoubleAnimation x:Name="TimerWidth"
Storyboard.TargetProperty="Width" Duration="0:0:.1" />
</Storyboard>
<Storyboard x:Name="PositionPlayBack"
Storyboard.TargetName="TBStatusBarPlayProgress" >
   <DoubleAnimation x:Name="PositionPlayBackWidth"
Storyboard.TargetProperty="Width" To="0" Duration="0:0:.1" />
   <DoubleAnimation x:Name="PositionIconLeft"
Storyboard.TargetName="ScrubIcon"
Storyboard.TargetProperty="(Canvas.Left)" To="0" Duration="0:0:.1" />
</Storyboard>
</Canvas.Resources>
<MediaElement x:Name="MediaElement" Canvas.Left="0"
Canvas.Top="0" Width="361" Height="280" ></MediaElement>
Listing Six: Storyboard as a resource

Now the hard part is actually to wire things up. All the eventing we are doing is in XAML, but you could do it programmatically as well. In the root of our page class that is our media player, we will create the values or objects we need.

private bool _ScrubIconMouseCapture = false;
private bool _MediaOpened = false;
private SilverlightHost SLHost = new SilverlightHost();
private System.Windows.Threading.DispatcherTimer MyTimer = new
System.Windows.Threading.DispatcherTimer();
Listing Seven: Setting values and binding events and other initialization in code.

Here we can see a number of values. The _ScrubIconMouseCapture value is used to determine if the mouse is captured so we can perform the drag operation on the MouseMove event. The _MediaOpened is used to see if the video is open or not and if so then we know we can scrub the video timeline. We also have a reference to the Silverlight host and an instance of the DispatcherTimer object that we can use to fire a thread. Now we take a look at our constructor code in Listing Eight.

MyTimer.Interval = new TimeSpan(0, 0, 0 , 0, 1);
MyTimer.Tick += new EventHandler(Timer_Completed);
MyTimer.Start();
Listing Eight: Code in the constructor needed for scrubing.

In Listing Eight, we see us setup a timer or thread that fires at the set interval, in this case 1 millisecond. We also define the event that is fired at every interval and start our thread.

Next let's look at the timer completed event (Listing Nine).

private void Timer_Completed(object sender, EventArgs e)
   {
   if (_MediaOpened && !_ScrubIconMouseCapture)
   {
       double Width = (this.TBStatusBarBg.Width /
       ThisMediaElement.NaturalDuration.TimeSpan.Seconds) * ThisMediaElement.Position.Seconds;
       if (Width > TBStatusBarBg.Width)
   {
       Width = TBStatusBarBg.Width;
   }
       else if (Width < 10)
   {
       Width = 10;
   }
       PositionPlayBackWidth.To = Width;
       PositionIconLeft.To = Width + 10;
       PositionPlayBack.Begin();
   }
}
Listing Nine: Timer completed event.

In this event, we check that the MediaOpened flag has been set and that that mouse is captured or not and starts the timer again. The media opened event that is bound to the media element sets the media opened flag when the video can start playing. Now if the media opened flag is set and the mouse is not captured, then it calculates the width based on the width of the background of the timeline vs. the length of the video taken from the natural direction based on seconds and then the current position. Next we set the two values, one for the position bar and the other for the "icon" or fourth rectangle we used as our scrub icon. After we set the critical values, we then start the animation again so the change is smooth as implemented in this sample.

Now we are ready to setup our scrub timeline. First we go over the download progress changed event. We bound this to show on our timeline how much of the video is downloaded, so this event then sets the width of the download progress bar on our timeline, once downloaded it starts the media element playing if our property auto play we created in our class is set to True.

private void ThisMediaElement_DownloadProgressChanged(object sender, RoutedEventArgs e)
  {
     TBStatusBarDownloadProg.Width =
     ThisMediaElement.DownloadProgress * TBStatusBarBg.Width;
       if (ThisMediaElement.DownloadProgress == 1)
  {
      if (AutoPlay)
     {
     Play();
     }
  }
}
Listing Ten: Download progress changed event code

A scrub operation starts once a user clicks on our mask rectangle. This fires off the event Mouse Left Button Down. Here we first set the mouse capture flag, and then we call the pause method we created on the application class. We could just "pause" the video ourselves, but since we might want to call this method in a number of places and in our application we also want to change the play icon; we just wrap it in a method called pause which we call here. Once the flag is set and pause is executed, we actually "capture" the mouse. Remember on this method we lose the capture without being able to track it if the user moves off the Silverlight control. If we do nothing else, if the user clicks somewhere on the timeline to scrub then position will stay somewhere else until they move the mouse. So to make sure this effect doesn't happen we also fire our mouse move event so that the position immediately goes to the correct position on the timeline and the video play back.

private void ScrubMask_MouseLeftButtonDown(object sender, MouseButtonEventArgs e)
   {
   _ScrubIconMouseCapture = true;
   Pause();
   ScrubMask.CaptureMouse();
   ScrubMask_MouseMove(sender, e);
   }
     private void ScrubMask_MouseMove(object sender, MouseEventArgs e)
     {
        if( _ScrubIconMouseCapture )
        {
           double x = e.GetPosition(null).X;
           if( x > TBStatusBarBg.Width + 10 )
        {
           x = TBStatusBarBg.Width;
        }
        else if( x < 10)
     {
          x = 10;
     }
        PositionPlayBackWidth.To = x - 10;
        PositionIconLeft.To = x;
        PositionPlayBack.Begin();
        double NewTime = x / ( TBStatusBarBg.Width /
        ThisMediaElement.NaturalDuration.TimeSpan.Seconds);
        if (NewTime >
        ThisMediaElement.NaturalDuration.TimeSpan.Seconds)
     {
        NewTime = ThisMediaElement.NaturalDuration.TimeSpan.Seconds;
     }
        try
     {
        double TotalSeconds = Math.Round(NewTime);
          //get time
        int Minutes = int.Parse( Math.Floor(TotalSeconds / 60).ToString() );
        int Seconds = int.Parse( (TotalSeconds - (Minutes * 60)).ToString() );
        ThisMediaElement.Position = new TimeSpan(0, Minutes, Seconds);
     } catch(Exception) { }
  }
}
Listing Eleven: Mouse events for the drag operation

The mouse move event this code implements as the user holds down the mouse button affects our timeline and video and is actually "scrubbing" the video.

First, this method fires any time the mouse moves over the mask> So as to not mess with the user too much, we want to make sure we have the mouse capture before doing anything. If we do, we then grab the x value of the mouse and change it by our offset of the control. Next, we need to make sure the value is not too big or too small. For example, if the user moves the mouse way past either end of the timeline we would effectively be past the end or before the beginning, so we reset our x value that accordingly by using an "if else if" statement based on the bounds of the timeline.

Since the position change animation is currently paused during the capture event and because we called pause on mouse down we just set the two double animation values. Then we restart the animation. Also on regular mouse moves we just set the values and call the begin method. After this we create a new time value conversion for what the new play back position will be. This is based on our x value divided by the line width divided by natural duration of the video. If the new time is greater than the natural duration, we set it to the end of the media; otherwise we go ahead and set the new position and use some error handling to make sure this operation doesn't blow up.

When the user is done with scrubbing the video with our mouse move event and lifts the mouse button up, we call our mouse up button to let the video continue playing. In this event, we set our mouse capture flag back to False, call the play method which we wrapped so we could change icons, and so on -- like we did with the pause method we implemented. We then can release our mouse capture.

private void ScrubMask_MouseLeftButtonUp(object sender, MouseButtonEventArgs e)
   {
      _ScrubIconMouseCapture = false;
      Play();
      ScrubMask.ReleaseMouseCapture();
   }
Listing Twelve: Left button up event that ends the drag operation.

You can implement this a number of ways but this is how I hacked it together and it works well. You can scrub the video timeline and become dangerous to yourself and others. This actually provides us with the ability to build the complete media player experience.

Terms of Service | Privacy Statement | Copyright © 2024 UBM Tech, All rights reserved.