SVG At the Movies

October 13, 2004

Antoine Quint

Synergies

If you're a regular reader of this column, or if you just read the specification carefully, you would know that SVG is more than just a vector graphics XML vocabulary. While I won't bore you here with the list of application contexts SVG is suited for, I will point out that since Day One there have been synergies between the work that took place at W3C around multimedia (SMIL) and SVG. For example, as we saw in the first article of this column, SVG supports SMIL Animation, using elements like <animate>, and can bring a whole bunch of SVG primitives to life.

You can animate a rectangle's width, its alpha transparency, some gradient's second color-stop, the standard deviation of some gaussian blur filter, and so on. The SMIL Animation framework is generic and applies to any XML attribute or CSS property for which a variation makes sense (i.e., the spec makes it clear what happens). While SMIL Animation integration in SVG 1.1 already brought forward some outstanding SMIL features, it did not really live up to the "M" in the acronym. SVG 1.2 however moves forward by providing SMIL-inspired support for audio and video. In this column I will present the latter's integration in SVG 1.2.

Adaptation

All along, SVG has been adapting SMIL so that it fits perfectly in the SVG picture. For instance, the animation elements were enhanced so that you could animate the SVG transform attribute with semantics reflecting the different transform primitives supported in SVG (the <animateTransform> element). The same rationale is being applied for the inclusion of new SMIL elements in SVG 1.2.

Of course, SMIL has a <video> element; and, looking at it, we see that it really is just a subclass of a more general media container. The element is further defined by a media type (the type attribute) and a link to a resource containing the video (the src attribute). SVG chose not to require the potentially redundant specification of the video media-type, but, rather, to rely on the resource's mime type and its available metadata when retrieved. This should simplify authoring. As for pointing to a resource, SVG chose to promote consistency and use the xlink:href attribute already used by the <image> element.

Another major adaptation for SVG is the way a video clip is laid out in the SVG canvas. SMIL has its own layout mechanisms, compatible with CSS2, with the big principle being the definition of media regions. These regions are containers that you use to specify the position and the size of the view port into which a media clip (like a video) can be rendered. The media object itself (like SMIL's <video> element) does not define any of its layout, it has to be included within a region.

The way layout is done in SVG is slightly different. You could take a very similar approach with SVG by using <svg> elements instead of a SMIL region. However, in SVG elements usually define themselves where they should be drawn (although relative to an inherited transformations). In that light, SVG adds the same attributes as those shared by rectangles and images: x, y, width and height.

Citizenship

You can lay out a video clip much like you would lay out simpler graphics like a rectangle or a raster image. In fact, the similarities between the three go further and SVG's <video> element actually integrates seamlessly with the rendering model of SVG. You can composite a video clip just like an image would get composited, making the video half-transparent, and have other SVG graphics (or even another video) render on top of it. The transformation primitives available in SVG are also applicable to video; thus, you can apply a rotation or any other transformation you see fit. You can also use the many filter effects available in SVG, if that's what you want. The attributes defining the position, the size or the volume of the <video> element can also be animated.

In other words, you can go beyond rendering video in a simple black box, which was the model with the <foreignObject> in the pre-SVG 1.2 world -- although no mainstream implementation ever combined SVG and SMIL with that model. Video is just a regular and respectable SVG citizen. The black magic going on behind the scene can be achieved by using an off-screen buffer in which your video clip is rendered, and then composited on the SVG canvas frame after frame.

It is good to note, too, how the <video> element fits in the SMIL timing model used in SVG. In SMIL every media object has a beginning and an end, whereas in SVG every graphic object is continuous and you don't have to use a begin attribute on any of the graphic objects in SVG (only on animation elements). This is where the <video> element differs since the resource it points to has an intrinsic duration and thus cannot fit in the persistent display mode of SVG. That meant we had to allow timing attributes on this media element, which shouldn't be seen as an inconsistency in SVG but rather an ad-hoc procedure.

Formats

But what video format do you have to use? I guess you're already scared that SVG mandates some patent-encumbered format that requires overpriced encoders. Or maybe SVG went the extreme opposite and chose some obscure patent-free format with no decent encoding software for my platform Well, in fact it's neither of these extremes.

SVG has set a precedent in mandating external formats, like JPEG and PNG for raster images. For raster images it was a good thing since we had two very good formats with broad support in tools; very good support via free rendering libraries, thought to be patent-free; and also offering two distinct approaches (JPEG being lossy and PNG lossless). Mandating those two was a good step in the direction of interoperability.

In the video area, however, there wasn't any straightforward choice to make; and, so far, the W3C SVG Working Group prefers not to mandate a given format since none of those proposed offer the same balance as JPEG and PNG offered. The idea is that the <video> element is an abstract layer on top of all the video capabilities of your system, and an SVG 1.2 implementation should try to integrate with as many video decoders available on the client.

With that in mind, what SVG offers doesn't solve the problem that you might face today when having to choose between the major vendors out there battling for the video market, but at least you can use formats and tools you are familiar with.

Possibilities

This SVG-and-video integration opens up a world of possibilities for mixing different kinds of rich content. Basically, with this new feature you can use SVG as a comprehensive presentation environment for multimedia. All the components are there: vector graphics, raster graphics, filters, transforms, audio, and video.

So now we can see SVG going places it wouldn't have gone before. Maybe a set-top box for instance, where SVG would be used to power the user interface and various screen bits and bobs for applications like an electronic program guide, display of statistics for a sports event, etc. Obviously this also has implications in desktop applications. For example, you could create a nice video jukebox with SVG subtitles. I'll let you figure out how video support fits best in your own type of applications.

Beautifully Integrated Demos

I couldn't resist hacking a couple of demos for showing off how beautifully video integrates with SVG. The first "simple" demo just shows off using video in a SMIL-aided animation context, where a good old SVG path gets morphed into a scaling rectangle, itself morphing into a video rendering box with slight semi-transparent curvy borders.

The "resize" demo might really gets some "ooohs" and "aaahs" from whoever is peeping at your screen while you view it. The point of that one was to create a piece of script that would let a video clip be resized dynamically, like you would resize a rectangle in Illustrator. If you play with it, you'll notice that I also took the time to add a "mirror" behavior for when you drag one of the corners beyond another. You'll find both demos in the resources archive for this article.

Dig up the source code and view the demos with the Adobe SVG Viewer 6.0 Technology Preview, which is Windows-only for now, or the latest QuickTime plugin.