_{(*Desktop Silverlight for Windows)}

There haven’t been too many articles written about the rendering subsystems of WPF and Silverlight, at least not in great detail. This leaves room for quite a bit of speculation and leads to some turf war augments between some Silverlight and WPF developers. I want to document (brain barf) at least what I have gathered from qualified sources, observations, educated guesses and plain old snooping around public symbols. I did not acquire knowledge of any of this under NDA and I cannot guarantee all information here is accurate, so I encourage others to let me know what is incorrect.

Windows Presentation Foundation
A Brief History

The first version of WPF was RTM’d around the time of Vista and was included in .NET 3.0. It touted a fully GPU accelerated user interface. The native-code part of WPF was called the “milcore”, also known as the “Media Integration Layer”. This part of the code was shared with the Vista DWM (aka Aero). Since .NET 3.5(sp1?), WPF’s “milcore” and the Vista DWM had split off. Presumably because major and possibly breaking changes had to be made to WPF’s native rendering stack (e.g., pixel shaders).

With Windows 7, the DWM has moved on to Direct3D 10 and WPF 3.5 and 4 are still using Direct3D9 (9Ex on Vista and higher) so there is little or no relation to the DWM and WPF’s native rendering stack.

If you have ever peeked at the managed shared source code for WPF, you will find a lot of references to “DUCE”. DUCE stands for “Dynamic Unified Composition Engine”. From here on out I’ll refer to WPF’s rendering stack as DUCE…hopefully that is correct.

Rendering Stack Features

Full GPU Acceleration – Unfortunately the term hardware acceleration has become generic marketing lingo. It is very important to ask “What is being accelerated and how?” For instance, there was a Flash 10 update that came around that enabled “Hardware accelerated video”. What was accelerated? Did it decode the video in the GPU then render it (and I found some believed)? Nope. It just used the GPU to scale video playback so your CPU didn’t have to run a scaling algorithm on every frame...when in full screen mode. So again, what is GPU accelerated in WPF? Well, just about every pixel you see is generated and composed by the GPU, with a few minor exceptions. My best guess how DUCE renders WPF visuals, colors, gradients is with shaders. Consider each shader to be a simple CPU core. Today’s low end GPUs have a couple hundred, while some high ends there are 700+! This allows WPF to have complex animations, while maintaining high frame rates at relatively low CPU usage. What I felt gave the DUCE performance a bad name in the early days was twofold. Low-end GPUs were still wide spread and there was no cached composition API that is in 4.0.

Threaded - The DUCE was built to support a retained mode rendering system which has a render thread and a UI thread. Since DUCE could run independently of the UI thread, even out of process and even out of system (e.g. network), it allowed for remote desktop to leverage the client’s GPU. This optimized remote desktop functionality because instead of transferring dirty area bitmaps which are quite large, it could simply send smaller DUCE commands and have the client GPU render them. This architecture is also why in Vista with .NET 3.0, a WPF application would be scary fast over remote desktop.

There were also claims of better performance by using multiple threads. This is usually not so much true when dealing with an immediate mode drawing system, but with a retained mode system, this may be true.

Pixel Shaders – WPF supports the ability to apply pixel shaders on any WPF visual. These will all run on the GPU, which are incredibly fast on modern graphics cards.

Software Rendering Fallback – I’m not very familiar with WPF’s software rendering capabilities. Mostly because if it was rendering in software it meant something was wrong with my computer. I will say that is very slow. A user interface would have to be designed very carefully if they were targeting computers that only supported software rendering.

Multimedia

MediaElement – Don’t let the simplistic API of this class fool you. Underneath it is very complex and has tremendous amounts of moving parts. Most of those moving parts are actually reuse of other media frameworks that come with windows. Let me walk you through what happens when you set “MediaElement.Source = ‘My.wmv’”.

The first thing that happens is an instance of the Windows Media Player ActiveX is created. Then WMP is told the media Uri to play. Based on the media type and file type, WMP will make a decision to use the DirectShow media pipeline or the newer MediaFoundation. You won’t ever actually see the WMP ActiveX because it is never displayed. Instead, DUCE initializes the WMP ActiveX to use the Enhanced Video Renderer or EVR. The EVR is in turned configured to use something called a custom “Presenter”. This custom presenter allows DUCE to receive a Direct3D surface with the entire decoded video frame, which then can be composited in the WPF user interface.

There are some hidden advantages here. The first is that WMP can use MediaFoundation or DirectShow. This allows MediaElement to play nearly any media the computer supports. If media is not supported, installing a DirectShow filter or a MediaFoundation transform would be enough to enable playback. The second advantage is the use of the EVR. Because the EVR supports DXVA, when possible it can decode the video right on the GPU (usually h.264 and VC1), leaving much more cycles for your CPU.

Text – I’m not an expert on text what so ever. But WPF 4.0 uses DirectWrite to render text. DirectWrite is Microsoft’s latest and greatest in terms of text. It is what is used to give IE9 beautiful anti-aliased text and does the same for WPF.

Silverlight
A Brief History

Like most others, when I talk about Silverlight, I usually skip over Silverlight 1.0 as it wasn’t much more than Javascriptable-XAML (IMHO). Silverlight 2.0 was a subset of WPF. It had most of the same UI capabilities, such as retained mode graphics, vectors and animations. Silverlight’s original name was WPF/Everywhere. The keyword “everywhere” is so broad, that to me, it really implied “lightweight and portable”.

Rendering Stack Features

Software Render – Because WPF/E…er…Silverlight needed to run “everywhere” it was probably an important design goal to have a fast software renderer for graphics. The main reason is because the CPU is the lowest common denominator. Not all systems that Silverlight would end up running on would have an accelerated GPU. As of Silverlight 4.0, every single visual, color and gradient and shader is rendered by the CPU. Because of this, operations like animating a gradient in Silverlight might be a pretty intensive task. Luckily Silverlight’s renderer is multicore ready and will use all CPU cores to render a visual.

Pixel Shaders – Silverlight supports pixel shaders, but they are also rendered via the CPU and take advantage of multiple cores. The Silverlight shader implementation is highly optimized and is most likely parts are written in assembly.

Partial GPU Acceleration – As of Silverlight 3.0, partial GPU acceleration was added. This means some operations can be offloaded to the GPU to do, therefore increasing performance. These operations are transforms (e.g. translate, scale, perspective added in 4.0). When “EnableGPUAcceleration” is set to true, the entire Silverlight user interface is still rendered by the CPU, but it is presented with Direct3D9. If a visual has been configured to use BitmapCache, the visual will copy (cached) to a Direct3D surface. When a supported hardware accelerated operation happens on the BitmapCache, the GPU will just apply the change and rerender the scene without the CPU having to rerender any dirty regions.

Multimedia – The multimedia pipeline in Silverlight is essentially a ported version of MediaFoundation found in Windows and possibly includes the same codebase for some media codecs and demuxers (e.g. , vc-1, h.264, wmv, mp4). The desktop Silverlight as of 4.0 does not include any GPU decoding functionality.

Webcam/Mic ­– Silverlight 4.0 included the ability to get access to the webcamera and microphone. Internally this is done via DirectShow. DirectShow was probably picked for compatibility reasons (like XP) as Windows 7 only just got webcam support in MediaFoundation. I imagine there was probably a custom DirectShow renderer used to do color space conversions before the video frame bytes are copied to managed memory.

Conclusion/Commentary

WPF and Silverlight have very different design goals and this is very evident in their constituent rendering stacks. Where WPF’s focus is performance and richness, Silverlight is a balance between richness and reach. Silverlight must be cross platform and stay some-what consistent, so in the context of rendering stack, it takes a lot longer to develop abstraction layers that work across operating systems. For example, it might be a while until you see Silverlight use DirectWrite for text.

In the end, both serve their different purposes extremely well. In my opinion, there should never be a reason to see WPF and Silverlight “converge” or to see Silverlight take over WPF. Silverlight, which runs on OSX, surely wouldn’t beat Cocoa in a shootout…why would it beat out WPF?  On the other side, WPF on the web sure wouldn’t make too much sense (sorry XBAP).  Why would you want a desktop sized UI platform on a mobile phone?  I think you wouldn’t

-Jer