WPF Hackery!

WPF MediaKit, SlimDX and 64bit

Sun, 07 Sep 2008 22:42:55 GMT

I run Vista x64 on my development computer, so it's pretty easy for me to test compatibility with 32bit and 64bit when I'm coding. While coding some more on the WPF Media Kit last night, I decided that I wanted the WPF MediaKit to support 64bit too. This of course is impossible with the MDX libraries. The MDX libraries are written in .NET 1.1 and compiled for 32bit only. Weak huh? So based on a suggestion from one of you guys, I checked out SlimDX, which is a managed wrapper for DirectX. The drawback was they didn't seem to have any 9Ex support, which is a must for Vista performance. Since it was open-source, I was just going to add my own support. While downloading the source from their SVN, I noticed that they have indeed added 9Ex support!

I ported all the MDX code over to SlimDX, which was fairly painless, though the API is slightly different, it was easy to navigate. Now the WPF MediaKit supports 64bit and no more .NET 1.1 MDX! Now if there were only more 64bit DirectShow filters...

Oh yeah, no more MDA LoaderLocks "exceptions" either :)

-Jer

D3DImage, Direct3D - The device is lost? Where'd it go?

Fri, 05 Sep 2008 11:16:08 GMT

One big bug in the "WPF MediaKit" that I've really been putting off has handling the Direct3D device when it's lost. Well I think I've got it licked! See, a standard IDirect3DDevice9 will be lost (meaning you can not use it anymore and all resources in the default pool have to be recreated) in many different scenarios. For instance if you change your display resolution, or if you switch to another user or even hit ctrl-alt-del. In Vista, I use the IDirect3DDevice9Ex, which is simply amazing. The device is only lost when very bad things happen, such as a hung video driver or a new video driver has been installed.

Naturally, I thought that I wouldn't have to do any work for Vista if the device is pretty much never lost. But when I changed my video resolution my video disappeared! After ripping some hair out, it seems as if the D3DImage drops it's Surface backbuffer. With some help from D3DImage's event IsFrontBufferAvailableChanged, all was fixed with Vista.

For good form's sake...and the sake of XP, I wanted to handle the IDirect3DDevice9 when it is lost. With a handful of strategically placed lines of code, all looks good. I don't really have an XP machine to test it on, but I forced Vista to not use the 9Ex, changed the resolution. It caught the lost device and reinitialized it and the video continued on.

Also, I was forced to move the DirectShow stuff from STA or MTA, to MTA only. This is because a DirectShow has it's own thread running the PresentImage in the allocator that needed to talk to other COM and STA wasn't cutting it.

I'll try to upload a new version this weekend so you guys can tell me how it works :)

-Jer

WPF MediaKit Preview

Thu, 04 Sep 2008 14:11:36 GMT

I'm notorious for showing off things before they are done. So keeping with my notoriety, I have decided to give a sneak preview of the WPF MediaKit. Keep in mind most things are largely untested. I don't restore a lost D3D Device just yet and don't expect the sample application to be pretty. I did spend a lot of time commenting everything, so it shouldn't be too bad to navigate.

Right now I have two "example" controls, the MediaUriElement and the VideoCaptureElement. I just threw the VideoCaptureElement together very quickly, so hopefully it works off the bat.

Ignore or turn off (in VS Debug options) the MDA LoaderLock "exceptions". These are not really exceptions!

http://jmorrill.hjtcentral.com/portals/21/blog/files/wpfdshow.zip

Refactoring until you can't abstract no mo'

Sat, 30 Aug 2008 19:48:47 GMT

In my last post about the WPF MediaKit library, I mentioned a little about the design goals and architecture. I have two distinct, but complimentary class hierarchies. The "MediaPlayerBase" class tree wraps DirectShow COM functionality into a DispatcherObject, running in it's own Dispatcher. The "MediaElementBase" wraps the "MediaPlayerBase" into a nice XAML friendly, data-bindable FrameworkElement.

Today the class inheritance looks like this:

On the DirectShow centric side

DispatcherObject->MediaPlayerBase->MediaSeekingPlayer

On the WPF centric side

FrameworkElement->D3DRenderer->MediaElementBase->MediaSeekingElement

If you notice, I have a "MediaSeekingPlayer". I decided to put the basic functionality of the DirectShow graph (Play, Pause, Stop) into the base class and the media position functionality into it's own sub class. I did this because it doesn't make sense to have seeking functionality on a live or unseekable source, such as a web cam.

So with these classes I wanted to write a MediaElement clone (aka a MediaUriElement). Well, not a total clone, but the same concept of playing any Uri, with maybe some extra features, like being able to set the out output device or seeking by frames, time, bytes, samples, etc (great for super-duper smooth rewind). I wanted to show how much source code it will really take to implement your own custom video. Here source code to make this screenshot happen.

<DShow:MediaUriElement 
    x:Name="mediaPlayer" 
    Source="c:\users\jmoney\desktop\sensor.wmv" 
    LoadedBehavior="Pause" Stretch="Uniform" 
    PreferedPositionFormat="Frames"/>

On the DirectShow Side

public class MediaUriPlayer : MediaSeekingPlayer
    {

#if DEBUG
        /// 
        /// Used to view the graph in graphedit
        /// 
        private DsROTEntry m_dsRotEntry;
#endif

        /// 
        /// The DirectShow graph interface.  In this example
        /// We keep reference to this so we can dispose 
        /// of it later.
        /// 
        private IGraphBuilder m_graph;

        /// 
        /// The VMR9 video renderer
        /// 
        private IBaseFilter m_renderer;

        /// 
        /// The media Uri
        /// 
        private Uri m_sourceUri;

        /// 
        /// Gets or sets the Uri source of the media
        /// 
        public Uri Source
        {
            get
            {
                VerifyAccess();
                return m_sourceUri;
            }
            set
            {
                VerifyAccess();
                m_sourceUri = value;
                OpenSource();
            }
        }

        /// 
        /// Opens the media by initializing the DirectShow graph
        /// 
        private void OpenSource()
        {
            string fileSource = m_sourceUri.OriginalString;

            /* Make sure we clean up any remaining mess */
            FreeResources();

#if DEBUG
            /* Remove me - just testing stuff */
            GC.Collect();
            GC.WaitForPendingFinalizers();
#endif
           
            try
            {
                /* Creates the GraphBuilder COM object */
                m_graph = new FilterGraph() as IGraphBuilder;

                /* Create and initialize a video renderer with a
                 * custom allocator.  Right now we only support VMR9
                 * but the EVR will be supported in the future */
                m_renderer = CreateVideoRenderer(RendererType.VideoMixingRenderer9);

                /* We add the renderer to the graph first
                 * and the GB will connect it automagically 
                 * when we render the file */
                int hr = m_graph.AddFilter(m_renderer, "VMR9 Renderer");
                DsError.ThrowExceptionForHR(hr);

                /* DirectShow now automatically assembles the
                 * graph and filters based on the media type */
                hr = m_graph.RenderFile(fileSource, null);
                DsError.ThrowExceptionForHR(hr);

#if DEBUG
                /* Adds the GB to the ROT so we can view
                 * it in graphedit */
                m_dsRotEntry = new DsROTEntry(m_graph);
#endif
                /* Configure the graph in the base class */
                SetupFilterGraph(m_graph);

                /* Remove and dispose of renderer if we do not
                 * have a video stream */
                if(!HasVideo)
                {
                    m_graph.RemoveFilter(m_renderer);

                    /* Tells the base class to unregister and
                     * free the custom allocator */
                    FreeCustomAllocator();

                    Marshal.FinalReleaseComObject(m_renderer);
                    m_renderer = null;
                }
            }
            catch (Exception ex)
            {
                /* This exection will happen usually if the media does
                 * not exist or could not open due to not having the
                 * proper CODECs installed */
                FreeResources();

                /* Fire our failed event */
                InvokeMediaFailed(new MediaFailedEventArgs(ex.Message, ex));
            }

            InvokeMediaOpened();
        }

        /// 
        /// Frees all unmanaged memory and resets the object back
        /// to its initial state
        /// 
        protected override void FreeResources()
        {
            /* Let's clean up the base class's stuff first */
            base.FreeResources();

#if DEBUG
            /* Remove us from the ROT */
            if (m_dsRotEntry != null)
            {
                m_dsRotEntry.Dispose();
                m_dsRotEntry = null;
            }
#endif
            if (m_renderer != null)
            {
                Marshal.FinalReleaseComObject(m_renderer);
                m_renderer = null;
            }
            if (m_graph != null)
            {
                Marshal.FinalReleaseComObject(m_graph);
                m_graph = null;

                InvokeMediaClosed();
            }
        }

        /// 
        /// Creates a new MediaUriPlayer, 
        /// running on it's own Dispatcher
        /// 
        public static MediaUriPlayer CreateMediaUriPlayer()
        {
            MediaUriPlayer engine = null;

            var reset = new ManualResetEvent(false);

            var t = new Thread((ThreadStart)delegate
            {
                engine = new MediaUriPlayer();

                /* We queue up a method to execute
                 * when the Dispatcher is ran. 
                 * This will wake up the calling thread. */
                engine.Dispatcher.Invoke((Action)(() => reset.Set()));

                Dispatcher.Run();
            }) { 
                Name = "MediaUriPlayer Graph",
                IsBackground = true
            };

            /* We can run in MTA also */
            t.SetApartmentState(ApartmentState.MTA);

            /* Starts the thread and creates the object */
            t.Start();

            /* We wait until our object is created and
             * the new Dispatcher is running */
            reset.WaitOne();

            return engine;
        }
    }

And on the WPF Side

public class MediaUriElement : MediaSeekingElement
{
    public MediaUriElement() : base()
    {}

    protected MediaUriPlayer MediaUriPlayer { get; set; }

    #region Source

    public static readonly DependencyProperty SourceProperty =
        DependencyProperty.Register("Source", typeof(Uri), typeof(MediaUriElement),
            new FrameworkPropertyMetadata(null,
                new PropertyChangedCallback(OnSourceChanged)));

    public Uri Source
    {
        get { return (Uri)GetValue(SourceProperty); }
        set { SetValue(SourceProperty, value); }
    }

    private static void OnSourceChanged(DependencyObject d, DependencyPropertyChangedEventArgs e)
    {
        ((MediaUriElement)d).OnSourceChanged(e);
    }

    protected void OnSourceChanged(DependencyPropertyChangedEventArgs e)
    {
        MediaPlayer.Dispatcher.BeginInvoke((Action)delegate
        {
            MediaUriPlayer.Source = e.NewValue as Uri;

            bool isLoaded = false;

            Dispatcher.Invoke((Action)(() => isLoaded = IsLoaded));
            if (isLoaded)
            {
                MediaState state = MediaState.Manual;
                Dispatcher.Invoke((Action)(() => state = LoadedBehavior));
                ExecuteMediaState(state);
            }
        });
    }

    #endregion

    /// 
    /// Gets the instance of the media player to initialize
    /// our base classes with
    /// 
    protected override MediaPlayerBase OnRequestMediaPlayer()
    {
        MediaUriPlayer = MediaUriPlayer.CreateMediaUriPlayer();
        return MediaUriPlayer;
    }
}

So hopefully I'll have this pretty much finished by this weekend. We'll see. I'll keep ya posted.

-Jer

WPF Multimedia Library, COM and DispatcherObject

Wed, 27 Aug 2008 22:13:11 GMT

A few of you have emailed, wondering if I'm still alive, or if I'm working more D3DImage video player. The answer is "YES!".

I have been very busy with work, but I found some time here and there to work on what I am calling (for the lack of a better name), the "WPF Multimedia Library". Sounds pretty self explanatory, no? The quick synopsis, it's a lib to quickly and easily create custom video/audio players in WPF. And hopefully some XAML enabled sound processing controls, though I have to check to see if I can release that part(it may be part of a closed source app).

So, yeah, the multimedia library, I mentioned quick and easy? I suppose those are both subjective terms, but I have been spending a lot of time refactoring and massaging the architecture to make it as easy as possible. Let me explain a little of where I'm at.

COM and DispatcherObject sitting in a tree...

The first thing I wanted to tackled was a tidy way of communicating with our DirectShow COM objects. See, COM is subject to threading apartment rules of Single-Threaded-Apartment or Multi-Threaded-Apartment. To be said simply an object created in STA can only be accessed via the same thread that created it. An object created in MTA can be called from any *MTA* thread. An STA thread can not talk to an MTA object and vise-versa.

So this MTA/STA ish sounds like a problem in WPF because it uses an STA UI thread and you don't want to create the DirectShow COM objects on the UI thread for threat of blocking the UI thread (bad!) when you make blocking calls to the COM objects. But couldn't we create all the COM objects under an MTA thread? Yes, but the WPF STA thread won't be able to access them (see previous paragraph). The problem is solved by sub-classing DispatcherObject and running the Dispatcher on it's own thread. To access your COM safely, you use the familiar WPF Dispatcher->Invoke/BeginInoke. So now all COM functionality (DirectShow in this case) is neatly wrapped up in it's own DispatcherObject. Here is an example factory method:

/// 
/// Creates a new DirectShowEngine, 
/// running on it's own Dispatcher
/// 
public static DirectShowEngine CreateDirectShowEngine()
{
    DirectShowEngine engine = null;

    var reset = new ManualResetEvent(false);

    var t = new Thread((ThreadStart) delegate
     {
         engine = new DirectShowEngine();

         /* We queue up a method to execute
          * when the Dispatcher is ran. 
          * This will wake up the calling thread. */
         engine.Dispatcher.Invoke((Action)(() => reset.Set()));

         Dispatcher.Run();
     }) {Name = "DirectShow Graph"};

    /* We can run in MTA also */
    t.SetApartmentState(ApartmentState.STA);

    t.IsBackground = true;

    /* Starts the thread and creates the object */
    t.Start();

    /* We wait until our object is created and
     * the new Dispatcher is running */
    reset.WaitOne();

    return engine;
}

Interfaces and Base Classes

Now that the COM crap was taken care of, I had to come up with a extensible way to make custom WPF DirectShow players. This was sort of tough considering the different ways one would want to use or implement DirectShow. So I started writing a WPF abstract control (eh, FrameworkElement), that was similar to the MediaElement API (minus the Source DP). I chose to start there because MediaElement's usage is pretty simple and it covered the most common usages of media playback in WPF. Next, I refactored around the aforementioned DispatcherObject base class to encapsulate all the functionality of my FrameworkElement control. To put it simply, the FrameworkElement control is a wrapper for the custom DispatcherObject that houses the COM.

Putting it Together

So now that I have the FrameworkElement, MultimediaElementBase, and the DispatcherObject, MediaPlayerBase, that it wraps, I was ready to build a player that plays media files. The first step was to sub-class MediaPlayerBase and create a property to send the Uri name and the DirectShow functionality to play the Uri. Next I sub-classed the MultimediaElementBase and added a "Source" Uri, registered the custom MediaPlayerBase with the MultimediaElementBase. Wired in the "Source" Uri DP to change the source on the custom MediaPlayerBase, and viola! It may sound like a lot, but really it's ~70 lines of code (not counting comments) for your own custom "MediaElement".

Not quite there...

There's still a bit more work to do, lots of refactoring left. There's still other features I need to add, such as being able to select the audio output device or getting a list of available audio devices, etc. But I'll get this out as soon as I can and it'll be cool. I promise!

-Jer

XNA Quest of IDirect3DDevice9Ex

Sun, 17 Aug 2008 14:34:09 GMT

I have been toiling more with the XNA libs, trying to get the IDirect3DDevice9Ex injected in...And boy, its not easy! Specially since there is so much unmanaged code in the XNA libs and the unmanaged code it's calling isn't so obvious via Reflector.

I started off with the ILDASM tool to find the byte patterns of the IL I wanted to change. Then with my trusty hex editor, I searched for the bytes and replaced them.

So far I have been replacing the hard-coded D3DPOOL_MANAGED to D3DPOOL_DEFAULT where applicable. This involves changing the IL command "ldc.i4.1" (byte value 0x17) with the IL command "ldc.i4.0" (byte value 0x16). Though I have to be careful because the XNA content manager seems to also use the D3DPOOL_SYSTEMMEM. For some reason using the content manager seems to make it crash right now, but if I don't use it, things seem to be fine so far! The screenshot (though not perfect yet) is XNA running a 9Ex device!

I think if I can get the content manager to work, I should be a step closer.

XNA, Meet WPF

Tue, 12 Aug 2008 10:28:40 GMT

The new D3DImage in WPF opens up worlds for WPF. Direct3D content rendered with WPF...it doesn't get any better! The downside is it seems (correct me if I'm wrong) that Microsoft was only targeting unmanaged Direct3D. Yes there is Managed DirectX, but it seems as if MDX is no longer being maintained and it's not so obvious how to get a D3D surface pointer from the API. It really seems MDX has been left in the dust in favor for XNA, which is a very intuitive D3D wrapper. XNA further exacerbates the issue with XNA and WPF.

A couple of the issues are:

1.) XNA hides all the nasty D3D COM pointers, so you can't get the surface pointer via the XNA API.

2.) XNA creates a IDirect3DDevice9 internally. WPF D3DImage needs a IDirect3DDevice9Ex for maximum performance on Vista.

Initially, my plan was create a 9Ex device, inject it into the XNA GraphicsDevice class via reflection, which *worked* until you tried to load a texture. The XNA libraries are hard coded for a D3DPOOL_MANAGED pool, which is incompatible with the 9Ex device (Thanks Doc!). Ok, so we can live without break-neck performance of 9Ex...for now. So I just focused on getting the surface pointer and setting the render target. Here is a quick snippet for the Microsoft.Xna.Framework.Game subclass. You will need to add reference to the MDX assembly Microsoft.DirectX.Direct3D. Ignore all "LoaderLock" exceptions or just turn them off in Visual Studio. You will also need to throw an event on the Draw override to notify your D3DImage that there is a dirty rect. You should probably also add an event for when a new surface is available also, for when the D3D device is lost.

private Surface m_surface;
private int WIDTH = 300;
private int HEIGHT = 300;

unsafe protected override void BeginRun()

{
    Type deviceType = typeof(GraphicsDevice);
    FieldInfo fi = deviceType.GetField("pComPtr", 
                                        BindingFlags.NonPublic | BindingFlags.Instance);
    object ptr = fi.GetValue(graphics.GraphicsDevice);
    IntPtr pComPtr = new IntPtr(Pointer.Unbox(ptr));

    Microsoft.DirectX.Direct3D.Device dev = new Microsoft.DirectX.Direct3D.Device(pComPtr);

    m_surface = dev.CreateRenderTarget(WIDTH, 
                                       HEIGHT, 
                                       Format.X8R8G8B8, 
                                       Microsoft.DirectX.Direct3D.MultiSampleType.None, 
                                       0, 
                                       System.Environment.OSVersion.Version.Major >= 6);

    dev.SetRenderTarget(0, m_surface);

    base.BeginRun();
}

unsafe public IntPtr GetSurfacePointer()
{
    /* TODO:  Figure out how to convert Surface.UnmanagedComPointer to IntPtr */
    return m_surface.GetObjectByValue(unchecked((int)0xd2b543af));
}

D3DImage VideoPlayer

Mon, 11 Aug 2008 19:52:49 GMT

Everyone grab a copy of .NET 3.5 SP1, its RTM. For this RTM release I have a new DirectShow video player that utilizes the D3DImage, for pretty seamless WPF->DShow interop. The code is a mess, there are big bugs, leaks and I'm using D3DImage slightly incorrectly, but besides that, This is just a rough preview that I'd like anyone who is interested to give it a run and see what the performance characteristics are like or any other feedback.

The demo application has a DVD player, Webcam and file player. SP1 RTM is required.

Download here: http://jmorrill.hjtcentral.com/Portals/21/Blog/Files/D3DImageExample.zip

New WriteableBitmap vs. VideoRendererElement

Wed, 21 May 2008 22:41:37 GMT

I've been playing around with the new WPF 3.5 SP1 beta lately and I wanted to test the performance of the new WriteableBitmap in comparison to the VRE.

The test simply copies the same pixel buffer to the WriteableBitmap or the VRE at a specific interval. This way we can see the raw CPU it takes to just update pixels.

For this test, I use a 1280x720 (720p) pixel buffer at around 30FPS. On my quad-core system, the VRE eats an average of ~1% CPU (it jumps to 2% then to 0% sometimes) and the new WriteableBitmap did an average of ~4.5% CPU usage, which is very good in my opinion. So if we put this in single-core numbers (multiply usage by number of cores), as WPF is STA, we can say that the VRE uses around 4% of the CPU core the rendering/ui thread are running on and the WriteableBitmap uses ~18%.

There are some pros and cons to consider. The upside to the WriteableBitmap is you are supposed to get tear-free video all the time, whereas the MediaElement (the driving force of the VRE) has had tearing problems on some systems (though I consider this a bug in WPF). The WriteableBitmap is also included in the framework and should be perfect for simple pixel updates. The upside to the VRE is obviously CPU usage and the VRE has the ability to update the pixel buffer on any thread, not just UI (Dispatcher) thread. The downside to the VRE is that there is a DShow filter that must be registered and of course, its not built into the framework. But MediaElement is...

Here is the test application I built so everyone can test it out for themselves and check my methods :). To use the VRE make sure to register the .ax file w/ admin privs. You will also need .NET 3.5 SP1 installed.

-Jer

Pavan's ElementFlow - Upgrade

Mon, 28 Apr 2008 16:24:58 GMT

Pavan released his now popular ElementFlow control about a month ago. Dissecting his code was both a pleasure and very educational (It was written much nicer than my version). The coolest part was how he sub-classed a Panel and overlayed a Viewport3D on top. Being a sub-class of Panel, you have all the nice built in WPF functionality that goes along with it, such as using ItemsControl class and data-binding. One feature that I felt was missing from ElementFlow was having interactive 2D in 3D space.

Then Dr. WPF released some powerful code and article on the CodeProject site. Without delving too deep into his article, it is a way to have a Panel with children who can be connected or disconnected in your UI trees. Josh made a nice sample (Panel3D) to show off what you can do with it.

I figured Dr. WPF's LogicalPanel would be a great fit for upgrading Pavan's ElementFlow to support Viewport2DVisual3D classes because without it, you will find some big conflicts because the Visual needs to be a part of Viewport2DVisual3D's tree and the Panel's tree.

After a grueling couple hours, I got everything moved over to be interactive. It wasn't as easy as just renaming GeometryModel3Ds to ViewPort2DVisual3D. Quite a bit of code had to be tracked down and modified! After that was accomplished, the real fun began. Here's a list of changes I have made:

Interactive2D on 3D
Fixed issue where you could only data-bind once
Removed visual reflection support in the control - Reflection is done in the control/data-templates
Added support in ViewState sub-classes to select which animation-prototype to use. Might be bad form, but its decent for now.
Modified animations to use keyframe animation with KeySplines. Animation looks *way* cooler that way ;)
Camera and the light are set via XAML. You can now animate their properties.
Added Spotlight to default example in showcase.

The new ElementFlow is called UIFlow3D. You can download it here: http://jmorrill.hjtcentral.com/Portals/21/blog/fluidkit.zip

Disclaimer: This code is not endorsed by Pavan. He has not even seen it as he is indisposed probably until next week.

-Jer