|
|
May
20
Written by:
Jeremiah Morrill
5/20/2007 5:27 PM
Native Pointer to Bitmapsource's Buffer Last post I covered how to get reference to the WPF DirectShow graph and inject your own filter chain. Like I stated, I think this approach is sort of dirty. I say that because it still lacks the control and flexibility needed. So what if we could get direct access a BitmapSource's native image buffer? You know one exists, though the WPF API hides it (probably for good reason). If we could efficiently update a BitmapSource's buffer, we could do fast image processing (brightness, contrast, hue, etc) or even make our own MediaPlayer in WPF without having to even use DirectShow! Well Friday night, after the bar, I set out with the goal of getting that BitmapSource's buffer. After about six hours and a hangover later, I got what I was looking for. Let me tell you, it was a crash course in Reflection and WIC APIs. Let me give a short explaination on how this is done. WPF uses the unmanaged WIC in the backend for imaging, so it would make sense that WPF keeps reference somewhere to the WIC handle. At least this is what I was guessing. Poking around with Reflection, I found the WIC handle I was looking for in a private field in the BitmapSource class. I then did some research into WIC on how to lock the bitmap's pixels and get the native buffer. Plugging in all the pieces, I was able to get the pixel's pointer! Doing some tests, writing to the buffer, I found that my xaml Image that held my hacked BitmapSource, was not changing. :(. I ran a Image.InvalidateVisual() and the change rendered! I did some tests, writing a 640x480 at 30 times/sec resolution buffer to the BitmapSource's buffer. There were zero page faults, RAM usage didn't move at all, but the CPU usage was around 7%. This leads me to believe that I am indeed copying over the same buffer each time and the CPU usage is due to either my lack of WIC/WPF knowledge or that is the hit at which WPF rendering takes for doing such operations. Now we can create high-performance imaging routines in WPF. We can make custom DirectShow renderers that take our native BitmapSource buffer and write to it. We can implement our own players w/o DirectShow. So many implementations! The only caveat as of now is I seem to only be able to make this work with RGB32 pixel format. **UPDATE: Other's have claimed to be able to use other color spaces. Play around, you might get lucky Also DO NOT FORGET TO RUN Image.InvalidateVisual() after you update the BitmapSource's buffer or you will not see anything change! **UPDATE: After Image.InvalidateVisual, make sure you get the pointer from the BitmapBuffer.BufferPointer property. The get{} relocks the bitmap pixels. Here is the example usage of the class (Source for WPFUtil.BitmapBuffer class in file attached at the bottom) int width = 640;
int height = 480;
int bpp = 4;
//These are 'dummy' pixels
//only used to create our bitmap
//further editing after bitmap creation
//of these pixels does nothing, that is
//what this hack is for
byte[] pixels = new byte[width*height*bpp];
//Create a new bitmap source
BitmapSource bmpsrc = BitmapSource.Create(width,
height,
96,
96,
PixelFormats.Bgr32,
null,
pixels,
width * bpp);
//Set our Image in our Xaml to our
//new bitmap
TestImage.Source = bmpsrc;
//Create our helper class
buf = new WPFUtil.BitmapBuffer(bmpsrc);
//Woo a pointer!
IntPtr buffPointer = buf.BufferPointer;
BitmapBuffer.cs - Get native pointer to BitmapSource's buffer
Tags:
31 comments so far...
Re: WPF Hackery! Part II
I just tried it awhile ago and it works great! We needed alphablending so I changed the PixelFormats.Bgr32 to PixelFormats.Bgra32 and it worked like perfectly.
I wish we didn't need to do hacks like this, but until Microsoft gives us better access to what is under the covers we are going to have to.
-Eric
By Eric Meyer on
5/21/2007 10:13 AM
|
Re: WPF Hackery! Part II
I have a co-worker who showed me how you could access private fields in C# via reflection. I come mostly from a Java background were this would be a security violation, is this really allowed by the CLR, you have acess to any private fields if you use reflection?
By Augusto on
5/21/2007 11:23 AM
|
Re: WPF Hackery! Part II
I'm not really a Java expert, but quick Google search shows that Java has the same behavior with reflection.
I do not believe this to be a security violation (unless you see it as an application level security violation for 3rd party libraries). DotNet (and presumably Java), seem to treat access modifiers as language constructs, not a runtime principal.
In any case, I think you can configure CAS to turn off reflection, disallowing such access (or not, I know very little about CAS and probably shouldn't be remarking about it).
By JMoney on
5/21/2007 5:52 PM
|
Re: WPF Hackery! Part II
I'm glad it worked for you Eric! I would double check my work a little and make sure there is no memory/handle leak. 90% of that class was done via guess-and-check, so there's no guarentees.
-Jer
By JMoney on
5/21/2007 5:58 PM
|
Re: WPF Hackery! Part II
Nice code, here's something that might improve performance, you're doing a call to SetBufferInfo within your BufferPointer getter. Also, you should store your MethodInfo and Type objects in static variables so you only need to initialize them once. Other than that really great.
Also, the only security concern with reflection is having code that is not privileged reflecting and executing a function that is. (For instance a service that is running with rights allowing it for instance to reset user passwords for your application). Like Jeremiah noted, this can be prevented using CAS.
Java does indeed allow reflection, it's how EJBs and JavaBeans in general are done. Indeed there were a few interesting quirks in the first version of J2EE where code installed in an EJB container would be able to use it to raise its privileges through reflection.
By Mike Brown on
5/22/2007 2:53 PM
|
Re: WPF Hackery! Part II
We want to make sure that the platform let's you do what you want without having to resort to things like this. It cannot be guaranteed that non-publc API access will continue to work compatibly in future versions.
I'll make sure that I point this out to the right people on our team so that we can handle your scenarios better in the futre.
Thanks, Rob Relyea
Program Manager, WPF Team
http://rrelyea.spaces.live.com
By Rob Relyea on
5/22/2007 3:38 PM
|
Re: WPF Hackery! Part II
Hi Rob,
I frequent your blog quite often!
I know this type of code isn't really condoned, but I felt the functionality is so important and necessary that I had to share.
Thanks for listening to this "cry-for-help". I think there are a lot of people out there starving for low-level access to WPF/MIL. Before I wrote this code, I think I would have sold my soul for a buffer pointer.
-Jer
By JMoney on
5/22/2007 8:29 PM
|
Re: WPF Hackery! Part II
I was wondering if you were able to post an actual example of this in use (ie following on from getting the pointer to the buffer). I am struggling to get it working.
By Steve Cassidy on
6/2/2007 10:09 PM
|
Re: WPF Hackery! Part II
I make use of this class in the other project I posted about putting Win32 controls in the WPF compositing engine. It should provide for a decent example.
Here is the URL.
http://jmorrill.hjtcentral.com/Home/tabid/428/EntryID/20/Default.aspx
By JMoney on
6/4/2007 8:47 AM
|
Re: WPF Hackery! Part II
There is another way.
You can derive from BitmapSource, and write your own code to compute the output image on the fly.
Such an object could be very lightweight, so instead of manual invalidating, you can just overwrite the pointer to the BitmapSource to a new one.
You are required to write three copy-out functions,
you must make the object clonable.
You must be prepared to run your code from another thread than the UI thread (i consider this an advantage).
By Jan de Vaan on
6/28/2007 1:00 PM
|
Re: WPF Hackery! Part II
Congratulation for this fantastic trick.
I'm playing with a webcam and with a DTV on Windows XP SP2, and I'm getting some tearing (vsync) issues :(
Here my frame update code:
unsafe int ISampleGrabberCB.BufferCB(double sampleTime, IntPtr buffer, int bufferLength)
{
System.Windows.Controls.Image image = this.hostingControl.WPFImage;
image.Dispatcher.BeginInvoke(DispatcherPriority.Render, new ThreadStart(delegate
{
CopyMemory(wpfBitmapBuffer.BufferPointer, buffer, bufferLength);
image.InvalidateVisual();
}));
return 0;
}
Do you no a way to synchronize the "InvalidateVisual()" with the monitor vsync?
By Dgis on
8/3/2007 4:00 AM
|
Re: WPF Hackery! Part II
Under Vista it seems to be synchronised!!!
No vsync issue on vista.
By Dgis on
8/3/2007 5:28 AM
|
Re: WPF Hackery! Part II
Under Vista it seems to be synchronised!!!
No vsync issue on vista.
By Dgis on
8/3/2007 5:29 AM
|
Re: WPF Hackery! Part II
Under Vista it seems to be synchronised!!!
No vsync issue on vista.
By Dgis on
8/3/2007 5:29 AM
|
Re: WPF Hackery! Part II
Under Vista it seems to be synchronised!!!
No vsync issue on vista.
By Dgis on
8/3/2007 5:29 AM
|
Re: WPF Hackery! Part II
I have not tested this on XP, but others using this code on XP have not reported any issues. If you are using a DirectShow based solution, maybe you should try the WPF Media Bridge (http://jmorrill.hjtcentral.com/Home/tabid/428/EntryID/15/Default.aspx). It provides the best performance and (sometimes) the best compatibility. Can you describe the tearing? Is it flickering on you?
By Jeremiah Morrill on
8/3/2007 12:01 PM
|
Re: WPF Hackery! Part II
I understood what's happen. Basically the issue is not from your trick, this is WPF which is not vertically synchronized on Windows XP SP2. On Vista, WPF is well synchronized. By "not vertically synchronized", I mean the WPF animation can be updated in the middle of the screen, inducing abreaking in the picture (like a game with the vsync switched off) . So maybe the fact of waiting for the vertical blank before calling InvalidateVisual() could solve this issue. I'm investigating the WaitForVerticalBlank() directx function to fix it. I do not try your "WPF Media Bridge" which seems to be another good idea, but I think it will produce the same issue, because the problem is in WPF.
By Dgis on
8/4/2007 2:45 PM
|
Re: WPF Hackery! Part II
This is a nice hack, but it still requires you to copy an entire bitmap to the BitmapSource. Has anyone found a way to invalidate just a portion of the bitmap so you don't have to move all that data every frame? I incorporated this technique in a project, and it works like a charm. But, even with the smallest changes in the bitmap, there's a lag caused by all the copying going on in the background. If the user wants feedback, for instance, if you have icons on a background image, and wanted a mouseover selection rectangle to show up, it takes about a second for it to appear. Right now, making each icon a WPF image object is not an option, so I was wondering if anyone here has found a way to do GDI-speed drawing (and region invalidation) in WPF.
By d on
8/22/2007 11:23 AM
|
Re: WPF Hackery! Part II
This hack does not require you to copy an entire bitmap. That is only done once on initialization.
You can update portions of the bitmap if you want. The greatest overhead is with WPF when you run the Invalidate() on the image.
By Jeremiah Morrill on
8/22/2007 11:45 AM
|
Re: WPF Hackery! Part II
This example shows me rendering directly on to a bitmap:
http://jmorrill.hjtcentral.com/Home/tabid/428/EntryID/72/Default.aspx
By Jeremiah Morrill on
8/22/2007 11:53 AM
|
Re: WPF Hackery! Part II
Ok, obviously I need to take another look at my code. The way I wrote the code was probably based on some wrong assumptions about what WPF was doing with its internal image buffer after an invalidate. I'll have to do more investigating. Thanks for the input.
Yes, I saw the water demo. It's very impressive, especially given WPF's limitations. But I wonder how well it would perform on, say, a 1280x1024 backdrop... In one of my tests, I'm blitting a 4000x4000 image to ~1500x900 sized wpf Image, and dragging it around with a mouse. I'm getting around 5 fps or so on my P4 3ghz laptop.
By d on
8/22/2007 2:24 PM
|
Re: WPF Hackery! Part II
Jeremiah, may I ask you how you are drawing to the buffer? The way I'm doing it is by creating a Bitmap using the IntPtr to the internal buffer, and then creating a graphics object out of that. When you're just calling DrawImage, it works perfectly. However, when you call functions like Clear or FillRect before you call DrawImage, there are times when the Image doesn't draw at all. This occurs more often the larger the window you're trying to draw on. I hope it's just the method I'm using to create the Graphics object. Otherwise, this technique might not be usable in most general use cases. Has anyone else noticed this behavior?
By d on
8/22/2007 8:34 PM
|
Re: WPF Hackery! Part II
Out of curiousity, when pixels are copied to the image's buffer and InvalidateVisual is called, would the buffer then be copied up to the graphics card (or some other location) for blitting? You mentioned that 7% CPU usage, and I'm wondering if that's caused by the full buffer being copied to the graphics card. Not sure if anyone here would know the details of how WPF handles its buffers, but just throwing the question out there.
By Dan on
8/28/2007 1:31 PM
|
Re: WPF Hackery! Part II
I'm only going to guess here...I THINK I heard on a Channel9 video that WPF keeps two copies of the media. In ram and on the video ram. I think that copying it to the GPU on Invalidate() might be where the overhead is. [Guessing again,] I think that the MediaElement renders directly to the the video card and is why it uses less CPU.
By Jeremiah Morrill on
8/28/2007 1:38 PM
|
Re: WPF Hackery! Part II
Jeremiah,
I've been using the bitmap hack for several months now and it's been working OK (although I really wish I could improve the performance a little more).
I've just discovered that the latest .NET 3.0 makes InteropBitmap a public class. I'm pretty sure you can now accomplish much the same thing as your bitmap hack without resorting to reflection. The basic steps are as follows:
1.) You need to create an unmanaged memory buffer to store the bitmap data. Create a memory section using the API call CreateFileMapping, and then get a pointer to the buffer using MapViewOfFile. The resultant pointer will always be the location of the bitmap data.
2.) Create a bitmap from the buffer using System.Windows.Interop.Imaging.CreateBitmapSourceFromMemorySection. This returns a BitmapSource which will store it's pixel data at the address you specify. It can never move the buffer on you.
3.) The BitmapSource is actually an InteropBitmap. Calling the Invalidate function on InteropBitmap forces the display to update. Copy your frame data into the memory you allocated, and call Invalidate on the InteropBitmap and it all works.
...the resultant performance seems identical to your bitmap hack...but it's nice not to have to resort to reflection and internals that we have no control over.
If you search the WPF forums you'll find a more detailed example.
By Todd on
12/8/2007 10:17 AM
|
Re: WPF Hackery! Part II
Jeremiah,
I've been using the bitmap hack for several months now and it's been working OK (although I really wish I could improve the performance a little more).
I've just discovered that the latest .NET 3.0 makes InteropBitmap a public class. I'm pretty sure you can now accomplish much the same thing as your bitmap hack without resorting to reflection. The basic steps are as follows:
1.) You need to create an unmanaged memory buffer to store the bitmap data. Create a memory section using the API call CreateFileMapping, and then get a pointer to the buffer using MapViewOfFile. The resultant pointer will always be the location of the bitmap data.
2.) Create a bitmap from the buffer using System.Windows.Interop.Imaging.CreateBitmapSourceFromMemorySection. This returns a BitmapSource which will store it's pixel data at the address you specify. It can never move the buffer on you.
3.) The BitmapSource is actually an InteropBitmap. Calling the Invalidate function on InteropBitmap forces the display to update. Copy your frame data into the memory you allocated, and call Invalidate on the InteropBitmap and it all works.
...the resultant performance seems identical to your bitmap hack...but it's nice not to have to resort to reflection and internals that we have no control over.
If you search the WPF forums you'll find a more detailed example.
By Todd on
12/8/2007 10:17 AM
|
Re: WPF Hackery! Part II
Hi,
I wanted to develop a WPF project that has very intense performance requirements, and I thought it was best to consult you on this before designing the app.
The application would be rendering data obtained from a medical hardware as charts. For this, the requirements are:
1) Requires rendering of almost 250 points every 4 milliseconds in a chart (to be connected by a line or Bezier curve). The rendered points will also have a spectrum of colors (as in a heatmap).
2) The rendered points would move when new data is pumped by the hardware and rendered (as in a ECG)
3) At any given point there could be over a 150,000 moving points on the chart (because the chart window would display data of 3-5 seconds).
4) Also later we would like to retain the history of data points of past 60 seconds or so.
5) There could be upto 12 such chart displays running parallel at the same time in a window on the application.
I seek your thoughts on what could be the best approach for rendering these points and animating them. Some kind of selectively invalidating the UI would be desirable here. Also what are the recommended options for storing this much data for history purpose. The problem demands something at a very low level in the WPF stack.
Any Help would be highly appreciated,
Thanks in Advance
By Nirupama on
5/6/2008 5:53 AM
|
Re: WPF Hackery! Part II
Hi..
Thsi is a gentle reminder..for the above query...
Please do give in your inputs...That would be of great help..
Thanks and Regards,
Nirupama
By Nirupama on
5/26/2008 6:13 AM
|
Re: WPF Hackery! Part II
Hello
downloading mp3s
mp3 download free
love songs
By hell on
6/21/2008 1:20 AM
|
Re: WPF Hackery! Part II
Beware of the first sight conclusions, often they can be wrong.tramadol
By mesa on
7/25/2008 5:49 AM
|
Re: WPF Hackery! Part II
I think you worry too much on this subject but you posted a good question there !
acomplia
By acomplia on
8/18/2008 5:18 AM
|
|
|
Text/HTML
|
 |
|
|
|
|
|