unexpected WM_PAINT message reentry when waiting for COM request under Vista/7

Recently, we had a weird problem: when CLR is waiting for the COM request result, especially a remoting one, the WM_PAINT message can be processed before it returns.

We are waiting the COM request in rendering thread, and assumes that the code should be blocked and does not consider reenter problem. This really cause lots of trouble.

When the problem happened, the callstack trace looks like below:

 

   1: UIFramework.dll!UIFramework.MainForm.viewport_Paint

   2: System.Windows.Forms.dll!System.Windows.Forms.Control.ControlNativeWindow.WndProc(ref System.Windows.Forms.Message m) + 0x31 bytes    

   3: System.Windows.Forms.dll!System.Windows.Forms.NativeWindow.DebuggableCallback(System.IntPtr hWnd, int msg = 15, System.IntPtr wparam, System.IntPtr lparam) + 0x57 bytes    

   4: ...

   5: user32.dll!_DispatchMessageW@4()  + 0xf bytes    

   6: ole32.dll!CCliModalLoop::HandleWakeForMsg()  + 0x48c30 bytes    

   7: ole32.dll!CCliModalLoop::BlockFn()  - 0x5df7 bytes    

   8: ole32.dll!ModalLoop()  + 0x52 bytes    

...
>   

Initially, I thought it may be related to COM's IMessageFilter interface, which prevents COM deadlock problem when client/server calls each other, just like SendMessage can be interrupted by the non-queue messages when being blocked. But after registering the interface, the problem is still there. After checking manual, finding the interface can deal with COM requests (also message based) and windows activation related messages, but not including WM_PAINT.

Then, jumping to ole32.dll!CCliModalLoop::HandleWakeForMsg(), we can find below codes:

   1: 75DA612A  call        CCliModalLoop::MyPeekMessage (75D5D3D3h) 

   2: 75DA612F  test        eax,eax 

   3: 75DA6131  je          CCliModalLoop::HandleWakeForMsg+95h (75DA613Dh) 

   4: 75DA6133  lea         eax,[ebp-44h] 

   5: 75DA6136  push        eax  

   6: 75DA6137  call        dword ptr [__imp__DispatchMessageW@4 (75D417ACh)] 

   7: 75DA613D  or          dword ptr [edi+0Ch],80000h 

   8:  

And below the function we can see "_DisableNewWM_PAINTDispatch@0:" symbol, meaning this feature can be disabled.

Finally, I found this is a new feature after windows vista. When being blocked in any COM request, the thread can still process WM_PAINT messages, and this new feature has brought a lot of troubles for software developers. Fortunately, it can be disabled for specific program via Windows Application Compatibility Toolkit. For instance, for my sample, I can create a specific setting file for my program disabling this feature, then use "sdbinst.exe" to install it, then the program can work as before.