I believe that Windows minimizes your fullscreen application, because dialog doen't have your 3D window set as a parent window for it.

Another hint: watch for FlipToGDISurface. :!:

PS. Generally combining fullscreen 3D applications and Windows GDI dialogs never worked fine for anyone. I'm afraid what you will loose too much time with no 100% working solution in the end. So, if you definetly looking for WinAPI dialogs using with D3D application - it's better to just use maximized window without borders to simulate "fullscreen mode".