in D3D9 it's called SetDialogBoxMode. But it's has some restrictions:

If the method succeeds, the return value is D3D_OK. If the method fails, the return value can be D3DERR_INVALIDCALL unless all of the following are true.

  • The application specified a back buffer format compatible with GDI, in other words, one of D3DFMT_X1R5G5B5, D3DFMT_R5G6B5, or D3DFMT_X8R8G8B8.
    The application specified no multisampling.
    The application specified D3DSWAPEFFECT_DISCARD.
    The application specified D3DPRESENTFLAG_LOCKABLE_BACKBUFFER.
    The application did not specify D3DCREATE_ADAPTERGROUP_DEVICE.
    The application is not between BeginScene and EndScene.