SuperBoy, the posted error's strange indeed.. Not DialogBox's fault though
Umm.. anyway. I've just hacked a simple app, which creates a modal dialog as its main window. It might give you some ideas about what you could be doing wrong, or you can use it as a starting point . Here it is:
example.pas:
[pascal]program example;

{$APPTYPE GUI}
{$R '*.res'}

uses
Windows,
Messages;

const
MainDlg = 1000;
IDC_TEXT = 1001;
IDC_BUTTON = 1002;

var
//----- Window vars -----
hInst,
hWindow: HWND;

Buffer: array[0..100] of char;

function DlgWindowProc(hWnd,Msg: LongWord; wParam,lParam:Longint):LongInt; stdcall;
begin
case Msg of
WM_CLOSE:
begin
PostQuitMessage (0);
end;
WM_COMMAND:
begin
case LoWord(wParam) of
IDC_BUTTON:
begin
Buffer[0] := #0;
GetDlgItemText(hWnd, IDC_TEXT, Buffer, 100);
MessageBox(hWindow, Buffer, 'Wazzat', MB_OK or MB_ICONINFORMATION);
end;
end;
end;
else
Result := 0;//false
Exit;
end;
Result := 1;//true
end;

//----- WinMain --------
begin

hInst := GetModuleHandle(nil);

hWindow := DialogBox (hInst, MAKEINTRESOURCE (MainDlg), 0, @DlgWindowProc);

end.[/pascal]
example.rc:
Code:
#define MAINDLG 1000
#define IDC_TEXT 1001
#define IDC_BUTTON 1002

MAINDLG DIALOG 11, 32, 206, 62
STYLE WS_POPUP | WS_VISIBLE | WS_CAPTION | WS_SYSMENU |
WS_THICKFRAME | WS_MINIMIZEBOX | WS_MAXIMIZEBOX
CAPTION "example"
FONT 8, "MS Sans Serif"
BEGIN
  EDITTEXT         IDC_TEXT,   15,17,111,13, ES_AUTOHSCROLL | ES_LEFT 
  DEFPUSHBUTTON    "Do Work", IDC_BUTTON, 141,17,52,13
END
Hope this helps
Fragle

P.S.: is it me, or do resources not get built automatically by fpc (its default ide - Rhino(?) to be more precise) when compiling an application? Did i forget some option or something? Now it just fails with linker error unless i build the resources myself... Downloaded FPC just an hour ago, so i might be missing something obvious ops:. Thanks in advance for any hints on that!