The UI is sort of standalone. Only thing that atm ties it to nxGL is automatic resizing (this happens only in TUIPanel.Draw()). It is possible to make a function do that manually by values, but it's very fast to check window sizes from nxGL itself.
And another thing is textures and fonts. Neither of these propably need the engine, but depend on the base units.

All i'm saying, OpenGL is almost same no matter how you create window, with minor changes you would be able to integrate whole Next3D in your current engine. It is overall light package. If i was to detach UI separately aswell they wouldn't necessarily stay updated same way and it would conflict with the structure of Next3D...

Menus did cross my mind and they are possible to make. So far i have skipped them though, being not so much needed feature and a bit complex thing to do. After making dropdown-list it may become more clear how to do it, they are similar.

The units seem to have their own modes set at beginning so your project wouldn't effect them. nxGL uses objfpc at the moment, which if changed to delphi seems to (at least) enrage about procedure pointers done differently ^^ (i have later become to like delphi mode more though, old relics these things...)

Edit: About menus i might add that, as seen in the screenshot the normal Lazarus menus work well with it. Popup menus won't have issues either. Still i may add them later for they can be more graphical.