Hm.. It's just a matter of learning DirectX/OpenGL and you're ready to go.
I don't see any problem, except for the fact that the Lazarus executables are not optimized. But that's a minor problem. Although I hate it~! But do not forget the power of Lazarus - cross platform. It's a huge undertaking!
And the speed Lazarus versus Delphi- my experience is that the free pascal compiler is way better than Delphi's... just imho
But real speed difference, in a final game product, shows itself in the quality of programming of course.