Do you really need ray casting? If you only want to select objects at e.g. a given cursor position, just go for color selection. Especially with shaders that's pretty simple and fast. I'm using this technique for selection too (see here). It's easy to implement and extremly fast.

It works like this :
- Render your scene to the backbuffer (so it's not visible), but only with colors and basic geometry. E.g. one building red, another yellow
- Limit the viewport to the cursor position (for performance)
- Read the pixel under the cursor (glReadPixels)
- Compare read colors with your color-to-object-table

It's very fast and can be implemented in just a few minuts. Especially when using only shaders you just need a new shader the outputs selection colors to the fragment buffer.