I don't understand what you mean by steps even, using mouse-ray is not iteration. Usually you would use line-to-triangle intersect, line-to-sphere intersect and so on functions. It's normally not at all about pixels. For that you may be able to do something different, like glReadPixels at cursor's screen coordinates? I don't know, very much depends on what the app needs to do.