Collision detection on a large map is often slow. Could that be the problem? You could try to use a physics engine. Most of them have good optimazations for ray intersection on large scale maps.