Are you working in 2D or 3D? your statement that models only include convex polygons is incorrect, at the highest order a triangle is convex in flat euclidean space but concave in curved space in comparison.

Anyway, a model can be made up of concave elements including individual concave faces, And that's not even going into curved manifolds.

Because *most* hardware is only suitable for rendering 3 sided polytope (triangles) 3D data will normally be tessellated into triangles, either by a 3rd party process (export from/function in from your favorite modeler) or in your engine before the resultant data is uploaded to graphics memory.

Now the distinction between the data and it's representation here is that you may want to retain faces with more than 3 sides (like a professional modeler would) but need to tessellate in order to display.

If you're not bothered about loosing information then tessellate and be done with it.

As you'd want to know exactly where on the model your ray would intersect, you're going to have to test against all the front facing polygons (from the rays origin) as these will most likely be triangles, your problem has now become 'Ray - Triangle Intersection' which you'll find in any basic maths textbook.

Such tests should be optimized by performing a bounding volume check on the given model first (or bounding volume on some tree nodes etc whatever makes sence with the data you're rendering) and further optimized by using some form of spatial partitioning (oct-tree/bsp etc) and their corresponding ray-cast optimizations, if you have many separate elements to test against.

Obviously, you'll need to ensure that your vertex winding is the same for all faces on a model otherwise you'll run into problems fast. (this should be be the case anyway from any half serious 3D modeling application)

And then you just itterate thru the faces and test every single one.

Some modeling programs employ spatial partitioning with the vertices/faces of the model itself, such applications include BSP map editors where the geometry is sparse and can benefit from such optimizations. Also programs like Z-Brush which deal with highly complex models also use partitioning schemes to speed up operations.

But from the 3D game, models and maps kind of setup, it's rare that you'll see partitioning used past the 'map' paradigm.

There exists screen-space solutions to collision detection as well, some people use OpenGL picking for example, rendering individual polygons with different unique colours to some off-screen buffer then sampling the pixel at projected screen coordinates of the ray. Then by taking the colour value, determine the polygon. This is a perfectly viable method but becomes increasingly uneconomical from a pre-processing perspective the more data you wish to operate on and doesn't instantly provide you with information such as the normal of the face the ray hits.

So I'd recommend classic ray-triangle intersection, optimized with spatial partitioning and testing only front facing polygons.