I'm not sure how much help this will be. It is a function that I wrote several years ago just after starting here at Krome that helped to work out which 3D object was selected with the mouse. Basically, it works out the world coordinate on the near clip plane, the world coordinate on the far clip plane, then casts a ray between those points from near to far. If the ray collides with an object, it returns the world position and normal of that collision point. It could also return the object that it collided with if it was extended a bit more.

BG was the name of our proprietary engine back then. BGU was my little utility library for common functions. Yes, it's in C.

vpRight and vpBottom were the right and bottom edges of the viewport, in this case, the width and height of the viewport.

Code:
//---------------------------------------------------------------------------
//  Function: BGU_ProjectScreenToWorld
//  Purpose:  Projects a ray from the screen coordinate into the world and
//            checks for a collision
//  Input:    sx, sy      Screen coordinate
//            world       Pointer to a vector that will receive the world
//                        coordinate of the collision point
//            normal      Pointer to a vector that will receive the normal of
//                        the polygon that was involved in the collision. Can
//                        be NULL if no normal is required
//  Output:   int         0 if no collision, !0 if a collision

int BGU_ProjectScreenToWorld(int sx, int sy, Vector *world, Vector *normal)
{
  Vector nearPoint, farPoint, farView;
  float px, py;
  float fovx, fovy;
  BG_GetFOV(&fovx, &fovy);

  // Get near point in worldspace
  px = (((float)sx / (float)vpRight) * 2 - 1) * (nearPlane + 1.0f) / fovx;
  py = (((float)sy / (float)vpBottom) * 2 - 1) * (nearPlane + 1.0f) / fovy;

  BG_WorldPoint(px, py, (nearPlane + 1.0f), &nearPoint.x, &nearPoint.y, &nearPoint.z);

  // Get far point in worldspace
  px = (((float)sx / (float)vpRight) * 2 - 1) * (farPlane - 1.0f) / fovx;
  py = (((float)sy / (float)vpBottom) * 2 - 1) * (farPlane - 1.0f) / fovy;

  BG_WorldPoint(px, py, (farPlane - 1.0f), &farPoint.x, &farPoint.y, &farPoint.z);

  // Now we have the near point and the far point, test for collision between them
  if (BG_SurfaceHit(WORLD_OBJECT, nearPoint.x, nearPoint.y, nearPoint.z, farPoint.x, farPoint.y, farPoint.z))
  {
    if (world)
      BG_GetLastPoint(&world->x, &world->y, &world->z);
    if (normal)
      BG_GetLastVector(&normal->x, &normal->y, &normal->z);
    return 1;
  }
  else
  {
    return 0;
  }
}