This is one of the things that's been bothering me, it just felt silly to have those Right/Up/Look vectors to help with rotating and I think I've finally figured out a way to rotate without them.

This is code from my camera test class:
[pascal]
if (Pitch <> 0) or (Yaw <> 0) or (Roll <> 0) then
begin
q := QuaternionRotateObjectToInertial(DegToRad(Pitch), DegToRad(Yaw), DegToRad(Roll));
Orientation := QuaternionCrossProduct(Orientation, q);
Matrix := MatrixCreateOpenGLTranslation(Orientation, Position);
Pitch := 0;
Yaw := 0;
Roll := 0;
end;
[/pascal]

Here are the quaternion functions:
[pascal]
function MatrixCreateOpenGLTranslation(const Orientation: TQuaternion; const Position: TVector3f): TMatrix4x4f;
begin
Result[0,0] := 1 - 2*Orientation.y*Orientation.y - 2*Orientation.z*Orientation.z;
Result[1,0] := 2*Orientation.x*Orientation.y - 2*Orientation.w*Orientation.z;
Result[2,0] := 2*Orientation.x*Orientation.z + 2*Orientation.w*Orientation.y;
Result[3,0] := 0;

Result[0,1] := 2*Orientation.x*Orientation.y + 2*Orientation.w*Orientation.z;
Result[1,1] := 1 - 2*Orientation.x*Orientation.x - 2*Orientation.z*Orientation.z;
Result[2,1] := 2*Orientation.y*Orientation.z - 2*Orientation.w*Orientation.x;
Result[3,1] := 0;

Result[0,2] := 2*Orientation.x*Orientation.z - 2*Orientation.w*Orientation.y;
Result[1,2] := 2*Orientation.y*Orientation.z + 2*Orientation.w*Orientation.x;
Result[2,2] := 1 - 2*Orientation.x*Orientation.x - 2*Orientation.y*Orientation.y;
Result[3,2] := 0;

Result[0,3] := Position.x;
Result[1,3] := Position.y;
Result[2,3] := Position.z;
Result[3,3] := 1;
end;

function QuaternionRotateObjectToInertial(const Pitch, Yaw, Roll: TFloat): TQuaternion;
var
sp, sb, sh: TFloat;
cp, cb, ch: TFloat;
begin
// Compute sine and cosine of the half angles
sp := Sin(Pitch * 0.5);
cp := Cos(Pitch * 0.5);
sb := Sin(Roll * 0.5);
cb := Cos(Roll * 0.5);
sh := Sin(Yaw * 0.5);
ch := Cos(Yaw * 0.5);
// Compute values
Result.w := ch*cp*cb + sh*sp*sb;
Result.x := ch*sp*cb + sh*cp*sb;
Result.y := -ch*sp*sb + sh*cp*cb;
Result.z := -sh*sp*cb + ch*cp*sb;
end;

function QuaternionCrossProduct(const q1, q2: TQuaternion): TQuaternion;
begin
Result.w := q1.w*q2.w - q1.x*q2.x - q1.y*q2.y - q1.z*q2.z;
Result.x := q1.w*q2.x + q1.x*q2.w + q1.z*q2.y - q1.y*q2.z;
Result.y := q1.w*q2.y + q1.y*q2.w + q1.x*q2.z - q1.z*q2.x;
Result.z := q1.w*q2.z + q1.z*q2.w + q1.y*q2.x - q1.x*q2.y;
end;
[/pascal]

This seems to work, but I still need to do some more testing. The quaternion used for orientation should probably be normalized every now and then.

Does anyone else use this scheme for object/camera rotation, any comments on this?