I've made a false assumption that you store camera orientation as a vector.

But if you store orientation as a quaternion, isn't it enough to create orientation from Euler angles, convert to matrix and send to OpenGL, without the multiplication?