After a bit more playing it turns out the ODE matries are not liin rows, but in columns.

elements 2,6 and 10 are for view, 1,5,9 are up vectors.

If you are wondering what I'm working on here is a sneek preview.