Cracked it - unscrambled the quaternion into a 3x3 rotation matrix and I can look around properly in VR now