Seems that https://github.com/bkaradzic/bgfx does something close to what your goal is, so you may find some inspiration there perhaps. As for API: provide low level functions and build the high level stuff on it, so it would be optional to use?