But then everytime I change scaling, I'd have to re-build the object's VAO and VBOs... That is certainly not what I want.
yeah.. That's why matrices are THE way to do transformations.

Can you tell me what doesn't work?