I don't think you should expect much of a performance gain by using Freetype directly because SDL_ttf uses FreeType internally.