Offcourse the texture size matters. Polyon is actualy just an surface on witch the texture is being drawn. So if you have small texture and larg polygon you will have to upscale the texture so it can be drawn on whole polygon (just like increasing picture size). If you have big texture and small polygon on other hand the texture itself would be scaled down so that it could be properly drawn on the polygon (just as if you would reduce the picture size).
The only difference is that when using polygons the texture scaling is done by the graphic card itself. And since most graphics cards have hardware support for textre scaling there is a considerable performance boost.
Bookmarks