Your original solution was almost perfect, but you are right, on larger maps it would get lower performance. Thus you can divide your map into chunks.

Code:
const CHUNKSIZE = 32;

TChunk = class
public
  posX, posY: integer;
  Tiles: array[0..CHUNKSIZE-1, 0..CHUNKSIZE] of TTile;
  Objects: array of TObject;
end;

TMap = class
public
  Chunks: array of array of TChunk;
  function GetTile(x, y: integer): TTile;
end;

function TMap.GetTile(x, y: integer): TTile;
var cx, cy: integer;
procedure
  cx:=x div CHUNKSIZE;
  cy:=y div CHUNKSIZE;
  result:=Chunks[cx, cy].Tiles[x-cx, y-cy]
end;
Or something along these lines... Of course you could do range checking in GetTile() to avoid critical errors on negative values or out of boundaries.