Code:
{$IFDEF FPC}
  {$packset 1}
{$ENDIF}
This fixed the problem! I don't know why but SizeOf(Set) was returning 4 to me in FPC but 1 in Delphi.