There are many performance advantages going for 64-bit which are too often overlooked. For instance, you have many more CPU registers so the compiler can optimize the code better and floating-point operations typically use SSE instead of FPU, which also gives more registers and is faster.

As for porting to 64-bit, you need to adjust the code to take increased pointer size into consideration. Note that the integer size may not necessarily change since typical OSes are either LLP64 or LP64 as opposed to ILP64 as one may expect; this might be weird at first especially if you recall the move from 16-bit to 32-bit OS.

In my opinion, as long as your CPU supports it, 64-bit is the way to go. On all my systems including ones in the office we use 64-bit OS exclusively and most applications we use have been ported to 64-bit too. For instance, Asphyre has been ported to 64-bit almost a year ago both for Delphi and FPC. I don't think you'll have much trouble porting Allegro too.