Endian issue with file encryption procedure on PPC Mac OS X?

[Christian Knudsen]

Hi all!

I'm in the process of porting a FreePascal/SDL game of mine from Windows to Mac OS X. I've already successfully ported the game to Linux 32 and 64-bit with no code changes whatsoever, and I believe I've got the game running on Intel Mac OS X systems as well (I haven't received any crash reports from Intel Mac users and the game runs perfectly on my own Intel Mac). However, I've received crash reports from PowerPC Mac OS X users and I believe the issue is with a procedure that encrypts/decrypts game files by using byte shifting:

[pascal]
FUNCTION Encrypt(Const S : String; Key : Word) : String;
VAR
I : Byte;
TempString : String;
BEGIN
TempString := '';
FOR I := 1 TO Length(S) DO BEGIN
TempString := TempString + Char(Byte(S[I]) xor (Key shr );
Key := (Byte(TempString[I]) + Key) * C1 + C2;
END;
Encrypt := TempString;
END;


FUNCTION Decrypt(Const S : String; Key : Word) : String;
VAR I : Byte;
TempString : String;
BEGIN
TempString := '';
FOR I := 1 TO Length(S) DO BEGIN
TempString := TempString + Char(Byte(S[I]) xor (Key shr );
Key := (Byte(S[I]) + Key) * C1 + C2;
END;
Decrypt := TempString;
END;
[/pascal]

The game would crash on PowerPC's when first using the decrypt function to get info from an encrypted data file. So, my first thought was that it's due to PowerPC's being big-endian whereas my x86 is little-endian, so I added the built-in FreePascal endian "conversion" functions:

[pascal]
FUNCTION Encrypt(Const S : String; Key : Word) : String;
VAR
I : Byte;
TempString : String;
BEGIN
TempString := '';
FOR I := 1 TO Length(S) DO BEGIN
TempString := TempString + Char(NtoLE(Byte(S[I])) xor (Key shr );
Key := (NtoLE(Byte(TempString[I])) + Key) * C1 + C2;
END;
Encrypt := TempString;
END;


FUNCTION Decrypt(Const S : String; Key : Word) : String;
VAR I : Byte;
TempString : String;
BEGIN
TempString := '';
FOR I := 1 TO Length(S) DO BEGIN
TempString := TempString + Char(LEtoN(Byte(S[I])) xor (Key shr );
Key := (LEtoN(Byte(S[I])) + Key) * C1 + C2;
END;
Decrypt := TempString;
END;
[/pascal]

The game still crashes at the same point, though, so I'm obviously not doing something right. (I've since read that bytes don't have endian issues?) Any ideas as to how to fix this?

[JSoftware]

A byte shouldn't be the source of endianess problems

Try this instead:
TempString := TempString + Char(Byte(S[I]) xor hi(Key));

[Christian Knudsen]

A byte shouldn't be the source of endianess problems

Yeah, after reading up on endianess a bit I realize how stupid that sounded.

Try this instead:
TempString := TempString + Char(Byte(S[I]) xor hi(Key));

How about doing:
[pascal]TempString := TempString + Char(Byte(S[I]) xor (NtoLE(Key) shr );
[/pascal]
Shouldn't that convert the Key to little-endian before doing the encrypt/decrypt?

[JSoftware]

I guess it would. I haven't worked with other systems with endians other than little, so I can't say precisely what would work

[VilleK]

Key shr 8 should be portable because it refers to a value in a register not in memory. Compare with C-syntax: i>>8 always shifts out the lowest bits regardless of endianess.

I would try to track down the error by adding logging messages to see exactly where it crashes. Maybe try a version that bypasses encryption and see if it works. Else you might risk keep guessing forever where the problem is

[Christian Knudsen]

Well, the thing is that it only seems to crash on PowerPC Macs not Intel, and according to the traceback, it crashes in the first procedure that uses the decrypt function in the game. That function simply just decrypts a file that contains information about the level to load, so I can't see what else should get it to crash -- and with only the PPC's crashing, it seems to actually be an endian issue.

Anyway, I've sent both the hi(Key) and NtoLE(Key) versions to some of the people that have been getting crashes. I'm waiting to hear back from them.

[Christian Knudsen]

I've just heard back from one of the testers. The hi(Key) version seems to work!