A byte shouldn't be the source of endianess problems

Try this instead:
TempString := TempString + Char(Byte(S[I]) xor hi(Key));