PDA

View Full Version : data format incompatibilities (FPC vs Delphi)



Chebmaster
03-06-2007, 04:07 PM
The Information was gathered the hard way during my work on this (http://www.pascalgamedevelopment.com/viewtopic.php?t=3489).

You can encounter these pitfalls if you try to link a DLL compiled in FreePascal to application compiled in Delphi and exchange data between, or you can just flush a data structure to disc directly, and expect it to load in Delphi nicely...

Believe me, there's a fat chance it won't.

1). The dynamic array format (I didn't check, but it could relate to strings as well). As you know, in Delphi a dynamic array is in fact a pointer pointing to the actual data, with some additional info at negative offset (while an empty array is always just a nil pointer).

There is reference count (dword) at -8 and the array length(dword) at -4, (given in array elements, not in bytes).
Now a trap: in FPC it's almost the same, except it keeps the High() value at -4. I.e. length - 1. You try to pas an array.. and boom.

2). The set size. In Delphi sets take no more bytes than necessary, having length of (elements div 8 ) bytes. In FPC, sets have either 4 bytes in size (those with 32 elements or less) or 32 bytes, there are no other options.

3). The records size. Unless you use {$alignrecords XXX} directive or packed records, a record in FPC does always take 32 bytes as a minimum -- even if it consists of a single byte. Delphi arranges them differently.

4). The default field alignment in Delphi is 8 bytes (i.e. 64 bits). That is, it wastes half your memory if you use exclusively 32-bit fields in your classes. FPC employs some complex scheme I couldn't figure out (especially when you mix 32-bit and 64-bit fields), so I just forcibly set it to 4 in my system.

...Still feel brave enough to try and mix those two...?

dmantione
03-06-2007, 05:20 PM
Records aren't rounded up:

program recsize;

type rec=record
x:byte;
end;

begin
writeln(sizeof(rec));
end.

... returns 1.

dmantione
03-06-2007, 05:24 PM
Field alignment in FPC is done by adding a few padding bytes until natural alignment is achieved. Bytes need to be byte aligned, so declaring a byte in your record never wastes a byte. If you declare a word though, and there was a byte in front of it, one padding byte is added to align the word according to a word. For a cardinal, up to 3 bytes can be added in front of it, in order to obtain natural alignment.

Lastly padding bytes at the end of a record are added to ensure an array of the record can have natural alignment.

JernejL
03-06-2007, 07:17 PM
4). The default field alignment in Delphi is 8 bytes (i.e. 64 bits). That is, it wastes half your memory if you use exclusively 32-bit fields in your classes. FPC employs some complex scheme I couldn't figure out (especially when you mix 32-bit and 64-bit fields), so I just forcibly set it to 4 in my system.


ever heard of packed records and / or $ALIGN compiler define in delphi?

aniway, nobody shares data directly like that, and the pascal doesn't even define how the data should be stored internaly for that purpose.

Chebmaster
03-06-2007, 09:30 PM
ever heard of packed records and / or $ALIGN compiler define in delphi?
Heard? I *use* them to solve this problem.
Question is how does this affect compiler's ability to optimize code.



Lastly padding bytes at the end of a record are added to ensure an array of the record can have natural alignment.
Hmmm... That's strange. Previously, whernever I tested it, the minimal record size came out as 32 bytes... I got that result with a record of two bytes, I am sure of that.

Might have been some forgotten compiler switch or code optimization combination. What compiler version was that?.. maybe some bug in a beta that they have long fixed? :?

Chebmaster
03-06-2007, 09:38 PM
Though I remember, I had to invoke {$packrecords 4} to avoid that, and with this switch, they have exactly as many bytes in size as their fields require.

JernejL
04-06-2007, 04:34 PM
Lastly padding bytes at the end of a record are added to ensure an array of the record can have natural alignment.
Hmmm... That's strange. Previously, whernever I tested it, the minimal record size came out as 32 bytes... I got that result with a record of two bytes, I am sure of that.

maybe you think 32 bits and not bytes?

aniway, packed records solve all your problem, since you can pad structures in any way you want. but actually, i'd love to see some proper way for padding inside structures in pascal syntax.

Chebmaster
04-06-2007, 05:41 PM
aniway, packed records solve all your problem,
Except that speed may die: processor chokes with unaligned reads/writes, stalling and performing a lot of unnecessary work.

The other way is to use the perversion named
{$align off}
Yuck. :x


maybe you think 32 bits and not bytes?
No, I'm positive: these were bytes, not bits. Anyway, since the quirk is not repeatable, let's think it was a glitch of 2.0.0 (I long since updated to 2.0.4).

The incompatibility of dynamic array format, on the other hand... I'm sure, I'tll bite someone's in his tender parts someday. :twisted:

JernejL
05-06-2007, 05:20 PM
aniway, packed records solve all your problem,
Except that speed may die: processor chokes with unaligned reads/writes, stalling and performing a lot of unnecessary work.

The other way is to use the perversion named
{$align off}
Yuck. :x


maybe you think 32 bits and not bytes?
No, I'm positive: these were bytes, not bits. Anyway, since the quirk is not repeatable, let's think it was a glitch of 2.0.0 (I long since updated to 2.0.4).

The incompatibility of dynamic array format, on the other hand... I'm sure, I'tll bite someone's in his tender parts someday. :twisted:

you know, even in delphi itself, loading and storing a dynamic array directly is like asking for trouble, and expecting FPC data format for arrays to be same as delphi's is silly. the only way to have identical data storage is to use packed records and reading and storing arrays in files properly (array item by item)

captainproton
17-07-2007, 09:19 AM
Am I confused? Did someone say that expecting Delphi and FPC to do something the same was silly! I thought both were PASCAL - or are you referring to the way the compiler handles this stuff.

JernejL
17-07-2007, 09:26 AM
Am I confused? Did someone say that expecting Delphi and FPC to do something the same was silly! I thought both were PASCAL - or are you referring to the way the compiler handles this stuff.

Ofcourse they are both pascal with same syntax and functionality, but nothing dictates on how the compiler has to be written internally or how it has to store helper structures for strings and dynamic arrays, and the compiler can store the data internaly as it pleases, so FPC and delphi stores strings and dynamic arrays internally differently, so if you analyse pointers to specific blocks to data in arrays (@array[1, 2]), and to array itself (@array), you will see they differ a lot, and the data at those location differs as well.

captainproton
17-07-2007, 08:51 PM
Thanks Delfi - I knew I was confused, it happens a lot to me.

Craig

Almindor
20-07-2007, 01:57 PM
You cannot depend on internal binary structure of managed types.

ansistrings and arrays (dynamic) are managed types, they both have some voodoo, and even between versions of FPC or Delphi itself they can change these voodoo hidden parts.

You should only save the "data" not the "metadata". I'm pretty positive alot has changed between delphi itself in this regard in all the versions, and I bet noone whined about it :)

If you save a dynarray as a whole you're doing something terribly wrong.

marcov
28-10-2007, 02:59 PM
2). The set size. In Delphi sets take no more bytes than necessary, having length of (elements div 8 ) bytes. In FPC, sets have either 4 bytes in size (those with 32 elements or less) or 32 bytes, there are no other options.


It could be that this is closer to Delphi with FPC 2.2