PDA

View Full Version : Application/DLL Memory Issues



AthenaOfDelphi
20-03-2007, 04:35 PM
Hi all,

I've been scratching my head over this problem for what seems like an age.

Basically, I'm working on a complete, start from scratch, rewrite of our competition entry from last year. One of the issues we had was with the scripts executing a little too slowly causing everything to just go *clunk* stop *clunk* start again.

Now, my cunning plan to stop this, is to put the 'scripts' in a DLL and have the IDE generate the source for this DLL which can then be compiled and loaded by the play engine. Sounds great doesn't it ;-)

One of my favourite mechanisms for data storage is INI files. I already have an INI implementation that is totally memory based, so its nice and fast, BUT, it relies on TStringList so I can't use this to pass data across the app/DLL boundary.

So, I've created a new variety of TStringList called TExposedStringList. This is basically is a rewrite of TStringList, but with some subtle differences. Where TStringList uses private variables (fCount, fList, fDuplicates etc.) these are held in a packed record in memory by TExposedStringList. This allows me to create an instance on one side of the boundary, and simply pass across a pointer to this data structure allowing an instance on the other side of the boundary to connect to this datastore.

This is then used as the basis for my new favourite toy... TExposedMemoryINI. This basically uses one of my TExposedStringLists to hold the list of sections. The section name is in the items and the list of keys is created by creating an instance of TExposedStringList and stuffing its datastore pointer into the sections objects list. I can then detach the object from its datastore and free it so the object gets cleaned up, but the data store remains intact.

All of this is working great in isolation. But its not so hot when I actually use it as its going to be used in the finished product.

I create an instance of TExposedMemoryINI, write a string into it, load a DLL, pass its data store into the DLL, which reads the value I just wrote into it and then proceeds to write another value in and read it back. I then drop out of the DLL, read the value I wrote into the INI inside the DLL, free up the INI and finally unload the library.

The first run through is sweet. No problems whatsoever.

The second run (and all subsequent runs) however, I get an access violation when I try and read the value written into the INI by the DLL.

The problem goes away when I build the exe and library with ShareMem.

So, my question is this... if I'm using getMem and freeMem for all my memory allocation/deallocation, and I'm only transferring strings using pchars (with data allocated by getmem)... why is it that this problem occurs?

Any thoughts, comments or suggestions? I know its a bit tricky without source code, but I've followed the instructions about shared memory and long strings and I am using pointers only, so I just don't get it.

Robert Kosek
20-03-2007, 05:30 PM
I like the idea behind the implementation very much; it is absolutely devious in my own opinion.

I haven't done much memory sharing between DLLs, but I do know that anything I do between a DLL and the application (in my experience at least) is tenuous if I don't include memory manager. Before I really got into looking around online a bunch about plugin systems and whatnot, I messed around with dynamic functions and wrote a sort of simplistic plugin "demo" for myself before I got bored with it. The whole idea behind the signifigance is that I too passed PChars from function calls, only, I never had to use ShareMem to clean up exceptions. In fact, my project never had a single exception.

To hypothesize, I think the "problem" is just a nitpick on the IDE's part. See, I think you are using real strings in the DLL and application, but passing memory and PChars through. So maybe it is just picky about the fact that you are using any strings in the DLL without ShareMem?

That's about the only possible reason I can come up with though. That's a real puzzler in my opinion.

AthenaOfDelphi
20-03-2007, 05:43 PM
Chebmaster raised the point about shared memory and the fact that the application and the DLL will have their own memory managers... Technomage confirmed it... and the tests I've done also suggest its just a simple case of the left hand doesn't know what the right hand is doing once you start allocating memory on both sides of the boundary.

So I'm going to be forced to use a shared memory manager... which in itself isn't a huge deal providing both sides of boundary can cope. If they can, then I can forget about using pointers... I've just tested passing an object across the boundary using Borlands shared memory manager and it works a treat. Strings and all :-)

Robert Kosek
20-03-2007, 06:18 PM
I had a feeling that was the problem, but my experience with this subject is meager so I wasn't about to point a rather certain finger ... when I don't know (for sure) what I'm talking about. But hey, I was pretty close. :D

I'm glad you have it all sorted out; let me know how it goes with your system's performance! I wouldn't have thought to split it off into a DLL to increase speed, but if it works I just might do that in my projects in the future. Assuming I find time to program games in Delphi again.

AthenaOfDelphi
20-03-2007, 07:04 PM
The speed increase will only come as a result of running compiled code instead of scripted code... admittedly I could do this by simply hard coding the game, but the beauty with this approach is that I hope to retain the flexibility offered by scripting whilst gaining the advantages of real compiled code.

But... this does bring with the complexities that have been brought to light.

I'm just experimenting with a D5 compiled EXE and a FreePascal compiled DLL.... its not going well :?

Edit:- My original reply was practically nonsense, so I've edited it :-)

Robert Kosek
20-03-2007, 08:11 PM
I'm just experimenting with a D5 compiled EXE and a FreePascal compiled DLL.... its not going well :?As far as I know, which isn't very far, the ShareMem constraints require both the DLL and Executable to have the unit required as its first unit; I don't know if FPC's "ShareMem" is compatible with Borland's because they can't just rip it off Delphi. You might have to require Delphi compiled DLLs just to get it working fully.

And I understood the gist of your initial reply anyway, so... I really don't know what pointlessness you removed. :P

I know the speed differences between compiled code and run-time interpreted, but I also know that some scripting engines support compilation of their code, giving some performance gains. Fortunately, or unfortunately, I think DWS2 has this feature so that might help a little.

The only danger I possibly foresee is that in coding an extension DLL for a game, is that someone could inject malicious code into it rather easily. So for the sake of safety against the myriads of viruses out there, you should probably build in a validity check like an MD5 hash file for each extension that gets checked each launch of the game. Potentially the risk is low, most viruses can inject their code into a system DLL, but it is there nonetheless.

AthenaOfDelphi
20-03-2007, 08:38 PM
The shared memory manager is the fly in the ointment I think.

I've just butchered QMem's shared memory manager a little and hooked that into the FPC memory manager of the DLL and rebuilt the EXE to use the same memory manager... BLERGH! *Crash* *Boom*

I did find an example online from someone who reads these boards (L505 I think) that illustrated having the application hook the DLL's memory manager. Thats an option too.

More tinkering is needed... much more :-)

technomage
20-03-2007, 09:39 PM
FPC's Memory Manager is similar to Borlands in the way you define it, but there are extra fields in the TMemoryManager record that you need to define for it to work.

I think the HeapStatus and FPCHeapStatus functions need to be assigned as well.

I have not tried to use a FPC dll memory manager with Delphi, I always compile all the binaries in the same app. Not sure if it will work, but in theory if your delphi app, implements all the things the FPC memory manager requires there should not be a problem.

Another thing, rather than trying to change the dll memory manager to be the same as the applications, I do it the other way around. I have a core memory management dll whcih all the other dll's and exe's use. It made things allot easier to implement.