files2pas data 'resource' creator

[paul_nicholls]

See this post for the latest usage details:
http://www.pascalgamedevelopment.com/showthread.php?3913-files2pas-data-resource-creator&p=43270&viewfull=1#post43270

Hi all,

I have made a program that someone may find useful.

As I am using freepascal, I was wanting to do similar things as the borland command line resource compiler program that processes resource script files (.rc) and creates windows resource files (.res) that can be compiled into your programs, but in a cross-platform way.


For example I can take a resource script file like so:

SpritesResource.rc


Background BITMAP "data\background.bmp"
mario BITMAP "data\sprites_mario.bmp"

this is in the format of:

resource_name resource_type resource_filename (absolute or relative to the resource script file as in this example)

Resource_name has to be a valid pascal identifier.
Resource_type can be anything but shouldn't contain spaces. May be a validator or similar in the future.
Resource_file only needs "" around it if it has spaces in the path or name.

Pass it into my files2pas.exe program as the first parameter like this:


files2pas c:\myprograms\SpritesResource.rc

It will then spit out a .pas file for every resource from the script file like so containing the file in hex digit format as a large Const declaration:



Unit mario_Resource;
// Created by files2pas.exe
// on 16-Mar-2007, 03:26:46p

{$IFDEF FPC}
{$Mode Delphi}
{$ENDIF}

Interface

Const
cmario_type = 'BITMAP';
cmario_data: Array[0..120054 - 1] of Byte =
(
$42,$4D,$F6,$D4,$01,$00,$00,$00,$00,$00,$36,$00,$0 0,$00,$28,$00,$00,$00,$C8,$00,$00,$00,$C8,$00,$00,
$00,$01,$00,$18,$00,$00,$00,$00,$00,$C0,$D4,$01,$0 0,$C4,$0E,$00,$00,$C4,$0E,$00,$00,$00,$00,$00,$00,
$00,$00,$00,$00,$FF,$FF,$FF,$FF,$FF,$FF,$FF,$FF,$F F,$FF,$FF,$FF,$FF,$FF,$FF,$FF,$FF,$FF,$FF,$FF,$FF,
$FF,$FF,$FF,$FF,$FF,$FF,$FF,$FF,$FF,$FF,$FF,$FF,$F F,$FF,$FF,$FF,$FF,$FF,$FF,$FF,$FF,$FF,$FF,$FF,$FF,
$FF,$FF,$FF,$FF,$FF,$FF,$FF,$FF,$FF,$FF,$FF,$FF,$F F,$FF,$FF,$FF,$FF,$FF,$FF,$FF,$FF,$FF,$FF,$FF,$FF,
$FF,$FF,$FF,$FF,$FF,$FF,$FF,$FF,$FF,$FF,$FF,$FF,$F F,$FF,$FF,$FF,$FF,$FF,$FF,$FF,$FF,$FF,$FF,$FF,$FF,
$FF,$FF,$FF,$FF,$FF,$FF,$FF,$FF,$FF,$FF,$FF,$FF,$F F,$FF,$FF,$FF,$FF,$FF,$FF,$FF,$FF,$FF,$FF,$FF,$FF,
$FF,$FF,$FF,$FF,$FF,$FF,$FF,$FF,$FF,$FF,$FF,$FF,$F F,$FF,$FF,$FF,$FF,$FF,$FF,$FF,$FF,$FF,$FF,$FF,$FF,

<SNIP>
$FF,$FF,$FF,$FF,$FF,$FF,$FF,$FF,$FF,$FF,$FF,$FF,$F F,$FF,$FF,$FF,$FF,$FF,$FF,$FF,$FF,$FF,$FF,$FF,$FF,
$FF,$FF,$FF,$FF,$FF,$FF,$FF,$FF,$FF,$FF,$FF,$FF,$F F,$FF,$FF,$FF,$FF,$FF,$FF,$FF,$FF,$FF,$FF,$FF,$FF,
$FF,$FF,$FF,$FF
);

Implementation

End.


It will also spit out a .pas file with the same name as the resource script file that allows you to get the pointer to each resource data + the size + type using a function:


Unit SpritesResource;
// Created by files2pas.exe
// on 16-Mar-2007, 03:26:43p

{$IFDEF FPC}
{$Mode Delphi}
{$H+}
{$ENDIF}

Interface

Function GetResourceByName(AResourceName: AnsiString;
Var AData: Pointer;
Var ASize: LongInt;
Var AType: AnsiString): Boolean;

Implementation

Uses
SysUtils,
Background_Resource,
mario_Resource;

Function GetResourceByName(AResourceName: AnsiString;
Var AData: Pointer;
Var ASize: LongInt;
Var AType: AnsiString): Boolean;

Begin
Result := False;
AResourceName := UpperCase(AResourceName);
If AResourceName = 'BACKGROUND' Then
Begin
AData := @cBackground_data[0];
ASize := SizeOf(cBackground_data);
AType := cBackground_type;
Result := True;
End
Else
If AResourceName = 'MARIO' Then
Begin
AData := @cmario_data[0];
ASize := SizeOf(cmario_data);
AType := cmario_type;
Result := True;
End;
End;

End.


So all you have to do is include the .pas file that has the information in it for retrieving the resource data info (SpritesResource.pas in this case) and use it.

Viola! now all your nice data files can be compiled into the executable file and not 'float' around for everyone to see.

It also makes it easy as you only have to distribute 1 file :)

EDIT:14-Sep-2012 - I have uploaded it to my dropbox folder now (includes source code):
https://dl.dropbox.com/u/1805932/files2pas.7z

cheers,
Paul.

[paul_nicholls]

Hi everyone,
I'm curious as to what people think of this tool, as no-one at all has replied to this thread!!

cheers,
Paul

[technomage]

Hi Paul

Well done :). Very nice work, I did something similar to embed images into my game engine (the engine logo) but I'm sure people will find this useful (I certainly will).

You could make the system a bit more dynamic by having each resource unit register it's resource with a central manager, that way the GetResourcebyName would just do a lookup in an array or string list rather than hard coding the resource names (if you have allot of resources that procedure would get very big).

Good work :)

[JSoftware]

{$R mario.bmp}
FindResource
LoadResource?

[arthurprs]

usefull when you need to hide a resource :)

[paul_nicholls]

{$R mario.bmp}
FindResource
LoadResource?

This utility is a cross-platform tool for 'resources' so I am not dependent on Windows routines, and I already know it works on arm-linux (GP2X) as well as Windows without modifications :-)

Besides this, I have more control over how it does things, and may even add compression to the resources when the resource pas files are generated to save on space.

cheers,
Paul

[paul_nicholls]

Hi Paul

Well done :). Very nice work, I did something similar to embed images into my game engine (the engine logo) but I'm sure people will find this useful (I certainly will).

You could make the system a bit more dynamic by having each resource unit register it's resource with a central manager, that way the GetResourcebyName would just do a lookup in an array or string list rather than hard coding the resource names (if you have allot of resources that procedure would get very big).

Good work :)

Thanks :-)

I will look at changing the way it looks up the resources by name as you suggested.

I am also upgrading it slightly to be even more handy, like being able to get a count of the number of resources, and get the resources by index too :-)

Also, resource names won't have to be a valid pascal identifier in the end either :)

cheers,
Paul

[harrypitfall]

I really like it, is a good way to embed resources.
I use a component to put resources into dfm (delphi), but i notice that large resources waste time, and lead to "stream read errors"
You tool is clean, and fast... create the .pas file to hold and retrieve the resource...
Is a inspiration for me... :) really... i think that i'll building something related to delphi...

[De-Panther]

brilliant
thanks

[paul_nicholls]

I'm glad to be a help to you :-)
cheers,
Paul

[paul_nicholls]

Hi all,
Since my last post, I have finally redone the files2pas converter to make it more flexible.

The latest files can be downloaded from here:

7-zip file containging the files2pas.exe + source code for any one interested in how I did it.

http://fpc4gp2x.eonclash.com/downloads/files2pas.7z

an example .rc file + the outputs that were created
http://fpc4gp2x.eonclash.com/downloads/file2pas_example.7z

The example .rc file that was used looks like this:



"background res" BITMAP "data\background.bmp" True
"mario res" BITMAP "data\mario.bmp" True


The first column now can be anything for the resource name used to retrieve the data as long as the name doesn't contain " characters.

The second column is the resource type.

The third column is the source file for this resource relative to the .rc file (with no " characters in the name).

The fourth column has been added and can be "True" or "False". If this value is true, then the binary data generated in the .pas file for that "resource" will be compressed using the TCompressionStream from either the FPC zstream unit, or the Delphi zlib unit (see below conditional defines below) prior to converting to hex format.

{$IFDEF FPC}
zstream,
{$ELSE}
zlib,
{$ENDIF}

For each resource in the .rc file, it will generate a single resource<number>.pas file.

A main resource pas file which uses the individual resource files will be generated.

It will also create a resource_unit.pas used by the generated units, but this doesn't need to be included in the project if you want to use the resources (see example below).

To retrieve a "resource", just include the resource file name .pas file in your code and also the Classes unit.

For example:



Uses
Classes,
Mario_Bros_Resource,

{...}

Var
ResourceType : AnsiString;
ResourceStream : TMemoryStream;
Begin
Result := False;
Sprite := TSDLSprite.Create;
If GetResourceByName('background res',ResourceType,ResourceStream) Then
Begin
Sprite.LoadFromStream_BMP(ResourceStream);
ResourceStream.Free;
End;


It will automatically decompress the data if necessary prior to returning the Resource Type string and a TMemoryStream containing the raw resource data.

Enjoy :)

cheers,
Paul

[jdarling]

Nice work Paul, just downloaded the latest version to take a look at it. This could be very useful for embedding resources especially now with the compression. Have you thought about simple encryption as well to make resources a bit more difficult to "hack over"?

[paul_nicholls]

Nice work Paul, just downloaded the latest version to take a look at it. This could be very useful for embedding resources especially now with the compression. Have you thought about simple encryption as well to make resources a bit more difficult to "hack over"?

Hi Jeremy,
Thanks for the kind words :)

Wouldn't it be already be very difficult to get at the 'resources' as they are in a very non-standard format?

On the other hand, I could always add XOR encryption/decryption as a starting point as that is the easiest and very hard to crack.

EDIT:
Off-topic: I read your blog http://eonclash.blogspot.com/ about your children who have allergies and I can totally relate.

My oldest daughter Kendra (2 years 9 months) is quite intolerance to AMINES and SALICYLATES http://www.cs.nsw.gov.au/rpa/Allergy/resources/foodintol/salicylates.cfm.

Amines can cause her to get bad stomach pains and make her get very bad eczema + other symptoms.

Salicylates give her very very bad 'nappy rash', burns actually around her nether regins.

Perhaps you may be interested in this book (Friendly Food) from the above site:
http://www.cs.nsw.gov.au/rpa/allergy/resources/foodintol/friendlyfood.cfm


Friendly Food is a recipe book and a complete guide to avoiding allergies, additives and problem chemicals. It is available to all members of the general public for $38.50 (price includes GST and postage and handing within Australia) and can be purchased here from the Allergy Unit, or from all major book stores throughout Australia.
(Murdoch Books® ISBN 1-74045-376-X)

cheers,
Paul

[Ñuño Martínez]

Nice tool. How many platforms did you test? I'm just curious.

[paul_nicholls]

Nice tool. How many platforms did you test? I'm just curious.

It works on my WindowsXP PC and the previous version worked on both my PC and gp2x hand-held (arm-linux) computer, but I haven't tested this newest version on the gp2x yet.

I am hopeful this one will work there too.

cheers,
Paul

[jdarling]

Paul, yes allergies can be a complete pain especially with the current labeling laws around food. At least now they have to call out the top 10, but that doesn't help anyone with a "local" allergy. Unfortunately they didn't expand the laws to cover derivative's such as SOY Oil (commonly called Vegetable Oil) and other such things to be labeled as an allergic warning.

We are looking into BHA and BHT allergies for my oldest son now, as we have started to notice some bad things there as well. It seems that what it all adds up to is that parents like us need to make sure our manufactures are up front and honest with us.

PS: I feel for you on the allergies, I can only imagine how difficult those are to find in product listings.

[paul_nicholls]

Paul, yes allergies can be a complete pain especially with the current labeling laws around food. At least now they have to call out the top 10, but that doesn't help anyone with a "local" allergy. Unfortunately they didn't expand the laws to cover derivative's such as SOY Oil (commonly called Vegetable Oil) and other such things to be labeled as an allergic warning.

We are looking into BHA and BHT allergies for my oldest son now, as we have started to notice some bad things there as well. It seems that what it all adds up to is that parents like us need to make sure our manufactures are up front and honest with us.

PS: I feel for you on the allergies, I can only imagine how difficult those are to find in product listings.

Yeah, it is a real pain to find those in product listings; instead we have to rely on a 'safe' product list given to us by a dietitian with regards to the low allergy diet. On the other hand, we do get 'used' to what things to avoid on labels :-)

BACK ON TOPIC: I have tested the latest version of the files2pas and it works on my gp2x arm-linux hand-held computer too as well as under Win32!

cheers,
Paul

[paul_nicholls]

Hi all,
since the original download link for files2pas has expired, and at least one person is still interested in it, I have uploaded it to my dropbox folder now (includes source code):
https://dl.dropbox.com/u/1805932/files2pas.7z

[code_glitch]

Make that one more person who's interested... :D

[paul_nicholls]

Make that one more person who's interested... :D

LOL! Thanks mate :)

[paul_nicholls]

hmm...I might try upgrading it to make a version that can create Oxygene for Java 'pascal' output too :)

[Cybermonkey]

Hi Paul, does this tool still work as described in the first post? Very useful tool it seems ... Maybe I can use this with my interpreter?

[hwnd]

thank you Paul.

[paul_nicholls]

Hi Paul, does this tool still work as described in the first post? Very useful tool it seems ... Maybe I can use this with my interpreter?

Hi Cybermonkey, it still works as described in post #11
Fee free to use it as you will, just don't say that you wrote it :)

[paul_nicholls]

thank you Paul.

You're welcome mate :)

[phibermon]

Paul, this is actually genius! :) yes resources can be included in an executable in other ways, but this is a very versatile method that allows for fine grain control of platform specific data with {$IFDEF} etc. Platform situations where you have no file system or hiding your executables resources from conventional eyes spring to mind.

An idea (if you've not already thought of it) :

For each file that is generated you could create an initialization section in which the data pointer/name etc is added to the global list/accessor.

The tool could also generate a 'files.inc' file that contained the main unit and any number of files :

VirtualFileSystem, {contains global instance of a list of data pointers}
image1_bmp, sound_wav, music_mp3

then to make use of it in a project, all the user needs do is :

uses
{$i files.inc};

So then no changes are needed to program code when your tool added new files (updating the .inc) and they would be automatically registered in a global list, accessible at run time.

[LP]

The idea of putting binary data into source code is well-known and widely used. For instance, Asphyre uses DirectX 10, DirectX 11 and GLSL shader binaries/source code embedded in the source code very similarly to this tool, except that in some cases the data is also compressed to reduce resulting source code size. Also, if you look at Delphi XE 2 source code included with installation, you'll see that they're putting some shaders in similar fashion.

By the way, why are you putting $mode delphi command in the generated source files? They don't seem to need it.

Also, I'd suggest updating the links in the very first post as they are broken and you have to read through 3-page thread to find updated link. Why not adding the file as a forum attachment in the first post?

...and keep up great work! :) Any chance to see GUI-based version of this tool?

[paul_nicholls]

The idea of putting binary data into source code is well-known and widely used. For instance, Asphyre uses DirectX 10, DirectX 11 and GLSL shader binaries/source code embedded in the source code very similarly to this tool, except that in some cases the data is also compressed to reduce resulting source code size. Also, if you look at Delphi XE 2 source code included with installation, you'll see that they're putting some shaders in similar fashion.

My tool also compresses files too if you hadn't noticed ;)


By the way, why are you putting $mode delphi command in the generated source files? They don't seem to need it.
I just wanted to be safe since the code works with Delphi and freepascal...I always work with Delphi mode code :)


Also, I'd suggest updating the links in the very first post as they are broken and you have to read through 3-page thread to find updated link. Why not adding the file as a forum attachment in the first post?
Good idea, I will update them right away!


...and keep up great work! :) Any chance to see GUI-based version of this tool?
Thanks mate! hmm...a GUI is a nice idea, I will think on it...

[Ñuño Martínez]

Any chance to see GUI-based version of this tool?

Thanks mate! hmm...a GUI is a nice idea, I will think on it...
I like a front-end program that uses the console program. Much like Lazarus itself. :)