Pak Files?

[xGTx]

Ok, you guys know how some games use .pak files to store all their game resources? From images to sounds... scripts to maps.... Well i was wondering if there's maybe already a way out there to do this with delphi. I'd like to compile all my games content into 1 .pak file and just read all the stuff out of that.

[WILL]

I think your looking at making a game engine. In essence your game executuable will be running the .pak file of yours and all you'd need to do then would be to make tools that create and modify new .pak files.

I'd reccomend looking into how games like Doom and Quake were designed. Also compression and file and memory stream functions would be of possitive use to you in this.

I hope that at least has given you some ideas where to look. Sorry I couldn't be more helpful at the moment.

[Mrwb]

You're looking for resource managament in a game engine ;). A google search will probably give you some generall ideas on what to do in case you want some in-depth theory on how to write efficient managers with cache etc.

For a way to pack files in delphi, check out this thread: http://terraqueous.f2o.org/dgdev/viewtopic.php?t=148&

It's not hard to pack all your files into one file, but it can be hard to do this efficient. I tried searching around a bit, but couldn't find anything special. I believe www.gamedev.net has a few articles on the subject, but I couldn't find them right now. Anyway, I suggest you search around. ;)

[xGTx]

Thanks alot guys, i'll have a look. And this IS a game engine im developing :)

[xGTx]

I've been searching the internet and google for some information but am coming up short. FileStreams... MemoryStreams... are the weakest link in my coding abilities. So im tryin my best. Alimonster apparently had some kind of example on how to do this, but the download is gone... It was posted over 2 years ago... I wish i could get a hold of that one.

What information would i need about each file in the header? FileOffset, FileSize, FileName, FileType? And how do i store how many headers i need to read?

[Alimonster]

[quote="xGTx"]I've been searching the internet and google for some information but am coming up short. FileStreams... MemoryStreams... are the weakest ]
I'll grab the tutorial and code from home tonight (if my floppy disc holds up...) and will try to email it to you tomorrow or on the weekend.

[Harry Hunt]

I posted this before, maybe I forgot to press submit or something :P :

You might want to check out the Package Editor/package loader tool/unit that comes with xcess. It's pretty straightforward and you don't need to use the other xcess components in order to be able to use the package editor and loader.

[xGTx]

Would creating a class with data objects in it and using WriteComponentRes be a good way to accomplish this.. Is this how you did it harry hunt? I still wants to see alimonster's way though.

[Harry Hunt]

I only used WriteComponentRes to save the package-file index because I was too lazy to use records (which would've been the more elegant way of doing it). In that index I store information about the number of files and the size of each file as well as the file names, etc.
From the file size I can then calculate the position of each file in the package by simply summing up the sizes (Storing an offset in the index table is more effective though).

[xGTx]

How do you accomplish reading and writing to a table?

[Harry Hunt]

It's not really a table, it's a collection which is like a list of components. You might know collections from TListView (Items property), TTreeNode (dito), TMainMenu...
You can add to these lists by creating a new TCollectionItem and setting the collection as the owner

MyItem := TCollectionItem.Create(MyCollection);


The problem with all package formats is the fact that in order to add a file to the package (and also when deleteting or modifying files), the package file has to be re-written completely. There are several workarounds for this. You could save the file table at the end of your file which would at least allow you to add files without having to re-write the entire file.

Alternatively (but this is much harder to implement), you could have fixed-size package files. Which means even an empty package file would be like 100 MB. You'd then have to write your own managing routines that split up files added to the package into clusters. When a file is deleted you simply mark all clusters that file occupied with "available". (This is how FAT/NTFS/etc. work) Again, this is very difficult to implement but of course possible.

[Harry Hunt]

If I have a little time today, I will write a tutorial about it.

[xGTx]

Thank you. Im rooting for the tutorial... File streams are so hard to understand.

[Alimonster]

Here's the text of the tutorial from my old site. I'm looking forward to Harry Hunt's one because his packing utility is pretty sweet.

I don't have anywhere to put the source code so see my PM and I'll send it your way by email.


Introduction

Sometimes you don't want to supply 5,000 different files with your project. It's untidy and it uses up quite a lot of space (for example, the images\buttons folder for a default Delphi install has 59K of bitmaps that take up over 5 megs on my FAT32 drive!). The obvious solution here is to crunch all those files together into one file and read each into memory as necessary - much cleaner!

How To Do It

The technique is relatively straightforward, though it presents many annoying traps. We separate the packed file into two sections:

1. The data (each file to be packed)
2. Information about that data (name, size)

The first part is used to store each file's contents (e.g. entire bitmaps). The files will be stored one after another without any space in between. As a result, we need some way to figure out the size of each file and where it is in the overall packed file - this is where the second part comes in!

Let's take a look at the first step, which is to splat each file together into one big gloopy mess. If you're not sure of your file handling, it might be time for you to brush up.

This is the procedure we follow. Note that the following is pseudocode - I'll be adding to it as we go along, so code won't be shown. If you want code, you can download the source code at the end. The pseudocode below will be expanded on, so don't use this version:


create a new output file
for each file in your list
if current input file exists then
get size of current input file
open current input file
resize buffer to wanted amount
read contents of the input file into the buffer
write the buffer to the output file
close input file
end if
end for
close the output file

(Note: do not use the above pseudocode!)

We want to copy each file over into one gigantic packed file, so we create a new file to store everything. For each to-be-packed file, we read the contents and write to the still-opened output file. This has the result of creating a big file with each individual item squashed together. There's a step in between - we have to use a buffer to store the read information first before we can write it. The bigger the buffer here, the better, up to the file's size. If that's too large, though, you can repeatedly read fixed amounts into a buffer until no more is left for the current file.

However, you might have noticed a small problem with the above pseudocode: there's no way to tell where one file starts in the overall pack, making it less than useful!

Storing File Info

We want to store information about each file somewhere, which will let us figure out the starting position and sizes. My chosen technique is to store the absolute position of each file (offset in bytes from the start of the file, position 0). This can be figured out as you write each to the pack by keeping an integer total and adding file_size to it for each input file. Uh, did I make that less-than-clear enough? :P

I'm going to refer this file-specific info as a header. It can equally be a footer, of course, depending on where you decide to stick it in the packed file ;).

The first problem is determining the format of the header. What info needs to be stored for each entry?

* An ID (string)
* The offset in bytes to this file

Anything else is just frills. The ID is necessary so that you can read the file - you may want to say "load somepic.bmp" instead of "load file number 2," because you may not always know the exact index of each file in the pack. The offset is required so that you know where the file starts. Any other details, such as the size, can be calculated later from this info.

The advantage of using a filename is that the type of the file is easily found - simply check the file extension (e.g. ".bmp" for a bitmap). You can add any other data required as additional fields in the record, though, so bear that in mind. The example record above is the minimum you'd get away with for most cases, but the format is entirely up to you.

Next up, there's a minor question. Should each entry be a fixed size? This is simply for the purposes of reading/writing the header at appropriate times. If each entry is a fixed size then the entire header can be read at once, instead of having a for loop reading each individual entry. It's your call again. I'm choosing to be lazy, so I will use a fixed size record. In fact, I'll go one further and use a shortstring instead of a string for the ID :).

type
TPackedFileEntry = record
Name: String[32];
Offset: Int64; // offset in *bytes* to this file
end;

The above is info about each file in the pack. You can adjust the size of "name" there - I used 32, but you can increase or shorten it if you want or use normal, resizable strings instead (which can introduce small complications to the unwary). [The sample code uses a slightly smaller string, in case you were wondering, plus a packed record.]

At this point, it's time for a little update of our wanted format. This is the new format:

Original file - whatever was sitting there, optional
Packed data - our extra files
Header - n entries, one for each file we packed and added [it's a footer actually]
Header size - an Int64 giving the header size (n entries * sizeof(each entry))
A signature - the string 'pack'

What's that? Yep, the header has now become a footer, plus we have some extra stuff at the start and end! This change makes the system more adaptable.

The original file refers to your executable, or indeed any other file onto which a pack can be added. This is optional (because you may want your pack files separate for clarity) but the format allows exes to have extra stuff embedded afterwards. The exe will still load and run, but with the added bonus that it's self-contained with its data.

Think about what happens if we want to add our packed file to the end of something else (for example, our main executable). We wouldn't have any idea of how large the original file was because we've padded it with our extra crap! The obvious (or not) solution is to check backwards from the end of the file. We always know where the end of the file is ;).

When reading the file, we have to check whether it contains packed stuff first, since that's not a certainty. The simplest method is to write a fixed number of bytes in a signature pattern (in this case, making the word 'pack', but feel free to use whatever you want) to the end. It's very unlikely that an unpacked file would end in those exact bytes, hopefully, so it serves as a good enough check.

The next step is to read the header info - once we know that, it's plain sailing! The first problem is figuring out where that header starts, though, since it can contain different amounts of files! This can be solved by reading in an offset in the file giving the header size, rather than a fixed position. We can seek back however many bytes from the end (skipping the packed-file ID) and then read the header all at once! From that, we know the position of each file and how big the original file was (the offset for the first file entry).

Phew!

Take a breath and re-read the above. The concept is straightforward enough; the only problem is side-stepping possible implementation difficulties (aka, "fiddly file handling").

Implementation

Here are some pseudocode functions (note the absence of type info, for example):

Generating the packed file


procedure make pack(filename, files to pack);
begin
if FileExists(filename) then
begin
current pos := file size(filename)
append to file(filename)
end
else
begin
current pos := 0
create new file(filename)
end if

// assume max entries possible - resize later
set header entry count(header, count(files to pack))

next entry := 0

for all files to pack do
begin
if file-to-be-packed exists then
begin
// step one: dump each file into a big messy gloop
open current file-to-be-packed
get file size
resize buffer
read data into buffer from file-to-be-packed
write buffer to output file

// maintain some info for the header
header[next entry]'s name := file-to-be-packed name
header[next entry]'s offset := current pos

close to-be-packed file

next entry := next entry + 1
current pos := current pos + input file size
end if
end for

// resize to the actual amount used
set header entry count(header, next entry)

// step two: write out the header info now
write file header
write file header size
write file signature('pack')

close output file
end

Reading a packed file

function is packed file(somefile): Boolean
begin
Result := (size(somefile) > 4 chars) and
last four chars(somefile) = 'pack'
end

procedure get header(somefile, header)
var
header_size: Int64
begin
if not is packed file(somefile) then
explode horribly

move to end of file(somefile)

// find the header size - ignore the last 4 bytes
seek backwards(4 bytes + SizeOf(Int64))

read header size(somefile, header size)
seek(end of file - 4 bytes - SizeOf(Int64) - header size) // yeesh!

set header entry count(header,
header size div SizeOf(TPackedFileEntry))

read all header(somefile, header, header size)
end

function packed file size(header, which_file): Int64
begin
if which_file is_not_last then
Result := header[which_file + 1].offset - header[which_file].offset
else
Result := (packed_file_size - 4 - sizeof(int64) - header_size) - header[which_file].offset
end

Getting the size is a little fiddly. If we're not dealing with the last file then we can simply take the difference between two entries, since there's no space in between each file (the next file starts after the previous file's entirety). The last one, though, has extra bits after it for the header, etc., so it's not quite as clean. We can calculate it as "remove the extra gunk - the offset for the last file". The extra gunk in this case is the entire header (however many bytes), header size (an Int64) and 4 bytes for the signature.

Reading each file itself is so straightforward that it doesn't even merit pseudocode! You simply seek to the specified offset for that file then do a BlockRead/whatever for the given file size into your chosen output format. I find that streams are pretty handy for this btw, because most objects have a "LoadFromStream" method. You can read the values into a memory buffer of some sort (either a dynamic array or a TMemoryStream).

The Code

The above pseudocode will be enough to get you started. You can wrap the functions in a class. Alternatively, you can simply download the source code: packing.zip (11 K) for this tutorial instead, saving you the effort :).

You might want to store a CRC32 checksum per-file or to the overall pack, which would allow you to check whether a pack (or a particular file) is corrupt. EFG has CRC32 covered somewhere on his site (I forget exactly where, unfortunately.)

The above method *should* let you create packed files added onto exe files. I tested this theory with notepad; the exe ran properly and the packed data could be viewed in the example program. There may be some issues to do with sharing (i.e., reading from an exe while it's running). Well, finding out for yourself is half the fun, right?

[Harry Hunt]

excellent article, Alimonster!
I decided that since your article pretty much covers everything there is to say about creating package files I will write a little program that loads/saves package files to demonstrate how it is implemented in Delphi and maybe give away one or two secrets of how the xcess package editor works. But I won't be done with that before tomorrow...

[xGTx]

Wow, I cannot thank you guys enough. Every time i got a question, this forum never fails to teach me until I understand. You people are absolutely awsome. I wish there was something I could do for this forum...

[Harry Hunt]

You already do a lot for this forum. This forum needs people asking questions just as much as people who answer them. And maybe in a few months/years, you'll come here answering questions yourself...

[xGTx]

Yea when people ask questions about topics i know! FileStreams and MemoryStreams are new to me. Same with graphics stuff like DelphiX but im getting the hang of it.

[WILL]

Wow, I cannot thank you guys enough. Every time i got a question, this forum never fails to teach me until I understand. You people are absolutely awsome. I wish there was something I could do for this forum...

Thanks for your words xGTx. It's good to know that this site has been of help to you in your ventures. Like HH says, you will someday be able to tutor those that need to know something that you are quite familiar with. The best thing you can do to help is continue to be a part of our community. We need all the people like you we can get. ;)

Best of luck in your future projects.

[Harry Hunt]

http://hosted.xcessred.com/articles/packageditor.htm

Here's an article I wrote on how to write a package editor. I also put this in the tutorial section of DGDev.

[xGTx]

Well, i've been messing with file handling trying to learn it real well, and im curious... When I write a string to the filestream like:

Stream.Write('asd',66785);

What's all those random characters it writes to the file after 'asd' since the buffer is 3 chars long, not 66785.?

[Paulius]

Write writes the number of bytes you tell it to, if you're variable is only 3 bytes long and you attempt to write 66785 bytes then if not causing an access violation you'll also write 66782 bytes of whatever is in memory after your variable.

[xGTx]

Thanks.

Back onto my pak file question and from what alimonster wrote:

How do i find the location of my file headers? Since every file is suppose to be put in the top of the package file one after another and I add all the file information at the end like you said, how do i locate those at the end?

[cairnswm]

Out of interest have you thought of using a Zip file to store the information. The dzlib components have a method to read the zipped file directly into a stream thereby not leaving anything on the hard disk.

[Paulius]

Store you're file header count at the end of the file, then when reading seek to the end of the stream ?¢_" sizeof(integer) and read the header count, seek back NumberOfHeaders * sizeof(THeader) and there you are.

[xGTx]

Thanks Paulius.

Cairns, I have thought about it, but would just much rather learn this part my own.

[xGTx]

Ok i think i have most this technique down. But 1 question left.

When adding a file to a pack, I write the actual file data at the end of all the other file data, but above the headers. This position can be found by reading all the existing headers and just adding together the filesizes. After adding that data, you then add a header for that data at the very bottom of the file, leaving a few bytes to store how many headers there are at the very end of the file. Correct?

My question is, more of a clarification.... To write a file to a certian spot in a file, i simple to Stream.SeekTo(0, AllOfHeaderSizesAdded); then Stream.WriteBuffer(NewFileStream, Length(NewFileStream); ?

[Harry Hunt]

Almost correct, the Seek method takes two parameters, the first one is the Position in the stream (so AllOfHeaderSizesAdded) and the second parameter tells the method where to start counting. The second parameter can for example be soFromBeginning or soFromEnd so
Seek(23653, soFromBeginning) seeks to the 23653rd byte from the beginning of the file.

Also, your package files consist of three individual parts: the file data at the beginning, a list of all files and a footer which stores the file count.

When you want to append files to your package, you'll have to seek to the position right after the last file in the package and then write the new file there. You will then have to re-build the entire file index and the footer so it's a good idea to create a copy of the index list before appending a file.

Here's an image that should clarify the structure of a package file for you
http://hosted.xcessred.com/articles/img1.gif

[xGTx]

And how do i write a file in at an exact position? (Just so i dont mess this up)

[Harry Hunt]

var
PackageStream, FileStream: TFileStream;
BehindLastFile: Int64;
begin
{...}
FileStream.Create('C:\somefile.abc', fmOpenRead);
FileStream.Seek(0, soFromBeginning);
PackageStream.Seek(BehindLastFile, soFromBeginning);
PackageStream.CopyFrom(FileStream, FileStream.Size);
FileStream.Free;
ReBuildFileIndex;
ReBuildFooter;
end;

[M109uk]

I have written a pak editor that uses the same structure as Harry Hunt's, to compile the application you need delphi 7 and Jvcl component set, but the zip also contains the exe and a package.

The source code is a bit of a mess, i havnt really had much time to clear it yet. And their is still a few problems i need to sort out with in it!

The zip is 2.04mb : http://www.biocoders.org/kPak.zip

I hope it helps.

[xGTx]

Hey thanks alot.

[xGTx]

Why will this not work:

Function IsAPack(FileName:String):Boolean;
Var Stream : TFileStream;
Signature : String[6];
Begin
Result := False; // Prove me wrong!
Stream := TFileStream.Create(FileName, fmOpenRead);
Stream.Position := Stream.Size-6;
Stream.ReadBuffer(Signature, 6);
Stream.Free;
If Signature = 'GTPACK' Then
Result := True;
End;

[xGTx]

The above gives out GTPAC

I dont know why.

And also, how do i know how many headers I have?

Currently i've come up with doing it like this... Im not sure how efficient it is:

public
bob : Array of TPackageItem;
end;


n := 1;
While n <> 0 do
begin
SetLength(Bob, High(Bob)+2);
Stream.ReadBuffer(Bob[High(Bob)],SizeOf(TPackageItem));
If (Stream.Size - Stream.Position) < SizeOf(TPackageItem) Then n := 0;
end;

[Harry Hunt]

a)
try

function IsAPack(FileName: string): Boolean;
var
Stream: TFileStream;
Signature: string[6];
begin
Result := False; // Prove me wrong!
Stream := TFileStream.Create(FileName, fmOpenRead);
Stream.Seek(SizeOf(Signature), soFromEnd);
Stream.ReadBuffer(Signature, SizeOf(Signature));
Stream.Free;
if Signature = 'GTPACK' then
Result := True;
end;


b)
Your second question (how do you know how many headers you have) is answered by the image in my last post:
Your package file should have the following structure: the actual files followed by a file table follow by a footer. The footer can consist of only two values, the Signature and a value "FileCount" which tells you how many files there are in your package. You'd then do something like


PackageStream.Seek(SizeOf(Signature) + SizeOf(Int64) + (FileCount * SizeOf(TPackageItem)), soFromEnd);
for I := 1 to FileCount do
begin
Stream.Read(PackageItem, SizeOf(TPackageItem);
// Add the package item to a list
end;

[xGTx]

I am not sure why, but when i do:

Stream.Seek(SizeOf(Signature), soFromEnd);

The stream.position then becomes 125310 ... The file size is 125303

So it's obviously adding 7 wich is sizeof(signature) ... This shouldnt be happening right?

[Paulius]

This should be happening, soFromEnd does not reverse the direction ar anything, to seek back add a minus, Stream.Seek(-SizeOf(Signature), soFromEnd);

[Harry Hunt]

damn, I forgot the minus. Sorry 'bout that!

[xGTx]

Thanks paulius... and DAMN YOU Harry! Lol j/k! :P I was freaking out like my delphi was screwy or something... But yea, I started using a minus to see if it worked and it did... And anyway, I believe I have a pretty good understanding of filestreams in Delphi now, for wich I greatly thank you guys. I expect my packaging code to be finished by either tonight or tommarow. Thanks again!

Just curious... I tried:

Var Signature :String[6]
Begin
Signature := 'GTPACK';
ShowMessage(IntToStr(SizeOf(Signature)));
End;
And it returns '4' ... Shouldnt it return 6? 1 byter per character?

[cairnswm]

It should return 7 one for each Character and a single byte storing the lengeth of the string.

Short Strings come from old Turbo Pascal days where the maximum length of a string was 255 characters. Each string was actually an array of characters with the zero index being a byte representing the length of the string. (This also allows you to do something like Password[1] to return the first character in the Password string)

[xGTx]

Ok, sorry I havn't posted an update, but I have infact built a very cool packaging system and now have a great understanding about file streams. I appreciate all the help.

Now for a quicky, Does anyone know of a free, easy to use compression delphi code or component? I would like to implement compression into my packaging now. THanks :)

[Harry Hunt]

There's a zlib.pas on the Delphi CD

[Ultra]

And a link to show how to use it. :wink:
http://www.swissdelphicenter.ch/torry/showcode.php?id=1617

[edit this was the one I thought I was linking to at first, just show that there are many good thing over at torry.net]
http://www.swissdelphicenter.ch/torry/showcode.php?id=822

[cairnswm]

(I just wish the things at Torry.net were easier to find...)

I use ZipMaster etc:
http://www.geocities.com/rjpeters_au/zipmaster.html

Still being maintained which is nice - can also zip and unzip from a stream so no footprint on the harddisk.

However does require DLLs to be installed.

[xGTx]

Well, my packer works great... But I tried moving some of the techniques I learned over to making a complete GUI config file that would load/save in a type... Problem is, it will save the type... then i can load it fine. But if i close the app out, get back in and try and load the type it wont load... But if i save something else, then i can go back and load the type just fine... I have no idea what's going on, maybe one of you can take a look:

http://www.gtrpg.com/downloads/GUIConf.rar

[cairnswm]

I'm currently using something similar for load and save but I store each field to the stream. I'm using this to enable the encryption of user information and some config information using my CryptoUnit I published under tutorials.


TFavorite = Record
User : TName;
Name : TName;
FileName : TFileName;
End;
TFavorites = Class(TObject)
Private
FavoritesList : TList;
...
Public
Procedure Load;
Procedure Save;
...
End;

procedure TFavorites.Load;
Var
I : Integer;
Fav : TFavorite;
Mem : TMemoryStream;
begin
// Load all Favorites into list
If Not FileExists(FileName) then
Exit; // Blank Favorites file - will save on close
Crypto.Password := 'Favorites for '+Application.Title;
Mem := TMemoryStream.Create;
Mem := TMemoryStream.Create;
Crypto.Decrypt(Mem,FileName);
Mem.Seek(0,soFromBeginning);
While Mem.Position < Mem.Size do
Begin
Mem.Read(Fav.User,SizeOf(TName));
Mem.Read(Fav.Name,SizeOf(TName));
Mem.Read(Fav.FileName,SizeOf(TFileName));
FavoritesList.Add(@Fav);
End;
Mem.Free;

UserName := UserName; // Initialize Current User List
end;

procedure TFavorites.Save;
Var
I : Integer;
Fav : TFavorite;
Mem : TMemoryStream;
begin
Crypto.Password := 'Favorites for '+Application.Title;
Mem := TMemoryStream.Create;
For I := 0 to FavoritesList.Count - 1 do
Begin
Fav := TFavorite(FavoritesList[I]^);
Mem.Write(Fav.User,SizeOf(TName));
Mem.Write(Fav.Name,SizeOf(TName));
Mem.Write(Fav.FileName,SizeOf(TFileName));
End;
Crypto.Encrypt(Mem,FileName);
Mem.Free;
end;



Hope that helps.

[Paulius]

You try to take the name from the save dialog when both saving and loading. On the side note, you should really name you're components, when you're program gets big it's much nicer to see names like SaveButton rather than Button1. And you should really read the Object Pascal Style Guide http://community.borland.com/soapbox/techvoyage/article/1,1795,10280,00.htm