MinGRo game engine

[Thyandyr]

Looking forward to the beta

[Ñuño Martínez]

Looking forward to the beta

Me too.

Right now I'm thinking in "data containers". I have an idea of a "packed file" container format similar to Ken Silverman's GRP files but using IFF I think it will work. I'm not sure if a "data container" component that keeps all things in memory is actually needed though.

[Chebmaster]

Why not pk3 aka zip ? You can open and operate them using any file manager. And the code is already there in standard Free Pascal package.
I made me a modified version that can use streams, not just open files directly. Want me to share? (LGPL)

[Ñuño Martínez]

Why not pk3 aka zip ? You can open and operate them using any file manager. And the code is already there in standard Free Pascal package.
I made me a modified version that can use streams, not just open files directly. Want me to share? (LGPL)

Thank-you. Of course, share it. I was thinking about this a lot but I'm still open to new ideas. I'm using zlib/libpng license though and AFAIK it has some issues with LGPL.

Package systems I know forces a sequential access for files. I mean, you have to go file by file until you find the one you want (For example, TTarArchive.FindNext). If the package is compressed then it is mandatory. I don't want to load all data at once but allow to load the data when it is needed, so direct access is desirable.

Also, I need to bind the package with the Allegro file system so I can use Allegro to load data. Of course I can load data in a memory stream then interface the memory stream with the Allegro file system then use Allegro to "actually load" data, but why do it that way if I can bind directly the package with Allegro?

Finally, I like simple things and I find modern programming unnecessarily complex. IFF is simple, flexible and easy to parse. I think an IFF-based package system would be great.

[SilverWarior]

Package systems I know forces a sequential access for files. I mean, you have to go file by file until you find the one you want (For example, TTarArchive.FindNext).

So why not use some different package/archive format instead of Tar which indeed does not support random read. You do know that Tar was initially developed to be used for storing data to magnetic tapes which by themselves also did not support random access.

I don't want to load all data at once but allow to load the data when it is needed, so direct access is desirable.

There are numerous package systems that do allow you to read any of the contained file individually at any time. It isn't even so hard to make a custom one by yourself.

Also, I need to bind the package with the Allegro file system so I can use Allegro to load data.

Wasn't there an extension for Allegro game library which added support for working with various packages? Or do I perhaps just remember some game developers to make one for their own game?

Finally, I like simple things and I find modern programming unnecessarily complex. IFF is simple, flexible and easy to parse. I think an IFF-based package system would be great.

Well the problem of the IFF is that it still requires sequential reading since the file is divided into multiple chunks of various sizes. And in order to determine the size of first chunk you need to read its "header". In order to read the size of the second chunk you first need to know where it starts in order to read its "header" and that means the need to read the size of the first chink first.

Now what you would want is a package which has some sort of map (like file allocation table on disk drive) from which you can quickly read where certain file data begins and how big it is.

[Chebmaster]

What is IFF ?
Classes are *simpler* than procedural programming because they allow for *much* easier modifications.
I've never looked at Allegro, so I cannot tell how hard coupling them would be.

Ok, here is my conversion of the unzip.pp straight from Free Pascal RTL.

Note that I did not check it for last-minute syntax errors as my engine is currently in complete disarray (not compiles). Another downside is that I haven't tried it in FPC 3 yet, may require replacement of AnsiString with RawByteString or something.
I tried removing the calls to the API of my engine but I may have missed some

It worked perfectly last year, compiled using FPC 2.6.4. I tested it on .pk3 files from OpenArena and my own .zip files. The average unpacking speed on my laptop was ~30 Mb/sec

http://chentrah.chebmaster.com/downloads/un_unzip.pp

So, what I did? I took the working but horribly outdated (API-wise) code and made it flexible by wrapping it into a class. The most important feature is it takes TStream as the source allowing you to support Unicode paths (if you have custom Unicode-supporting streams), work with your entire zip file loaded into memory (TMemoryStream) and even create support for nested zips with some extra effort.

I left *all* previous code commented out but not deleted, in case I encounter a rare bug resulting from improper conversion (none so far).

Its interface is not super easy to use, but! When you begin using pack files of any format you quickly realize you need a *directory* of files so that newer pack files can overwrite files from older ones and separate files in external folders should take precedence over filen in your pack files. Just like any game out there is programmed to do, from Quake 3 to fallout 4.
I cannot help with that, it's up to you to create directory you need.

What does this class do?
- Lets you iterate over the zip file's directory ti get file list
- Jump to specific file by name (It does iterate over the directory, in fact, but that is fast)
- unpack the next chunk of the currently selected file

To get file unpacked into memory (and you *have* to do that because unpacking is sequential and you applied routine would probably want to seek all over the unpacked file), I use code like this:

// this is optimized for my engine, lots of overhead due to multi-threaded asset loading
// you may want to use something more elegant than the array of byte
// f_zip is the TUnzip class instance (field of TZipFile)
function TZipFile.ReadFile(fname: TFileNameString): TArrayOfByte;
var
  ufi: unz_file_info;
  fn, comment: AnsiString;
  i, read_how_much: integer;
  err: boolean;
  q: Qword;
begin
[...]
  if not f_unzip.LocateFile(fname)
  then begin
    AddLog(RuEn(
      'Загрузка прервана: файл не найден в контейнере!'#10#13' файл %0'#10#13' контейнер %1',
      'Aborting the load: file not found in the container!'#10#13' file %0'#10#13' container %1'
      ), ['{$MODULE}' + fname, UnmangleFileName('{$MODULE}' + f_name)]);
    Exit(nil);
  end;

  Mother^.Timer.UsecDelta(@q);
  err:= not f_unzip.GetCurrentFileInfo (@ufi, fn, nil, 0, comment);
  if not err then err:= not f_unzip.OpenCurrentFile();
  if not err then begin
    SetLength(Result, ufi.uncompressed_size);
    i:= 0;
    while i < Length(Result) do begin
      read_how_much:= f_unzip.ReadCurrentFile(@Result[i], Length(Result) - i);
      if read_how_much <=0 then begin
        err:= true;
        break;
      end;
      i+= read_how_much;
    end;
  end;
  if err then begin
    AddLog(RuEn(
      'Загрузка прервана: ошибка чтения из контейнера!'#10#13' файл %0'#10#13' контейнер %1',
      'Aborting the load: error reading from the container!'#10#13' file %0'#10#13' container %1'
      ), [fname, UnmangleFileName('{$MODULE}' + f_name)]);
    SetLength(Result, 0);
  end;
  if Mother^.Debug.Verbose then AddLog('      ..unpacked %0 bytes in %1 μs', [Length(Result), round(Mother^.Timer.UsecDelta(@q))]);
end;

and, directory building example:

function TZipFile.Initialize: boolean; //is ONLY called by threaded tasks from their thread!
var
  ufi: unz_file_info;
  i: integer;
  fn, comment: AnsiString;
begin
  Result:= inherited Initialize;
  if Result and not Assigned(f_unzip) then begin
    if Assigned(f_unzip) then Exit(true);
    try
      if Mother^.Debug.Verbose then AddLog('  Attaching a zlib object to the stream ...');
      f_unzip:= TUnzip.Create(f_stream);
      repeat
        if not f_unzip.GetCurrentFileInfo (@ufi, fn, nil, 0, comment) then break;
        if ufi.uncompressed_size <= 0 then continue; //there are folders and shit
		
		//now there is my engine adding that file to its own directory 
        SetLength(f_list, Length(f_list) + 1);
		with f_list[High(f_list)] do begin
          name:= '{$MODULE}' + fn;
          pak_file:= Self;
          Hash.FileAge:= FileDateToDateTime(ufi.dosDate);
          Hash.NameMd5:= Md5String(name);
          Hash.FileSize:= ufi.uncompressed_size;
          if Mother^.Debug.Verbose then AddLog('    %0 , size=%1, ratio=%2%',
            [name, ufi.uncompressed_size, round (100.0 * ufi.compressed_size / max(1, ufi.uncompressed_size))]);
        end;
      until not f_unzip.GoToNextFile();
      if Mother^.Debug.Verbose then AddLog('  Parsed successfully, found %0 files.', [Length(f_list)]);
    except
      AddLog(RuEn(
        'Не удалось инициализировать распаковщик unzip'#10#13#10#13'%0',
        'Failed to initialize unzip unpacker'#10#13#10#13'%0')
        ,[StopDying()]);
      Result:= false;
    end;
  end;
end;

[Ñuño Martínez]

So why not use some different package/archive format instead of Tar which indeed does not support random read. You do know that Tar was initially developed to be used for storing data to magnetic tapes which by themselves also did not support random access.

(...)

There are numerous package systems that do allow you to read any of the contained file individually at any time. It isn't even so hard to make a custom one by yourself.

I like this.

Wasn't there an extension for Allegro game library which added support for working with various packages? Or do I perhaps just remember some game developers to make one for their own game?

Yes, there is: The PhysicsFS add-on but I have no idea how to use it. I suppose PhysicsFS is a library you install in your Windows/Linux system then it calls it or something.

I'm used to the "old" API (Allegro 4.x) and I'm still a bit confused with some parts of the new one (5.x), it is mostly designed as a state machine now but the file IO looks like a state machine mixed with other stuff. Not sure how to use it.

Well the problem of the IFF is that it still requires sequential reading since the file is divided into multiple chunks of various sizes. And in order to determine the size of first chunk you need to read its "header". In order to read the size of the second chunk you first need to know where it starts in order to read its "header" and that means the need to read the size of the first chink first.

Now what you would want is a package which has some sort of map (like file allocation table on disk drive) from which you can quickly read where certain file data begins and how big it is.

Yes, I know about that IFF limitation, but I have two solutions:

• Include a chunk at the begginning with the map of the file telling where each file is, in a similar way Ken Silverman's GRP files does.
• Scan the whole file when opening building the map in memory.

What is IFF ?

It is a standard container file format developed by Electronic Arts and Commodore/Amiga inspired by some old Apple MacOS API widely used by AmigaOS. It's still in use by a few formats like Apple's AIFF and Microsoft's WAV (this one using a modified version called RIFF though). PNG file format uses a similar approach too.

You can read the complete description in this paper.

Classes are *simpler* than procedural programming because they allow for *much* easier modifications.
I've never looked at Allegro, so I cannot tell how hard coupling them would be.

Ok, here is my conversion of the unzip.pp straight from Free Pascal RTL.

(...)

Thanks for sharing this.

Didn't know ZIP files allows random access to files. May be I'll rethink this.

[SilverWarior]

Didn't know ZIP files allows random access to files. May be I'll rethink this.

Yes they do. In fact I believe this is one of the main features why ZIP archives became so popular even thou they were other arhive types which offered better compression ratios like RAR archives for instance. That is also the reason why it was possible to extend Windows Explorer to allow working with ZIP files like they are just regular folders.