MinGRo game engine

[Ñuño Martínez]

Last month I didn't work as much as I would like on MinGRo. Anyway I've added a complete command-line simulation so it is now quite simple to add an in-game console, even use the command interpreter to define a batch-like scripting language. It isn't fast and lacks any memory management though.

Also, I've done some internal improvements, bug fixing and documentation updates.

Next LudumDare is near so I decided to make a first alpha release so I can use it in the contest. You can test it too.

https://sourceforge.net/projects/mingro/files/1.alpha/

[Ñuño Martínez]

So I'm back, and I decided to go straight on to finish this engine, or at least a "beta" version.

The first I've done is to rewrite the animation manager and the sprite engine subsystem. In both cases I used classes for everything so each time the number of animations or sprites changed the engine has to destroy and/or create a lot of tiny objects. So I've replaced some classes by records (as discussed here): Less memory fragmentation and less spreading.

Also I've decided to use more ideas from Diana Gruber's "Action Arcade Aventure Kit" book in the sprite engine. As a side effect the change has reduced the code complexity: I think the code (both the engine and the examples) are cleaner and more easy to read and follow. I don't know if it is faster now, but I like it.

[Thyandyr]

Looking forward to the beta

[Ñuño Martínez]

Looking forward to the beta

Me too.

Right now I'm thinking in "data containers". I have an idea of a "packed file" container format similar to Ken Silverman's GRP files but using IFF I think it will work. I'm not sure if a "data container" component that keeps all things in memory is actually needed though.

[Chebmaster]

Why not pk3 aka zip ? You can open and operate them using any file manager. And the code is already there in standard Free Pascal package.
I made me a modified version that can use streams, not just open files directly. Want me to share? (LGPL)

[Ñuño Martínez]

Why not pk3 aka zip ? You can open and operate them using any file manager. And the code is already there in standard Free Pascal package.
I made me a modified version that can use streams, not just open files directly. Want me to share? (LGPL)

Thank-you. Of course, share it. I was thinking about this a lot but I'm still open to new ideas. I'm using zlib/libpng license though and AFAIK it has some issues with LGPL.

Package systems I know forces a sequential access for files. I mean, you have to go file by file until you find the one you want (For example, TTarArchive.FindNext). If the package is compressed then it is mandatory. I don't want to load all data at once but allow to load the data when it is needed, so direct access is desirable.

Also, I need to bind the package with the Allegro file system so I can use Allegro to load data. Of course I can load data in a memory stream then interface the memory stream with the Allegro file system then use Allegro to "actually load" data, but why do it that way if I can bind directly the package with Allegro?

Finally, I like simple things and I find modern programming unnecessarily complex. IFF is simple, flexible and easy to parse. I think an IFF-based package system would be great.

[SilverWarior]

Package systems I know forces a sequential access for files. I mean, you have to go file by file until you find the one you want (For example, TTarArchive.FindNext).

So why not use some different package/archive format instead of Tar which indeed does not support random read. You do know that Tar was initially developed to be used for storing data to magnetic tapes which by themselves also did not support random access.

I don't want to load all data at once but allow to load the data when it is needed, so direct access is desirable.

There are numerous package systems that do allow you to read any of the contained file individually at any time. It isn't even so hard to make a custom one by yourself.

Also, I need to bind the package with the Allegro file system so I can use Allegro to load data.

Wasn't there an extension for Allegro game library which added support for working with various packages? Or do I perhaps just remember some game developers to make one for their own game?

Finally, I like simple things and I find modern programming unnecessarily complex. IFF is simple, flexible and easy to parse. I think an IFF-based package system would be great.

Well the problem of the IFF is that it still requires sequential reading since the file is divided into multiple chunks of various sizes. And in order to determine the size of first chunk you need to read its "header". In order to read the size of the second chunk you first need to know where it starts in order to read its "header" and that means the need to read the size of the first chink first.

Now what you would want is a package which has some sort of map (like file allocation table on disk drive) from which you can quickly read where certain file data begins and how big it is.

[Chebmaster]

What is IFF ?
Classes are *simpler* than procedural programming because they allow for *much* easier modifications.
I've never looked at Allegro, so I cannot tell how hard coupling them would be.

Ok, here is my conversion of the unzip.pp straight from Free Pascal RTL.

Note that I did not check it for last-minute syntax errors as my engine is currently in complete disarray (not compiles). Another downside is that I haven't tried it in FPC 3 yet, may require replacement of AnsiString with RawByteString or something.
I tried removing the calls to the API of my engine but I may have missed some

It worked perfectly last year, compiled using FPC 2.6.4. I tested it on .pk3 files from OpenArena and my own .zip files. The average unpacking speed on my laptop was ~30 Mb/sec

http://chentrah.chebmaster.com/downloads/un_unzip.pp

So, what I did? I took the working but horribly outdated (API-wise) code and made it flexible by wrapping it into a class. The most important feature is it takes TStream as the source allowing you to support Unicode paths (if you have custom Unicode-supporting streams), work with your entire zip file loaded into memory (TMemoryStream) and even create support for nested zips with some extra effort.

I left *all* previous code commented out but not deleted, in case I encounter a rare bug resulting from improper conversion (none so far).

Its interface is not super easy to use, but! When you begin using pack files of any format you quickly realize you need a *directory* of files so that newer pack files can overwrite files from older ones and separate files in external folders should take precedence over filen in your pack files. Just like any game out there is programmed to do, from Quake 3 to fallout 4.
I cannot help with that, it's up to you to create directory you need.

What does this class do?
- Lets you iterate over the zip file's directory ti get file list
- Jump to specific file by name (It does iterate over the directory, in fact, but that is fast)
- unpack the next chunk of the currently selected file

To get file unpacked into memory (and you *have* to do that because unpacking is sequential and you applied routine would probably want to seek all over the unpacked file), I use code like this:

// this is optimized for my engine, lots of overhead due to multi-threaded asset loading
// you may want to use something more elegant than the array of byte
// f_zip is the TUnzip class instance (field of TZipFile)
function TZipFile.ReadFile(fname: TFileNameString): TArrayOfByte;
var
  ufi: unz_file_info;
  fn, comment: AnsiString;
  i, read_how_much: integer;
  err: boolean;
  q: Qword;
begin
[...]
  if not f_unzip.LocateFile(fname)
  then begin
    AddLog(RuEn(
      'Загрузка прервана: файл не найден в контейнере!'#10#13' файл %0'#10#13' контейнер %1',
      'Aborting the load: file not found in the container!'#10#13' file %0'#10#13' container %1'
      ), ['{$MODULE}' + fname, UnmangleFileName('{$MODULE}' + f_name)]);
    Exit(nil);
  end;

  Mother^.Timer.UsecDelta(@q);
  err:= not f_unzip.GetCurrentFileInfo (@ufi, fn, nil, 0, comment);
  if not err then err:= not f_unzip.OpenCurrentFile();
  if not err then begin
    SetLength(Result, ufi.uncompressed_size);
    i:= 0;
    while i < Length(Result) do begin
      read_how_much:= f_unzip.ReadCurrentFile(@Result[i], Length(Result) - i);
      if read_how_much <=0 then begin
        err:= true;
        break;
      end;
      i+= read_how_much;
    end;
  end;
  if err then begin
    AddLog(RuEn(
      'Загрузка прервана: ошибка чтения из контейнера!'#10#13' файл %0'#10#13' контейнер %1',
      'Aborting the load: error reading from the container!'#10#13' file %0'#10#13' container %1'
      ), [fname, UnmangleFileName('{$MODULE}' + f_name)]);
    SetLength(Result, 0);
  end;
  if Mother^.Debug.Verbose then AddLog('      ..unpacked %0 bytes in %1 μs', [Length(Result), round(Mother^.Timer.UsecDelta(@q))]);
end;

and, directory building example:

function TZipFile.Initialize: boolean; //is ONLY called by threaded tasks from their thread!
var
  ufi: unz_file_info;
  i: integer;
  fn, comment: AnsiString;
begin
  Result:= inherited Initialize;
  if Result and not Assigned(f_unzip) then begin
    if Assigned(f_unzip) then Exit(true);
    try
      if Mother^.Debug.Verbose then AddLog('  Attaching a zlib object to the stream ...');
      f_unzip:= TUnzip.Create(f_stream);
      repeat
        if not f_unzip.GetCurrentFileInfo (@ufi, fn, nil, 0, comment) then break;
        if ufi.uncompressed_size <= 0 then continue; //there are folders and shit
		
		//now there is my engine adding that file to its own directory 
        SetLength(f_list, Length(f_list) + 1);
		with f_list[High(f_list)] do begin
          name:= '{$MODULE}' + fn;
          pak_file:= Self;
          Hash.FileAge:= FileDateToDateTime(ufi.dosDate);
          Hash.NameMd5:= Md5String(name);
          Hash.FileSize:= ufi.uncompressed_size;
          if Mother^.Debug.Verbose then AddLog('    %0 , size=%1, ratio=%2%',
            [name, ufi.uncompressed_size, round (100.0 * ufi.compressed_size / max(1, ufi.uncompressed_size))]);
        end;
      until not f_unzip.GoToNextFile();
      if Mother^.Debug.Verbose then AddLog('  Parsed successfully, found %0 files.', [Length(f_list)]);
    except
      AddLog(RuEn(
        'Не удалось инициализировать распаковщик unzip'#10#13#10#13'%0',
        'Failed to initialize unzip unpacker'#10#13#10#13'%0')
        ,[StopDying()]);
      Result:= false;
    end;
  end;
end;