Looking forward to the beta
Looking forward to the beta
Me too.
Right now I'm thinking in "data containers". I have an idea of a "packed file" container format similar to Ken Silverman's GRP files but using IFF I think it will work. I'm not sure if a "data container" component that keeps all things in memory is actually needed though.
No signature provided yet.
Why not pk3 aka zip ? You can open and operate them using any file manager. And the code is already there in standard Free Pascal package.
I made me a modified version that can use streams, not just open files directly. Want me to share? (LGPL)
Thank-you. Of course, share it. I was thinking about this a lot but I'm still open to new ideas. I'm using zlib/libpng license though and AFAIK it has some issues with LGPL.
Package systems I know forces a sequential access for files. I mean, you have to go file by file until you find the one you want (For example, TTarArchive.FindNext). If the package is compressed then it is mandatory. I don't want to load all data at once but allow to load the data when it is needed, so direct access is desirable.
Also, I need to bind the package with the Allegro file system so I can use Allegro to load data. Of course I can load data in a memory stream then interface the memory stream with the Allegro file system then use Allegro to "actually load" data, but why do it that way if I can bind directly the package with Allegro?
Finally, I like simple things and I find modern programming unnecessarily complex. IFF is simple, flexible and easy to parse. I think an IFF-based package system would be great.
No signature provided yet.
So why not use some different package/archive format instead of Tar which indeed does not support random read. You do know that Tar was initially developed to be used for storing data to magnetic tapes which by themselves also did not support random access.
There are numerous package systems that do allow you to read any of the contained file individually at any time. It isn't even so hard to make a custom one by yourself.
Wasn't there an extension for Allegro game library which added support for working with various packages? Or do I perhaps just remember some game developers to make one for their own game?
Well the problem of the IFF is that it still requires sequential reading since the file is divided into multiple chunks of various sizes. And in order to determine the size of first chunk you need to read its "header". In order to read the size of the second chunk you first need to know where it starts in order to read its "header" and that means the need to read the size of the first chink first.
Now what you would want is a package which has some sort of map (like file allocation table on disk drive) from which you can quickly read where certain file data begins and how big it is.
What is IFF ?
Classes are *simpler* than procedural programming because they allow for *much* easier modifications.
I've never looked at Allegro, so I cannot tell how hard coupling them would be.
Ok, here is my conversion of the unzip.pp straight from Free Pascal RTL.
Note that I did not check it for last-minute syntax errors as my engine is currently in complete disarray (not compiles). Another downside is that I haven't tried it in FPC 3 yet, may require replacement of AnsiString with RawByteString or something.
I tried removing the calls to the API of my engine but I may have missed some
It worked perfectly last year, compiled using FPC 2.6.4. I tested it on .pk3 files from OpenArena and my own .zip files. The average unpacking speed on my laptop was ~30 Mb/sec
http://chentrah.chebmaster.com/downloads/un_unzip.pp
So, what I did? I took the working but horribly outdated (API-wise) code and made it flexible by wrapping it into a class. The most important feature is it takes TStream as the source allowing you to support Unicode paths (if you have custom Unicode-supporting streams), work with your entire zip file loaded into memory (TMemoryStream) and even create support for nested zips with some extra effort.
I left *all* previous code commented out but not deleted, in case I encounter a rare bug resulting from improper conversion (none so far).
Its interface is not super easy to use, but! When you begin using pack files of any format you quickly realize you need a *directory* of files so that newer pack files can overwrite files from older ones and separate files in external folders should take precedence over filen in your pack files. Just like any game out there is programmed to do, from Quake 3 to fallout 4.
I cannot help with that, it's up to you to create directory you need.
What does this class do?
- Lets you iterate over the zip file's directory ti get file list
- Jump to specific file by name (It does iterate over the directory, in fact, but that is fast)
- unpack the next chunk of the currently selected file
To get file unpacked into memory (and you *have* to do that because unpacking is sequential and you applied routine would probably want to seek all over the unpacked file), I use code like this:
and, directory building example:Code:// this is optimized for my engine, lots of overhead due to multi-threaded asset loading // you may want to use something more elegant than the array of byte // f_zip is the TUnzip class instance (field of TZipFile) function TZipFile.ReadFile(fname: TFileNameString): TArrayOfByte; var ufi: unz_file_info; fn, comment: AnsiString; i, read_how_much: integer; err: boolean; q: Qword; begin [...] if not f_unzip.LocateFile(fname) then begin AddLog(RuEn( 'Загрузка прервана: файл не найден в контейнере!'#10#13' файл %0'#10#13' контейнер %1', 'Aborting the load: file not found in the container!'#10#13' file %0'#10#13' container %1' ), ['{$MODULE}' + fname, UnmangleFileName('{$MODULE}' + f_name)]); Exit(nil); end; Mother^.Timer.UsecDelta(@q); err:= not f_unzip.GetCurrentFileInfo (@ufi, fn, nil, 0, comment); if not err then err:= not f_unzip.OpenCurrentFile(); if not err then begin SetLength(Result, ufi.uncompressed_size); i:= 0; while i < Length(Result) do begin read_how_much:= f_unzip.ReadCurrentFile(@Result[i], Length(Result) - i); if read_how_much <=0 then begin err:= true; break; end; i+= read_how_much; end; end; if err then begin AddLog(RuEn( 'Загрузка прервана: ошибка чтения из контейнера!'#10#13' файл %0'#10#13' контейнер %1', 'Aborting the load: error reading from the container!'#10#13' file %0'#10#13' container %1' ), [fname, UnmangleFileName('{$MODULE}' + f_name)]); SetLength(Result, 0); end; if Mother^.Debug.Verbose then AddLog(' ..unpacked %0 bytes in %1 μs', [Length(Result), round(Mother^.Timer.UsecDelta(@q))]); end;
Code:function TZipFile.Initialize: boolean; //is ONLY called by threaded tasks from their thread! var ufi: unz_file_info; i: integer; fn, comment: AnsiString; begin Result:= inherited Initialize; if Result and not Assigned(f_unzip) then begin if Assigned(f_unzip) then Exit(true); try if Mother^.Debug.Verbose then AddLog(' Attaching a zlib object to the stream ...'); f_unzip:= TUnzip.Create(f_stream); repeat if not f_unzip.GetCurrentFileInfo (@ufi, fn, nil, 0, comment) then break; if ufi.uncompressed_size <= 0 then continue; //there are folders and shit //now there is my engine adding that file to its own directory SetLength(f_list, Length(f_list) + 1); with f_list[High(f_list)] do begin name:= '{$MODULE}' + fn; pak_file:= Self; Hash.FileAge:= FileDateToDateTime(ufi.dosDate); Hash.NameMd5:= Md5String(name); Hash.FileSize:= ufi.uncompressed_size; if Mother^.Debug.Verbose then AddLog(' %0 , size=%1, ratio=%2%', [name, ufi.uncompressed_size, round (100.0 * ufi.compressed_size / max(1, ufi.uncompressed_size))]); end; until not f_unzip.GoToNextFile(); if Mother^.Debug.Verbose then AddLog(' Parsed successfully, found %0 files.', [Length(f_list)]); except AddLog(RuEn( 'Не удалось инициализировать распаковщик unzip'#10#13#10#13'%0', 'Failed to initialize unzip unpacker'#10#13#10#13'%0') ,[StopDying()]); Result:= false; end; end; end;
I like this.
Yes, there is: The PhysicsFS add-on but I have no idea how to use it. I suppose PhysicsFS is a library you install in your Windows/Linux system then it calls it or something.
I'm used to the "old" API (Allegro 4.x) and I'm still a bit confused with some parts of the new one (5.x), it is mostly designed as a state machine now but the file IO looks like a state machine mixed with other stuff. Not sure how to use it.
Yes, I know about that IFF limitation, but I have two solutions:
- Include a chunk at the begginning with the map of the file telling where each file is, in a similar way Ken Silverman's GRP files does.
- Scan the whole file when opening building the map in memory.
It is a standard container file format developed by Electronic Arts and Commodore/Amiga inspired by some old Apple MacOS API widely used by AmigaOS. It's still in use by a few formats like Apple's AIFF and Microsoft's WAV (this one using a modified version called RIFF though). PNG file format uses a similar approach too.
You can read the complete description in this paper.
Thanks for sharing this.
Didn't know ZIP files allows random access to files. May be I'll rethink this.
No signature provided yet.
Yes they do. In fact I believe this is one of the main features why ZIP archives became so popular even thou they were other arhive types which offered better compression ratios like RAR archives for instance. That is also the reason why it was possible to extend Windows Explorer to allow working with ZIP files like they are just regular folders.
Before to work in the data containers I was working on the graphics subsystem.
The old way to initialize the graphics display was quite complex. There were a method to create it telling the size, depth and if we want a window or full screen. Then I added a method that implements a few pre-defined "legacy graphics modes".
I didn't liked it so I've finally rewrote the initialization code with a simpler one. Now there's only one method that receives a single parameter: an identifier that tells the kind of display we want (i.e. the bigger definition available, or simulate an old graphics mode as the PC-VGA or the C64-multicolor mode). Then the class decides the actual size and mode (windowed or full screen) of the display from the system information and the configuration. It is possible to drive the way the display will be created by using the configuration file. For example, we can force to create a window of a specific size.
It also defines most methods as virtual so the default behavior can be modified as needed. For example, current code creates the display using true color pixels allways. I'm planning to add to the engine a display class that initializes the code using a shader that simulates 8bpp palette modes (i.e. VGA/MCGA) so it will allow to use color animations and the actual color values of old computers. I just have to figure how to load the color palette information from image files as Allegro 5 doesn't do it.
No signature provided yet.
Ok, I got it.
I've just finished the data package subsystem. There's an abstract base class that defines the interface and defines a few basic tasks. Then you can extend it to use any package you want. Right now I've included a simple custom IFF-based package system, format description has been included in documentation.
I had to add a binding between Object Pascal TStream and Allegro's file system too. I had a few problems with this (the way C and Pascal use pointers and parameters makes binding stuff a bit tricky) but now it works seamlessly and you can use Allegro's functions to load stuff (bitmaps, fonts, sounds...) from my package subsystem.
I think it's time to release a new alpha version of the engine. May be this week-end or sooner.
No signature provided yet.
Bookmarks