virtual file system in pascal/delphi

[noeska]

Thanks to a question of Brainer on the pgd forum ( http://www.pascalgamedevelopment.com/viewtopic.php?t=5537&postdays=0&postorder=asc&start=0 ) i started writing a virtual file system in pascal. Things now become useable.

Now it is alpha 7 time already. Now it is also possible to delete files. Deleted blocks are reused again.
This only works for tvirtualfilestream bases access to the tvirtualfilesystem. So writedata (and readdata) need to be rewritten to use a tvirtualfilestream. Besides making reusing deleted blocks work with those it save some double code.
With reusing deleted block the tvirtualfilesystem may end up to be defragmented. Hmm how to write such an thing?

Who are willing to give this alpha version a test run? It comes with an basic example i use for testing.
I still have to give the virtualfilesystem a proper name and i am out of ideas so suggestions are welcome.

Download: http://www.noeska.net/downloads/VirtualFileSystemAlpha7.zip

[Brainer]

Nice you're progressing with your code, I'm too busy right now to do anything, that's why I didn't respond earlier. As soon as I get some time (as soon as some motivation as well) I'll post my code, too.

Anyway, congrats noeaska for your constant development, thanks for that. :)

[noeska]

A short tutorial:

Opening a virtual file system.


TestDataFile := TVirtualFileSystem.Create();
TestDataFile.FileName := 'myfilesystem.dat';
TestDataFile.Open;


Writing a tmemo (mTest) inside the virtual file system:


MyStream:= TvfsFileStream.Create('test.txt', TestDataFile);
mTest.Lines.SaveToStream(MyStream);
MyStream.Free;


Reading a tmemo (mTest) from inside the virtual file system:


MyStream:= TvfsFileStream.Create('test.txt', TestDataFile);
mTest.Lines.LoadFromStream(MyStream);
MyStream.Free;


Also you can download alpha8 here: http://www.noeska.net/downloads/VirtualFileSystemAlpha8.zip
ReadData and WriteData now use TVirtualFileStream and are only 4 lines long now :-)

[chronozphere]

wow... awesome stuff. Looks really easy to use. :)

Are you going to implement directories?

Keep up the good work!

[noeska]

for now there is only one main directory. But you can cheat by using / in the filename. Just make sure it fits within 255 chars total length. I am not sure if giving each directory its own file within the vfs is an advantage or just complicates things.

Also i discovered an bug that does not allow a file withing the vfs to become smaller. So i need to add code for that first.

[Brainer]

For those who thought my project is dead - it's back on tracks. :) I have nothing to show yet, but there is going to be a directories support and many more like compression and encryption per file (means you don't have to encrypt/compress all files, you can choose which files you want to affect).

Stay tuned. :)

[noeska]

2008-12-21: Alpha9 is released. When a (virtual)file becomes smaller additional blocks are marked as deleted. Should work, but needs additional testing. If this goes ok it is almost time for an real beta version.

Download: http://www.noeska.net/downloads/VirtualFileSystemAlpha9.zip

My plan on encryption is make it pluggeable. I do not want to implement encryption support in the virtualfilesystem code. So the vfs class gets a method called SetEncryptionMethod(AEncryptMethod: Object);
For now i am puzzled on should i make AEncryptMethod an Object Pointer or TEncryptProcedure?

[JSoftware]

I would make the argument a class type, so the VFS can instantiate the encrypter class at will

[noeska]

The vfs has a name. Not the most original, but it has a name now and it is: nvfs . Or in full: Noeska's Virtual File System .
The project site is being setup now. (ps it almost took me more time to setup wordpress mu then to write nvfs )

Project Site: http://www.noeska.net/projects/nvfs/
Download version 1.0b: http://www.noeska.net/projects/files/2008/12/nvfs.zip

The basics are working now so i took it out of alpha and into beta.

I need beta testers now. So test it if you dare...

[chronozphere]

There seems to be a problem with your website:


Fatal Error
If your blog does not display, please contact the owner of this site.

If you are the owner of this site please check that MySQL is running properly and all tables are error free.

Could Not Find Blog!
Searched for www.noeska.net/ in wp_noeska_net::wp_blogs table. Is that right?

What do I do now?
Read the bug report page. Some of the guidelines there may help you figure out what went wrong.
If you're still stuck with this message, then check that your database contains the following tables:

* wp_blogs
* wp_users
* wp_usermeta
* wp_site
* wp_sitemeta
* wp_sitecategories

If you suspect a problem please report it to the support forums but you must include the information asked for in the WPMU bug reporting guidelines!

I always get to see this, no matter on which link i click.

[noeska]

It should work again now...

I was trying to make a blog at /projects/nvfs/ instead of /projects/ and wordpress does not like that. Using mod rewrite did not help either. So it is back to defaults now.

[Brainer]

Hello there! :D

noeska got ahead of me, so I can't give as good as he gets. Here's what I've got with my Alpha1 release of Brain Virtual File System. You can get it here (http://www.pateman.net76.net/BrainVFS_1.0_Alpha1.zip). What's missing are:
:arrow: encryption and compression
:arrow: loading archives
:arrow: removing files
It is distributed under the BSD license.

The following code is an example of usage:

const
TestFile = 'C:\Users\Patryk\Documents\Anime List.txt';
TestFile1 = 'C:\Mal Updater\license.txt';
var
VFS: TBrainVirtualFileSystem;
begin
VFS := TBrainVirtualFileSystem.Create('C:\test.dat');
try
VFS.NewFolder(VFSRootDir, 'New folder\');
VFS.NewFile(TestFile1, '.\New folder\');
VFS.NewFile(TestFile, VFSRootDir, 'File 2');
VFS.ExtractFile('.\New folder\New file', 'C:\X.txt'); // you have to specify a full path if you want to get a file from a subdirectory
VFS.ExtractFile('File 2', 'C:\Y.txt');
finally
VFS.Free();
end;
end;


I'm awaiting any feedback from you. :)

[Brainer]

Hi again. :)

This time I'm back with a new release of my Brain Virtual File System! :) You can see a couple of improvements as well as a GUI layout of the file manager application. Here's a screenshot:
http://i43.tinypic.com/vq2qkg.jpg

The full source code and the demo application can be downloaded from here (http://pateman.net76.net/BrainVFS_Alpha2.zip). :)

What's still missing
:arrow: Encryption/compression of files
:arrow: Searching for files and folders
:arrow: Minor improvements like a checksum of every file

To preview a file added into a package, just click it twice.

That's all from me. :) I hope you can test it. Please report any bugs to me!

[noeska]

The screenshot looks nice. I see you implemented a 'real' folder sytem.
I wanted to give it a test, but my 7zip does not seem to recognize the .7z archive. Can you post a .zip file instead? Not everyone is having 7zip installed.

[Brainer]

Yip, I changed the link now. :)

[leniz]

Worked perfectly. Though had to drag my ass to google to find myself FastMM_FullDebugMode.dll. Don't know whether it was my problem or Your application's.

Anyways, good job

Best regards,
Leniz

[leniz]

Ah.. How to delete posts? :oops:

Best regards,
Leniz

[Brainer]

Oops, sorry! I forgot to include that DLL to the ZIP. :oops: I updated the link.

I'm very glad you like it. :) The next release will bring compression and maybe encryption too.

[Brainer]

After a few tests I discovered serious design flaws in my project, so I'm going to start it over. It'll take some time before I release another version for testing as I'm busy with other things so I might not be able to re-design it.

If you're interested in this project, feel free to visit my website. I'll try to frequently post updates and a full documentation of my VFS should be available there for downloads very soon.

Last, but not least, the project is not dead! I'm going to work on it as soon as I find enough spare time to dedicate myself to it.

That's all for now. :)

[Rahakasvi]

Looks freaking sweet..

What license do you have this library? I would be interested to add this to my Dage engine :)

[Brainer]

Nice you're interested in it, but please don't use it just now as it's really bad and needs to be fixed. :) It's released under BSD license.

[Rahakasvi]

Great :D I have no need to use it now, but perhaps later. I hope you can finish this project :)

I see that with this technique you will be generating a lot of fragmentation, especially when using that to game development ;) One wishlist for features would be defragment button, that completely rewrites the archive.

[noeska]

So we have 2 vfs for delphi/fpc now? Nice :D

I am currently waiting for feedback before making changes to it again. One planned feature is closing the archive file after a user is finished reading/writing and opening again when needed. Also my example should be updated to something nice looking as Brainer's. Now some featurs available are not visible.

And yes defragmentation is an issue.

[Brainer]

@noeska: Yip, it seems like it's getting interesting now. :)

@Rahakasvi: The idea of a VFS is to simulate a real hard disk, and AFAIK fragmentation cannot be avoided, so yes, sooner or later it'll be necessary to defragment your files. Defragmentation will be available in the new release of the VFS.

The main design flaw of my VFS was that blocks information is stored in memory, so it's impossible to have huge drives. Now I'm working on a new design and a new file structure to make the VFS more flexible and faster. Also, I'd like to use that VFS for real data backups in the future, so I must make it reliable enough. I'll try to keep you posted - to anyone who's interested. :)

[noeska]

You read the complete archive in memory first? In my first version i also played with the idea of using a real record structure for an block and read that as a whole in memory but decided against it. Now if you need one byte from an file inside an archive only that byte is read not the complete block that the byte is stored in. The same goes for writing.



function TVirtualFileSystem.ReadBlock(ablockid: int64; var buffer; beginpos: integer; endpos: integer): int64;
var
nextblockid: int64;
begin
{$R+}
FDataFile.Position :=(ablockid * FBlockSize); //goto begin of block
FDataFile.ReadBuffer(nextblockid, sizeof(nextblockid));

FDataFile.Position :=FDataFile.Position+beginpos; //advance to position within block
FDataFile.ReadBuffer(buffer, (endpos-beginpos) ); //only read bytes needed

result := nextblockid; //return the next block
end;

[Brainer]

Nope, files aren't loaded into memory, only data about blocks (i.e. block's flag and size). And you can't have a High(Cardinal) of them when you store it in memory. :)