BESEN - A complete ECMAScript 5th edition implementation with a JIT

[BeRo]

I want to post here my newest project named BESEN. BESEN is an acronym for "Bero's EcmaScript Engine", and it is a complete ECMAScript Fifth Edition Implemention in Object Pascal, which is compilable with Delphi >=7 and FreePascal >= 2.5.1 (maybe also 2.4.1).

BESEN is licensed under a RubyLike+AGPLv3 dual-license.

Features:


Complete implementation of the ECMAScript Fifth Edition standard
Own bytecode-based ECMA262-complaint Regular Expression Engine
Incremental praise/exact mark-and-sweep garbage collector
Unicode UTF8/UCS2/UTF16/UCS4/UTF32 support (on ECMAScript level, UCS2/UTF16)
Compatibility modes, for example also a facile JavaScript compatibility mode
Bytecode compiler
Call-Subroutine-Threaded Register-based virtual machine
Context-Threaded 32-bit x86 Just-in-Time Compiler (x64/AMD64 JIT is in the work and a ARMv7 EABI JIT for ARM CPUs with VFPUv1 instruction set is planned)
Constant folding
Dead code elimination
Abstract-Syntax-Tree based optimizations
Type inference (both exact and speculative)
Property Inline Cache
Perfomance optimized hash maps
Self balanced trees (for example to sort on-the-fly linked list items of hash maps for very fast enumeration of array objects)
Easy native ObjectPascal class integration (properties per RTTI and published methods per by-hand-parsing of the native virtual method table)


And here is the link to BESEN: http://besen.sourceforge.net/

Have fun with it ;)

EDIT: Moved to SF.NET

[Stoney]

Wow, I've been looking for something like this to be integrated into my game engine.
At the moment I'm using the Thorium scripting language (http://delphigl.com/forum/viewtopic.php?f=13&t=7570) (german link) for this task.

Just took a quick look at it, but it failed to compile on Mac OS X 10.6.3 (Intel) with FreePascal 2.4.0 (just wanted to test if it would work with 2.4.0, I will use a more recent FPC version tomorrow) with some errors about wrong parameters for FormatDateTime and StrToDateTime.

Well, as I said I will take a more thorough look at BESEN tomorrow.

[jdarling]

Ok, this is truly awesome. Downloading now and will give it a shot in the versions of Delphi and FPC/Lazarus I have installed. Is this a full hand implementation or did you use a compiler builder (lex, coco, etc) for it?

PS: How long till we see the docs :)

- Jeremy

[BeRo]

Yeah, BESEN is a full hand implementation. I hate lexer and parser generators :D Only the unicode lookup table constant arrays are auto-generated.

I've fixed two bugs today (due to a >>= >>>= and <<= code generator bug), so a new version is online now. And i've added some hints to the BESEN webpage due to the code runtime perfomance.

As said, it's is for FPC >=2.5.1 and Delphi >=7, so that FPC 2.4.0 isn't usable for it, because of old Delphi7-incompatible FormatDateTime definitions and so on.

[VilleK]

Impressive! :)

You keep making "impossible" projects with Pascal, BeRo! :)

[phibermon]

Excellent work! I shall be having a play :)

[chronozphere]

This looks quite interesting. I'll give it a shot when i find the time. :)

[arthurprs]

100% impressive! :yes:

[BeRo]

Thanks for all your feedback. :) I've made BESEN even compatible for to FPC 2.4.0 now, and fixed some bugs and moved to Sourceforge.

[JernejL]

Why would anyone write something like this is beyond my logic, but you really did write something very VERY nice here. Despite my hate for javascript from web programming i do every day something like this in a simplier form would be pretty suitable as a game scripting programming language, the only issue i see would be license (GPL/Custom), but i'll try to evalute it and see where i go from there.

Really nice work, i was pleasantly suprised to see something this advanced on this forum.

[Stoney]

After testing BESEN a bit, I'm still impressed with your work. Thanks for adding support for FPC 2.4.0.

It works without any problems on my Windows machine, but unfortunately I'm having some issues getting it to work on a Mac.
First off, you something like this

{$ifdef fpc}
{$ifdef darwin}
{$pic off}
{$endif}
...

since -fPic switch is set by default when compiling on a Mac.

BESEN loads dynamically from libc.so. Well, libc.so doesn't exist on Mac, it's called libc.dylib. So around line 41287 please change it to following:

{$ifdef darwin}
fpmprotect:=dlsym(dlopen('libc.dylib',RTLD_NOW),'m protect');
{$else}
fpmprotect:=dlsym(dlopen('libc.so',RTLD_NOW),'mpro tect');
{$endif}


With those two changes it compile and BESENTest will at least start when being executed, but as soon as I load any script into the sample, I get this error:


Exception object An unhandled exception occurred at $00043114 :
Illegal instruction



I would love to see some more examples especially in regards on how to integrate BESEN into existing applications, like how to access certain aspects of the application from the script and vice versa.

[BeRo]

After testing BESEN a bit, I'm still impressed with your work. Thanks for adding support for FPC 2.4.0.


But support for 2.4.0 is only inoffically :D For example the 64-bit x64/AMD64 codegenerator of FPC 2.4.0 was/is very buggy, so BESEN compiles there only with -O1 or without any code optimizations.



It works without any problems on my Windows machine, but unfortunately I'm having some issues getting it to work on a Mac.
First off, you something like this

{$ifdef fpc}
{$ifdef darwin}
{$pic off}
{$endif}
...

since -fPic switch is set by default when compiling on a Mac.

BESEN loads dynamically from libc.so. Well, libc.so doesn't exist on Mac, it's called libc.dylib. So around line 41287 please change it to following:

{$ifdef darwin}
fpmprotect:=dlsym(dlopen('libc.dylib',RTLD_NOW),'m protect');
{$else}
fpmprotect:=dlsym(dlopen('libc.so',RTLD_NOW),'mpro tect');
{$endif}



Fixed :)


With those two changes it compile and BESENTest will at least start when being executed, but as soon as I load any script into the sample, I get this error:


Exception object An unhandled exception occurred at $00043114 :
Illegal instruction



Try the new BESEN.pas from the Sourceforge SVN with -dDisableJIT to disable the JIT testwise, and say me then, if it had work or not. The JIT stuff works at least under Windows and LInux with FPC 2.5.1 and Delphi 7 + Kylix.



I would love to see some more examples especially in regards on how to integrate BESEN into existing applications, like how to access certain aspects of the application from the script and vice versa.


Yeah, this example stuff comes later. :)

[Stoney]

Unfortunately I didn't work with the DisableJIT switch. I still get the same error :(
Also when I recompile the latest SVN sources I get


BESEN.pas(19,24) Fatal: Internal error 200208151
Fatal: Compilation aborted

Only if I do a clean build (delete all *.o and *.ppu) compilation works without problems.

[BeRo]

Unfortunately I didn't work with the DisableJIT switch. I still get the same error :(
Also when I recompile the latest SVN sources I get


BESEN.pas(19,24) Fatal: Internal error 200208151
Fatal: Compilation aborted

Only if I do a clean build (delete all *.o and *.ppu) compilation works without problems.


Which CPU target? 32-bit x86, 64-bit x64/AMD64, PowerPC, etc.?

Maybe you can try to remove the Set8087CW($133f); stuff from the BESENTest.dpr. Or otherwise try to get the exact code line, where the "Illegal instruction" is triggered with the -g parameter to the FPC compiler to generate debug infos for GDB. and then run "./gdb --args ./BESENTEST.dpr myfile.js", type run, and check the output.

Anyway a new version is on the SVN again, some bugs in the RegEx Stuff fixed. :)

[jdarling]

BESEN.pas(19,24) Fatal: Internal error 200208151
Actually this exception happens with the latest SVN version on all Lazarus/FPC 2.4.1 builds. Not sure exactly what it is, but I know it has to do with Inlining (http://bugs.freepascal.org/view.php?id=15909) typically.

I posted this to the mail group as well, but I finally started playing with BESEN. Great, excellent, amazing work! Except...

I want to import existing objects that do not inherit from TBESENNativeObject and some that do, and make them accessible to (some/all of) the running scripts . I found that you can register the prototype for a TBESENNativeObject via RegisterNativeObject, but can't figure out how to register the actual instance (btw: how can you create a proper instance of a TBESENNativeObject without passing in the BESEN base?) I'm guessing this is done via RegisterNativeFunction for instances that derive from TBESENNativeObject, but can't figure out the exact details.

So, in short, lets say I have a playing field that has "smart objects" placed on it. Each object has a script assigned to it, and there is a global master script that needs access to these objects as well. How do I achieve this?

This brings me to a great example app that could be put together: Tic-Tac-Toe. Two separate AI scripts loaded and one "master" script loaded. Each AI gets ran for 1 turn, then the master gets ran to check game state and report back. The game board is a standard object and each "player" is a TBESENNativeObject.

This would should most scenarios where you need to interface with existing and custom objects.

Thanks, and hopefully someone is still reading this thread,
- Jeremy

[jdarling]

BESEN.pas(19,24) Fatal: Internal error 200208151

PPS: If you want to get rid of the error building with Lazarus on the latest version, change line 160 to:

{.$define caninline}
This removes the inlining (slowing down execution a bit) and lets you do a regular build with no exceptions.

- Jeremy

[BeRo]

BESEN is moved to Google Code http://code.google.com/p/besen/ now. And it's also relicensed as LGPL with static-link-exception now.

[Rodrigo Robles]

Any special reason to move to Google code? Many people are doing it nowadays.

[BeRo]

Any special reason to move to Google code? Many people are doing it nowadays.

The reasons:


The SourceForge servers are often very slow, even with my 64mbit/s down 5mbit/s up internet connection here in Germany. Google's server are simply reaction-faster and better load-balanced.
The whole GoogleCode site (the enduser site backend and the admin web interface) is more cleaner and better usable than SourceForge. And GoogleCode has a nice Wiki integration, what is somewhat better than the stuff at SourceForge, at least for my usage.
SourceForge forces an additional SSH Layer for R/W SVN Connections over Putty/TortoisePLink under Windows as TortoiseSVN user (and without a option to save the SVN user password, argh omg ??? ), if you upgrade to their new platform version, what I've did, but what was a mistake from me at the end. Short: The SVN usage at GoogleCode is better than at SourceForge. And if you're on SF, then don't upgrade to their new platform version, at least, if you're using a SVN repo for your project.
And some other smaller not-so-important reasons.

[Rodrigo Robles]

Looks like Sourceforge will lose their position soon...