Per Bothner's blog

Scripting API for compiled JavaFX

The new read-eval-print-loop (REPL) for JavaFX required some major changes in the scripting API for JavaFX. JavaFX had for a while supported the javax.script API specified by JSR-223. However, that JavaFX implementation didn't allow you to declare a variable in one "eval" and make it visible in future commands, which made it rather useless for a REPL. This turns out to be somewhat tricky for a compiled language with static name binding and typing. Unfortunately, JSR-223 has some fundamental design limitations in that respect. (Embarrassingly, I was a not-very-active member of the JSR-223 expert group. It would have been better if we could have thought a little more about scripting staticly-typed languages, but we didn't have time.)

Design limitations in javax.script

Top-level name-to-value bindings in JSR-223 are stored in a javax.script.Bindings instance, which is basically a Map. Unfortunately, Bindings is a concrete class that the programmer can directly instantiate and add bindings to, and there is no way to synchronize a Bindings instance with the internal evaluator state. (Note that top-level bindings have to be maintained in a language-dependent data structure if the language supports static typing, since typing is language-dependent.) This means the script engine has to explicitly copy every binding from the Bindings object to the language evaluator - or the script evaluator has to be modified to search the Bindings. Worse, if a script modifies or declares a binding, that has to be copied back to the Bindings, which is an even bigger pain.

This problem could have been avoided if Bindings were an abstract class to be implemented by the language script engine. An alternative solution could allow the evaluator to register itself as as listener on the Bindings object.

JSR-223 does support compilation being separate from evaluation. Unfortunately, it assumes that name-to-value bindings are needed during evaluation but not during compilation. This is problematic in a language with lexical binding, and fails utterly in a language with static typing: The types of bindings have to be available at compile-time, not just run-time.

The Invocable interface is also problematical. It invokes a function or method in a ScriptEngine, without passing in a ScriptContext, even though invoking a function is basically evaluation, which should require a ScriptContext.

The ScriptContext interface has an extra complication in that it consists of two Bindings objects, of which the global one seems to have limited utility. Making it worse is that some methods use the default ScriptContext of a ScriptEngine and some take an explicit ScriptContext parameter.

The new JavaFX scripting API

Because of these problems with javax.script, I implemented a new JavaFX-specific API. These classes are currently in the com.sun.tools.javafx.script package; after some shaking down they might be moved to some other package, perhaps javafx.script.

The class JavaFXScriptCompiler manages the compilation context, including the typed bindings preserved between compilations, and a MemoryFileManager pseudo-filesystem that saves the bytecode of previously compiled commands.

The main method of JavaFXScriptCompiler is compile. This takes a script, compiles it, and (assuming no errors) returns a JavaFXCompiledScript.

The main method of JavaFXCompiledScript is eval, which takes a JavaFXScriptContext, evaluates the script, and returns the result value.

The JavaFXScriptContext contains the evaluation state, which is primarily a ClassLoader. Creating a JavaFXScriptContext automatically creates a JavaFXScriptCompiler, so in the current implementation there is a one-to-one relationship between JavaFXScriptContext and JavaFXScriptCompiler. (In the future we might allow multiple JavaFXScriptContext instances to share a single JavaFXScriptCompiler.)

This API is very much subject to change.

The REPL API

The actual REPL is provided by the ScriptShell class, which is implemented using the above API. It first reads in commands from the command line (the -e option), or a named file (the -f option). It then enters the classic REPL loop: Print a prompt, read a line, evaluate it, and print the result.

Most of these behaviors can be customized by overriding a method. For example when the compiler finds an error it creates a Diagnostic object, which is passed to a report method. The default handling is to print the Diagnostic, but a sub-class of ScriptShell could override report to do something else with the Diagnostic.

The ScriptShell API is also very much subject to change as we get more experience with it.

The new javax.script wrapper

The class JavaFXScriptEngineImpl is the JavaFX implementation of the standard javax.script.AbstractScriptEngine class. It was re-written to be a wrapper on top of the API discussed earlier. Because of the previously-mentioned limitations of javax.script it is not completely faithful implementation of JSR-223. It uses a WeakHashMap to translate from the Bindings used by the javax.script API into the JavaFXScriptContext objects used by the internal API. Compilation and evaluation of a script need to use JavaFXScriptContext consistently. This means there are basically two usage modes that should work:

• It is best to always use the default ScriptContext as returned by the ScriptEngine's getContext method. Don't call createBindings, getBindings, setBindings, or setContext.
• If you really need to use any of the above four methods, don't try to use either of the compile methods. The reason for this is that compilation does need to use the script context, and it should be the same as used for evaluation.