Mixing Lisps in Kawa

Per Bothner

What Is Kawa?

• Originally (1996-) an implementation of Scheme.
• Compiles Scheme into Java bytecodes.
• Includes run-time environment.
• Written in Java with some Scheme.
• Convenient integration and scripting for Java.
• Has been extended to support other languages.

Why multiple languages?

• Different people need/prefer different languages.
• Share development - a compiler is a lot of work.
• Use libraries written in one language from another.
• Groups using different languages can collaborate.
• Allows migrating from one language to another.
  Example: Emacs potentially migrating from Emacs Lisp to Scheme.

Kawa Lisp languages

• This talk concentrates on the Lisp family of languages.
• Specifically these languages have at least partial implementations:
  • Scheme: Full-featured, in production use.
  • Common Lisp: A small subset works.
  • Emacs Lisp: Enough implemented to run some existing Emacs code.

Other Kawa languages

• XQuery: A large useful subset of the new XML-processing language from W3C.
• XSLT: A recent primitive but promising development.
• BRL (Beautiful Report Language): A JSP-like template language embedding Scheme in HTML. In production use.
• KRL (Kawa Report Language): An experimental dialect of BRL.
• Nice: A strongly-typed language.
• EcmaScript (JavaScript): Never got very far.

Execution

1. Read/parse source code, yielding internal Expression data structure.
2. Tree-walk the Expression for analysis, optimimization, and selecting representations.
3. If immediate mode and tree is simple: interpret it.
4. Else, code generation: Create one or more Java classes with byte-code instructions.
5. Load-and-execute generated code, or save for future use.

Lisp reading

• For Lisp languages: reading is first pass of parsing.
• Result is S-expression, with line+column-number enhanced conses.
• Reader uses Common Lisp parsing rules, controlled by a readtable.
• Result of reading is converted to Expression, which includes syntax/macro-expansion and lexical name resolution.
• Complications include hygiene (for Scheme), forward references, and modules.

Values and Objects

• Java is a hybrid class-based language.
  • Most values are heap-allocated objects.
  • All classes extend the root Object class.
  • There are also primitive (unboxed) values.
• Implementor must map Scheme/Lisp types to Java classes.
• Sometimes use standard Java classes, sometimes custom ones.
• Different languages may use different classes.

Symbols

• Scheme symbols have no state.
• Lisp symbols have value, function, and plist cells.
• Traditional value cell cannot be thread-specific.
• Therefore Kawa symbols are immutable, without cells.
• An Environment maps (symbol, property) to cell.
  • Property can be arbitary object; address hashed with symbol.
  • Property is null to get value cell.
  • Special FUNCTION property to get function cell.
• Environments can be thread-specific, with inheritance.

Sequences

• Common Lisp (but not Scheme) defines a hierarchy of sequence types.
• Kawa implements a compatible hierarchy.
• Kawa has multi-dimensional arrays.
• Arrays generalize displayed arrays, allowing an arbitrary affine mapping.

The Nil problem

• In Scheme the empty list, the symbol nil, and Boolean false are 3 distinct objects.
• In Common Lisp and ELisp they're a single object.
• This complicates calling between the languages and sharing libraries.
• We must pick one of the Scheme objects, and use it for Common Lisp nil.
• We chose to use the Scheme empty list for nil.

Streams

• Native Unicode support for characters, strings, and streams.
• Programmable readtable-based reader.
• Format, with most CL features.
• Efficient pretty-printer.

Multiple values

• Scheme, Lisp, and XQuery expressions can return multiple values.
• CLisp (unlike Scheme) allows coercing to a single value.
• XQuery expressions evaluate to sequences which are similar to multiple values.
• However, XQuery sequences can be quite long.
• Implict push representation: for each value the producer calls a method in a Consumer object.
• Output streams implement Consumer, so values can be written as generated.
• Also explicit representation using a TreeList class.
• Latter is also used for a compact XML representation.

Functions

• A function is normally compiled to a method in a Java class.
• Optional parameters yield multiple overloaded methods.
• Kawa also supports keyword and rest parameters.
• Call to known function usually compiled to direct method invocation.
• A function value is an instance of a class that extends Procedure.
• Calling an unknown function calls an apply method that invokes function's method.
• Some extra magic to handle tail-calls, and generic functions.
• Support planned for full continuations.

Defining classes

• Kawa provides define-class; similar to defclass.
• Generates a Java interface to implement multiple-inheritance.
• define-simple-class generates a plain (single-inheritance) Java class.
• Can define methods as part of the class.
• There is also support for generic (overloaded) functions, and selecting the closest match.
• Most of CLOS can be implemented efficiently.
• Some MOP features (e.g. change-class) may be impractical.

Modules

• Modules can be compiled separately.
• Normally a source file is compiled to a module class.
• Each top-level definition translates to a Java field.
• Some definitions may be module-private.
• Compiler imports a module using reflection.
• Macros are compiled into field that have type Macro.
• Macro hygiene works across modules.

Types

• Java type inspection and creation facilities are insufficient.
• Need representation for compile-time (static) types.
• XQuery has an specially complex type system.
• Kawa extends Scheme with optional type declarations.

Type declarations

(define (double-abs (x :: <double>))
    :: <double>
  (if (> x 0) x (- x)))

• This declares that x and the returned value are 64-bit unboxed floating-point values.
• Generates optimal bytecode using native arithmetic.
• No heap-allocation or access is done.

Exceptions

• Kawa provides various forms of try-catch using Java exceptions.
• Supports limited (exit-only) continuations.
• Hope to implement full call-with-current-continuation soon.
• Latter based on mechanism used for tail-calls.
• Currently doesn't support CL-style conditions, but no problems expected.

Development: Editing and debugging

• Kawa preserves line+column information internally.
• Standard line numbers are emitted in generated bytecode.
  Gives us file+line numbers in exception traces.
• Dominque Boucher wrote an Eclipse plugin; very nice Scheme editing.
• Usable debugging of Scheme using Eclipse:
  • Stack traces; can set breakpoints etc.
  • Examining Scheme values requires familiarity with Java representation.

Scheme status

• A mature and full-featured Scheme implementation.
• Numerous companies, groups, and individuals use it.
• People like it for features, performance, Java integration.
• Merced Systems depends on on it for their performance management product (and have paid generously for consulting/support).

Emacs Lisp and JEmacs status

• Can compile and execute non-trivial ELisp packages.
• Working Swing-based buffers, windows, mode-line, etc.
• Lots of "cool!" feedback but little support.
• Years of stagnation.
• Recent progress: received port to Eclipse's SWT toolkit.
• Interest in integration with Eclipse framework.

Common Lisp status

• A lot of Common Lisp functionality exists.
• However, only a small amount is accessible using Common Lisp syntax.
• Getting a useful subset of Common Lisp requires writing syntax transformers and wrapper functions.
• This should be a fairly modest project, but time and priority has been missing - so far.

Final words

• Main web site: http://www.gnu.org/software/kawa/
• Greatest Free Java project nobody's heard of.
• Version 1.8 out soon.
• - or use the very latest code from CVS.
• Users and contributors both welcome.