Per's blog

The DomTerm terminal emulator now supports sub-windows (panes) and tabs. These are resizable and draggable. This is implemented using the GoldenLayout JavaScript library.

The screenshot below shows 3 sub-windows. The lower one has two tabs, domterm-2 and domterm-3. The latter has focus (indicated by the magenta border), and is running the mc file manager. The upper left pane (domterm-1) shows how to embed html from the shell. The domterm-4 pane shows using Kawa to create composable picture values which are displayed using embedded SVG.

You can resize panes by dragging the separator between them, and you can re-arrange panes or tabs by dragging a title bar. The screenshot shows as we are dragging the splitter between the two upper panes; the blue rectangles temporarily display the resulting sizes.

domterm-panes-2.png

Compared to GNU screen or tmux, DomTerm supports more flexible layouts and easier manipulation using either mouse or keyboard. However, DomTerm does not yet support sessions that can be detached or accessed by multiple users at once, though I do hope to add it. Until then, I suggest using abduco or dtach to handle session management.

Created 5 Jul 2017 20:46 PDT. Last edited 6 Jul 2017 10:18 PDT. Tags:
Many programming languages have some variant of list comprehension syntax. For example Haskell:
[ 2 * x | x <- [2,3,5,7,11]]
The part after the | is the scan specifier x <- [2,3,5,7,11], where the scan variable x ranges over the elements of the scan sequence [2,3,5,7,11]. The result expression 2 * x is evaluated for each element of the resulting sequence, using the scan variables from the scan specifier(s).

Kawa's new (and experimental) List Comprehension Patterns separate these two parts by making the scan specifier part of the more general pattern-matching feature. The result expression can be widely separated from the scan specifier, as long is it within the lexical scope of any scan variable it uses. For example they can be in separate REPL commands (the #|kawa:N# is the REPL prompt).

First, some simple variable definions, using the Kawa-specific ! (which is like define extended to patterns):

#|kawa:1|# (! A [2 3 5 7 11])
#|kawa:2|# (! [A0 A1 A2 A3 A4] A)
The first line binds the variable A to the result of evaluating the sequence literal [2 3 5 7 11]. The second illustrates matching the pattern [A0 A1 A2 A3 A4] against the value of A, which causes A0...A4 to be matched against corresponding elements of A:
#|kawa:3|# (+ A1 A4)
14
Finally, here is a list comprehension (or scan) pattern:
#|kawa:4|# (! [a ...] A)
This declares a to be a scan variable that ranges over the elements of the scan sequence A. You can only use a scan variable inside a scan context:
#|kawa:5|# a
/dev/stdin:2:1: using scan variable a while not in scan context
A scan context is an expression that is followed by ellipsis. The expression is evaluated once for each value of the corresponding scan variable(s), and the result sequence is spliced into its context. Here the scan context is the expression (* 2 a), which doubles the corresponding element a:
#|kawa:6|# [(* 2 a) ...]
[4 6 10 14 22]

This use of ellipsis is inspired by the syntax-rules macro transformers of Scheme.

A scan variable can be used multiple times in the same scan context (ellipsis expression):

#|kawa:7|# [a ... a ...]
[2 3 5 7 11 2 3 5 7 11]
#|kawa:8|# [(* a a) ...]
[4 9 25 49 121]
Ellipses expressions are useful not just in sequence literals, but in the argument list of a procedure call, where the resulting sequence is spliced into the argument list. This is especially useful for functions that take a variable number of arguments, because that enables a convenient way to do fold/accumulate/reduce operations. For example:
#|kawa:9|# (+ a ...)
28
because 28 is the result of (+ 2 3 5 7 11).

An elegant way to implement dot product:

(define (dot-product [x ...] [y ...])
  (+ (* x y) ...))
When an ellipse expression references two or more distinct scan variables then they are scanned in parallel. That does not (necessarily) imply muliple threads, but that the first element of the scan result is evaluated using the first element of all the scan sequences, the second element of the result uses the second element of all the scan sequences, and so on.

Sub-patterns in ellipsis patterns

The form before ... in a pattern is itself a (sub-)pattern. For example, the sub-pattern may include a type specifier, which applies to each element:

#|kawa:11|# (define (isum [x::integer ...]) (+ x ...))
#|kawa:12|# (isum [4 5 6])
15
#|kawa:12|# (isum [4 5.1 6])
Argument #1 (null) to 'isum' has wrong type
	at gnu.mapping.CallContext.matchError(CallContext.java:189)
	at atInteractiveLevel-6.isum$check(stdin:11)
	...
(The stack trace line number stdin:11 is that of the isum definition.)

You can nest ellipsis patterns, allowing matching against sequences whose elements are sequences.

#|kawa:31|# (define (fun2 [[x ...] ...] [y ...])
#|.....32|#   [[(+ x y) ...] ...])
#|kawa:33|# (fun2 [[1 2 3] [10 11 12]] [100 200])
[[101 102 103] [210 211 212]]

Note that x is double-nested, while y is singly-nested.

The following does not work at time of writing, but will soon:

(! [[x y] ...] sequence-of-pairs)  ;;Not yet working

Filtering the scan sequence

Often you only want to use some of the elements from the scan sequence. For example SQL's select has a where with a boolean expression; only elements for which the expession is true are included. With scan patterns we have the option to filter the values either where the scan variables are introduced (the pattern), or when they are used.

(The examples in this section are not working - and the design is undecided.)

Kawa has a splice operator @expr that takes a sequence and spreads it out to multiple arguuments. (It is similar to JavaScripts's spread syntax ...expr.) It is useful to allow splice operators in scan expressions, and even more useful to allow them in one or more branches of an if. The following expressions selects the odd elements from a, doubling them, but drops even elements. It does the latter by returning a 0-length splice for each even element:

[(if (odd? a) (* 2 a) @[]) ...]

Some reasonable syntatic sugar would be to allow leaving out the else expression of an if, having it default to a zero-length splice:

[(if (odd? a) (* 2 a)) ...]

APL has an expand operator that repeats elements of an array based on another count array. This functionality can be achived using splices in scan expressions. For example, the following repeats each odd element, and skips even elements:

[(if (odd? a) @[a a] @[]) ...]
Equivalently (parsing issues aside):
[@(if (odd? a) [a a] []) ...]

It is also useful to filter at the pattern level. The following hypothetical syntax would create a scan variable a from those elements of the scan sequence A that satisfy the boolean expression (P a).

(! [a ...] A #!if (P a)) ;; hypothetical
Note the sequence of a values used to evaluate (P a) would be different than the values in scan expressions. That might be an issue.

Outer product

Instead of processing multiple sequences in parallel, sometimes you want the multiplicative combination of elements from all the sequences. This is called Cartesian product (or outer product in APL), or an cross join in database query languages.

This could be handled with a (not implemented) outer-product function:

(outer-product [10 20] [1 2 3]) ⇒
  [[10 1] [10 2] [10 3] [20 1] [20 2] [20 3] [30 1] [30 2] [30 3]] 
We could use it like this:
(! [[x y] ...) (outer-product X Y))

The above syntax is a bit awkward. Some syntatic sugar would help. If so, it should include an option for filtering only the desired combinations based on a boolean expression.

Comprehension as looping

Scan expression can form the basis of a looping framework in a programming language. Assume a function ignore that ignores its arguments. Then this is one way to write a for loop, using its scan expression for its side effects:

(let (([i ...] [0 <: 10]))
  (ignore (do-something-with i) ...))
The scan valuable i range of the range expression [0 <: 10]. The scan expression (do-something-with i) is evaluated for each i.

A general looping construct needs a way to exit the loop based on a condition, as in a while loop. Than can be done by having a scan expression return a special #!done value.

This assumes the scan elements are evaluated sequentially, which is a reasonable default. It would be useful to have some way to request that evaluation be done in parallel. This could be implemented by translating the code to use Java 8 streams, though for performance reasons we only want to do that when parallel execution is wanted.

Implementation notes

A scan variable is implemented as either a java.util.List or a (Java) array. Using an array may be more efficient, partly because it easier to avoid boxing; on the other hand it may require more copying.. Scan patterns are expanded by the compiler to a loop which initializes the array. Scan expressions expand to map calls, which are generally inlined to loops.

Numerous optimizations and performance improvements are possible.

Created 14 May 2017 20:22 PDT. Last edited 20 May 2017 13:52 PDT. Tags:

The DomTerm terminal emulator has a number of unique features. In this article we will explore how it enables dynamic re-flow for Lisp-style pretty-printing.

The goal of pretty-printing is to split a text into lines with appropriate indentation, in a way that conforms to the logical structure of the text.

For example if we need to print the following list:

((alpha-1 alpha-2 alpha-3) (beta-1 beta-2 beta-3 beta-4))
in a window 40 characters wide, we want:
((alpha-1 alpha-2 alpha-3)
 (beta-1 beta-2 beta-3 beta-4))
but not:
((alpha-1 alpha-2 alpha-3) (beta-1
 beta-2 beta-3 beta-4))

Pretty-printing is common in Lisp environments to display complicated nested data structures. Traditionally, it is done by the programming-language runtime, based on a given line width. However, the line width of a console can change if the user resizes the window or changes font size. In that case, previously-emitted pretty-printed lines will quickly become ugly: If the line-width decreases, the breaks will be all wrong and the text hard to read; if the line-width increases we may be using more lines than necessary.

Modern terminal emulators do dumb line-breaking: Splitting a long lines into screen lines, but regardless of structure or even word boundaries. Some emulators remember for each line whether an overflow happened, or whether a hard newline was printed. Some terminal emulators (for example Gnome Terminal) will use this to re-do the splitting when a window is re-sized. However, that does not help with pretty-printer output.

Until now. Below is a screenshot from Kawa running in DomTerm at 80 colutions.
pprint-1.png

We reduce the window size to 50 columns. The user input (yellow background) is raw text, so its line is split non-pretty, but the output (white background) gets pretty re-splitting. (Note the window size indicator in the lower-right.)
pprint-2.png

We reduce the window size to 35 columns:
pprint-3.png

It also works with saved pages

DomTerm allows you to save the current session as a static HTML page. If the needed DomTerm CSS and JavaScript files are provided in the hlib directory, then dynamic line-breaking happens even for saved log files. (The lazy way is to create hlib as a symbolic link to the hlib directory of the DomTerm distribution.)

Try it yourself on a saved session. The --debug-print-expr flag causes Kawa to print out each command before it is compiled and evaluated. The result (shown in red because it is sent to the standard error stream) is pretty-printed dynamically.

Structured text

This is how it works.

When an application pretty-prints a structure, it calls special output procedures to mark which parts of the output logically belong together (a logical block), and where line-breaks and indentation may be inserted. In Kawa the default print formatting for lists and vectors automatically calls these procedures when the output is a pretty-printing stream. The pretty-printing library calculates where to put line-breaks and indentation, based on these commands and the specified line length.

However, when the output stream is a DomTerm terminal, Kawa's pretty-printing library does not actually calculate the line-breaks. Instead it encodes the above-mentioned procedure calls as special escape sequences that get sent to DomTerm.

When DomTerm receives these escape sequences, it builds a nested DOM structure that corresponds to the orginal procedure calls. DomTerm calculates how to layout that structure using a variant of the Common Lisp pretty-printing algorithm, inserting soft left-breaks and indentation as needed.

When DomTerm detects that its window has been re-sized or zoomed, it first removes old soft line-breaks and identation. It does re-runs the layout algorithm.

When a page is saved, the nested DOM structure is written out too. If the saved page is loaded in a browser, and the necessary JavaScript libraries are available, then the pretty-printing algorithm is run on the saved page, both on initial load, and whenever the window is re-sized.

Hyphenation and word-breaking

DomTerm supports general soft (optional) line breaks: You can specify separate texts (optionally with styling) for each of the following: The text used when the line is not broken at that point; the text to use before the line-break, when broken (commonly a hyphen); the text to use after the line-break (following indentation). One use for this words that change their spelling when hyphenated, as may happen in German. For example the word backen becomes bak-ken. You can handle this using ck as the non-break text; k- as the pre-break text; and k as the post-break text.
Created 30 Jan 2017 13:23 PST. Last edited 3 Feb 2017 12:13 PST. Tags:
Created 22 Nov 2016 23:07 PST. Last edited 22 Nov 2016 23:07 PST.

I have recently spent a lot of time on DomTerm, which is becoming a fairly decent terminal emulator. Being based on HTML5 technologies lets you do many interesting things, including embed graphics. The new qtdomterm standalone terminal emulator is especially nice; alternatively, you can also use the domterm --browser command to start a process that uses a window/tab in your default desktop browser.

The gnuplot package is a powerful graphing package. It has a command-line that lets you specify complicated functions and plots, and then give a plot command to display the resulting plot. Unfortunately, it is difficult to find a a good front-end that is usable for both command-line interaction and graphics. People usually have to specify output to either a file or an external viewer. Would that one could insert the plots directly in the REPL output stream!

The development version of gnuplot (i.e. 5.1, available from CVS) has native support for DomTerm. It has domterm as a new terminal type, which you can select explicitly (with the command set term domterm), or use by default (since gnuplot checks the DOMTERM environment variable, which is set by DomTerm).

This works by using the pre-existing svg output driver to generate SVG, but surrounding the SVG by escape sequences, and then printing the result to standard output. DomTerm recognizes the escape sequence, extracts the SVG (or more generally clean HTML) and inserts it at the cursor.

You can save the session output to an html file. Here is an example. In qtdomterm you can use the File / Save As menu item; otherwise ctrl-shift-S should work. This is a single html file, with embedded SVG; images are embedded with a data: URL. The file is actually XML (xhtml), to make it easier to process. The saved file does need to css and js (JavaScript) files to be readable, so you need to link or copy the hlib directory in the DomTerm source distribution.

1.png

Created 18 Aug 2016 20:45 PDT. Last edited 21 Sep 2016 23:06 PDT. Tags:
Let's say we want to get the contents of a file as a value, using a simple function, without using a port or looping. Kawa has a function to do that:
(path-data path)
The path can be a Path object, or anything that can be converted to a Path, including a filename string or a URL.

You can also use the following syntactic sugar, which is an example of SRFI-108 named quasi-literals:

&<{pname}
This syntax is meant to suggest the shell input redirection operator <pname. The meaning of &<{pname} is the same as (path-data &{pname}), where &{pname} is a SRFI-109 string quasi-literal. (This is almost the same as (path-data "pname") using a traditional string literal, except for the rules for quoting and escaping.)

What kind of object is returned by &<{pname}? And what is printed when you type that at the REPL? Fundamentally, in modern computers the contents of a file is a sequence of uninterpreted bytes. Most commonly, these bytes represent text in a locale-dependent encoding, but we don't always know this. Sometimes they're images, or videos, or word-processor documents. It's like writing assembly code: you have to know the types of your values. At best we can guess at the type of a file based on its name or extension or looking for magic numbers. So unless we have more information, we'll say that path-data returns a blob, and we'll implementing it using the gnu.lists.Blob type.

$ cat README
Check doc directory.
$ kawa
#|kawa:1|# (define readme &<{README})
#|kawa:2|# readme:class
class gnu.lists.Blob
You can explicitly coerce a Blob to a string or to a bytevector:
#|kawa:3|# (write (->string readme))
"Check doc directory.\n"
#|kawa:4|# (write (->bytevector readme))
#u8(67 104 101 99 107 32 100 111 99 32 100 105 114 101 99 116 111 114 121 46 10)
#|kawa:5|# (->bytevector readme):class
class gnu.lists.U8Vector

The output of a command (which we'll discuss in the next article): is also a blob. For almost all programs, standard output is printable text, because if you try to run a program without re-direction, and it spews out binary data, it may mess up your terminal, which is annoying. Which suggests an answer to what happens when you get a blob result in the REPL: The REPL should try to print out the contents as text, converting the bytes of the blob to a string using a default encoding:

#|kawa:6|# &<{README}
Check doc directory.
It makes sense look at the bytes to see if we can infer an encoding, especially on Windows which doesn't use a default encoding. Currently Kawa checks for a byte-order mark; more sniffing is likely to be added later.

What if the file is not a text file? It might be reasonable to be able to configure a handler for binary files. For example for a .jpg image file, if the the console can display images, it makes sense to display the image inline. It helps if the blob has a known MIME type. (I believe a rich text console should be built using web browser technologies, but that's a different topic.)

Writing to a file

The &<{..} operation can be used with set! to replace the contents of a file:

(set! &<{README} "Check example.com\n")

If you dislike using < as an output operator, you can instead using the &>{..} operation, which evaluates to function whose single argument is the new value:

(&>{README} "Check example.com\n")

You can use &>> to append more data to a file:

(&>>{README} "or check example2.com\n")

The current directory

Functions like path-data or open-input-file or the sugar we seen above all use the current directory as a base directory for relative pathname. You get the current value of the current directory with the expression (current-path). This returns a path object, which prints as the string value of the path.

The initial value of (current-path) is the value of the "user.dir" property, but you change it using a setter:

(set! (current-path) "/opt/myApp/")
A string value is automatically converted to a path, normally a filepath.

The procedure current-path is a parameter, so you can alternatively call it with the new value:

(current-path "/opt/myApp/")
You can also use the parameterize form:
(parameterize ((current-path "/opt/myApp/"))
  (list &<{data1.txt} &<{data2.txt}))
Created 8 Mar 2014 17:01 PST. Last edited 25 Mar 2014 23:43 PDT. Tags:

Oracle's Java group uses the JIRA bug tracker. I've been experimenting with processing and filtering the XML backup file that JIRA can generate. The obvious choice seemed to be XSLT, and it worked pretty well on an evaluation instance of JIRA. Then I got a copy of our 1.5GB production instance. Ouch. XSLT works by building an in-memory DOM of the XML document and then matching templates. There is no chance that is going to work.

Next I tried Joost, an implementation of Streaming Transformations for XML (STX). This has the look of XSLT, but is designed to support one-pass streaming. This worked, but it took about 45 minutes on my laptop, which was discouraging, though better than nothing. (It's also worth noting that STX and Joost don't seem to be actively developed, perhaps because people are focusing on the streaming mode that XSLT 3.0 will have.)

Next I tried the StAX cursor classes that ship with Java 6. StAX is claimed to support fast XML processing. It took about 20 minutes. Better, but hardly fast.

Later (after the Kawa experiments discussed below), I tried an alternative StAX processor: The Aalto XML Processor took just under 1 minute for a full copy. This is pretty good. While not as fast as Kawa, Aalto is constrained by the StAX API, which isn't optimized for performance as well as Kawa's private APIs. This also doesn't seem to be actively maintained.

The Kawa source code includes a smattering of XML tools (including an implementation of XQuery 1.0). I wrote the XML parser with careful attention to performance. For example it avoids copying or needless object allocation. So I was curious how it would handle the 1.5GB JIRA dump. Needless to say - it didn't, at first. But I tracked down and fixed the bugs.

The result was quite gratifying: About 20 seconds for a full copy. I then modified the application to skip Issue, Action, Label, ChangeItem, and CustomFieldValue elements, including their children. That run took about 15 seconds, writing a 296MB output file.

This is the entire program:

import gnu.text.*;
import gnu.xml.*;
import gnu.lists.*;
import java.io.*;

public class KawaFilter extends FilterConsumer {
    int nesting = 0;
    int skipNesting = -1;

    public boolean skipElement(XName name) {
        String local = name.getLocalName();
        if (local=="Issue" || local=="Action" || local=="Label"
            || local=="ChangeItem" || local=="CustomFieldValue")
            return true;
        return false;
    }

    public KawaFilter(Consumer base) {
        super(base);
    }

    @Override
    public void startElement(Object type) {
        nesting++;
        if (! skipping && type instanceof XName
                && skipElement((XName) type)) {
            skipNesting = nesting;
            skipping = true;
        }
        super.startElement(type);
    }

    @Override
    public void endElement() {
        super.endElement();
        if (skipNesting == nesting) {
            skipping = false;
            skipNesting = -1;
        }
        nesting--;
    }

    public static void main(String[] args) throws Throwable {
        String inname = args[0];
        OutputStream outs = new FileOutputStream(args[1]);
        XMLPrinter xout = new XMLPrinter(outs);
        xout.setPrintXMLdecl(true);
        KawaFilter filter = new KawaFilter(xout);
        SourceMessages messages = new SourceMessages();
        xout.beginEntity(inname);
        XMLParser.parse(inname, messages, filter);
        xout.endEntity();
        messages.printAll(System.err, 20);
        xout.close();
    }
}

Most of this is boiler-plate (and should be abstracted out). The only context-specific code is the skipElement method, which is called when the start of an element is seen. The method takes an element tag (QName), and returns true if the element and it's children should be skipped. The startElement and endElement are called by the parser. They work by setting the protected skipping field in the FilterConsumer class.

One could easily modify KawaFilter to rename elements or attributes, or to remove attributes. Changing skipElement so it could base its decision on the names of ancestor elements would also be easy. More challenging would be to allow skipElement to base its decision on the existance or values of attributes. The reason is that startElement is called before the attributes is called: This is for performance, so we don't have to build a data structure for the attributes. However, it would not be difficult to add an option to remember the attributes, and defer the skipElement call to the end of the element start-tag, rather than the beginning.

You run KawaFilter thus:

$ java java -cp .:kawa.jar KawaFilter input.xml output.xml

For kawa.jar you need to build Kawa from the Subversion repository (until I make a binary release containing the fixed XML classes).

Created 4 Jan 2014 21:31 PST. Last edited 12 Jan 2014 17:16 PST. Tags:
A number of programming languages have facilities to allow access to system processes (commands). For example Java has java.lang.Process and java.lang.ProcessBuilder. However, they're not as convenient as as the old Bourne shell, or as elegant in composing commands.

If we ignore syntax, the shell's basic model is that a command is a function that takes an input string (standard input) along with some string-valued command-line arguments, and whose primary result is a string (standard output). The command also has some secondary outputs (including standard error, and the exit code). However, the elegance comes from treating standard output as a string that can be passed to another command, or used in other contexts that require a string (e.g. command substitution). This article presents how Kawa solves this problem.

Process is auto-convertable to string

To run a command like date, you can use the run-process procedure:

#|kawa:2|# (define p1 (run-process "date --utc"))
Equivalently you can use the &`{command} syntactic sugar:
#|kawa:2|# (define p1 &`{date --utc})

But what kind of value is p1? It's an instance of gnu.kawa.functions.LProcess, which is a class that extends java.lang.Process. You can see this if you invoke toString or call the write procedure:

#|kawa:2|# (p1:toString)
gnu.kawa.functions.LProcess@377dca04
#|kawa:3|# (write p1)
gnu.kawa.functions.LProcess@377dca04

An LProcess is automatically converted to a string or a bytevector in a context that requires it. More precisely, an LProcess can be converted to a blob because it implements Lazy<Blob>. This means you can convert to a string (or bytevector):

#|kawa:9|# (define s1 ::string p1)
#|kawa:10|# (write s1)
"Wed Jan  1 01:18:21 UTC 2014\n"
#|kawa:11|# (define b1 ::bytevector p1)
(write b1)
#u8(87 101 100 32 74 97 110 ... 52 10)

However the display procedure prints it in "human" form, as a string:

#|kawa:4|# (display p1)
Wed Jan  1 01:18:21 UTC 2014

This is also the default REPL formatting:

#|kawa:5|# &`{date --utc}
Wed Jan  1 01:18:22 UTC 2014

Command arguments

The general form for run-process is:

(run-process keyword-argument... command)

The command is the process command-line. It can be an array of strings, in which case those are used as the command arguments directly:

(run-process ["ls" "-l"])
The command can also be a single string, which is split (tokenized) into command arguments separated by whitespace. Quotation groups words together just like traditional shells:
(run-process "cmd a\"b 'c\"d k'l m\"n'o")
   ⇒ (run-process ["cmd"   "ab 'cd"   "k'l m\"no"])

Using string templates is more readable as it avoids having to quote quotation marks:

(run-process &{cmd a"b 'c"d k'l m"n'o})
You can also use the abbreviated form:
&`{cmd a"b 'c"d k'l m"n'o}
This syntax is the same as of SRFI-108 named quasi-literals. In general, the following are roughly equivalent (the difference is that the former does smart quoting of embedded expressions, as discussed later):
&`{command}
(run-command &{command})

Similarly, the following are also roughly equivalent:

&`[keyword-argument...]{command}
(run-command keyword-argument... &{command})

A keyword-argument can specify various properties of the process. For example you can specify the working directory of the process:

(run-process directory: "/tmp" "ls -l")
You can use the shell keyword to specify that we want to use the shell to split the string into arguments. For example:
(run-process shell: #t "command line")
is equivalent to:
(run-process ["/bin/sh" "-c" "command line"])
You can can also use the abbreviation &sh:
&sh{rm *.class}
which is equivalent to:
&`{/bin/sh -c "rm *.class"}

In general, the abbreviated syntax:

&sh[args...]{command}
is equivalent to:
&`[shell: #t args...]{command}

Command and variable substitution

Traditional shells allow you to insert the output from a command into the command arguments of another command. For example:
echo The directory is: `pwd`
The equivalent Kawa syntax is:
&`{echo The directory is: &`{pwd}}

This is just a special case of substituting the result from evaluating an expression. The above is a short-hand for:

&`{echo The directory is: &[&`{pwd}]}

In general, the syntax:

...&[expression]...
evaluates the expression, converts the result to a string, and combines it with the literal string. (We'll see the details in the next section.) This general form subsumes command substitution, variable substitution, and arithmetic expansion.

Tokenization of substitution result

Things gets more interesting when considering the interaction between substitution and tokenization. This is not simple string interpolation. For example, if an interpolated value contains a quote character, we want to treat it as a literal quote, rather than a token delimiter. This matches the behavior of traditional shells. There are multiple cases, depending on whether the interpolation result is a string or a vector/list, and depending on whether the interpolation is inside a quotes.

  • If the value is a string, and we're not inside quotes, then all non-whitespace characters (including quotes) are literal, but whitespace still separates tokens:
    (define v1 "a b'c ")
    &`{cmd x y&[v1]z}  ⟾  (run-process ["cmd" "x" "ya" "b'c" "z"])
    
  • If the value is a string, and we are inside single quotes, all characters (including whitespace) are literal.
    &`{cmd 'x y&[v1]z'}  ⟾  (run-process ["cmd" "x ya b'c z"])
    

    Double quotes work the same except that newline is an argument separator. This is useful when you have one filename per line, and the filenames may contain spaces, as in the output from find:

    &`{ls -l "&`{find . -name '*.pdf'}"}
    

    If the string ends with one or more newlines, those are ignored. This rule (which also applies in the previous not-inside-quotes case) matches traditional shell behavior.

  • If the value is a vector or list (of strings), and we're not inside quotes, then each element of the array becomes its own argument, as-is:
    (define v2 ["a b" "c\"d"])
    &`{cmd &[v2]}  ⟾  (run-process ["cmd" "a b" "c\"d"])
    
    However, if the enclosed expression is adjacent to non-space non-quote characters, those are prepended to the first element, or appended to the last element, respectively.
    &`{cmd x&[v2]y}  ⟾  (run-process ["cmd" "xa b" "c\"dy"])
    &`{cmd x&[[]]y}  ⟾  (run-process ["cmd" "xy"])
    
    This behavior is similar to how shells handle "$@" (or "${name[@]}" for general arrays), though in Kawa you would leave off the quotes.

    Note the equivalence:

    &`{&[array]}  ⟾  (run-process array)
    
  • If the value is a vector or list (of strings), and we are inside quotes, it is equivalent to interpolating a single string resulting from concatenating the elements separated by a space:
    &`{cmd "&[v2]"}  ⟾  (run-process ["cmd" "a b c\"d"])
    

    This behavior is similar to how shells handle "$*" (or "${name[*]}" for general arrays).

  • If the value is the result of a call to unescaped-data then it is parsed as if it were literal. For example a quote in the unescaped data may match a quote in the literal:
    (define vu (unescaped-data "b ' c d '"))
    &`{cmd 'a &[vu]z'}  ⟾  (run-process ["cmd" "a b " "c" "d" "z"])
    
  • If we're using a shell to tokenize the command, then we add quotes or backslashes as needed so that the shell will tokenize as described above:
    &sh{cmd x y&[v1]z}  ⟾  (run-process ["/bin/sh" "-c" "cmd x y'a' 'b'\\'''c' z'"])
    
Smart tokenization only happens when using the quasi-literal forms such as &`{command}. You can of course use string templates with run-process:
(run-process &{echo The directory is: &`{pwd}})
However, in that case there is no smart tokenization: The template is evaluated to a string, and then the resulting string is tokenized, with no knowledge of where expressions were substituted.

Input/output redirection

You can use various keyword arguments to specify standard input, output, and error streams. For example to lower-case the text in in.txt, writing the result to out.txt, you can do:

&`[in-from: "in.txt" out-to: "out.txt"]{tr A-Z a-z}
or:
(run-process in-from: "in.txt" out-to: "out.txt" "tr A-Z a-z")

These options are supported:

in: value
The value is evaluated, converted to a string (as if using display), and copied to the input file of the process. The following are equivalent:
&`[in: "text\n"]{command}
&`[in: &`{echo "text"}]{command}

You can pipe the output from command1 to the input of command2 as follows:

&`[in: &`{command1}]{command2}
in-from: path
The process reads its input from the specified path, which can be any value coercible to a filepath.
out-to: path
The process writes its output to the specified path.
err-to: path
Similarly for the error stream.
out-append-to: path
err-append-to: path
Similar to out-to and err-to, but append to the file specified by path, instead of replacing it.
in-from: 'pipe
out-to: 'pipe
err-to: 'pipe
Does not set up redirection. Instead, the specified stream is available using the methods getOutputStream, getInputStream, or getErrorStream, respectively, on the resulting Process object, just like Java's ProcessBuilder.Redirect.PIPE.
in-from: 'inherit
out-to: 'inherit
err-to: 'inherit
Inherits the standard input, output, or error stream from the current JVM process.
out-to: port
err-to: port
Redirects the standard output or error of the process to the specified port.
out-to: 'current
err-to: 'current
Same as out-to: (current-output-port), or err-to: (current-error-port), respectively.
in-from: port
in-from: 'current
Re-directs standard input to read from the port (or (current-input-port)). It is unspecified how much is read from the port. (The implementation is to use a thread that reads from the port, and sends it to the process, so it might read to the end of the port, even if the process doesn't read it all.)
err-to: 'out
Redirect the standard error of the process to be merged with the standard output.

The default for the error stream (if neither err-to or err-append-to is specifier) is equivalent to err-to: 'current.

Note: Writing to a port is implemented by copying the output or error stream of the process. This is done in a thread, which means we don't have any guarantees when the copying is finished. A possible approach is to have to process-exit-wait (discussed later) wait for not only the process to finish, but also for these helper threads to finish.

Here documents

A here document is a form a literal string, typically multi-line, and commonly used in shells for the standard input of a process. Kawa's string literals or string quasi-literals can be used for this. For example, this passes the string "line1\nline2\nline3\n" to the standard input of command:
(run-process [in: &{
    &|line1
    &|line2
    &|line3
    }] "command")

The &{...} delimits a string; the &| indicates the preceding indentation is ignored.

Pipe-lines

Writing a multi-stage pipe-line quickly gets ugly:

&`[in: &`[in: "My text\n"]{tr a-z A-Z}]{wc}
Aside: This would be nicer in a language with in-fix operators, assuming &` is treated as a left-associative infix operator, with the input as the operational left operand:
"My text\n" &`{tr a-z A-Z} &`{wc}

The convenience macro pipe-process makes this much nicer:

(pipe-process
  "My text\n"
  &`{tr a-z A-Z}
  &`{wc})

All but the first sub-expressions must be (optionally-sugared) run-process forms. The first sub-expression is an arbitrary expression which becomes the input to the second process expression; which becomes the input to the third process expression; and so on. The result of the pipe-process call is the result of the last sub-expression.

Copying the output of one process to the input of the next is optimized: it uses a copying loop in a separate thread. Thus you can safely pipe long-running processes that produce huge output. This isn't quite as efficient as using an operating system pipe, but is portable and works pretty well.

Setting the process environment

By default the new process inherits the system environment of the current (JVM) process as returned by System.getenv(), but you can override it.
env-name: value
In the process environment, set the "name" to the specified value. For example:
&`[env-CLASSPATH: ".:classes"]{java MyClass}
NAME: value
Same as using the env-name option, but only if the NAME is uppercase (i.e. if uppercasing NAME yields the same string). For example the previous example could be written:
(run-process CLASSPATH: ".:classes" "java MyClass")
environment: env
The env is evaluated and must yield a HashMap. This map is used as the system environment of the process.

Process-based control flow

Traditional shell provides logic control flow operations based on the exit code of a process: 0 is success (true), while non-zero is failure (false). Thus you might see:

if grep Version Makefile >/dev/null
then echo found Version
else echo no Version
fi

One idea to have a process be auto-convertible to a boolean, in addition to be auto-convertible to strings or bytevectors: In a boolean context, we'd wait for the process to finish, and return #t if the exit code is 0, and #f otherwise. This idea may be worth exploring later.

Currently Kawa provides process-exit-wait which waits for a process to exit, and then returns the exit code as an int. The convenience function process-exit-ok? returns true iff process-exit-wait returns 0.

(process-exit-wait (run-process "echo foo")) ⟾ 0
The previous sh example could be written:
(if (process-exit-ok? &`{grep Version Makefile})
  &`{echo found}
  &`{echo not found})
Note that unlike the sh, this ignores the output from the grep (because no-one has asked for it). To match the output from the shell, you can use out-to: 'inherit:
(if (process-exit-ok? &`[out-to: 'inherit]{grep Version Makefile})
  &`{echo found}
  &`{echo not found})
Created 30 Dec 2013 01:34 PST. Last edited 1 Nov 2014 14:50 PDT. Tags:
Kawa 1.14 is now evailable from the usual places:
ftp://ftp.gnu.org/pub/gnu/kawa/kawa-1.14.tar.gz (source code)
ftp://ftp.gnu.org/pub/gnu/kawa/kawa-1.14.jar (runnable jar)

For a extended list of changes, see the news page. Below are some highlights.

More R7RS functions

The proposed new standard for Scheme R7RS has a number of new functions, and also extensions to existing functions. Kawa 1.14 implements (and documents) most of the new functions and extensions, and also implements a number of the syntactic changes:

  • After adding list procedures make-list, list-copy, list-set! all the R7RS list procedures are implemented.
  • Other added procedures: square, boolean=?, string-copy!, digit-value, get-environment-variable, get-environment-variables, current-second, current-jiffy, jiffies-per-second, and features.
  • The predicates finite?, infinite?, and nan? are generalized to complex numbers.
  • The procedures write, write-simple, and write-shared are now consistent with R7RS.
  • String and character comparison functions are generalized to more than two arguments (but restricted to strings or characters, respectively).
  • The procedures string-copy, string->list, and string-fill! now take optional (start,end)-bounds. All of the R7RS string functions are now implemented.
  • Support => syntax in case form.
  • Support backslash-escaped special characters in symbols when inside vertical bars, such as '|Hello\nworld|.
  • The new functions and syntax are documented in the Kawa manual; look for the functions in the index.
  • The most important functionality still missing is support for the library (module) system, and specifically the define-library form. Of course Kawa has its own powerful module system which is in spirit quite compatible, but we still need to decide on the pragmatic issues, including how modules are resolved to files (probably using a module search path), and translation to class names.

    Another feature to ponder is how to handle the R7RS exception mechanism, and how it maps to JVM exceptions. We probably won't implement all of the R7RS semantics, but it would be desirable if (for example) some simple uses of guard would work on Kawa.

    Better support for multi-line strings and templates

    SRFI-109 specifies a syntax for quasi-literal strings. These have a numbers of advantages over traditional double-quote literals. The are quasi-literals in the sense that they can contain embedded expressions. For example, if name has the value "John", then:
    &{Hello &[name]!}
    
    evaluates to "Hello John!". Other benefits include control over indentation (layout), built-in support for format specifiers, and a huge number of named special characters. See the documentation and the SRFI-109 specification for details.

    Named quasi-literals (SRFI-108)

    SRFI-108 can be viewed as a generalization of SRFI-109, in that the constructors have names, and you can define new constructors. This could be used for writing documentation (similar to the Scribble system), or constructing complex nested data structures (a generalization of XML).

    Better support for command scripts

    A number of little improvements have been made to make it easier to write Kawa command scripts (simple one-file applications). Most importantly the documentation has been re-written. The value returned by (car (command-line)) (i.e. the "name" of the script/command invoked) is now accurate and useful in many more cases; you also have the option of setting explicitly it with the -Dkawa.command.name option.

    Experimental Java 8 support

    There is some very preliminary support for next year's Java 8. These are enabled if you run configure with the --with-java-source=8 (or if ant auto-detects java.util.stream.Stream). This flag doesn't yet do very much:

    • It sets the classfile major version number.
    • It adds an optimized version of the Stream#forEach method for gnu.lists.SimpleVector and its subclasses, which includes the FVector used to implement Scheme vectors.
    • Enhance gnu.lists.Consumer to implement java.util.function.Consumer and the specialized classes java.util.function.{Int,Long,Double}Consumer.

      Because output port implements gnu.lists.Consumer, this means you can pass a port to the forEach method:

      #|kawa:11|# ((['a 'b 'c 'd]:stream):forEach (current-output-port))
      a b c d
      

    Of course a large extent streams just Work Out of the Box with Kawa data types, because they implement the apropriate Java interfaces. For examples lists and vectors both implement java.util.List, and thus they automatically have the stream method.

    Created 4 Oct 2013 21:44 PDT. Last edited 5 Oct 2013 14:30 PDT. Tags:

    JavaFX 1.0 was a next-generation GUI/client platform. It had a new Node-based GUI API and used a new language, JavaFX Script, whose goal was to make it easier to program Rich Client applications. (Yours truly was hired by Sun to work on the JavaFX Script compiler.) In 2010 the JavaFX Script language was cancelled: JavaFX would still refer to a new GUI based API based on many of the same concepts, but the primary programming language would be Java, rather than JavaFX Script.

    Java is a relatively low-level and clumsy language for writing Rich Client appliations, though it's not too painful. Still, there was a reason we worked on JavaFX Script: It had a number of features to make such programs more convenient. Luckily, other JVM languages - and specifically Kawa-Scheme - can take up the slack. Below I'll show you a simple Hello-World-type example, and then explain how you can try it yourself. In later acticles I'll show different examples.

    Simple buttons and events

    Our first example is just 3 buttons and 2 trivial event handlers. It is translated from HelloButton.java written in Java by Richard Bair.

    (require 'javafx-defs)
    (javafx-application)
    
    (javafx-scene
     title: "Hello Button"
     width: 600 height: 450
     (Button
      text: "Click Me"
      layout-x: 25
      layout-y: 40
      on-action: (lambda (e) (format #t "Event: ~s~%" e))
      on-key-released: (lambda (e) (format #t "Event: ~s~%" e)))
     (Button
      text: "Click Me Too"
      layout-x: 25
      layout-y: 70)
     (Button
      text: "Click Me Three"
      layout-x: 25
      layout-y: 100))
    

    For those new to Scheme, the basic syntactic building block has the form:

    (operator argument1 ... argumentN)
    
    The operator can be a function (like format), an arithmetic operator in prefix form (like (+ 3 4)), a command, a keyword (like lambda), or a user-defined macro. This general format makes for a lot of flexibility.

    The first two lines in HelloButton.scm are boiler-plate: The require imports various definitions and aliases, while the (javafx-application) syntax declares this module is a JavaFX Application.

    The javafx-scene form (a macro) creates a scene, which is a collection of graphical objects. The Scene has certain named properties (title, width, and height), specified using keyword arguments. The Scene also has 3 Button children. Finally, the make-scene command puts the scene on the stage (the window) and makes it visible.

    Each Button form is an object constructor. For example:

    (Button
     text: "Click Me Three"
     layout-x: 25
     layout-y: 100)
    
    is equivalent to the Java code:
    javafx.scene.control.Button tmp = new javafx.scene.control.Button();
    tmp.setText("Click Me Three");
    tmp.setLayoutX(25);
    tmp.setLayoutY(100);
    return tmp;
    

    The on-action and on-key-released properties on the first Button bind event handlers. Each handler is a lambda expression or anonymous function that takes an event e as a parameter. The Kawa compiler converts the handler to a suitable event handler object using SAM-conversion features. (This conversion depends on the context, so if you don't have a literal lambda expression you have to do the conversion by hand using an object operator.)

    Getting it to run

    Downloading JavaFX 2.x beta

    For now JavaFX is only available for Windows, but Mac and GNU/Linux ports are being worked on and mostly work. (I primarily use Fedora Linux.) The primary JavaFX site has lots of information, including a link to the download site. You will need to register, as long the software is beta. Download the zipfile and extract it to some suitable location.

    In the following, we assume the variable JAVAFX_HOME is set to the build you've installed. For example (if using plain Windows):

    set JAVAFX_HOME=c:\javafx-sdk2.0-beta
    
    The file %JAVAFX_HOME%\rt\lib\jfxrt.jar should exist.

    Downloading and building Kawa

    The JavaFX support in Kawa is new and experimental (and unstable), so for now you will have to get the Kawa source code from SVN.

    There are two ways to build Kawa. The easiest is to use Ant - on plain Windows do:

    ant -Djavafx.home=%JAVAFX_HOME%
    
    or on other platforms (including Cygwin):
    ant -Djavafx.home=$JAVAFX_HOME
    

    Alternatively, you can use configure and make (but note that on Windows you will need to have Cygwin installed to use this approach):

    $ KAWA_DIR=path_to_Kawa_sources
    $ cd $KAWA_DIR
    $ ./configure --with-javafx=$JAVAFX_HOME
    $ make
    

    Running the example

    On Windows, the easiest way to run the example is to use the kawa.bat created when building Kawa. It sets up the necessary paths for you.

    %KAWA_HOME%\bin\kawa.bat HelloButton.scm
    

    On Cygwin (or Unix/Linux) you can use the similar kawa.sh. I suggest setting your PATH to find kawa.bat or kawa.sh, so you can just do:

    kawa HelloButton.scm
    

    Using the kawa command is equivalent to

    java -classpath classpath kawa.repl HelloButton.scm
    
    but it sets the classpath automatically. If you do it by hand you need to include %JAVAFX_HOME%\rt\lib\jfxrt.jar and %KAWA_DIR%\kawa-version.jar.

    This is what pops up:

    HelloButton.png

    If you click the first button the action event fires, and you should see something like:

    Event: javafx.event.ActionEvent[source=Button@3a5794[styleClass=button]]
    

    If you type a key (say n) while that button has focus (e.g. after clicking it), then when you release the key a key-released event fires:

    Event: KeyEvent [source = Button@3a5794[styleClass=button],
    target = Button@3a5794[styleClass=button], eventType = KEY_RELEASED,
    consumed = false, character = , text = n, code = N]
    

    Note: Running a JavaFX application from the Kawa read-eval-print-loop (REPL) doesn't work very well at this point, but I'm exploring ideas to make it useful.

    Compiling the example

    You can compile HelloButton.scm to class files:

    kawa --main -C HelloButton.scm
    

    You can execute the resulting application in the usual way:

    java -classpath classpath HelloButton
    
    or use the kawa command:
    kawa HelloButton
    

    Next

    JavaFX-using-Kawa-animation
    Created 28 Jul 2011 11:06 PDT. Last edited 16 Aug 2011 15:36 PDT. Tags:
    Tags: