tag SchemePer Bothner's bloghttp://per.bothner.com/blog/tag/Scheme/Per Bothner's blogikiwiki2014-11-01T21:50:49ZIs it a binary file? Is it a text file? It's a Blobhttp://per.bothner.com/blog/2014/text-and-binary-data/2014-03-26T06:43:25Z2014-03-09T01:01:14Z
Let's say we want to <q>get the contents of a file</q> as a value,
using a simple function, without using a port or looping.
Kawa has a function to do that:
<pre>
(path-data <var>path</var>)
</pre>
The <var>path</var> can be a
<a href="http://www.gnu.org/software/kawa/Paths.html"><code> Path</code></a> object,
or anything that can be converted to a <code>Path</code>, including
a filename string or a URL.
<p>
You can also use the following syntactic sugar, which is an example
of <a href="http://srfi.schemers.org/srfi-108/srfi-108.html">SRFI-108 named quasi-literals</a>:
<pre>
&<{<var>pname</var>}
</pre>
This syntax is meant to suggest the shell input redirection operator
<code><<var>pname</var></code>.
The meaning of <code>&<{<var>pname</var>}</code> is the same as
<code>(path-data &{<var>pname</var>})</code>,
where <code>&{<var>pname</var>}</code> is a
<a href="http://www.gnu.org/software/kawa/Strings.html#string-quasi-literals">SRFI-109 string quasi-literal</a>. (This is almost the same as <code>(path-data "<var>pname</var>")</code>
using a traditional string literal, except for the rules for quoting and escaping.)
<p>
What kind of object is returned by <code>&<{<var>pname</var>}</code>?
And what is printed when you type that at the REPL?
Fundamentally, in modern computers the contents of a file is
a sequence of uninterpreted bytes.
Most commonly, these bytes represent text in a locale-dependent encoding,
but we don't always know this. Sometimes they're images, or videos,
or word-processor documents. It's like writing assembly code:
you have to know the types of your values.
At best we can guess at the type of a file
based on its name or extension or looking for magic numbers.
So unless we have more information, we'll say that
<code>path-data</code> returns a
<a href="http://en.wikipedia.org/wiki/Binary_large_object"><dfn>blob</dfn></a>,
and we'll implementing it using the <code>gnu.lists.Blob</code> type.
<pre>
$ cat README
Check doc directory.
$ kawa
#|kawa:1|# (define readme &<{README})
#|kawa:2|# readme:class
class gnu.lists.Blob
</pre>
You can explicitly coerce a Blob to a string or to a bytevector:
<pre>
#|kawa:3|# (write (->string readme))
"Check doc directory.\n"
#|kawa:4|# (write (->bytevector readme))
#u8(67 104 101 99 107 32 100 111 99 32 100 105 114 101 99 116 111 114 121 46 10)
#|kawa:5|# (->bytevector readme):class
class gnu.lists.U8Vector
</pre>
<p>
The output of a command (which we'll
<a href="http://per.bothner.com/blog/2014/Kawa-shell-programming/">discuss in the next article</a>):
is also a blob. For almost all programs,
standard output is printable text, because if you try to run a
program without re-direction, and it spews out binary data,
it may mess up your terminal, which is annoying.
Which suggests an answer to what happens when you get a
blob result in the REPL: The REPL should try to print out
the contents as text, converting the bytes of the blob
to a string using a default encoding:
<pre>
#|kawa:6|# &<{README}
Check doc directory.
</pre>
It makes sense look at the bytes to see if we can infer an encoding,
especially on Windows which doesn't use a default encoding.
Currently Kawa checks for a
<a href="http://en.wikipedia.org/wiki/Byte_order_mark">byte-order mark</a>;
more <a><q>sniffing</q></a> is likely to be added later.
<p>
What if the file is not a text file?
It might be reasonable to be able to configure
a handler for binary files.
For example for a <code>.jpg</code> image file,
if the the console can display images, it makes sense to display
the image inline. It helps if the blob has a known MIME type.
(I believe a rich text console should be built using web browser technologies,
but that's a different topic.)
<h3>Writing to a file</h3>
<p>The <code>&<{..}</code> operation can be used with <code>set!</code>
to replace the contents of a file:
<pre>
(set! &<{README} "Check example.com\n")
</pre>
<p>
If you dislike using <q><code><</code></q> as an output operator,
you can instead using the <code>&>{..}</code> operation,
which evaluates to function whose single argument is the new value:
<pre>
(&>{README} "Check example.com\n")
</pre>
<p>
You can use <code>&>></code> to append more data to a file:
<pre>
(&>>{README} "or check example2.com\n")
</pre>
<h3>The current directory</h3>
<p>
Functions like <code>path-data</code> or <code>open-input-file</code>
or the <q>sugar</q> we seen above all use the current directory
as a base directory for relative pathname. You get the
current value of the current directory with the expression
<code>(current-path)</code>.
This returns a <a href="http://www.gnu.org/software/kawa/Paths.html"><code>path</code></a> object, which prints as the string value of the path.
<p>
The initial value of <code>(current-path)</code> is the
value of the <code>"user.dir"</code> property,
but you change it using a setter:
<pre>
(set! (current-path) "/opt/myApp/")
</pre>
A string value is automatically converted to a <code>path</code>,
normally a <code>filepath</code>.
<p>
The procedure <code>current-path</code> is a
<a href="http://www.gnu.org/software/kawa/Parameter-objects.html">parameter</a>,
so you can alternatively call it with the new value:
<pre>
(current-path "/opt/myApp/")
</pre>
You can also use the <code>parameterize</code> form:
<pre>
(parameterize ((current-path "/opt/myApp/"))
(list &<{data1.txt} &<{data2.txt}))
</pre>
Shell-style programming in Kawahttp://per.bothner.com/blog/2014/Kawa-shell-programming/2014-11-01T21:50:49Z2013-12-30T09:34:38Z
A number of programming languages have facilities to allow access
to system processes (commands). For example Java has
<code>java.lang.Process</code> and <code>java.lang.ProcessBuilder</code>.
However, they're not as convenient as
as the old Bourne shell, or as elegant in composing commands.
<p>
If we ignore syntax, the shell's basic model is that a command is
a function that takes an input string (standard input)
along with some string-valued
command-line arguments, and whose primary
result is a string (standard output). The command also has some
secondary outputs (including standard error, and the exit code).
However, the elegance comes from treating standard output as a string
that can be passed to another command, or used in other contexts that
require a string (e.g. command substitution).
This article presents how Kawa solves this problem.
<h3>Process is auto-convertable to string</h3>
<p>
To run a command like <code>date</code>,
you can use the <code>run-process</code> procedure:
<pre>
#|kawa:2|# (define p1 (run-process "date --utc"))
</pre>
Equivalently you can use the <code>&`{<var>command</var>}</code>
syntactic sugar:
<pre>
#|kawa:2|# (define p1 &`{date --utc})
</pre>
<p>
But what kind of value is <code>p1</code>?
It's an instance of <code>gnu.kawa.functions.LProcess</code>,
which is a class that extends <code>java.lang.Process</code>.
You can see this if you invoke <code>toString</code> or call the
<code>write</code> procedure:
<pre>
#|kawa:2|# (p1:toString)
gnu.kawa.functions.LProcess@377dca04
#|kawa:3|# (write p1)
gnu.kawa.functions.LProcess@377dca04
</pre>
<p>
An <code>LProcess</code> is automatically converted to
a string or a bytevector in a context that requires it.
More precisely, an <code>LProcess</code>
can be converted to a <i><a href="http://per.bothner.com/blog/2014/text-and-binary-data/">blob</a></i>
because it implements <code>Lazy<Blob></code>.
This means you can convert to a string (or bytevector):
<pre>
#|kawa:9|# (define s1 ::string p1)
#|kawa:10|# (write s1)
"Wed Jan 1 01:18:21 UTC 2014\n"
#|kawa:11|# (define b1 ::bytevector p1)
(write b1)
#u8(87 101 100 32 74 97 110 ... 52 10)
</pre>
<p>
However the <code>display</code> procedure prints it in "human" form,
as a string:
<pre>
#|kawa:4|# (display p1)
Wed Jan 1 01:18:21 UTC 2014
</pre>
<p>
This is also the default REPL formatting:
<pre>
#|kawa:5|# &`{date --utc}
Wed Jan 1 01:18:22 UTC 2014
</pre>
<p>
<h3>Command arguments</h3>
<p>
The general form for <code>run-process</code> is:
<pre>
(run-process <var>keyword-argument</var>... <var>command</var>)
</pre>
<p>
The <var>command</var> is the process command-line.
It can be an array of strings, in which case
those are used as the command arguments directly:
<pre>
(run-process ["ls" "-l"])
</pre>
The <var>command</var> can also be a single string,
which is split (tokenized) into command arguments separated by whitespace.
Quotation groups words together just like traditional shells:
<pre>
(run-process "cmd a\"b 'c\"d k'l m\"n'o")
⇒ (run-process ["cmd" "ab 'cd" "k'l m\"no"])
</pre>
<p>
Using <a href="http://www.gnu.org/software/kawa/Strings.html#String-templates">string templates</a> is more readable as it avoids having
to quote quotation marks:
<pre>
(run-process &{cmd a"b 'c"d k'l m"n'o})
</pre>
You can also use the abbreviated form:
<pre>
&`{cmd a"b 'c"d k'l m"n'o}
</pre>
This syntax is the same as
of <a href="http://srfi.schemers.org/srfi-108/srfi-108.html">SRFI-108 named quasi-literals</a>.
In general, the following are roughly equivalent
(the difference is that the former does smart quoting of embedded expressions,
as discussed later):
<pre>
&`{<var>command</var>}
(run-command &{<var>command</var>})
</pre>
<p>
Similarly, the following are also roughly equivalent:
<pre>
&`[<var>keyword-argument</var>...]{<var>command</var>}
(run-command <var>keyword-argument</var>... &{<var>command</var>})
</pre>
<p>
A <var>keyword-argument</var> can specify various
properties of the process. For example you can
specify the working directory of the process:
<pre>
(run-process directory: "/tmp" "ls -l")
</pre>
You can use the <code>shell</code> keyword to
specify that we want to use the shell to
split the string into arguments.
For example:
<pre>
(run-process shell: #t "command line")
</pre>
is equivalent to:
<pre>
(run-process ["/bin/sh" "-c" "command line"])
</pre>
You can can also use the abbreviation <code>&sh</code>:
<pre>
&sh{rm *.class}
</pre>
which is equivalent to:
<pre>
&`{/bin/sh -c "rm *.class"}
</pre>
<p>
In general, the abbreviated syntax:
<pre>
&sh[args...]{command}
</pre>
is equivalent to:
<pre>
&`[shell: #t args...]{command}
</pre>
<h3>Command and variable substitution</h3>
Traditional shells allow you to insert the output
from a command into the command arguments of another command. For example:
<pre>
echo The directory is: `pwd`
</pre>
The equivalent Kawa syntax is:
<pre>
&`{echo The directory is: &`{pwd}}
</pre>
<p>
This is just a special case of substituting the result
from evaluating an expression. The above is a short-hand for:
<pre>
&`{echo The directory is: &[&`{pwd}]}
</pre>
<p>
In general, the syntax:
<pre>
...&[<var>expression</var>]...
</pre>
evaluates the <code><var>expression</var></code>,
converts the result to a string, and combines it with the literal string.
(We'll see the details in the next section.)
This general form subsumes command substitution,
variable substitution, and arithmetic expansion.
<h3 id="substitution-tokenization">Tokenization of substitution result</h3>
<p>
Things gets more interesting when considering the interaction
between substitution and tokenization.
This is not simple string interpolation.
For example, if an interpolated value contains
a quote character, we want to treat it as a literal quote,
rather than a token delimiter. This matches the
behavior of traditional shells. There are multiple cases,
depending on whether the interpolation result is a string
or a vector/list, and depending on whether the interpolation
is inside a quotes.
<ul>
<li>If the value is a string, and we're not inside
quotes, then all non-whitespace characters (including quotes) are literal,
but whitespace still separates tokens:
<pre>
(define v1 "a b'c ")
&`{cmd x y&[v1]z} ⟾ (run-process ["cmd" "x" "ya" "b'c" "z"])
</pre>
<li>If the value is a string, and we are inside
single quotes, all characters (including whitespace) are literal.
<pre>
&`{cmd 'x y&[v1]z'} ⟾ (run-process ["cmd" "x ya b'c z"])
</pre>
<p>
Double quotes work the same except that newline is an argument separator.
This is useful when you have one filename per line, and the filenames
may contain spaces, as in the output from <code>find</code>:
<pre>
&`{ls -l "&`{find . -name '*.pdf'}"}
</pre>
<p>
If the string ends with one or more newlines, those are ignored.
This rule (which also applies in the previous not-inside-quotes case)
matches traditional shell behavior.
<li>If the value is a vector or list (of strings), and we're
not inside quotes, then each element of the array becomes its own argument,
as-is:
<pre>
(define v2 ["a b" "c\"d"])
&`{cmd &[v2]} ⟾ (run-process ["cmd" "a b" "c\"d"])
</pre>
However, if the enclosed expression is adjacent to non-space non-quote
characters, those are prepended to the first element, or appended
to the last element, respectively.
<pre>
&`{cmd x&[v2]y} ⟾ (run-process ["cmd" "xa b" "c\"dy"])
&`{cmd x&[[]]y} ⟾ (run-process ["cmd" "xy"])
</pre>
This behavior is similar to how shells handle <code>"$@"</code>
(or <code>"${name[@]}"</code> for general arrays), though in Kawa
you would leave off the quotes.
<p>
Note the equivalence:
<pre>
&`{&[<var>array</var>]} ⟾ (run-process <var>array</var>)
</pre>
<li>
If the value is a vector or list (of strings), and
we <em>are</em> inside quotes, it is equivalent to interpolating
a single string resulting from concatenating the elements
separated by a space:
<pre>
&`{cmd "&[v2]"} ⟾ (run-process ["cmd" "a b c\"d"])
</pre>
<p>
This behavior is similar to how shells handle <code>"$*"</code>
(or <code>"${name[*]}"</code> for general arrays).
<li>
If the value is the result of a call to <code>unescaped-data</code>
then it is parsed as if it were literal. For example a quote
in the unescaped data may match a quote in the literal:
<pre>
(define vu (unescaped-data "b ' c d '"))
&`{cmd 'a &[vu]z'} ⟾ (run-process ["cmd" "a b " "c" "d" "z"])
</pre>
<li>
If we're using a shell to tokenize the command,
then we add quotes or backslashes as needed so that
the shell will tokenize as described above:
<pre>
&sh{cmd x y&[v1]z} ⟾ (run-process ["/bin/sh" "-c" "cmd x y'a' 'b'\\'''c' z'"])
</pre>
</ul>
Smart tokenization only happens when using the quasi-literal forms
such as <code>&`{command}</code>.
You can of course use <a href="http://www.gnu.org/software/kawa/Strings.html#String-templates">string templates</a> with <code>run-process</code>:
<pre>
(run-process &{echo The directory is: &`{pwd}})
</pre>
However, in that case there is no smart tokenization: The template
is evaluated to a string, and then the resulting string is tokenized,
with no knowledge of where expressions were substituted.
<h3>Input/output redirection</h3>
<p>
You can use various keyword arguments to
specify standard input, output, and error streams. For example
to lower-case the text in <code>in.txt</code>, writing
the result to <code>out.txt</code>, you can do:
<pre>
&`[in-from: "in.txt" out-to: "out.txt"]{tr A-Z a-z}
</pre>
or:
<pre>
(run-process in-from: "in.txt" out-to: "out.txt" "tr A-Z a-z")
</pre>
<p>
These options are supported:
<dl>
<dt><code>in: <var>value</var></code>
<dd>The <code><var>value</var></code> is evaluated,
converted to a string (as if using <code>display</code>),
and copied to the input file of the process. The following are equivalent:
<pre>
&`[in: "text\n"]{command}
&`[in: &`{echo "text"}]{command}
</pre>
<p>You can pipe the output from <code>command1</code>
to the input of <code>command2</code> as follows:
<pre>
&`[in: &`{command1}]{command2}
</pre>
<dt><code>in-from:</code> <var>path</var>
<dd>The process reads its input from the specified <var>path</var>,
which can be any value coercible to a <a href="http://www.gnu.org/software/kawa/Paths.html"><code>filepath</code></a>.
<dt><code>out-to:</code> <var>path</var>
<dd>The process writes its output to the specified <var>path</var>.
<dt><code>err-to:</code> <var>path</var>
<dd>Similarly for the error stream.
<dt><code>out-append-to:</code> <var>path</var>
<dt><code>err-append-to:</code> <var>path</var>
<dd>
Similar to <code>out-to</code> and <code>err-to</code>, but append
to the file specified by <var>path</var>, instead of replacing it.
<dt><code>in-from: 'pipe</code>
<dt><code>out-to: 'pipe</code>
<dt><code>err-to: 'pipe</code>
<dd>Does not set up redirection.
Instead, the specified stream is available using the methods
<code>getOutputStream</code>, <code>getInputStream</code>,
or <code>getErrorStream</code>, respectively,
on the resulting <code>Process</code> object,
just like Java's <code>ProcessBuilder.Redirect.PIPE</code>.
<dt><code>in-from: 'inherit</code>
<dt><code>out-to: 'inherit</code>
<dt><code>err-to: 'inherit</code>
<dd>
Inherits the standard input, output, or error stream from
the current JVM process.
<dt><code>out-to: <var>port</var></code>
<dt><code>err-to: <var>port</var></code>
<dd>Redirects the standard output or error of the process
to the specified <code><var>port</var></code>.
<dt><code>out-to: 'current</code>
<dt><code>err-to: 'current</code>
<dd>Same as <code>out-to: (current-output-port)</code>,
or <code>err-to: (current-error-port)</code>, respectively.
<dt><code>in-from: <var>port</var></code>
<dt><code>in-from: 'current</code>
<dd>Re-directs standard input to read from the
<var>port</var> (or <code>(current-input-port)</code>).
It is unspecified how much is read from the <var>port</var>.
(The implementation is to use a thread that reads from the port,
and sends it to the process, so it might read to the end of the port,
even if the process doesn't read it all.)
<dt><code>err-to: 'out</code>
<dd>
Redirect the standard error of the process to
be merged with the standard output.
</dl>
<p>The default for the error stream (if neither <code>err-to</code>
or <code>err-append-to</code> is specifier) is equivalent
to <code>err-to: 'current</code>.
<p><em>Note:</em>
Writing to a port is implemented by copying
the output or error stream of the process.
This is done in a thread, which means we don't have any
guarantees when the copying is finished.
A possible approach is to have to <code>process-exit-wait</code>
(discussed later) wait for not only the process to
finish, but also for these helper threads to finish.
<h3><q>Here</q> documents</h3>
A <a href="https://en.wikipedia.org/wiki/Here_document">here document</a>
is a form a literal string, typically multi-line, and
commonly used in shells for the standard input of a process.
Kawa's string literals or <a href="http://www.gnu.org/software/kawa/Strings.html#string-quasi-literals">string quasi-literals</a> can be used for this.
For example, this passes the string <code>"line1\nline2\nline3\n"</code>
to the standard input of <code>command</code>:
<pre>
(run-process [in: &{
&|line1
&|line2
&|line3
}] "command")
</pre>
<p>
The <code>&{...}</code> delimits a string;
the <code>&|</code> indicates the preceding indentation is ignored.
<h3>Pipe-lines</h3>
<p>
Writing a multi-stage pipe-line quickly gets ugly:
<pre>
&`[in: &`[in: "My text\n"]{tr a-z A-Z}]{wc}
</pre>
<blockquote>
<i>Aside:</i> This would be nicer in a language with in-fix operators,
assuming <code>&`</code> is treated as a left-associative infix operator,
with the input as the operational left operand:
<pre>
"My text\n" &`{tr a-z A-Z} &`{wc}
</pre>
</blockquote>
<p>
The convenience macro <code>pipe-process</code> makes this much nicer:
<pre>
(pipe-process
"My text\n"
&`{tr a-z A-Z}
&`{wc})
</pre>
<p>
All but the first sub-expressions must be (optionally-sugared)
<code>run-process</code> forms.
The first sub-expression is an arbitrary expression
which becomes the input to the second process expression;
which becomes the input to the third process expression;
and so on. The result of the <code>pipe-process</code>
call is the result of the last sub-expression.
<p>
Copying the output of one process to the input of the next
is optimized: it uses a copying loop in a separate thread.
Thus you can safely pipe long-running processes that produce
huge output. This isn't quite as efficient as using
an operating system pipe, but is portable and works pretty well.
<h3>Setting the process environment</h3>
By default the new process inherits the system environment of the
current (JVM) process as returned by <code>System.getenv()</code>,
but you can override it.
<dl>
<dt><code>env-<var>name</var>: <var>value</var></code>
<dd>
In the process environment, set the <code>"<var>name</var>"</code>
to the specified <code><var>value</var></code>.
For example:
<pre>
&`[env-CLASSPATH: ".:classes"]{java MyClass}
</pre>
<dt><code><var>NAME</var>: <var>value</var></code>
<dd>
Same as using the <code>env-<var>name</var></code> option,
but only if the <code><var>NAME</var></code> is uppercase
(i.e. if uppercasing <code><var>NAME</var></code> yields the same string).
For example the previous example could be written:
<pre>
(run-process CLASSPATH: ".:classes" "java MyClass")
</pre>
<dt><code><var>environment</var>: <var>env</var></code>
<dd>
The <code><var>env</var></code> is evaluated and must yield
a <code>HashMap</code>.
This map is used as the system environment of the process.
</dl>
<h3>Process-based control flow</h3>
<p>
Traditional shell provides logic control flow operations
based on the exit code of a process: 0 is success (true),
while non-zero is failure (false).
Thus you might see:
<pre>
if grep Version Makefile >/dev/null
then echo found Version
else echo no Version
fi
</pre>
<p>
One idea to have a process be auto-convertible to a boolean,
in addition to be auto-convertible to strings or bytevectors:
In a boolean context, we'd wait for the process to finish,
and return <code>#t</code> if the exit code is 0,
and <code>#f</code> otherwise. This idea may be worth exploring later.
<p>
Currently Kawa provides <code>process-exit-wait</code>
which waits for a process to exit, and then returns the
exit code as an <code>int</code>.
The convenience function <code>process-exit-ok?</code>
returns true iff <code>process-exit-wait</code> returns 0.
<pre>
(process-exit-wait (run-process "echo foo")) ⟾ 0
</pre>
The previous <code>sh</code> example could be written:
<pre>
(if (process-exit-ok? &`{grep Version Makefile})
&`{echo found}
&`{echo not found})
</pre>
Note that unlike the <code>sh</code>, this ignores the
output from the <code>grep</code> (because no-one has asked for it).
To match the output from the shell, you can use <code>out-to: 'inherit</code>:
<pre>
(if (process-exit-ok? &`[out-to: 'inherit]{grep Version Makefile})
&`{echo found}
&`{echo not found})
</pre>
Kawa 1.14 releasedhttp://per.bothner.com/blog/2013/Kawa-1.14-released/2013-10-05T21:30:41Z2013-10-05T04:44:14Z
Kawa 1.14 is now evailable from the usual places:
<blockquote>
<a href="ftp://ftp.gnu.org/pub/gnu/kawa/kawa-1.14.tar.gz">ftp://ftp.gnu.org/pub/gnu/kawa/kawa-1.14.tar.gz (source code)</a><br />
<a href="ftp://ftp.gnu.org/pub/gnu/kawa/kawa-1.14.jar">ftp://ftp.gnu.org/pub/gnu/kawa/kawa-1.14.jar (runnable <code>jar</code>)</a>
</blockquote>
<p>
For a extended list of changes, see the <a href="http://www.gnu.org/software/kawa/news.html">news page</a>. Below are some highlights.
<h3>More R7RS functions</h3>
<p>
The proposed new standard for Scheme <a href="http://www.r7rs.org/">R7RS</a>
has a number of new functions, and also extensions to existing functions.
Kawa 1.14 implements (and documents) most of the new functions
and extensions, and also implements a number of the syntactic changes:
<li>
After adding list procedures <code>make-list</code>, <code>list-copy</code>,
<code>list-set!</code> all the R7RS list procedures are implemented.
<li>Other added procedures: <code>square</code>,
<code>boolean=?</code>, <code>string-copy!</code>,
<code>digit-value</code>,
<code>get-environment-variable</code>, <code>get-environment-variables</code>,
<code>current-second</code>, <code>current-jiffy</code>,
<code>jiffies-per-second</code>, and <code>features</code>.
<li>
The predicates <code>finite?</code>, <code>infinite?</code>, and
<code>nan?</code> are generalized to complex numbers.
<li>
The procedures <code>write</code>, <code>write-simple</code>,
and <code>write-shared</code> are now consistent with R7RS.
<li>
String and character comparison functions are generalized to more
than two arguments (but restricted to strings or characters, respectively).
<li>
The procedures <code>string-copy</code>, <code>string->list</code>,
and <code>string-fill!</code> now take optional (start,end)-bounds.
All of the R7RS string functions are now implemented.
<li>
Support <code>=></code> syntax in <code>case</code> form.
<li>
Support backslash-escaped special characters in symbols
when inside vertical bars, such as <code>'|Hello\nworld|</code>.
<li>
The new functions and syntax are documented in
the <a href="http://www.gnu.org/software/kawa/index.html">Kawa manual</a>; look for the functions in the
<a href="http://www.gnu.org/software/kawa/Overall-Index.html">index</a>.
</ul></li>
<p>
The most important functionality still missing is support for the
library (module) system, and specifically the <code>define-library</code>
form. Of course Kawa has its own powerful
<a href="http://www.gnu.org/software/kawa/Module-classes.html">module system</a>
which is in spirit quite compatible, but we still need to decide on the
pragmatic issues, including how modules are resolved to files
(probably using a module search path), and translation to class names.
<p>
Another feature to ponder is how to handle the R7RS exception
mechanism, and how it maps to JVM exceptions. We probably won't
implement all of the R7RS semantics, but it would be desirable
if (for example) some simple uses of <code>guard</code> would work on Kawa.
<h3>Better support for multi-line strings and templates</h3>
<a href="http://srfi.schemers.org/srfi-109/srfi-109.html">SRFI-109</a>
specifies a syntax for quasi-literal strings.
These have a numbers of advantages over traditional double-quote literals.
The are quasi-literals in the sense that they can contain
embedded expressions. For example,
if <code>name</code> has the value <code>"John"</code>, then:
<pre>
&{Hello &[name]!}
</pre>
evaluates to <code>"Hello John!"</code>.
Other benefits include control over indentation (layout),
built-in support for <code>format</code> specifiers,
and a huge number of <a href="http://www.w3.org/2003/entities/2007/w3centities-f.ent">named special characters</a>.
See the <a href="http://www.gnu.org/software/kawa/Strings.html#String-templates">documentation</a> and the
<a href="http://srfi.schemers.org/srfi-109/srfi-109.html">SRFI-109 specification for details</a>.
<h3>Named quasi-literals (SRFI-108)</h3>
<a href="http://srfi.schemers.org/srfi-108/srfi-108.html">SRFI-108</a>
can be viewed as a generalization of SRFI-109, in that the
constructors have names, and you can define new constructors.
This could be used for writing documentation
(similar to the <a href="http://docs.racket-lang.org/scribble/">Scribble</a>
system), or constructing complex nested data structures (a generalization
of XML).
<h3>Better support for command scripts</h3>
A number of little improvements have been made to make it
easier to write Kawa command scripts (simple one-file applications).
Most importantly the
<a href="http://www.gnu.org/software/kawa/Scripts.html">documentation</a> has been re-written.
The value returned by <code>(car (command-line))</code>
(i.e. the "name" of the script/command invoked) is now accurate and
useful in many more cases; you also have the option of setting explicitly
it with the <code>-Dkawa.command.name</code> option.
<h3>Experimental Java 8 support</h3>
<p>There is some very preliminary support for next year's Java 8.
These are enabled if you run <code>configure</code> with
the <code>--with-java-source=8</code> (or if <code>ant</code>
auto-detects <code>java.util.stream.Stream</code>).
This flag doesn't yet do very much:
<ul>
<li>It sets the classfile major version number.
<li>It adds an optimized version of the <code>Stream#forEach</code>
method for <code>gnu.lists.SimpleVector</code> and its subclasses,
which includes the <code>FVector</code> used to implement Scheme vectors.
<li>Enhance <code>gnu.lists.Consumer</code> to implement
<code>java.util.function.Consumer</code> and the
specialized classes <code>java.util.function.{Int,Long,Double}Consumer</code>.
<p>
Because output port implements <code>gnu.lists.Consumer</code>,
this means you can pass a port to the <code>forEach</code> method:
<pre>
#|kawa:11|# ((['a 'b 'c 'd]:stream):forEach (current-output-port))
a b c d
</pre>
</ul>
<p>
Of course a large extent streams just Work Out of the Box
with Kawa data types, because they implement the apropriate
Java interfaces. For examples lists and vectors both implement
<code>java.util.List</code>, and thus they automatically have
the <code>stream</code> method.
Generalized literals for Kawahttp://per.bothner.com/blog/2012/GeneralLiterals/2012-08-30T23:12:00Z2012-08-29T00:32:19Z
<p>This note is a rationale and design discussion
of a number Kawa feature for more powerful string and other literals.
This feature aims to satisfy a number of related needs.
<ul>
<li>
<p>Using Kawa string literals for long multi-line strings is awkward.
One problem is that the same delimiter (double-quote) is used for both
the start and end of the string. This is error-prone and not robust:
adding or removing a single character changes the reading of the entire
rest of the program.
A related problem is that the delimiter needs to be quoted using
an escape character, which can get hard-to-read.
<p>
A common solution is a
<a href="http://en.wikipedia.org/wiki/Here_document"><q>here document</q></a>,
where distinct multi-character start and end delimiters are used.
For example the <a href="http://en.wikipedia.org/wiki/Unix_shell">Unix shell</a>
uses uses <code><<</code> followed by an arbitrary token
as the start delimiter, and then the same token as the end delimiter:
<pre>
tr a-z A-Z <<END_TEXT
one two three
uno dos tres
END_TEXT
</pre>
<p>
This proposal would use:
<pre>
(string-upcase #&[
one two three
uno dos tres
])
</pre>
<li>Commonly one wants to construct a string as a concatenation of
literal text with evaluated expressions.
Using explicit string concatenation (Scheme <code>string-append</code>
or Java's <code>+</code> operator)
is verbose and can be error-prone.
Using <code>format</code> is an alternative, but it is also a bit verbose,
and has the problem that the format specifier in the string is widely
separated from the expression.
Nicer is to be able to use
<a href="http://en.wikipedia.org/wiki/Variable_interpolation">Variable interpolation</a>, as in Unix shells:
<pre>
echo "Hello ${name}!"
</pre>
<p>
This proposal uses the syntax:
<pre>
#&[Hello &{name}!]
</pre>
<p>
Note that <code>&</code> is used for two different related purposes:
Part of the prefix <code>#&[</code> to mark the entire string,
and as an escape character for the variable interpolation.
This will be justified shortly.
<li>
Going one step further,
<a href="http://en.wikipedia.org/wiki/Template_processor">template processor</a>
has many uses.
Examples include <a href="http://brl.sourceforge.net/">BRL</a>
and <a href="http://en.wikipedia.org/wiki/JavaServer_Pages">JSP</a>,
which are both used to generate web pages.
<p>
The simple solution is to allow general Kawa expressions in substitutions:
<pre>
#&[Hello &{(string-capitalize name)}!]
</pre>
<p>
You can also leave out the curly braces when the expression is
a parenthesized expression:
<pre>
#&[Hello &(string-capitalize name)!]
</pre>
<p>
Note that this syntax for unquoted expressions matches that used in Kawa's
<a href="http://www.gnu.org/software/kawa/XML-literals.html">XML literals</a>.
<li>
The <a href="http://docs.racket-lang.org/scribble/">Scribble</a> system
defines a language for writing documents. It is a kind of template
processor with embedded expression in the Racket Scheme dialect.
The general Racket syntax for an embedded expression is:
<pre>
@<var>cmd</var>[<var>datum</var> ...]{<var>text-body</var>}
</pre>
Kawa switches the roles of <code>{}</code> and <code>[]</code> to
be compatible with XML-literals, and also because
<code>{}</code> is more commonly used for anti-quotation.
Kawa uses special characters, so it becomes:
<pre>
&<var>cmd</var>{<var>datum</var> ...}[<var>text-body</var>]
</pre>
<p>
This form translates to:
<pre>
(<var>cmd</var> <var>datum</var> ... #&[<var>text-body</var>])
</pre>
<li>
<a href="http://caml.inria.fr/pub/docs/tutorial-camlp4/tutorial004.html">Camp4 quotation</a>
SRFI-10
Example: #&sql[select * from person where firstname != ${ignore}]
</ul>
JavaFX using Kawa - introductionhttp://per.bothner.com/blog/2011/JavaFX-using-Kawa-intro/2011-08-16T22:36:59Z2011-07-28T18:06:26Z
<p>
JavaFX 1.0 was a next-generation GUI/client platform.
It had a new Node-based GUI API and used a new language,
JavaFX Script, whose goal was to make it easier to program Rich Client applications.
(Yours truly was hired by Sun to work on the JavaFX Script compiler.)
In 2010 the JavaFX Script language was cancelled:
JavaFX would still refer to a new GUI based API based on many
of the same concepts, but the primary programming language would be Java,
rather than JavaFX Script.</p>
<p>
Java is a relatively low-level and clumsy language for
writing Rich Client appliations, though it's not <em>too</em> painful.
Still, there was a reason we worked on JavaFX Script: It had
a number of features to make such programs more convenient.
Luckily, other JVM languages - and specifically Kawa-Scheme - can take up the slack. Below I'll show you a simple Hello-World-type example,
and then explain how you can try it yourself. In later acticles I'll show
different examples.</p>
<h3>Simple buttons and events</h3>
<p>
Our first example is just 3 buttons and 2 trivial event handlers.
It is translated from <a href="http://per.bothner.com/blog/2011/JavaFX-using-Kawa-intro/HelloButton.java">HelloButton.java</a> written in Java by Richard Bair.</p>
<pre>
(require 'javafx-defs)
(javafx-application)
(javafx-scene
title: "Hello Button"
width: 600 height: 450
(Button
text: "Click Me"
layout-x: 25
layout-y: 40
on-action: (lambda (e) (format #t "Event: ~s~%" e))
on-key-released: (lambda (e) (format #t "Event: ~s~%" e)))
(Button
text: "Click Me Too"
layout-x: 25
layout-y: 70)
(Button
text: "Click Me Three"
layout-x: 25
layout-y: 100))
</pre>
<p>
For those new to Scheme, the basic syntactic building block has the form:
<pre>
(<var>operator</var> <var>argument1</var> ... <var>argumentN</var>)
</pre>
The <code><var>operator</var></code> can be a function (like <code>format</code>),
an arithmetic operator in prefix form (like <code>(+ 3 4)</code>),
a command, a keyword (like <code>lambda</code>), or a user-defined macro.
This general format makes for a lot of flexibility.
<p>
The first two lines in <code>HelloButton.scm</code> are boiler-plate:
The <code>require</code> imports various definitions and aliases,
while the <code>(javafx-application)</code> syntax declares
this module is a JavaFX <code>Application</code>.</p>
<p>
The <code>javafx-scene</code> form (a macro) creates a <em>scene</em>,
which is a collection of graphical objects.
The <code>Scene</code> has certain named properties (<code>title</code>,
<code>width</code>, and <code>height</code>), specified using <em>keyword arguments</em>. The <code>Scene</code> also has 3 <code>Button</code> children.
Finally, the <code>make-scene</code> command puts the scene
on the <em>stage</em> (the window) and makes it visible.
<p>
Each <code>Button</code> form is an <a href="http://www.gnu.org/software/kawa/Allocating-objects.html">object constructor</a>. For example:
<pre>
(Button
text: "Click Me Three"
layout-x: 25
layout-y: 100)
</pre>
is equivalent to the Java code:
<pre>
javafx.scene.control.Button tmp = new javafx.scene.control.Button();
tmp.setText("Click Me Three");
tmp.setLayoutX(25);
tmp.setLayoutY(100);
return tmp;
</pre>
<p>
<p>The <code>on-action</code> and <code>on-key-released</code>
properties on the first <code>Button</code> bind event handlers.
Each handler is a <a href="http://en.wikipedia.org/wiki/Anonymous_function">lambda expression or anonymous function</a> that takes an event <code>e</code> as a parameter. The Kawa compiler converts the handler to a suitable
event handler object using
<q><a href="http://www.gnu.org/software/kawa/Anonymous-classes.html#SAM-conversion">SAM-conversion</a> features</q>.
(This conversion depends on the context, so if you don't have a
literal lambda expression you have to do the conversion by hand
using an <a href="http://www.gnu.org/software/kawa/Anonymous-classes.html"><code>object</code> operator</a>.)
<h3>Getting it to run</h3>
<h4>Downloading JavaFX 2.x beta</h4>
<p>
For now JavaFX is only available for Windows, but Mac and GNU/Linux ports
are being worked on and mostly work. (I primarily use Fedora Linux.)
The <a href="http://javafx.com">primary JavaFX site</a> has lots of information,
including a link to the
<a href="http://www.oracle.com/technetwork/java/javafx/downloads/index.html">download site</a>.
You will need to register, as long the software is beta.
Download the zipfile and extract it to some suitable location.
<p>
In the following, we assume the variable <code>JAVAFX_HOME</code>
is set to the build you've installed. For example (if using plain Windows):
<pre>
set JAVAFX_HOME=c:\javafx-sdk2.0-beta
</pre>
The file <code>%JAVAFX_HOME%\rt\lib\jfxrt.jar</code> should exist.
<h4>Downloading and building Kawa</h4>
<p>
The JavaFX support in Kawa is new and experimental (and unstable),
so for now you will have to
<a href="http://www.gnu.org/software/kawa/Getting-Kawa.html#Getting-the-development-sources-using-SVN">get the Kawa source code from SVN</a>.
<p>
There are two ways to build Kawa. The easiest is to use Ant - on plain Windows do:
<pre>
ant -Djavafx.home=%JAVAFX_HOME%
</pre>
or on other platforms (including Cygwin):
<pre>
ant -Djavafx.home=$JAVAFX_HOME
</pre>
<p>
Alternatively, you can use <code>configure</code> and <code>make</code>
(but note that on Windows you will need to have <a href="http://www.cygwin.org/">Cygwin</a> installed to use this approach):
<pre>
$ KAWA_DIR=path_to_Kawa_sources
$ cd $KAWA_DIR
$ ./configure --with-javafx=$JAVAFX_HOME
$ make
</pre>
<h4>Running the example</h4>
<p>
On Windows, the easiest way to run the example is to use the
<code>kawa.bat</code> created when building Kawa. It sets up
the necessary paths for you.
<pre>
%KAWA_HOME%\bin\kawa.bat HelloButton.scm
</pre>
<p>
On Cygwin (or Unix/Linux) you can use the similar <code>kawa.sh</code>.
I suggest setting your <code>PATH</code> to find
<code>kawa.bat</code> or <code>kawa.sh</code>, so you can just do:
<pre>
kawa HelloButton.scm
</pre>
<p>
Using the <code>kawa</code> command is equivalent to
<pre>
java -classpath <var>classpath</var> kawa.repl HelloButton.scm
</pre>
but it sets the <var>classpath</var> automatically. If you do it
by hand you need to include <code>%JAVAFX_HOME%\rt\lib\jfxrt.jar</code>
and <code>%KAWA_DIR%\kawa-<var>version</var>.jar</code>.
<p>
This is what pops up:
<p>
<img src="http://per.bothner.com/blog/2011/JavaFX-using-Kawa-intro/HelloButton.png" alt="HelloButton.png" />
<p>
If you click the first button the <code>action</code> event fires, and
you should see something like:
<pre>
Event: javafx.event.ActionEvent[source=Button@3a5794[styleClass=button]]
</pre>
<p>
If you type a key (say <code>n</code>) while that button <q>has focus</q> (e.g. after clicking it),
then when you release the key a <code>key-released</code> event fires:
<pre>
Event: KeyEvent [source = Button@3a5794[styleClass=button],
target = Button@3a5794[styleClass=button], eventType = KEY_RELEASED,
consumed = false, character = , text = n, code = N]
</pre>
<p>
<em>Note:</em> Running a JavaFX application from the Kawa read-eval-print-loop
(REPL) doesn't work very well at this point,
but I'm exploring ideas to make it useful.
<h4>Compiling the example</h4>
<p>
You can compile <code>HelloButton.scm</code> to class files:
<pre>
kawa --main -C HelloButton.scm
</pre>
<p>
You can execute the resulting application in the usual way:
<pre>
java -classpath <var>classpath</var> HelloButton
</pre>
or use the <code>kawa</code> command:
<pre>
kawa HelloButton
</pre>
<h3>Next</h3>
<a href="http://per.bothner.com/blog/2011/JavaFX-using-Kawa-animation/">JavaFX-using-Kawa-animation</a>
JavaFX using Kawa - simple animationhttp://per.bothner.com/blog/2011/JavaFX-using-Kawa-animation/2011-08-16T23:06:12Z2011-07-28T18:05:22Z
<p>
Read <a href="http://per.bothner.com/blog/2011/JavaFX-using-Kawa-intro/">JavaFX-using-Kawa-intro</a> first.
That introduces JavaFX and how you can use Kawa to write
rich GUI applications.
<p>
This example demonstrates simple animation: A rectangle that moves
smoothly left to right and back again continuously.
This is example is converted from <a href="http://per.bothner.com/blog/2011/JavaFX-using-Kawa-animation/HelloAnimation.java">HelloAnimation.java</a> written in Java by Kevin Rushforth. Here is the entire program <code>HelloAnimation.scm</code>:
<pre>
(require 'javafx-defs)
(javafx-application)
(define rect
(Rectangle
x: 25 y: 40 width: 100 height: 50
fill: Color:RED))
(javafx-scene
title: "Hello Animation"
width: 600 height: 450 fill: Color:LIGHTGREEN
rect)
((Timeline
cycle-count: Timeline:INDEFINITE
auto-reverse: #t
(KeyFrame
(Duration:millis 500)
(KeyValue (rect:xProperty) 200))):play)
</pre>
<p>
The first two lines are boilerplate, as in <a href="http://per.bothner.com/blog/2011/JavaFX-using-Kawa-intro/">JavaFX-using-Kawa-intro</a>.
The <code>(define rect (Rectange ...))</code> defines a variable
<code>rect</code>, and initializes it to a new <code>Rectangle</code>,
using an <a href="http://www.gnu.org/software/kawa/Allocating-objects.html">object constructor</a> as seen before.
<p>
The <code>(javafx-scene ...)</code> operations creates the scene with
the specified <code>title</code>, <code>width</code>, <code>height</code>
and <code>fill</code> (background color) properties.
It adds the <code>rect</code> to the scene, and then creates a window to make it visible.
<p>
Finally, we animate <code>rect</code>. This requires some background explanation.
<h4>Key values and key frames</h4>
<p>
To animate a scene you create a <code>TimeLine</code> data structure,
which describes what properties to modify and when to do so.
Once you have done so, call <code>play</code> on the <code>TimeLine</code>,
which instructs JavaFX's animation engine to start running the animation.
In this example, the animation continuous indefinitely:
When done, it reverses (because <code>auto-reverse</code> was set to true)
and starts over.
<p>The timeline is divided into a fixed number of
<a href="http://en.wikipedia.org/wiki/Key_frame"><dfn>key frames</dfn></a>,
which are associated with a specific point in time.
In the current example there is the implicit starting key-frame
at time 0, and an explicit key-frame at time 500 (milliseconds).
(The <code>KeyFrame</code> specifies a duration or ending time,
where the start time is the ending time of the previous key-frame,
or 0 in the case of the first key-frame.)
The animator (or programmer) specifies various properties (such
as positions and sizes of objects) at each key-frame, and then the
computer smoothly interpolates between the key-frames.
By default the interpolation is linear, but you can specify other
kinds of interpolation.
<p>
Each <code>KeyFrame</code> consists of some <code>KeyValue</code> objects.
A <code>KeyValue</code> species which property to modify
and the ending value that the property will have at the end of the key-frame.
In the example, we have a single <code>KeyValue</code> to modify
<code>rect:x</code>, ending at 200. (The start value is 25, as set when
<code>rect</code> was constructed.)
<p>
In JavaFX, the properties to animate are specified using
<code>Property</code> objects. Typeically, a property is a reference
to a specific field in a specific object instance. You register the
property with the animation engine (as shown in the sample), and then
the animation engine uses the property to modify the field.
In the example the expression <code>(rect:xProperty)</code>
(equivalent to the Java method call <code>rect.xProperty()</code>)
returns a property that references the <code>x</code> property
of the <code>rect</code> object.
The <code>Property</code> object also has ways to register dependencies
(rather like change listeners) so that things get updated
when the property is changed. (For example the rectangle is re-drawn
when its <code>x</code> property changes.)
<p>
Here is a static screenshot - obviously an animiated gif would be nicer here!
<img src="http://per.bothner.com/blog/2011/JavaFX-using-Kawa-animation/HelloAnimation.png" alt="HelloAnimation.png" />
<h4>More information</h4>
<i>ToDo: Links to documentation and other JavaFX examples.</i>
Using JAXB Annotations to read/write XMLhttp://per.bothner.com/blog/2011/Using-JAXB-annotations/2011-01-07T19:46:14Z2011-01-04T02:31:51Z
The <a href="http://jcp.org/en/jsr/detail?id=222">Java Architecture for XML Binding (JAXB) 2.0</a> lets you use
<a href="http://download.oracle.com/javase/1.5.0/docs/guide/language/annotations.html">annotations</a>
to how specify Java class instances get <q>printed</q> (serialized) as XML documents. Kawa recently gained support for annotations, so let us
look at an example.
You will need Java 6 for the JAXB 2.0 support, and Kawa
<a href="http://www.gnu.org/software/kawa/Getting-Kawa.html">from SVN</a> (or Kawa 1.12 when it comes out). Nothing else is needed.
<p>
Our example is a simple bibliography database of books and some information
about them.
The <a href="http://www.w3.org/TR/xquery-use-cases/#xmp-dtd">DTD</a>
and <a href="http://www.w3.org/TR/xquery-use-cases/#xmp-data">sample input data</a>
are taken from the <a href="http://www.w3.org/TR/xquery-use-cases/">XML Query Use Cases</a>. The complete example is in the
file <code>testuite/jaxb-annotations3.scm</code>
in the Kawa source distribution.
<h3>Defining the Java classes and their XML mapping</h3>
We start with some <code>define-alias</code> declarations, which serve
the role of Java's <code>import</code>:
<pre>
(define-alias JAXBContext javax.xml.bind.JAXBContext)
(define-alias StringReader java.io.StringReader)
(define-alias XmlRegistry javax.xml.bind.annotation.XmlRegistry)
(define-alias XmlRootElement javax.xml.bind.annotation.XmlRootElement)
(define-alias XmlElement javax.xml.bind.annotation.XmlElement)
(define-alias XmlAttribute javax.xml.bind.annotation.XmlAttribute)
(define-alias BigDecimal java.math.BigDecimal)
(define-alias ArrayList java.util.ArrayList)
</pre>
The bibliography is represented by a Java singleton class <code>Bib</code>,
which has a <code>books</code> field, which is
an <code>ArrayList</code> collection of <code>Book</code> objects:
<pre>
(define-simple-class Bib ( ) (@XmlRootElement name: "bib")
(books (@XmlElement name: "book" type: Book) ::ArrayList))
</pre>
<p>
The <code>XmlRootElement</code> annotation says that the <code>Bib</code>
is represented in XML as a root element (document element)
named <code><bib></code>. Each <code>Book</code> is represented
using a <code><book></code> element.
Note the annotation value <code>name: "book"</code> is needed
because I decided to use the plural <code>books</code> for the
field name (because it is an <code>List</code>), but the singular
<code>book</code> for the tag name for each book element.
<p>
Each book is represented using the class <code>Book</code>,
which has the fields <code>year</code>, <code>title</code>,
<code>publisher</code> (all strings),
<code>authors</code> and <code>editors</code> (both <code>ArrayList</code>s),
and <code>price</code> (a <code>BigDecimal</code>).
Note that the while the other fields get mapped to XML child elements,
the <code>year</code> is mapped to an XML attribute,
also named <code>year</code>.
<pre>
(define-simple-class Book ()
(year (@XmlAttribute required: #t) ::String)
(title (@XmlElement) ::String)
(authors (@XmlElement name: "author" type: Author) ::ArrayList)
(editors (@XmlElement name: "editor" type: Editor) ::ArrayList)
(publisher (@XmlElement) ::String)
(price (@XmlElement) ::BigDecimal))
</pre>
<p>
Finally, the <code>Author</code> and <code>Editor</code> classes,
both of which inherit from <code>Person</code>:
<pre>
(define-simple-class Person ()
(last (@XmlElement) ::String)
(first (@XmlElement) ::String))
(define-simple-class Author (Person))
(define-simple-class Editor (Person)
(affiliation (@XmlElement) ::String))
</pre>
<h3>Setting up the JAXB context</h3>
<p>
Next we need to specify a <a href="http://download.oracle.com/javase/6/docs/api/javax/xml/bind/JAXBContext.html"><code>JAXBContext</code></a>, which manages the
mapping between Java classes and the XML document schema (structure).
Once JAXB knows the classes involved, it can figure out the XML representation
by analyzing the classes and their annotations (using reflection).
A simple way to do this is to just list the needed classes
when creating the <code>JAXBContext</code>:
<pre>
(define jaxb-context ::JAXBContext
(JAXBContext:newInstance Bib Book Person Editor))
</pre>
<p>
(Actually, you don't need to list all the classes: if you just
list the root class <code>Bib</code>, JAXB can figure out the rest.)
<p>
An alternative is to make use of an <code>ObjectFactory</code>.
Specifically, if the element classes are in a package named <code>my.package</code>
then you need to create a class <code>ObjectFactory</code> (in the same
package), and specify the package name to the <code>JAXBContext</code> factory method <code>newInstance</code>. The <code>ObjectFactory</code> class
needs to have a factory method for at least the root class <code>Bib</code>:
<pre>
(define jaxb-context ::JAXBContext
(JAXBContext:newInstance "my.package"))
(define-simple-class ObjectFactory () (@XmlRegistry)
((createBib) ::Bib (Bib)))
</pre>
<p>
<h3>Unmarshalling - reading XML and creating objects</h3>
<p>At this point actually parsing the XML and creating objects
is trivial. You create an
<a href="http://download.oracle.com/javase/6/docs/api/javax/xml/bind/Unmarshaller.html"><code>Unmarshaller</code></a> from the <code>JAXBContext</code>,
and then just invoke one of the <code>Unmarshaller</code>
<code>unmarshall</code> methods.
<pre>
(define (parse-xml (in ::java.io.Reader)) ::Bib
((jaxb-context:createUnmarshaller):unmarshal in))
(define bb (parse-xml (current-input-port)))
</pre>
<h3>Update the values</h3>
<p>The whole point of unmarshalling the XML is
presumably so you can do something with the data.
First, an example of modifying existing <q>records</q>:
let's deal with inflation and increase all the prices by 10%:
<pre>
;; Multiply the price of all the books in bb by ratio.
(define (adjust-prices (bb::Bib) (ratio::double))
(let* ((books bb:books)
(nbooks (books:size)))
(do ((i :: int 0 (+ i 1))) ((>= i nbooks))
(let* ((book ::Book (books i)))
(set! book:price (adjust-price book:price ratio))))))
;; Multiply old by ratio to yield an updated 2-decimal-digit BigDecimal.
(define (adjust-price old::BigDecimal ratio::double)::BigDecimal
(BigDecimal (round (* (old:doubleValue) ratio 100)) 2))
(adjust-prices bb 1.1)
</pre>
<p>Next, let's add a new book:
<pre>
(bb:books:add
(Book year: "2006"
title: "JavaScript: The Definitive Guide (5th edtion)"
authors: [(Author last: "Flanagan" first: "David")]
publisher: "O'Reilly"
price: (BigDecimal "49.99")))
</pre>
<p>Notice the use of square brackets (a new Kawa feature)
for the <code>authors</code> single-element sequence.
<h3>Marshalling - writing XML from objects</h3>
<p>Finally, we might want to write the updated data out to an
XML file. For that we create a
<a href="http://download.oracle.com/javase/6/docs/api/javax/xml/bind/Marshaller.html"><code>Marshaller</code></a>. It has a number of <code>marshall</code>
methods that take a <q>jaxbElement</q> and a target specification,
and serializes the former to the latter as XML. For example, you could
just pass in a <code>File</code> to specify the name of the XML output file.
However, Kawa includes an XML pretty-printer, and it would be nice
to have pretty-printed and indented XML. To do that we have to use
a SAX2 <a href="http://download.oracle.com/javase/6/docs/api/org/xml/sax/ContentHandler.html"><code>ContentHandler</code></a> as the bridge between JAXB and
Kawa's XML tools. The Kawa <a href="http://www.gnu.org/software/kawa/api/gnu/xml/XMLFilter.html"><code>XMLFilter</code></a> implements
<code>ContentHandler</code> and can forward XML-like data to an
<a href="http://www.gnu.org/software/kawa/api/gnu/xml/XMLPrinter.html"><code>XMLPrinter</code></a>. The latter doesn't do pretty-printing by default -
you have to set the <code>*print-xml-indent*</code> fluid variable
to the symbol <code>'pretty</code>.
<pre>
;; Write bb as pretty-printed XML to an output port.
(define (write-bib (bb ::Bib) (out ::output-port))::void
(let ((m (jaxb-context:createMarshaller)))
(fluid-let ((*print-xml-indent* 'pretty))
;; We could marshal directly to 'out' (which extends java.io.Writer).
;; However, XMLPrinter can pretty-print the output more readably.
;; We use XMLFilter as glue that implements org.xml.sax.ContentHandler.
(m:marshal bb (gnu.xml.XMLFilter (gnu.xml.XMLPrinter out))))))
(write-bib bb (current-output-port))
</pre>
Kawa 1.11 releasedhttp://per.bothner.com/blog/2010/Kawa-1.11-released/2010-11-15T21:48:14Z2010-11-15T20:11:32Z
Kawa 1.11 is now evailable from the usual places:
<blockquote>
<a href="ftp://ftp.gnu.org/pub/gnu/kawa/kawa-1.11.tar.gz">ftp://ftp.gnu.org/pub/gnu/kawa/kawa-1.11.tar.gz (source code)</a><br />
<a href="ftp://ftp.gnu.org/pub/gnu/kawa/kawa-1.11.jar">ftp://ftp.gnu.org/pub/gnu/kawa/kawa-1.11.jar (runnable <code>jar</code>)</a>
</blockquote>
<p>
For a full list of changes, see the <a href="http://www.gnu.org/software/kawa/news.html">news page</a>. Below are some highlights.
<h3>Improved object initialization</h3>
<p>When constructing an object and there is no matching
constructor method, look for "<code>add</code>" methods in addition
to "<code>set</code>" methods. Also, allow passing constructor args
as well as keyword setters.
These features make it more convenient to allocate and initialize
complex objects.
<a href="http://www.gnu.org/software/kawa/Allocating-objects.html">See here</a> for details and examples, including a simple but complete Swing application.
<h3>SAM-conversion</h3>
<p>
In a context that expects a Single Abstract Method (SAM) type (for
example <code>java.lang.Runnable</code>), if you pass a lambda you will get an
object where the lambda implements the abstract method. For example,
you can just write:
<pre>
(button:addActionListener
(lambda (e) (do-something)))
</pre>
instead of the equivalent but more long-winded:
<pre>
(button:addActionListener
(object (java.awt.event.ActionListener)
((actionPerformed (e ::java.awt.event.ActionEvent))::void
(do-something))))
</pre>
See <a href="http://www.gnu.org/software/kawa/Anonymous-classes.html#SAM-conversion">here for more information.</a>
<h3>Enumerations</h3>
<p>A new <code>define-enum</code> form, contributed by Jamison Hope,
makes it easy to define enuration types and values compatible with Java5 enums.
<pre>
(define-enum colors (red blue green))
</pre>
For <a href="http://www.gnu.org/software/kawa/Enumerations.html">more details and examples, see here</a>.
<h3>New Kawa logo</h3>
Kawa new has a logo contributed by Jakub Jankiewicz:
<p>
<img src="http://www.gnu.org/software/kawa/style/kawa-logo.png" alt="Kawa logo" />
Q2 - extensible syntax with indentationhttp://per.bothner.com/blog/2010/Q2-extensible-syntax/2010-11-15T15:26:28Z2010-11-14T21:28:29Z
<code>Q2</code> is the (temporary?) name of a project
where I'm exploring some ideas in programming languages.
At this point it is not a complete or coherent language,
but you might find some ideas of interest.
This article looks at the overall syntactic structure of Q2 programs.
Future articles will look at other issues (declarations and patterns;
logic variables and unification; sequences and loop comprehensions) as they
get implemented.
<p>
To try the examples <a href="http://www.gnu.org/software/kawa/Getting-Kawa.html#Getting-the-development-sources-using-SVN">get the SVN development sources</a>
and <a href="http://www.gnu.org/software/kawa/Source-distribution.html">build them</a>. You can then run Q2 code by passing the <code>--q2</code>
language switch on the <code>kawa</code> command line, or using the <code>.q2</code> file extension.
There are some examples in the <code>gnu/q2/testsuite</code> directory.
<h3>Extensible syntax with no reserved identifiers</h3>
<p>
There are many virtues of traditional Lisp/Scheme syntax,
primarily a minimal fixed core syntax, combined with a very flexible
and extensible structural syntax. There are no reserved identifiers,
and the pre-defined syntax for definitional forms and control flow
is not hard-coded: they can in principle be defined in the language
itself, and you can replace and add to them.
<p>Q2 follows these goals: There is a fixed core (lexical) syntax of tokens
and delimiters, which is read by a (Lisp-like) <q>reader</q>.
Name-resolution and high-level syntactic forms are processed
later at <q>macro-expansion time</q>.
<h3>Support infix operators with precedence</h3>
Infix operators with multiple precedence levels may not be as elegant
as Lisp-style prefix notation, but most people are familiar with the former,
and it seems to make for somewhat more readable programs.
Thus Q2 supports it. This expression:
<pre>
b := x + 1 > 3 + y * 2
</pre>
means the same as:
<pre>
b := ((x + 1) > (3 + (y * 2)))
</pre>
Assignment (<code>set!</code>) uses the operator <code>:=</code>,
while numerical equality uses <code>==</code> (for now).
The plan is to use <code>=</code> for definitional equality
- and ultimately unification.
<p>
Infix operators appear to conflict with the goal <q>extensible syntax
with no reserved identifiers</q>, if we allow user-defined operator tokens,
operator tokens are just regular identifiers and which could be re-defined (as say a function),
and we allow forward references (at least of functions).
How can we tell that <code>*</code> is the built-in binary
operator, or if it gets redefined a function later in the script?
The solution is for the lexical parser (reader) to just return
a list of tokens. Splitting the token list according to
operator precedence is done later - just like macro-expansion is.
<h3>Use juxtaposition for function application</h3>
Haskell and ML use juxtaposition for function application,
rather than parentheses and commas.
For example to call the <code>atan</code> function with
two arguments <code>x</code> and <code>y</code>:
<pre>
atan x y
</pre>
<p>
This is compact, easy to type, and easy to read.
It is similar to the syntax of many command languages
including <code>sh</code>.
<h3>Naming a zero-argument function applies it</h3>
<p>
The complication is how do you distinguish a variable
reference from a call to a zero-argument function?
This can get tricky in a language with first-class functions,
since the value of a variable can be a function value.
(This is not an issue in Haskell or other pure lazy function language,
since there is no difference between applying a zero-argument function
and accessing a lazily-initialized variable.)
Some languages use a special syntax or operator for force function application
of zero-argument functions, but I think that is backwards, and
inconsistent with command languages.
For example, if <code>foo</code> is a function that can take
zero or one arguments, then
<pre>
foo1 # Called with zero arguments
foo1 10 # Called with one argument
</pre>
<p>
The exact rule for a distinguishing between a variable
reference and a zero-argument function application isn't decided yet.
For now it just depends on the run-time type of the identifier:
If the value of the variable is a function that can take zero arguments,
apply the function, otherwise return the value.
An alternative rule is to apply the function only if the identifier
is lexically bound to a function definition.
<h3>Flexible token format</h3>
<p>Expressions are constructed from lexical tokens.
A token can be (among other things) a name (an identifier),
a literal, or an operator.
Supporting an extensible syntax with infix operators suggests that an operator
should just be an identifier using non-alphanumeric characters.
Extensibility is also helped if programmers can add new kinds of literals.
So it seems like a good idea be be rather flexible and open-ended
in terms of what characters can comprise a token or an identifier.
Many languages only allow alphanumeric characters (with perhaps one or
two extra characters allowed), but
Lisp-family languages traditionally allows a wider class of characters,
include arithmetic symbols like <code>+</code>.
Q2 follows this tradition.
<p>
The downside is that more spaces are needed to separate adjacent tokens.
That seems a good trade-off - the spaces are probably a good idea
for readability, regardless. Furthermore, if we're using spaces
to separate function arguments, it seem reasonable to also require
spaces to separate infix operators from operands.
<h3>Use indentation for grouping</h3>
Some languages (including Haskell and Python) indicate
block structure based on the indentation of the lines of a program.
I think this makes sense - it allows for a cleaner layout with fewer
redundant delimiters (redundant because good programmers use
indentation anyway). It also avoids the common annoyance of noise
lines that only contain a closing delimiter.
<p>
However, it is more of a challenge to make use of indentation in a
Lisp-like language without a fixed syntax. There are some experiments
where essentially a pre-processor adds parentheses based on indentation,
but just using indentation just as syntactic <q>sugar</q>
for parentheses doesn't make for nice-looking programs.
<p>
The basic use of indentation is Q2 is simple.
For example:
<pre>
f1 a b c d
f2 e f
f3 g
h
f4 i j
</pre>
is equivalent to:
<pre>
f1 a b c d (f2 e f (f3 g) (h)) (f4 i j)
</pre>
I.e. there are implicit grouping parentheses around each line
<em>and</em> each following line that is indented more.
<p>
We'd like to add local variable and function definitions
inside an indented block, but first a digression.
<h3>Block expressions yield multiple values</h3>
<p>
Many languages provide blocks, consisting of zero or more
declarations, expressions, and other statements.
In an expression language, the value of the block
is usually the value of the final expression.
In Q2 instead the value of the block is the value of <em>all</em>
as a tuple or <q>multiple values</q>. For example:
<pre>
(10-7; 2+2; 4+1)
</pre>
The above is a block with 3 sub-expressions,
which evaluates to 3 values: 3, 4, and 5.
If this is a top-level expression then all 3 values are printed.
Note that the parentheses are just for grouping, so are optional
if this is the outermost expression.
<p>The semi-column is a special delimiter that cannot appear in identifiers,
but semantically it is a binary infix operator that <q>concatenates</q>
the values of the operands.
<p>
Note that multiple values are not the same as sequences or lists:
for one thing multiple values do not nest:
<pre>
((1;2); (3; 4))
</pre>
This evaluates to 4 values, not 2.
<p>
If we think of declaration, assignments, and other <q>statements</q>
as expressions that return zero results, then it all works out.
This example (which uses the old-style Scheme <code>define</code> for now)
evaluates to a vector with a single element valued 10:
<pre>
vector (define v 5; v + v)
</pre>
<p>Newlines are roughly equivalent to semi-colons in that they
concatenate multiple values. Here is an example of the
the top-level of a script, which is a block of 4 sub-expressions:
<pre>
define x 10
x + 2
define y (3 * x)
x + y
</pre>
This evaluates to 4 sub-results, where the two declarations yield
zero results, and the additions yield 12 and 40, for a total of 2 values.
<h3>Multiple values becomes multiple arguments</h3>
If an argument expression evaluates to multiple values,
these becomes multiple arguments in the function application:
<pre>
fun2 (3;4) (5;6)
</pre>
evaluates to the same result as:
<pre>
fun2 3 4 5 6
</pre>
<h3>Indentation as blocks</h3>Now let's slightly change a previous example to
include an inner local variable definition:
<pre>
f1 a b
f2 e f
define x (g + 10)
2 * x
f4 i j
</pre>
<p>
This works nicely if we slightly change the rewrite rules to
make use of blocks. Roughly, each newline becomes a semicolon.
The result is:
<pre>
f1 a b c d (f2 e f (define x (g + 10); 2 * x); f4 i j)
</pre>
Each <q>block</q> can evaluate to zero, one, or multiple values.
The inner block has two sub-expressions: The <code>define</code>
provides 0 values, and the other yields a single number.
Thus <code>f2</code> is called with 3 arguments: <code>e</code>,
<code>f</code>, and the value of <code>2 * x</code>.
The other block consist of the calls two <code>f2</code> and <code>f4</code>.
Assuming these each return a single value, as do <code>a</code> and <code>b</code>,
then <code>f1</code> is called with 4 parameters.
<h3>Works well with a REPL</h3>
While the read-eval-print-loop will need more tuning,
the current behavior is quite reasonable.
Typing in commands consisting of one or more space-separated tokens on a line
is convenient and familiar from shells.
(In fact building a command shell on top of Q2
is a tempting project.)
<p>
In an interactive REPL there is an issue of whether to read ahead
to check the indentation of the following line, or in general to see if
the current command is terminated.
Reading the following line before evaluating and printing
the current command of course makes for a confusing experience,
so the REPL uses the single-line mode of the parser: Reading the
following lines is avoided unless we're nested inside parentheses
or the current commands ends with a binary operator.
<h3>Scheme macro and special form compatibility</h3>
<p>
For now Q2 support most of the syntax of Scheme tokens,
and most Scheme builtins, including lexical forms.
This is not necessarily a long-term goal,
but it provides something we can use until Q2-specific replacements
are been designed and implemented.
<p>
For example here is a possible definition of the <code>foo1</code>
used above:
<pre>
define counter 0
define (foo1 #!optional (x 10)) (
counter := counter + 1
format #t "foo called x=~s counter=~s!~%~!" x counter)
</pre>
Hello world in Scheme for Androidhttp://per.bothner.com/blog/2010/AndroidHelloScheme/2011-04-10T18:55:46Z2010-10-18T21:27:26Z
<p>(This is an update of the <a href="http://per.bothner.com/blog/2009/AndroidHelloScheme/">2009 version</a>,
which was an update of the <a href="http://per.bothner.com/blog/2008/AndroidHelloScheme/">original 2008 version</a>.)</p>
<p>Google's phone operating system
<a href="http://code.google.com/android/">"Android"</a> is based on
a custom Java virtual machine on top of GNU/Linux.
So it occurred to me: How difficult would it be to
get a <a href="http://www.gnu.org/software/kawa/">Kawa</a>
application running on Android?
Not that difficult, it turns out.
<p>
Here is "Hello world" written in Kawa Scheme:
<pre>
(require 'android-defs)
(activity hello
(on-create-view
(android.widget.TextView (this)
text: "Hello, Android from Kawa Scheme!")))
</pre>
(A more interesting
<a href="http://androidscheme.blogspot.com/2010/10/text-to-speech-app.html">text-to-speech example app</a> is on
<a href="http://androidscheme.blogspot.com/">Santosh Rajan's Android-Scheme blog</a>.)
<p>
The following instructions have been tested on GNU/Linux,
specifically Fedora 13,
but if you've managed to build Android applications under (say) Windows,
you should be able to appropriately modify these instructions.
<h3>Getting and building Kawa and Android</h3>
<p>First <a href="http://code.google.com/android/download.html">download the Android SDK</a>. Unzip in a suitable location,
which we'll refer to as <code>ANDROID_HOME</code>.
<pre>
$ export ANDROID_HOME=/path/to/android-sdk-linux_86
$ PATH=$ANDROID_HOME/tools:$PATH
</pre>
<p>
Next you have to get the appropriate platform SDK:
<pre>
$ android update sdk
</pre>
Select <code>SDK Platform Android 2.2</code> or whatever
and click <code>Install</code>.
<p>
You need to select an Android <q>platform</q>.
Platform 8 corresponds to Android 2.2 (Froyo).
<pre>
ANDROID_PLATFORM=android-8
</pre>
<p>
You need to <a href="http://www.gnu.org/software/kawa/Getting-Kawa.html">get the Kawa source code</a> (version 1.11 or later).
<p>
Set <code>JAVA_HOME</code> to where your JDK tree is, for example:
<pre>
$ export JAVA_HOME=/opt/jdk/1.6.0
</pre>
<p>
If using Ant (as is recommended on Windows):
<pre>
$ ant -Denable-android=true
</pre>
<p>
Alternatively, you can use <code>configure</code> and <code>make</code>:
<pre>
$ KAWA_DIR=path_to_Kawa_sources
$ cd $KAWA_DIR
$ ./configure --with-android=$ANDROID_HOME/platforms/$ANDROID_PLATFORM/android.jar --disable-xquery --disable-jemacs
$ make
</pre>
<h3>Creating the application</h3>
<p>Next, we need to create a project or <q>activity</q>,
in the target directory <code>KawaHello</code>,
with the main activity being a class named <code>hello</code>
in a package <code>kawa.android</code>:
<pre>
$ android create project --target $ANDROID_PLATFORM --name KawaHello --activity hello --path ./KawaHello --package kawa.android
</pre>
<p>Replace the skeleton <code>hello.java</code> by the
Scheme code at the top of this note:
<pre>
$ cd KawaHello
$ HELLO_APP_DIR=`pwd`
$ cd $HELLO_APP_DIR/src/kawa/android/
$ rm hello.java
$ emacs hello.scm
</pre>
<p>
We need to copy/link the Kawa jar file so the Android SDK can find it:
<pre>
$ cd $HELLO_APP_DIR
$ ln -s $KAWA_DIR/kawa-1.10.jar libs/kawa.jar
</pre>
<p>Optionally, you can use <code>kawart-1.10.jar</code>, which is
slightly smaller, but does not support <code>eval</code>,
and does not get built by the Ant build:
<pre>
$ ln -s $KAWA_DIR/kawart-1.10.jar libs/kawa.jar
</pre>
<p>
We also need to modify the Ant <code>build.xml</code>
so it knows how to compile Scheme code:
<pre>
$ patch < <a href="http://per.bothner.com/blog/2010/AndroidHelloScheme/build-xml-patch.txt">build-xml-patch.txt</a>
</pre>
<p>
Finally, we can build our application:
<pre>
$ ant debug
</pre>
<h3>Running the application on the Android emulator</h3>
<p>First you need to
<a href="http://developer.android.com/guide/developing/other-ide.html#AVD">
create an Android Virtual Device (<code>avd</code>)</a>.
Start:
<pre>
android
</pre>
then select <code>Virtual Devices</code>, then click <code>New...</code>.
Pick a <code>Name</code> (we use <code>avd8</code> in the following),
a <code>Target</code> (to match <code>$ANDROID_PLATFORM</code>),
and optionally change the other properties, before clicking <code>Create AVD</code>.
<p>
Start up the Android emulator:
<pre>
$ emulator -avd avd8 &
</pre>
<p>
Wait until Android has finished booting (you will see the Android home screen),
click the menu and home buttons.
Now install our new application:
<pre>
adb install bin/KawaHello-debug.apk
</pre>
<p>The new <code>hello</code> application should show up.
Click it, and you should see something like:
<img src="http://per.bothner.com/blog/2010/AndroidHelloScheme/HelloKawa2.png" alt="HelloKawa2.png" />
<h3>Running the application on your phone</h3>
<p>If the emulator is running, kill it:</p>
<pre>
$ kill %emulator
</pre>
<p>On the phone, enable <code>USB debugging</code>. (This is settable from the
<code>Settings</code> application, under <code>Applications</code> / <code>Development</code>.)
<p>
Connect the phone to your computer with the USB cable.
Verify that the phone is accessible to <code>adb</code>:</p>
<pre>
$ adb devices
List of devices attached
0A3A560F0C015024 device
</pre>
<p>If you don't see a device listed, it may be permission
problem. You can figure out which device corresponds to the phone
by doing:
<pre>
$ ls -l /dev/bus/usb/*
/dev/bus/usb/001:
total 0
...
crw-rw-rw- 1 root wheel 189, 5 2010-10-18 16:52 006
...
</pre>
<p>The timestamp corresponds to when you connected the phone.
Make it readable:</p>
<pre>
$ sudo chmod a+w /dev/bus/usb/001/006
</pre>
<p>
Obviously if you spend time developing for an Androd phone you'll
want to automate this process;
<a href="http://sites.google.com/site/siteofhx/Home/android/drivers/udc">this link</a> or
<a href="http://groups.google.com/group/android-developers/browse_thread/thread/9d37e184f92d19f3">this link</a> may be helpful.
<p>
Anyway, once <code>adb</code> can talk to the phone, you
install in the same way as before:
<pre>
adb install bin/KawaHello-debug.apk
</pre>
<h3>Some debugging notes</h3>
<p>
You will find a copy of the SDK documentation in
<code>$ANDROID_HOME/docs/index.html</code>.
<p>If the emulator complains that your application
<q>has stopped unexpectedly</q>, do:</p>
<pre>
$ adb logcat
</pre>
<p>This shows log messages, stack traces, output from the <code>Log.i</code>
logging method, and other useful information.
(You can alternatively start <code>ddms</code>
(Dalvik Debug Monitor Service), click on the <code>kawa.android</code>
line in the top-left sub-window to select it, then from the
<code>Device</code> menu select <code>Run logcat...</code>.)
<p>To uninstall your application, do:
<pre>
$ adb uninstall kawa.android
</pre>