.c 11:06am  Thursday, 7 October 1982 -*- Bolio -*-

.chapter "Syntax"
.setq syntax-section-page section-page

.section "What the Reader Tolerates"

I will defer detailed discussion of reader input and printed
representation to the forthcoming lan{common lisp} manual
ref{CommonLisp}.  The lan{lispm} manual ref{LMMan} also
contains a good discussion of this.  What will be presented here is
basically a summary of what the current lan{nil} reader accepts,
utilizing lan{common lisp} syntax.

Basically, the lan{lisp} reader reads characters from a stream and
forms tokens out of them.  Certain characters cause additional actions
to take place;  for instance, the 3(* character will cause
multiple (but possibly zero) expressions up to a matching 3)* to
be formed into a list.  Some characters are significant only when they
are the first non-whitespace character;  the 3#* dispatch macro
character behaves like this:  3#o403* is the integer 3259*
(3#o* meaning "read in octal"), but 3foo#o403* is the symbol
whose name is the string 7"FOO#O403"*.  Aside from these,
the basic rule is that if a number can be formed from the characters
of the token, it is;  otherwise, it is a symbol.  The sole exception
is that the period character (3.*) is taken as a term_i{cons
dot}.  If a character is preceded by a backslash (3\*), then
2all* special significance is removed from it, including case
translation, and it is treated as a token constituent.

Thus:
.table 7 250 1000
.item foobar
the atomic symbol 7FOOBAR*
.item foo\bar
The atomic symbol whose print name is 7FOObAR*
.item 259.259
A floating point number.  (Currently this is a double-float.  But see
the later discussion on floating-point syntax,
(floating-point-syntax-section-page).)
.item 259\.259
The atomic symbol with print name 7259.259*.
.item |FooBar|
Vertical bars read a symbol with all characters (except for vertical
bar and backslash) interpreted as constituent characters.  So, this is
the symbol whose print name is 7FooBar*.  Backslash may be used to
include vertical bars and backslashes in the symbol.
.item |Foo\|Bar\\|
Similarly, the symbol whose print name is 7Foo|Bar\*.
.item 259
The decimal integer 3259*.
.item 259.
The decimal integer 3259*.  A trailing decimal point explicitly
forces decimal notation, not floating-point.
.item -259
The negative decimal integer 3-259*.
.item +259
The decimal integer 3259*.
.item 25\9
The symbol 7259*.
.item \259
The symbol 7259*.
.item |259|
The symbol 7259*.
.item 1.0d-5
One times ten to the minus fifth power, as a double-float.  See
(floating-point-syntax-section-page).
.item :foo
A colon in a token uses the characters on the left to name the
2package* the following symbol is to be read into.  The "null"
package name means the package into which keywords are read.
.item si:foo
The symbol 7FOO*, read into the package named 7SI* (the
system-internals package).
.item si:|Foo\|Bar\\|
The symbol 7Foo|Bar\*, read into the 7SI* package.
.item foo#o403
The symbol 7FOO#O403*.
.end_table

If the token consists entirely of 3.* characters (and none
have been slashified), then it is illegal unless there is exactly one;
that is a term_i{cons dot}, which is only legal in list/cons
formation.  Thus, 3.foo.* is the symbol whose print name is
7.FOO.*, but 3...* is an error.

The primitive syntax for a cons is
.lisp
(2car* . 2cdr*)
.end_lisp
In this notation, a list of items 3a*, 3b*, and 3c* would
be written as
.lisp
(a . (b . (c . nil)))
.end_lisp
term_i{List syntax} allows us to "elide" a cons dot with a following
cons;  the dot is eliminated, as are the parentheses of the following
cons:
.lisp
(a b . (c . nil))
(a b c . nil)
.end_lisp
and finally, because 3nil* is the same as 3()*,
.lisp
(a b c)
.end_lisp

The following characters terminate token formation, and do something
special when encountered:
.table 3 250 250
.item (
Starts a list or cons, as described above.
.item )
Terminates a list or cons, or some other construct which "matches"
with parentheses, such as 3#(*.
.item |
Vertical bar terminates token formation.
.item "
String syntax.  The characters up to the matchine 3"* form the
string;  3"* and 3\* may be included by preceding them
with 3\*.
.item '
Reads the following expression, and "wraps" it with the function
3quote*.  Thus, 3'foo* reads as 3(quote foo)*.
.item `
"Backquote".  This is used for constructing expressions.  Backquote is
described later ((backquote-section-page)), and also in
ref{Maclisp-Extensions}.
.item ,
Comma is used for performing substitution within backquoted
expressions (q.v.).
.end_table

The 3#* character is a term_i{dispatch macro character}.  It
reads (optional) digits as a numeric decimal argument, and then
dispatches off of the following character.  The following are defined:
.table 3 250 250
.item #'2expression*
Wraps 2expression* with 3function*, similar to 3'*.
Thus, 3#'car* reads as 3(function car)*.
.item #(2x1* 2x2* ... 2xn*)
Reads as a simple general vector 2n* elements long (2n* may be
zero), with those elements.
.item #*2bits*
Reads in as a simple bit vector whose elements are 2bits* (the
digits 0 or 1).  Thus, 3#*100* is a simple-bit-vector of length 3;
its element 0 is 31*, and its elements numbered 1 and 2 are both
30*.
.item #B2rational*
Reads 2rational*, the syntax for a rational number (which,
remember, may be an integer) in binary (base 2).
.item #O2rational*
Reads 2rational* in octal.
.item #X2rational*
Reads 2rational* in hexidecimal.
.item #2radix*R2rational*
Reads 2rational* in radix 2radix*.
.item #2font*\2character-or-character-name*
Read an object of type 3character*.  The 3\* may be followed
by either a single character (and then a delimiter), or by a token
(read as described above), which is interpreted as the name of a
character.  The returned object will have a font attribute of
2font*, which defaults to 30*.
3#\a* is lowercase 3a*, and 3#\|* is the character object
for vertical-bar.  A character name may have the names of character
bits prepended to it.  For instance, 3#\hyper-space* is the
character for space with the hyper bit.  If the long form is used, the
final character may need to be slashified to be interpreted correctly.
For instance, 3#\control-a* is uppercase 3a* with the control
bit, 3#\control-\a* is lowercase 3a* with the control bit, and
3#\control-(* is an error (the left-paren delimits) which should
have been typed as 3#\control-\(*.
.item #C(2real* 2imag*)
A complex number with real part of 2real* and imaginary part of
2imag*.
.item #2n*A2contents*
Reads in as an array of rank 2n*, with contents 2contents*
(see 3make-array*, (make-array-fun)).  This does not work yet.
.item #S(2name* 2kwd1* 2val1* ... 2kwdn* 2valn*)
General structure syntax, for structures defined by 3defstruct*.
2name* is the name of the 3defstruct*-defined structure.  This
does not work yet.
.item #+2conditional-expression* 2expression-to-conditionalize*
Read-time conditionalization.  See the Maclisp Extensions Manual.
.item #-2conditional-expression* 2expression-to-conditionalize*
Read-time conditionalization.  See the Maclisp Extensions Manual.
.item #.2expression*
Reads in as the 2evaluation* of 2expression*.
.item #,2expression*
Load-time evaluation.  If the expression is being read normally into
lan{nil}, this behaves like 3#.*.  However, if it is in a file
being compiled, the compiler will arrange to have 2expression*
evaluated at load (i.e., 3vasload*) time, when the containing
expression is being constructed.  This does not work yet in
lan{nil}.
.end_table

.subsection "Backquote"

The term_i{backquote} facility is used for constructing list
structure from a template.  Typically, most of the template is
constant, and the most common use is when what is being created is
lan{lisp} code, inside of macro definitions.  In the simplest case,
the backquote character (3`*) is used instead of quote
(3'*), and, where substitution inside the template is desired, a
form to be evaluated is preceded by a comma character.  For instance,
.lisp
(defmacro push (item place)
  `(setf ,place (cons ,item ,place)))
.end_lisp
This definition reads in as code which is functionally equivalent to
.lisp
(defmacro push (item place)
  (list 'setf place (list 'cons item place)))
.end_lisp
Simple substitution is often not enough.  If the comma in a backquoted
expression is immediately followed by an atsign character (3@*),
then, instead of simple substitution, the item being substituted is
"spliced" into the containing list:
.lisp
`(foo ,@bar mumble)
.end_lisp
expands into code functionally like
.lisp
(cons 'foo (append bar '(mumble)))
.end_lisp
For instance, the macro
.lisp
(defmacro always-returning-nil (&body body)
  `(progn ,@body nil))
.end_lisp
produces expansions
.lisp
(always-returning-nil)
  ==>  (progn nil)
(always-returning-nil (do-this))
  ==>  (progn (do-this) nil)
(always-returning-nil (do-this) (do-that))
  ==>  (progn (do-this) (do-that) nil)
.end_lisp
The splicing is non-destructive on the expression being substituted,
as if the splicing happens by use of 3append*.

In lan{nil} (but not lan{common lisp}), a comma immediately
followed by a dot instead of an atsign is treated similarly,
2except* that lan{nil} is free to use 2destructive*
operations for the splicing;  that is, the splicing is performed as if
by 3nconc* ((nconc-fun)) rather than 3append*.

Backquoted expressions may be nested.  They should be assumed to be
expanded from the inside out.  Consider the following macro
definition:
.lisp
(defmacro defhack (name fn &aux (place-var (gentemp)))
  `(defmacro ,name (,place-var)
     `(setf ,,place-var (,',fn ,,place-var))))
.end_lisp
In this example, the expansion of the outer backquote expression
causes the values of 3name*, 3fn*, and3place-var* to be
substituted in;  this essentially means that we get (using italics now
to show the substitutions)
.lisp
(defmacro 2name* (2place-var*)
  `(setf ,2place-var* (,'2fn* ,2place-var*)))
.end_lisp
Now, ",'2fn*" means that we substitute in the value of
"'2fn*", which is of course just 2fn*, which is what we want;
on the other hand, we 2do* want the value of 2place-var* to be
substituted in here, so it does not need that quote.

To see another way that this works, one can manually pseudo-expand the
2inner* backquote in the original 3defhack* definition
.lisp
(defmacro defhack (name fn &aux (place-var (gentemp)))
  `(defmacro ,name (,place-var)
     (list 'setf place-var (list ',fn place-var))))
.end_lisp
which indeed produces what we want.  Generally, then, one uses ",,"
when one wants the substituted form to itself be evaluated and
substituted with the expansion of the inner backquote expression, and
",'," when one wants to substitute in something which is "constant"
with respect to the inner backquote expansion.

As one final extension to backquote, lan{nil} allows its use with
general vector syntax.  This will probably be supported forever and
ever, if for no other reason than the lan{nil}
compiler/code-generator emits instructions as vectors and constructs
them that way.
.lisp
`#(foo ,(+ 3 7) bar)  =>  #(foo 10 bar)
.end_lisp
"Splicing" syntax (",@" and ",.") is not supported here.  Neither
lan{nil} nor lan{common lisp} support backquote use with array
syntax (3#A*) or structure syntax (3#S*).


.section "The Lisp Reader"

.subsection "Introduction"

The lan{nil} reader was designed to be
incrementally extensible and to support the implementation of other
languages in lan{nil}. It also addresses some efficiency issues to
to take advantage of, but to also hide, low-level considerations in disk 
and terminal I/O.

lan{Common Lisp} and lan{maclisp} compatible syntax extension functions 
are provided, along with readtables for the syntax of lan{nil},
lan{common lisp},lan{maclisp}, and lan{cgol}. The definition of these
is in the file 7[NIL.LISPIO]RTBSETUP.LSP*.

Note that the default readtable has been set to one conforming to the
lan{common Lisp} specification. The only significant difference between
this and what lan{maclisp} and lan{lispm} users have been using is
that the syntax escape character is backslash, instead of slash.
Some lan{maclisp} programs we have seen are also using what is now
the package prefix character ":" as a regular symbol-constituent character.
If any of this presents a code porting problem, then set the readtable to
one of the compatible readtables documented later, or specify a readtable
in the modeline of the source files in question. For example:

.lisp
;;-*-Mode:Lisp;Readtable:ML-*-
;; This code uses ":" and "/" as in maclisp.

;;-*-Mode:Lisp;Readtable:LM-*-
;; This code reads using the old lispmachine syntax.

C -*-Mode:Fortran;Readtable:Fortran-*-
C This would work if one defined a readtable for Fortran.

% -*-Mode:Lisp;Readtable:Cgol-*-
  This is lisp code in cgol syntax. Yow! %
define fib(x); if x<2 then 1 else fib(x-1)+fib(x-2)$

;; To get a maclisp readtable.
(setq readtable (si:lookup-readtable "ML"))
;; to get a lispmachine readtable.
(setq readtable (si:lookup-readtable "LM"))
.end_lisp

.subsection "Reader Extensions"

For exotic or extensive reader extensions, see the documentation
on the readtable, and how the various language readtables are set up,
in 7[NIL.LISPIO]RTBSETUP.LSP* and in 7[NIL.LISPIO]PARSER.LSP*.

.defun setsyntax character type value
This is a lan{maclisp} compatibility feature, altering the syntax of the
2character* in the current readtable.  2type* may be
3macro*, 3splicing*, or 3single*.  If it is 3macro* or
3splicing*, then 2value* is a function of no arguments which
is invoked when the 2character* is read.
.end_defun

.defun setsyntax-sharp-macro character type function &optional readtable
This is also a lan{maclisp} compatibility feature.  2type* can be
3macro*, 3peek-macro*, 3splicing*, or 3peek-splicing*.
2function* gets called with one argument, which is either null or
the number between the 3#* and the character.
.end_defun

.subsection "Readtable"

.defvar readtable
The value of this variable is a datastructure that controls the 
behavior of the function fun{read}.
.end_defvar
.defun si:lookup-readtable name
Returns the readtable corresponding to the syntax named by the string
2name*.
.end_defun
.defun si:enter-readtable name a-readtable
Enters 2a-readtable*  giving the syntax for 2name*.
2name* may then appear as a readtable specification in the mode-line of
a source-file.
.end_defun
.defun create-readtable
Returns a naked readtable, with syntax for reading whitepace-delimited symbols.
.end_defun
.defun si:add-number-syntax
Adds syntax for parsing numbers to the current readtable.
.end_defun
.defun si:add-list-syntax &optional (open #\() (close #\))
Adds syntax for parsing lists to the current readtable.
.end_defun
.defun si:add-package-syntax &optional (char #\:)
Adds syntax for specifing packages to the current readtable.
.end_defun
.defun si:add-escape-char-syntax char
Makes 2char* the syntax-escape-character in the current readtable.
.end_defun
.defun si:add-prefix-op-macro char operator
Makes 2char* a readmacro that returns a list of 2operator* and
the next thing read. For example, 
.lisp
(si:add-prefix-op-macro #/' 'quote)
.end_lisp
.end_defun

.subsection "Alternative Syntax"
The lan{cgol} syntax ref{Cgol} is available by loading the file 
7NIL$DISK:[NIL.LISPIO]CGOL.LOD*.  Further documentation is in the
file 7NIL$DISK:[NIL.MANUAL]CGOL.TXT*.  The implementors do not
recommend the extensive use of lan{cgol} or any lan{algol}-like
syntax for lan{lisp} programming, especially in environments where
program readability and editability are important long range
considerations.  However, some feel that syntactic variety taken in
moderation is good for the soul.

.defun cgolread
Reads a lan{cgol} syntax expression.
.end_Defun
.defun cgolprint expression
Prints an s-expression lisp program in the lan{cgol} syntax.
.end_defun

Another parser, for a language with syntax compatible with the
symbolic algebra system lan{macsyma} ref{Macsyma}, is available
by loading the file 7NIL$DISK:[NIL.VAS]PARSER*, which sets up a readtable
named 3infix*. The readmacro character "#$" has been set up to invoke
this parser in the "NIL" readtable. One could then write the following:
.lisp
(defun f (v a b x)
  #$(v[0,0]:COS((A-B)*X)/(2-2*(A-B)^2)+COS(V[1,1]*X),
     v[0,1]:COS((A+B)*X)/(2-2*(A+B)^2)-V[1,0]*V[0,0],
     v[1,0]:V[1,1]*V[0,0],
     v[1,1]:V[0,1]*V[1,0])$)
.end_lisp

