.c 	Wednesday, December 5, 1984   -*- bolio -*-


.chapter "Characters"
.setq character-chapter-number chapter-number

In lan{nil}, 2characters* are represented as a separate data
type.  This provides multiple benefits;  among them,  the object
maintains some semantic identity when it appears in code (it is
obvious that it is a "character"), and since it does maintain its
identity as a character, the read/print/read "fixed-point" is capable
of functioning across differing lan{lisp} implementations that
internally utilize different character sets (e.g., lan{ascii} vs.
lan{ebcdic}).

Characters in lan{nil} have three different attributes:  their
term{code},
'cindex character code
their term{bits},
'cindex character bits
and their term{font}.
'cindex character font

The code defines the basic ("root") character;  for instance, "A", "$",
and the formfeed or page-break character are all characters defined just
by their 2code*.

The bits are used as
modifiers.  Typically, an input processor (such as the editor, or even
the prescan for the toplevel Lisp read-eval-print loop) will treat a
character without any bits as "ordinary" and assume it is part of the
text being typed in, but treat a character with some bits as being a
command.  Four of the special bits are named:  they are 3control*,
3meta*, 3super*, and 3hyper*.  These all have one-character
abbreviations ("c", "m", "s", and "h").  When the character with
2code* for "A" and 2bits* modifiers of control and meta is being
referred to in text, it might be written in short form as "c-m-A" or
written out as "Control-Meta-A".

The font of the character defines how certain things about the
character are defined;  for instance, equating characters of different
fonts (3char-equal*), whether a character is upper or lower case,
alphabetic, or a digit (3upper-case-p* etc.), and how to do case
conversion (3char-upcase*).  While lan{nil} does not yet really
define "fonts", there is a mechanism for how such definitions can
interface to the low-level character manipulation primitives which
utilize such things ((primitive-font-section-page)).

lan{Nil} character objects are immediate-pointer structures;  they
require no storage.  Most of the routines which construct, dissect,
and compare characters are open-coded by the compiler.

The lan{nil} character set has not yet been cleaned up with respect
to the confusion between the lan{ascii} control characters and the
characters it uses with the 3control* bit.  See (nil-character-set-section-page).

.defconstant char-code-limit
.defconstant1 char-font-limit
.defconstant1 char-bits-limit
These constants have as their values the upper exclusive limits on
those attributes of characters.  The values should not be changed.
It happens that, in the lan{vax} implementation, all three are
3256* so that each quantity will fit into an 8-bit byte.
.end_defconstant


.section "Predicates on Characters"

.defun standard-char-p character
This returns 3t* if 2character* is one of the "standard"
lan{ascii} characters.  These are all the ordinary graphic
characters (alphanumerics and punctuation characters), plus
2Space* and 2Return*.  Only characters with 0 font and bits
attributes can be standard.
.end_defun

.defun graphic-char-p character
Returns 3t* if 2character* is a graphic (printing)
character;  that is, it has a single glyph representation.  No
character with non-zero bits attribute is graphic.  Whether or not a
character is graphic depends on its font.
.end_defun

.defun alpha-char-p character
.defun1 upper-case-p character
.defun1 lower-case-p character
.defun1 both-case-p character
.defun1 alphanumericp character
Predicates on character objects.  All are 3nil* for characters
with any 2bits*;  otherwise, they depend on the font.
.end_defun

.defun char= character &rest more-characters
.defun1 char< character &rest more-characters
.defun1 char<= character &rest more-characters
.defun1 char> character &rest more-characters
.defun1 char>= character &rest more-characters
.defun1 char/= character &rest more-characters
These routines collate or compare characters.  The comparison 2is*
dependent on the bits, font, and case of the characters.  Each routine
returns 3t* if all pairwise combinations of its arguments satisfy
the appropriate predicate, 3nil* otherwise;  if a single argument
is given, the result is 3t*.

3char=* is the primitive function for telling if the characters
are "the same character"--that is, if their code, bits, and font are
all the same.  It is what is used by 3eql* and 3equal* for
comparing characters.  3char/=* returns 3t* if 2none* of
the pairs of characters are 3char=*.

3char<*, 3char<=*, 3char>=*, and 3char>* collate the
characters according to some unspecified collating sequence.  What is
guaranteed is that the upper-case alphabetics, the lower-case
alphabetics, and the digits will not be interspersed within the
collating sequence, and, within each of these sets, they follow the
obvious collating order.
.end_defun

.defun char-equal character &rest more-characters
.defun1 char-lessp character &rest more-characters
.defun1 char-greaterp character &rest more-characters
.defun1 char-not-lessp character &rest more-characters
.defun1 char-not-greaterp character &rest more-characters
These routines compare characters independently of font, bits, and
case.  This does 2not* mean that only the 2code* portion of
the characters is compared, but rather a font-dependent
canonicalization is performed on the characters before they are
compared.  Thus, for two characters in different fonts, it is possible
for them to not be 3char-equal* even if they have the same
2code* (one might be a "greek" character and the other "roman"),
and conversely it is possible for them to be the same even with
different codes (one might be upper-case and the other lower-case).
.end_defun


.section "Character Construction and Selection"

.defun character frobozz
Coerces 2frobozz* to a 3character*.  It may be a
character, a fixnum, in which case 3char-int* is applied, or a
string or symbol of length 1, in which case that single character is
returned.  This is what is used by the 3coerce* function
((coerce-fun)).
.end_defun

.defun char-code character
.defun1 char-bits character
.defun1 char-font character
These three functions extract those attributes
(as fixnums).  All are efficiently open-coded by the compiler, and
accept only character objects.
.end_defun


.defun code-char code &optional (bits 0) (font 0)
Creates a character with code, bits, and font of 2code*,
2bits*, and 2font*, unless that is not possible in the
implementation, in which case 3nil* is returned.  In other words,
it is an error for 2code*, 2bits*, or 2font* to not be
non-negative integers; however, they need not be less than
3char-code-limit*, 3char-bits-limit*, and
3char-font-limit* respectively.
.end_defun

.defun make-char char &optional (bits 0) (font 0)
Creates a character with code of the the code of 2char*, and with
bits and font of 2bits* and 2font*, unless that is not
possible in the implementation, in which case 3nil* is returned.
3make-char* could have been defined as
.lisp
(defun make-char (char &optional (bits 0) (font 0))
  (code-char (char-code char) bits font))
.end_lisp
.end_defun


.section "Character Conversions"

.defun char-upcase char
.defun1 char-downcase char
Upper- or lower-casify 2char*, preserving the font and bits
attributes of it.  Note in particular that these functions just return
2char* if it has a non-zero 2bits* attribute, because such a
character is not alphabetic and hence not subject to having its case
changed.
.end_defun

.defun char-int char
Returns a non-negative integer (in lan{nil}, this will be a fixnum)
encoding of the character 2char*.  If the bits and font of
2char* are 0, this is the same as 3char-code*.  This is
useful for hashing, and certain character-fixnum conversions such as
those needed for the lan{maclisp} 3tyi* function are
defined in terms of 3char-int*.
.end_defun

.defun int-char integer
If there is some character 2c* for which 3(char-int 2c*)*
equals 2integer*, that character is returned;  otherwise,
3nil* is returned.
.end_defun

.defun char-name char
Returns the name of the character 2char*, if it has one.
Supposedly, all characters which have zero font and bits attributes
and which are non-graphic (see 3graphic-char-p*) have names.

In lan{nil}, the name of a character is by convention a symbol in
the keyword package.  This is incompatible with the lan{common lisp}
definition (which defines the name as being a string), and will be
fixed eventually.
.end_defun

.defun name-char sym
The argument 2sym* must be a symbol.  If the symbol is the name of
a character object, that object is returned;  otherwise 3nil* is
returned.

In lan{nil}, character name symbols are symbols in the keyword
package.  This is incompatible with the lan{common lisp} definition,
which states that the argument to 3name-char* may be anything
coercible to a string using the 3string* function, and that name
comparison is performed using 3string-equal*.  This will be
fixed eventually.
.end_defun

.defun digit-char-p digit &optional (radix 10)
3digit-char-p* is a semi-predicate.  If 2digit*, which must be
a character, is a digit in radix 2radix*, then the weight of that
digit is returned, otherwise 3nil* is returned.  By definition, a
character with non-zero 2bits* is not a digit, so for that
3digit-char-p* will always return 3nil*.
.end_defun

.defun digit-char weight &optional (radix 10) (bits 0) (font 0)
If it is possible to construct a character with the given bits and
font which has the weight 2weight* in radix 2radix*, then such
a character is returned, otherwise 3nil* is returned.
2weight* must be a valid weight for the radix (a non-negative
integer less than 2radix*).  Note the similarity to
3make-char* ((make-char-fun)) also.
.lisp
(digit-char 5)      => #\5
(digit-char 10)	    => nil
(digit-char 10 25)  =>  #\A
.end_lisp
Note that if 3digit-char* succeeds in constructing the character,
and 2bits* is not 30*, then the returned character is by
definition not a digit--3digit-char-p* would return 3nil* for
it.
.end_defun


.section "Low-Level Interfaces"
.setq low-level-character-section-page section-page

.defun %int-char fixnum
Non-check version of 3int-char*.  It is an error for 2fixnum*
to not be the integer encoding of a valid character object, as would
be returned by 3char-int*.
.end_defun

The following four routines define digitness in the lan{nil}
character set, at a low level.  These routines do 2not* depend on
the 2font* of the character (if they take a character as an
input);  rather, they define the basic conversions for the "standard"
lan{nil} character set (lan{nil}'s interpretation of
lan{ascii}).

.defun %valid-digit-radix-p radix
This defines the valid range of radices which
3%digit-char-in-radix* operates on.  The radix must be a fixnum.
.end_defun

.defun %digit-char-in-radix-p char radix
Primitive predicate for testing whether the character object
2char* is a digit character in the fixnum radix 2radix* (which
must be a valid numeric radix).
.end_defun

.defun %digit-char-to-weight char
For any char which satisfies 3%digit-char-in-radix-p*, this will
return the weight of that digit.
.end_defun

.defun %digit-weight-to-char weight
This inverts 3%digit-char-to-weight*.
.end_defun


The following two routines perform low-level case mapping for the
lan{nil} character set.

.defun %char-upcase-code code
.defun1 %char-downcase-code code
These routines perform low-level case mapping for the lan{nil}
character set.  2code* must be a valid character code;  the
returned value is a character with 30* bits and font attributes.

For example, if lan{nil} did not provide low-level support for
multiple fonts,
.lisp
(char-equal 2c1* 2c2*)
.end_lisp
would be the same as
.lisp
(char= (%char-upcase-code (char-code 2c1*))
       (%char-upcase-code (char-code 2c2*)))
.end_lisp
The actual definition of 3char-equal* is somewhat different in
that the translation and canonicalization of the characters depends on
their font.
.end_defun


.section "The NIL Character Set"
.setq nil-character-set-section-page section-page
.cindex character set
.cindex extended character set
.cindex ascii

As can be seen, the format of character objects in lan{nil} provides
for a basic eight-bit 2root* character (defined by the
2code*), which can then have both 2bits* and 2font*
attributes added to it.  However, the lan{i/o} devices lan{nil}
must deal with only handle (at most) eight-bit characters--often only
seven.

In principle, lan{nil} will utilize an eight-bit character set;
half of these will be 2graphic* characters (the normal
lan{ascii} graphics, plus special symbols), and the other half reserved,
format effectors (such as linefeed, backspace), and special
commands for things like the editor and debugger (2Abort*,
2Resume*, 2Clear-Screen*, that kind of thing).  All of these
characters will then be able to have 2bits* and 2font*
attributes added.

To deal with this on (for instance) an ordinary seven-bit lan{ascii}
terminal, lan{nil} will have to do three things in the future.
These are not done now, but are noted because they have bearing on the
representation of character objects and input from devices.
.table 2 250 250
.item 1
Turn ordinary lan{ascii} control characters into the lan{nil}
version of the control character.  For instance, the lan{ascii}
character with code of 31*, which is what you get when typing
2Control-A* on a standard lan{ascii} keyboard, will turn into
the character 2A* (uppercase A) with the 2Control* bit set.
.item 2
Provide "prefix" character-level commands in various places in order
that other characters with bits may be entered (this is what is done
in the editor).  For instance, in the current input processor used by
the reader, the character 2Control-\* is "prefix meta"--it
"reads" another character and returns that with the meta bit added.
.item 3
Provide some quoting or escape convention for inputting the extended
graphic characters, since the codes for them are now normally being
interpreted as characters with the 2Control* bit set.
.end_table
Secondarily, there will probably also be some translation of the
actual codes involved, but that is irrelevant unless one is looking at
the actual codes used in characters (which one generally should not).
For instance, of the 256 "normal" characters, the low 128 would be
graphics, and the high 128 the others.  Some sort of symmetry would be
maintained by having the lan{ascii} format effectors and
2Rubout* have their lan{ascii} values plus 3128*.  The
others would be new characters to be used as various sorts of
commands, but mostly left reserved for expansion.  This is, in fact,
approximately how the lan{lispm} character set is defined.

Unfortunately, none of this is done right now.  When the character
2Control-A* is typed on an lan{ascii} terminal, it is read in as
the character whose 2code* is 31*, not as what is actually the
character 2Control-A*.  The editor, for instance, does some of
the abovementioned transmutations on its input, and any prefixing
commands for adding bits supplied by other input processors would be
modeled after those used in the editor.

The algorithm which may be used to translate lan{ascii} into what is
2currently* used in lan{nil} is this.  Given an eight-bit
character, if the 2code* has the high bit set (it is greater than
3127* decimal), then subtract out that bit, and remember to add
the 2Meta* bit to the character which will be eventually obtained.
(This bit is what would be set by a terminal with a 2Meta* key;
such capability normally needs to be enabled by something like
.lisp
$ set term/eightbit
.end_lisp
to lan{dcl}.)  Now we have a seven-bit character.  If the seven-bit
code is less than 332*, and if it is 2not* one of the codes
for 2Backspace*, 3Tab*, 2Linefeed*, or 2Return*
(38*, 39*, 310*, and 313* respectively), then add
364* to it, and set the 2Control* bit.  Adding 364* forms
the corresponding uppercase-alphabetic or punctuation character.
Thus, 31* turns into 2Control-A* (365* plus
2Control*), and 3135* turns into 2Control-Meta-G*
(3135* = 3128* + 37*, = 2Meta* + 2Control* +
373* which is 3G*).


.section "Primitive Font Definitions"
.setq primitive-font-section-page section-page

This section details how "fonts" are described at a low level in
lan{nil} in order to be interfaced to the character primitives
described in this chapter.  The contents of this section can be safely
ignored by anyone who isn't going to try to define fonts and interface
them to lan{nil}.

First, remember that the 2code* portion of a character is exactly
8 bits.

There are several "attributes" of a font which we need to encode in
such a way that certain operations become trivial.  They are these:
.table 2 250 250 0 0 0
.item conversion to upper-case
.item1 conversion to lower-case
These want to be done rapidly
.item testing for certain attributes
the predicates 3upper-case-p*, 3graphic-char-p*,
3both-case-p*, etc.
.item comparison of characters of different fonts
Two goals are desired here.  It may be desirable for two characters in
different fonts with the same code to 2not* be 3char-equal*,
because they represent different things.  It might also be desirable
for two characters in different fonts with 2different* codes to be
3char-equal*--one might be 2p*, the other 3P*.
.item getting weights of digits
.item obtaining digit characters from weights
.end_table

For efficiency, all of these goals are implemented through table
lookup.  There are several special variables whose values are simple
vectors which contain one entry for each possible font (256 entries in
each vector).  In the initial lan{nil} environment, all of the
entries of each vector are initialized to the same thing.

Before these tables get listed, there are a couple of conventions
which must be explained first.  The first is what is called internally
a lan{translation table} for characters.  This is just a
simple-string or simple-bit-vector 256 bytes (2048 bits) long, for
doing code-to-code translation.  For instance, translation to
upper-case for a particular font utilizes such a table.  The lan{nil}
primitives are defined such that it does not matter whether this
datastructure is a simple-bit-vector or a simple-string.  Another such
datastructure is a 2syntax table*.  This is the same as a
translation table, but each byte within the table is treated instead
as a bit-mask with one bit set for each attribute about the code being
defined.  Both of these concepts will be abstracted away from
characters in a future lan{nil}.

The attributes about a font we define are these:
.table 2 250 250 0 0 0
.item upper-case translation table
.item1 lower-case translation table
Each font has one of each of these.  3char-upcase* works by
returning a new character of the same font, with a 2code*
which is the result of extracting the byte from the
upper-case translation table at the 2code* index.  (If the
character has non-zero 2bits* it is just returned however.)
.item syntax table
A syntax table (as described above) with syntax bits for the character
primitives 3graphic-char-p*, 3upper-case-p*,
3lower-case-p*, 3both-case-p*, 3alphanumericp*, and
3alpha-char-p*.  Constants for these bits are defined below.
.item digit weight table
This is a simple vector 256 long with one entry for each character code.
All entries are fixnums.  Those entries corresponding to characters
which cannot ever be interpreted as digits should be 3-1*;  all
others should have the weight of the digit.
.item digit char table
This is a simple-string (it must 2not* be a simple-bit-vector)
used for conversion from a digit weight to the corresponding
character.  The length of the string is the maximum radix which can be
handled in that font, and all entries in-between must be valid.  For
instance, in the standard such table, the entry at index 39* is
the character 39*, and the entry at index 312* is the
character 3C*.
.item the typeface
The format of this object is not defined;  the typefaces of different
fonts are compared with 3eq* by such predicates as
3char-equal*.  This is discussed later.
.item typeface canonicalization translation table
This is a translation table which is used for canonicalization of
characters of different fonts but the same typeface;  discussed later.
.end_table

.defun si:note-primitive-font font-number &key upper-case-translations syntax-table lower-case-translations typeface backwards-reference digit-weight-table digit-char-table typeface-canonicalization
This assigns all of the information described above to the font
2font-number*.  Appropriate defaults are assigned for those
entries which are 3nil*.
.end_defun

The following constants define syntax bits used in the syntax table
used.  For each, the "$v_" constant has as its value the position of
the bit within the field (byte), and the "$m_" constant a mask for
testing the bit (with 3logtest&*, for instance) or setting it
(with 3logior&*).

.defconstant si:charsyntax$v_graphic
.defconstant1 si:charsyntax$m_graphic
This bit tells if the character is graphic;  that is, if it should
satisfy 3graphic-char-p*.
.end_defconstant


.defconstant si:charsyntax$v_upper_case
.defconstant1 si:charsyntax$m_upper_case
This bit tells if the character is upper case (should satisfy
3upper-case-p*).
.end_defconstant

.defconstant si:charsyntax$v_lower_case
.defconstant1 si:charsyntax$m_lower_case
This tells if the character is lower case.
.end_defconstant

.defconstant si:charsyntax$v_both_case
.defconstant1 si:charsyntax$m_both_case
This tells if the character satisfies 3both-case-p*
((both-case-p-fun)).
.end_defconstant

.defconstant si:charsyntax$v_digit
.defconstant1 si:charsyntax$m_digit
This tells if the character is a (decimal) digit character.
.end_defconstant

.defconstant si:charsyntax$v_standard
.defconstant1 si:charsyntax$m_standard
This tells if the character, were it if font 0, is a standard char
(3standard-char-p*, (standard-char-p-fun)).
.end_defconstant

.defconstant si:charsyntax$v_alpha
.defconstant1 si:charsyntax$m_alpha
This tells if the character is alphabetic.  If the character is either
upper or lower case, then it must have this bit set also.  It is
hypothetically possible, however, for a character to be alphabetic but
neither upper or lower case.
.end_defconstant

Character comparison ala 3char-equal*, 3char-lessp*, etc., is
performed as follows.  We assume that it is possible to group
characters into logical "typefaces";  canonicalizations within a
typeface are valid, and comparisons of characters of different
typefaces are not.  Each font has associated with it one such
typeface, and the assumption is that many different fonts map to just
a few typefaces.  When two characters are compared, if their typefaces
are the same, then the codes of the characters are translated with the
typeface canonicalization translation table, and the results compared
in the appropriate manner (3=* for 3char-equal*, 3<* for
3char-lessp*, etc.).  If the typefaces are different, then the
characters are not equal;  for ordering, they are compared in a manner
consistent with 3char<*.  lan{nil} does not do anything at all
with the typeface object described here;  all that is done with
typefaces is to compare them with 3eq* to do this character
comparison.

