The mal Reader¶
- project:
mal-py
See also
In mal the reader exists “outside” the regular steps since once it is setup, it’s not going to change much.
That said many data types are optional to start with so, I’m likely to only add them on demand as I progress through the steps.
Tokenising¶
- filename:
reader.py
The first step is to chop up the input stream into a sequence of tokens, the suggested way to do this is with a fairly intimidating regular expression.
Thankfully, by compiling it with re.VERBOSE we can at least annotate it.
mal-py
reader.py
import contextlib
import json
import re
TOKEN = re.compile(
r"""
[\s,]* # tokens are separated by whitespace or commas
( # a token is ...
~@ # a literal '~@'
| [\[\]{}()'`~^@] # or a literal "[", "]", "{", "}", "(", ")", "'", "`", "~", "^", "@"
| "(?: # or a string i.e. '"' followed by
\\. # an escaped character e.g. '\n'
| [^\\"] # anything except '\' or '"'
)* # (repeated zero or more times)
"? # closed by a '"' (made optional to also handle invalid strings)
| ;.* # or a comment i.e. ';' followed by anything (except newlines) zero or more times
| [^\s\[\]{}()'"`,;]* # or a symbol i.e. a sequence of zero or more non-special characters
)
""",
re.VERBOSE,
)
The (?: syntax was new to me, apparently this defines a “non-capturing group”. It allows you to express A or B without the group showing up in Match.groups() - nice!.
Using this pattern, we can now define a function that takes a string and return a list of tokens.
mal-py
reader.py
def tokenise(text: str) -> list[str]:
return TOKEN.findall(text)
Reading¶
Now we have a stream of tokens, the next step is to construct a data structure representing the code inputted.
The Reader¶
To help manage the stream of tokens, the mal guide suggests creating a Reader object
mal-py
reader.py
class Reader:
def __init__(self, tokens):
self.tokens = tokens
self.pos = 0
{{ insert(slots['reader-methods'], indent=4) }}
Which provides the following methods:
A
nextmethod to return the current token and advances the position.mal-pyreader.pydef next(self): try: tok = self.tokens[self.pos] except IndexError: raise EOFError() self.pos += 1 return tok
A
peekmethod that simply returns the current tokenmal-pyreader.pydef peek(self): try: return self.tokens[self.pos] except IndexError: raise EOFError()
read_form¶
The generic read_form method dispatches to specialised read methods depending on what the current token is.
mal-py
reader.py
def read_form(reader: Reader):
match reader.peek():
case '(':
return read_list(reader)
case "'":
tok = reader.next()
return [S("quote"), read_form(reader)]
case "~":
tok = reader.next()
return [S("unquote"), read_form(reader)]
case "~@":
tok = reader.next()
return [S("splice-unquote"), read_form(reader)]
case "`":
tok = reader.next()
return [S("quasiquote"), read_form(reader)]
case _:
return read_atom(reader)
read_list¶
As you would expect read_list is responsible for constructing lists
mal-py
reader.py
def read_list(reader: Reader):
if (tok := reader.next()) != '(':
raise RuntimeError(f"Expected '(', got {tok!r}")
items = []
while reader.peek() != ')':
items.append(read_form(reader))
if (tok := reader.next()) != ')':
raise RuntimeError(f"Expected ')', got {tok!r}")
return items
read_atom¶
An atom is “everything else”
mal-py
reader.py
def read_atom(reader: Reader):
tok = reader.next()
if tok in {'(', ')'}:
raise RuntimeError(f"Expected atom, got {tok!r}")
with contextlib.suppress(ValueError):
return int(tok)
match tok:
case 'true':
return True
case 'false':
return False
case 'nil':
return None
case str(s) if s.startswith('"'):
return parse_string(s)
case _:
return S(tok)
I was having huge difficulties getting string based tests to pass… until I went back and read the derferrables from step 1…
mal-py
reader.py
def parse_string(s: str) -> str:
if not s.endswith('"'):
raise RuntimeError(f"unterminated string literal: {s!r}")
return (s[1:-1].replace('\\\"', '"')
.replace("\\n", "\n")
.replace("\\\\", "\\"))
read_str¶
Finally, read_str ties it all together.
mal-py
reader.py
def read_str(ins: str):
reader = Reader(tokenise(ins))
return read_form(reader)
Data Types¶
Atoms¶
Step 6 calls for atoms…
mal-py
reader.py
class A:
__match_args__ = ("v",)
def __init__(self, v):
self.v = v
Functions¶
Tail Call Optimization
Step 5 introduced TCO which resulted in the need for this class.
It’s to provide enough information to evaluate for the loop to apply TCO to fn calls.
The Fn class represents a function
mal-py
reader.py
class Fn:
__match_args__ = ("params", "body", "env", "is_macro")
def __init__(self, params, body, env, f):
self.params = params
self.body = body
self.env = env
self.f = f # not needed until step 6
self.is_macro = False # set by defmacro!
# Let's cheat so that core functions don't have to know which flavour of
# function they are calling
def __call__(self, *args):
return self.f(*args)
def __repr__(self):
return "#<macro>" if self.is_macro else "#<function>"
Symbols¶
The S class represents a symbol.
mal-py
reader.py
class S:
__match_args__ = ("name",)
def __init__(self, name: str):
self.name = name
def __repr__(self):
return self.name
The mal printer¶
- filename:
printer.py
Like the reader, the mal printer is handled “outside” of the steps.
mal-py
printer.py
import json
from reader import A
def print_form(form):
match form:
case A(v):
return f"(atom {print_form(v)})"
case [*fs]:
inner = " ".join(print_form(f) for f in fs)
return f"({inner})"
case True:
return 'true'
case False:
return 'false'
case None:
return 'nil'
case str(s):
return json.dumps(s)
case _:
return str(form)