:title: mal reader :date: 2026-09-04 :tags: mal, python :identifier: 20260904T122549 :signature: 10=3 The ``mal`` Reader ================== :project: mal-py .. seealso:: `Mal: Step 1 `__ In ``mal`` the reader exists "outside" the regular steps since once it is setup, it's not going to change much. That said many data types are optional to start with so, I'm likely to only add them on demand as I progress through the steps. Tokenising ---------- :filename: reader.py The first step is to chop up the input stream into a sequence of tokens, the suggested way to do this is with a fairly intimidating regular expression. Thankfully, by compiling it with ``re.VERBOSE`` we can at least annotate it. .. code-block:: python import contextlib import json import re TOKEN = re.compile( r""" [\s,]* # tokens are separated by whitespace or commas ( # a token is ... ~@ # a literal '~@' | [\[\]{}()'`~^@] # or a literal "[", "]", "{", "}", "(", ")", "'", "`", "~", "^", "@" | "(?: # or a string i.e. '"' followed by \\. # an escaped character e.g. '\n' | [^\\"] # anything except '\' or '"' )* # (repeated zero or more times) "? # closed by a '"' (made optional to also handle invalid strings) | ;.* # or a comment i.e. ';' followed by anything (except newlines) zero or more times | [^\s\[\]{}()'"`,;]* # or a symbol i.e. a sequence of zero or more non-special characters ) """, re.VERBOSE, ) The ``(?:`` syntax was new to me, apparently this defines a "non-capturing group". It allows you to express ``A or B`` without the group showing up in ``Match.groups()`` - nice!. Using this pattern, we can now define a function that takes a string and return a list of tokens. .. code-block:: python def tokenise(text: str) -> list[str]: return TOKEN.findall(text) Reading ------- Now we have a stream of tokens, the next step is to construct a data structure representing the code inputted. The Reader ^^^^^^^^^^ To help manage the stream of tokens, the mal guide suggests creating a ``Reader`` object .. code-block:: python class Reader: def __init__(self, tokens): self.tokens = tokens self.pos = 0 {{ insert(slots['reader-methods'], indent=4) }} Which provides the following methods: - A ``next`` method to return the current token and advances the position. .. code-block:: python :slot: reader-methods def next(self): try: tok = self.tokens[self.pos] except IndexError: raise EOFError() self.pos += 1 return tok - A ``peek`` method that simply returns the current token .. code-block:: python :slot: reader-methods def peek(self): try: return self.tokens[self.pos] except IndexError: raise EOFError() read_form ^^^^^^^^^ The generic ``read_form`` method dispatches to specialised read methods depending on what the current token is. .. code-block:: python def read_form(reader: Reader): match reader.peek(): case '(': return read_list(reader) case "'": tok = reader.next() return [S("quote"), read_form(reader)] case "~": tok = reader.next() return [S("unquote"), read_form(reader)] case "~@": tok = reader.next() return [S("splice-unquote"), read_form(reader)] case "`": tok = reader.next() return [S("quasiquote"), read_form(reader)] case _: return read_atom(reader) read_list ^^^^^^^^^ As you would expect ``read_list`` is responsible for constructing lists .. code-block:: python def read_list(reader: Reader): if (tok := reader.next()) != '(': raise RuntimeError(f"Expected '(', got {tok!r}") items = [] while reader.peek() != ')': items.append(read_form(reader)) if (tok := reader.next()) != ')': raise RuntimeError(f"Expected ')', got {tok!r}") return items read_atom ^^^^^^^^^ An atom is "everything else" .. code-block:: python def read_atom(reader: Reader): tok = reader.next() if tok in {'(', ')'}: raise RuntimeError(f"Expected atom, got {tok!r}") with contextlib.suppress(ValueError): return int(tok) match tok: case 'true': return True case 'false': return False case 'nil': return None case str(s) if s.startswith('"'): return parse_string(s) case _: return S(tok) I was having huge difficulties getting string based tests to pass... until I went back and read the derferrables from step 1... .. code-block:: python def parse_string(s: str) -> str: if not s.endswith('"'): raise RuntimeError(f"unterminated string literal: {s!r}") return (s[1:-1].replace('\\\"', '"') .replace("\\n", "\n") .replace("\\\\", "\\")) read_str ^^^^^^^^ Finally, ``read_str`` ties it all together. .. code-block:: python def read_str(ins: str): reader = Reader(tokenise(ins)) return read_form(reader) Data Types ---------- Atoms ^^^^^ Step 6 calls for atoms... .. code-block:: python class A: __match_args__ = ("v",) def __init__(self, v): self.v = v Functions ^^^^^^^^^ .. admonition:: Tail Call Optimization Step 5 introduced TCO which resulted in the need for this class. It's to provide enough information to ``evaluate`` for the loop to apply TCO to fn calls. The ``Fn`` class represents a function .. code-block:: python class Fn: __match_args__ = ("params", "body", "env", "is_macro") def __init__(self, params, body, env, f): self.params = params self.body = body self.env = env self.f = f # not needed until step 6 self.is_macro = False # set by defmacro! # Let's cheat so that core functions don't have to know which flavour of # function they are calling def __call__(self, *args): return self.f(*args) def __repr__(self): return "#" if self.is_macro else "#" Symbols ^^^^^^^ The ``S`` class represents a symbol. .. code-block:: python class S: __match_args__ = ("name",) def __init__(self, name: str): self.name = name def __repr__(self): return self.name The ``mal`` printer ------------------- :filename: printer.py Like the reader, the ``mal`` printer is handled "outside" of the steps. .. code-block:: python import json from reader import A def print_form(form): match form: case A(v): return f"(atom {print_form(v)})" case [*fs]: inner = " ".join(print_form(f) for f in fs) return f"({inner})" case True: return 'true' case False: return 'false' case None: return 'nil' case str(s): return json.dumps(s) case _: return str(form)