Alex Carney | Blog

Indexing Info Manuals

2026-03-09T00:00:00+00:00

Indexing Info Manuals¶

I’ve been working around this issue for a while now.

By default, the C-h i binding in Emacs opens the list of available info manuals for you to browse through. Unfortunately, I would only be shown manuals that had been included with third-party Emacs packages like denote. System provided manuals, such as the one for Emacs itself, were nowhere to be seen:

File: dir,        Node: Top       This is the top of the INFO tree

  This (the Directory node) gives a menu of major topics.
  Typing "q" exits, "H" lists all Info commands, "d" returns here,
  "h" gives a primer for first-timers,
  "mEmacs<Return>" visits the Emacs manual, etc.

  In Emacs, you can click mouse button 2 on a menu item or cross reference
  to select it.

* Menu:

Emacs
* Denote Sequence: (denote-sequence).
                                Sequence notes or Folgezettel with Denote.
* Denote: (denote).             Simple notes with an efficient file-naming
                                  scheme.
* Eat: (eat).                   Emulate A Terminal.
* Embark: (embark).             Emacs Mini-Buffer Actions Rooted in Keymaps.
* Forge: (forge).               Access Git Forges from Magit.
* Ghub: (ghub).                 Client library for the Github API.
* Magit: (magit).               Using Git from Emacs with Magit.
* Magit-Section: (magit-section).
                                Use Magit sections in your own packages.
* Orderless: (orderless).       Completion style for matching regexps in any
                                  order.
* Tempel: (tempel).             Simple templates for Emacs.
* Transient: (transient).       Transient Commands.
* With-Editor: (with-editor).   Using the Emacsclient as $EDITOR.

However, I could clearly see files like /usr/share/info/emacs.info.gz on my system and I could access specific info nodes by evaluating expressions like (info "(emacs)Displaying Buffers").

I’ve now reached the point where it’s bugging me enough to actually dig in and try to figure out what is going on.

`dir` files¶

It turns out that the buffer you see when invoking C-h i is assembled from all the places Emacs is configured to search for info manuals (via the Info-directory-list variable). Each directory is expected to contain a file called dir that would provide a list of all info manuals in the directory. For example in the denote package:

~/.emacs.d/elpa/denote-4.1.3/ $ cat ./dir

This is the file .../info/dir, which contains the
topmost node of the Info hierarchy, called (dir)Top.
The first time you invoke Info you start off looking at this node.

File: dir,    Node: Top       This is the top of the INFO tree

  This (the Directory node) gives a menu of major topics.
  Typing "q" exits, "H" lists all Info commands, "d" returns here,
  "h" gives a primer for first-timers,
  "mEmacs<Return>" visits the Emacs manual, etc.

  In Emacs, you can click mouse button 2 on a menu item or cross reference
  to select it.

* Menu:

Emacs misc features
* Denote: (denote).             Simple notes with an efficient file-naming
                                  scheme.

where (denote). instructs Emacs to look for a file called denote.info.

Armed with this knowledge, you would expect to find a similar file in /usr/share/info, but unsurprsingly, it’s missing in my case. The obvious solution then, would be to ensure that a dir file is present in the /usr/share/info directory however, my Linux flavour of choice (Aurora) makes this a bit more interesting:

$ touch /usr/share/info/dir
touch: cannot touch '/usr/share/info/dir': Read-only file system

Interestingly however, this didn’t turn out to be a big deal as manually editing the dir file provided by denote I proved to myself that you didn’t actually need the .info file to be in the same folder as the dir file, as long as it was somewhere listed in Info-directory-list

  * Menu:

  Emacs misc features
+ * Emacs: (emacs).               The Emacs manual.
  * Denote: (denote).             Simple notes with an efficient file-naming
                                  scheme.

Now all that’s left is to build a dir file indexing all the missing manuals.

Note

The “correct” way to do this is is to use the install-info command from the texinfo project.

But since I don’t have the texinfo package installed I thought it would be fun to write a small Python script to do the same.

`build-info-dir.py`¶

build-info-dir.py

"""build-info-dir.py <directory> <output>

Index all info manuals in a given directory and produce a ``dir`` index file.
"""
import argparse
import gzip
import pathlib
import sys

As we can see from the examples above, the format of a dir file is quite straightforward, the hardest part is figuring out the title and description of each manual along with which sections to group them under.

The texinfo documentation states that manuals export specific info nodes to the top-level dir using the following syntax (example taken from the info file for denote):

INFO-DIR-SECTION Emacs misc features
START-INFO-DIR-ENTRY
* Denote: (denote).     Simple notes with an efficient file-naming scheme.
END-INFO-DIR-ENTRY

Which can be extracted as follows:

build-info-dir.py

DIR_SECTION = "INFO-DIR-SECTION "

def extract_dir_nodes(filename: pathlib.Path) -> dict[str, list[str]]:
    nodes: dict[str, list[str]] = {}

    section: str = ""
    entry: list[str] | None = None

    with open_info_file(filename) as f:
        for line in f.readlines():
            text = line.decode(errors="ignore").rstrip()
            if text.startswith(DIR_SECTION):
                section = text.replace(DIR_SECTION, "")
                continue

            elif text == "START-INFO-DIR-ENTRY":
                entry = []
                continue

            elif text == "END-INFO-DIR-ENTRY":
                nodes.setdefault(section, []).extend(entry)
                entry = None

            elif isinstance(entry, list):
                entry.append(text)

    return nodes

open_info_file is a separate helper function as info manuals can be gzipped.

build-info-dir.py

def open_info_file(filename: pathlib.Path):
    if ".gz" in filename.suffix:
        return gzip.open(filename)

    return filename.open()

All that’s left is the infrastructure to find and aggregate the result of each info file.

build-info-dir.py

def index_info_dir(info_dir: pathlib.Path) -> str:
    nodes: dict[str, list[str]] = {}

    for filename in info_dir.glob("*.info*"):
        # Some manuals are split into mutliple files, we only need to index the top-level file
        if "info-" in filename.name:
            continue

        print(f"Indexing {filename.name}...", file=sys.stderr)
        index = extract_dir_nodes(filename)
        for section, items in index.items():
            nodes.setdefault(section, []).extend(items)

    lines = [
        chr(0x1f),
        "File: dir\tNode: Top\tThis is the top of the INFO tree",
        "",
        "* Menu:",
        ""
    ]

    for section, items in nodes.items():
        lines.append(section)
        lines.extend(items)

    return "\n".join(lines)

After much trial and error, I eventually figured out that the dir file must contain

the special character "\x1f" (rendered as ^_ by Emacs),
the File: dir... header and
the * Menu: string

in order to be parsed correctly.

Other than that, it appears like you can add any other text you’d like and it would be rendered in the final *info* buffer. I was tempted to add a count of all the manuals found, but since the file we’re generating here isn’t the only source, any counts are unlikely to be accurate.

Finally, to make the script useful let’s wrap it in a simple CLI interface.

build-info-dir.py

def main():
    cli = argparse.ArgumentParser()
    cli.add_argument("infodir", type=pathlib.Path)
    cli.add_argument("-o", "--output", type=argparse.FileType("w"), default="-")

    args = cli.parse_args()
    content = index_info_dir(args.infodir)
    print(content, file=args.output)


if __name__ == "__main__":
    main()

Building and Using the Index¶

To get Emacs to use the the index, we of course have to generate it:

$ mkdir -p ~/.local/share/info
$ python build-info-dir.py /usr/share/info -o ~/.local/share/info/dir
...
Indexing url.info.gz...
Indexing use-package.info.gz...
Indexing vhdl-mode.info.gz...
Indexing vip.info.gz...
Indexing viper.info.gz...
Indexing vtable.info.gz...
Indexing widget.info.gz...
Indexing wisent.info.gz...
Indexing woman.info.gz...

And make sure Emacs is configured to search the folder

init.el

(add-to-list 'Info-directory-list
             (concat (getenv "HOME") "/.local/share/info/"))

Success!

File: dir    Node: Top       This is the top of the INFO tree

* Menu:

Software development
* As: (as).                     The GNU assembler.
* Gas: (as).                    The GNU assembler.
* Bfd: (bfd).                   The Binary File Descriptor library.
* Binutils: (binutils).         The GNU binary utilities.
* bison: (bison).       GNU parser generator (Yacc replacement).
* Cpp: (cpp).                  The GNU C preprocessor.
* Cpplib: (cppinternals).      Cpplib internals.
* CTF: (ctf-spec).         The CTF file format.
* gcc: (gcc).                  The GNU Compiler Collection.
* g++: (gcc).                  The GNU C++ compiler.
* gcov: (gcc) Gcov.            ‘gcov’--a test coverage program.
* gcov-tool: (gcc) Gcov-tool.  ‘gcov-tool’--an offline gcda profile processing program.
* gcov-dump: (gcc) Gcov-dump.  ‘gcov-dump’--an offline gcda and gcno profile dump tool.
* lto-dump: (gcc) lto-dump.    ‘lto-dump’--Tool for
* flex: (flex).      Fast lexical analyzer generator (lex replacement).
dumping LTO object files.
* Gccgo: (gccgo).           A GCC-based compiler for the Go language
* gccinstall: (gccinstall).    Installing the GNU Compiler Collection.
* gccint: (gccint).            Internals of the GNU Compiler Collection.
* GNUstep Make: (gnustep-make).         The GNUstep build system.
* gprof: (gprof).                Profiling your program's execution
* Ld: (ld).                       The GNU linker.
* Ld-Internals: (ldint).     The GNU linker internals.
* Make: (make).            Remake files automatically.
* SFrame: (sframe-spec).         The Simple Frame format.
Basics
* Bash: (bash).                     The GNU Bourne-Again SHell.
* Coreutils: (coreutils).       Core GNU (file, text, shell) utilities.
* Common options: (coreutils)Common options.
* File permissions: (coreutils)File permissions.  Access modes.
* Date input formats: (coreutils)Date input formats.
* Ed: (ed).                     The GNU line editor
* Finding files: (find).        Operating on files matching certain criteria.
* Time: (time).                  GNU time utility.
Math
* bc: (bc).                    An arbitrary precision calculator language.
Individual utilities
* addr2line: (binutils)addr2line. Convert addresses to file and line.
* ar: (binutils)ar.               Create, modify, and extract from archives.
* c++filt: (binutils)c++filt.          Filter to demangle encoded C++ symbols.
* cxxfilt: (binutils)c++filt.     MS-DOS name for c++filt.
* dlltool: (binutils)dlltool.          Create files needed to build and use DLLs.
* nm: (binutils)nm.               List symbols from object files.
* objcopy: (binutils)objcopy.          Copy and translate object files.
* objdump: (binutils)objdump.     Display information from object files.
* ranlib: (binutils)ranlib.       Generate index to archive contents.
* readelf: (binutils)readelf.          Display the contents of ELF format files.
* size: (binutils)size.           List section sizes and total size.
* strings: (binutils)strings.     List printable strings from files.
* strip: (binutils)strip.         Discard symbols.
* elfedit: (binutils)elfedit.     Update ELF header and property of ELF files.
* windmc: (binutils)windmc.    Generator for Windows message resources.
* windres: (binutils)windres.          Manipulate Windows resources.
* arch: (coreutils)arch invocation.             Print machine hardware name.
* b2sum: (coreutils)b2sum invocation.           Print or check BLAKE2 digests.
* base32: (coreutils)base32 invocation.         Base32 encode/decode data.
* base64: (coreutils)base64 invocation.         Base64 encode/decode data.
* basename: (coreutils)basename invocation.     Strip directory and suffix.
* basenc: (coreutils)basenc invocation.         Encoding/decoding of data.
* cat: (coreutils)cat invocation.               Concatenate and write files.
* chcon: (coreutils)chcon invocation.           Change SELinux CTX of files.
* chgrp: (coreutils)chgrp invocation.           Change file groups.
* chmod: (coreutils)chmod invocation.           Change access permissions.
* chown: (coreutils)chown invocation.           Change file owners and groups.
* chroot: (coreutils)chroot invocation.         Specify the root directory.
* cksum: (coreutils)cksum invocation.           Print POSIX CRC checksum.
* comm: (coreutils)comm invocation.             Compare sorted files by line.
* cp: (coreutils)cp invocation.                 Copy files.
* csplit: (coreutils)csplit invocation.         Split by context.
* cut: (coreutils)cut invocation.               Print selected parts of lines.
* date: (coreutils)date invocation.             Print/set system date and time.
* dd: (coreutils)dd invocation.                 Copy and convert a file.
* df: (coreutils)df invocation.                 Report file system usage.
* dir: (coreutils)dir invocation.               List directories briefly.
* dircolors: (coreutils)dircolors invocation.   Color setup for ls.
* dirname: (coreutils)dirname invocation.       Strip last file name component.
* du: (coreutils)du invocation.                 Report file usage.
* echo: (coreutils)echo invocation.             Print a line of text.
* env: (coreutils)env invocation.               Modify the environment.
* expand: (coreutils)expand invocation.         Convert tabs to spaces.
* expr: (coreutils)expr invocation.             Evaluate expressions.
* factor: (coreutils)factor invocation.         Print prime factors
* false: (coreutils)false invocation.           Do nothing, unsuccessfully.
* fmt: (coreutils)fmt invocation.               Reformat paragraph text.
* fold: (coreutils)fold invocation.             Wrap long input lines.
* groups: (coreutils)groups invocation.         Print group names a user is in.
* head: (coreutils)head invocation.             Output the first part of files.
* hostid: (coreutils)hostid invocation.         Print numeric host identifier.
* hostname: (coreutils)hostname invocation.     Print or set system name.
* id: (coreutils)id invocation.                 Print user identity.
* install: (coreutils)install invocation.       Copy files and set attributes.
* join: (coreutils)join invocation.             Join lines on a common field.
* kill: (coreutils)kill invocation.             Send a signal to processes.
* link: (coreutils)link invocation.             Make hard links between files.
* ln: (coreutils)ln invocation.                 Make links between files.
* logname: (coreutils)logname invocation.       Print current login name.
* ls: (coreutils)ls invocation.                 List directory contents.
* md5sum: (coreutils)md5sum invocation.         Print or check MD5 digests.
* mkdir: (coreutils)mkdir invocation.           Create directories.
* mkfifo: (coreutils)mkfifo invocation.         Create FIFOs (named pipes).
* mknod: (coreutils)mknod invocation.           Create special files.
* mktemp: (coreutils)mktemp invocation.         Create temporary files.
* mv: (coreutils)mv invocation.                 Rename files.
* nice: (coreutils)nice invocation.             Modify niceness.
* nl: (coreutils)nl invocation.                 Number lines and write files.
* nohup: (coreutils)nohup invocation.           Immunize to hangups.
* nproc: (coreutils)nproc invocation.           Print the number of processors.
* numfmt: (coreutils)numfmt invocation.         Reformat numbers.
* od: (coreutils)od invocation.                 Dump files in octal, etc.
* paste: (coreutils)paste invocation.           Merge lines of files.
* pathchk: (coreutils)pathchk invocation.       Check file name portability.
* pinky: (coreutils)pinky invocation.           Print information about users.
* pr: (coreutils)pr invocation.                 Paginate or columnate files.
* printenv: (coreutils)printenv invocation.     Print environment variables.
* printf: (coreutils)printf invocation.         Format and print data.
* ptx: (coreutils)ptx invocation.               Produce permuted indexes.
* pwd: (coreutils)pwd invocation.               Print working directory.
* readlink: (coreutils)readlink invocation.     Print referent of a symlink.
* realpath: (coreutils)realpath invocation.     Print resolved file names.
* rm: (coreutils)rm invocation.                 Remove files.
* rmdir: (coreutils)rmdir invocation.           Remove empty directories.
* runcon: (coreutils)runcon invocation.         Run in specified SELinux CTX.
* seq: (coreutils)seq invocation.               Print numeric sequences
* sha1sum: (coreutils)sha1sum invocation.       Print or check SHA-1 digests.
* sha2: (coreutils)sha2 utilities.              Print or check SHA-2 digests.
* shred: (coreutils)shred invocation.           Remove files more securely.
* shuf: (coreutils)shuf invocation.             Shuffling text files.
* sleep: (coreutils)sleep invocation.           Delay for a specified time.
* sort: (coreutils)sort invocation.             Sort text files.
* split: (coreutils)split invocation.           Split into pieces.
* stat: (coreutils)stat invocation.             Report file(system) status.
* stdbuf: (coreutils)stdbuf invocation.         Modify stdio buffering.
* stty: (coreutils)stty invocation.             Print/change terminal settings.
* sum: (coreutils)sum invocation.               Print traditional checksum.
* sync: (coreutils)sync invocation.             Sync files to stable storage.
* tac: (coreutils)tac invocation.               Reverse files.
* tail: (coreutils)tail invocation.             Output the last part of files.
* tee: (coreutils)tee invocation.               Redirect to multiple files.
* test: (coreutils)test invocation.             File/string tests.
* timeout: (coreutils)timeout invocation.       Run with time limit.
* touch: (coreutils)touch invocation.           Change file timestamps.
* tr: (coreutils)tr invocation.                 Translate characters.
* true: (coreutils)true invocation.             Do nothing, successfully.
* truncate: (coreutils)truncate invocation.     Shrink/extend size of a file.
* tsort: (coreutils)tsort invocation.           Topological sort.
* tty: (coreutils)tty invocation.               Print terminal name.
* uname: (coreutils)uname invocation.           Print system information.
* unexpand: (coreutils)unexpand invocation.     Convert spaces to tabs.
* uniq: (coreutils)uniq invocation.             Uniquify files.
* unlink: (coreutils)unlink invocation.         Removal via unlink(2).
* uptime: (coreutils)uptime invocation.         Print uptime and load.
* users: (coreutils)users invocation.           Print current user names.
* vdir: (coreutils)vdir invocation.             List directories verbosely.
* wc: (coreutils)wc invocation.                 Line, word, and byte counts.
* who: (coreutils)who invocation.               Print who is logged in.
* whoami: (coreutils)whoami invocation.         Print effective user ID.
* yes: (coreutils)yes invocation.               Print a string indefinitely.
* cmp: (diffutils)Invoking cmp.                 Compare 2 files byte by byte.
* diff: (diffutils)Invoking diff.               Compare 2 files line by line.
* diff3: (diffutils)Invoking diff3.             Compare 3 files line by line.
* patch: (diffutils)Invoking patch.             Apply a patch to a file.
* sdiff: (diffutils)Invoking sdiff.             Merge 2 files side-by-side.
* find: (find)Finding Files.                    Finding and acting on files.
* locate: (find)Invoking locate.                Finding files in a database.
* updatedb: (find)Invoking updatedb.            Building the locate database.
* xargs: (find)Invoking xargs.                  Operating on many files.
* awk: (gawk)Invoking Gawk.                     Text scanning and processing.
* Gawk Work Flow: (gawkworkflow)Overview.         Participating in ‘gawk’ development.
* gunzip: (gzip)Overview.                       Decompression.
* gzexe: (gzip)Overview.                        Compress executables.
* zcat: (gzip)Overview.                         Decompression to stdout.
* zdiff: (gzip)Overview.                        Compare compressed files.
* zforce: (gzip)Overview.                       Force .gz extension on files.
* zgrep: (gzip)Overview.                        Search compressed files.
* zmore: (gzip)Overview.                        Decompression output by pages.
* tar: (tar)tar invocation.                     Invoking GNU ‘tar’.
Archiving
* Cpio: (cpio).                 Copy-in-copy-out archiver to tape or disk.
* Tar: (tar).                   Making tape (or disk) archives.
* Xorrecord: (xorrecord).           Emulates CD/DVD/BD program cdrecord
* Xorriso-dd-target: (xorriso-dd-target). Device evaluator and disk image copier for GNU/Linux
* Xorriso: (xorriso).           Burns ISO 9660 on CD, DVD, BD.
* Xorrisofs: (xorrisofs).           Emulates ISO 9660 program mkisofs

* dc: (dc).                   Arbitrary precision RPN "Desktop Calculator".
Text creation and manipulation
* Diffutils: (diffutils).       Comparing and merging files.
* Gawk: (gawk).                 A text scanning and processing language.
* Gawk Work Flow: (gawkworkflow).                 Participating in ‘gawk’ development.
* grep: (grep).                 Print lines that match patterns.
* M4: (m4).                     A powerful macro processor.
* pm-gawk: (pm-gawk).    Persistent memory version of gawk.
* sed: (sed).                   Stream EDitor.

GNU organization
* Maintaining Findutils: (find-maint).        Maintaining GNU findutils
Network applications
* awkinet: (gawkinet).          TCP/IP Internetworking With 'gawk'.
GNU Gettext Utilities
* gettext: (gettext).                          GNU gettext utilities.
* autopoint: (gettext)autopoint Invocation.    Copy gettext infrastructure.
* envsubst: (gettext)envsubst Invocation.      Expand environment variables.
* gettextize: (gettext)gettextize Invocation.  Prepare a package for gettext.
* msgattrib: (gettext)msgattrib Invocation.    Select part of a PO file.
* msgcat: (gettext)msgcat Invocation.          Combine several PO files.
* msgcmp: (gettext)msgcmp Invocation.          Compare a PO file and template.
* msgcomm: (gettext)msgcomm Invocation.        Match two PO files.
* msgconv: (gettext)msgconv Invocation.        Convert PO file to encoding.
* msgen: (gettext)msgen Invocation.            Create an English PO file.
* msgexec: (gettext)msgexec Invocation.        Process a PO file.
* msgfilter: (gettext)msgfilter Invocation.    Pipe a PO file through a filter.
* msgfmt: (gettext)msgfmt Invocation.          Make MO files out of PO files.
* msggrep: (gettext)msggrep Invocation.        Select part of a PO file.
* msginit: (gettext)msginit Invocation.        Create a fresh PO file.
* msgmerge: (gettext)msgmerge Invocation.      Update a PO file from template.
* msgunfmt: (gettext)msgunfmt Invocation.      Uncompile MO file into PO file.
* msguniq: (gettext)msguniq Invocation.        Unify duplicates for PO file.
* ngettext: (gettext)ngettext Invocation.      Translate a message with plural.
* xgettext: (gettext)xgettext Invocation.      Extract strings into a PO file.
* ISO639: (gettext)Language Codes.             ISO 639 language codes.
* ISO3166: (gettext)Country Codes.             ISO 3166 country codes.
GNU Utilities
* gpg2: (gnupg).           OpenPGP encryption and signing tool.
* gpgsm: (gnupg).          S/MIME encryption and signing tool.
* gpg-agent: (gnupg).      The secret key daemon.
* dirmngr: (gnupg).        X.509 CRL and OCSP server.
* dirmngr-client: (gnupg). X.509 CRL and OCSP client.
* pinentry: (pinentry).    Securely ask for a passphrase or PIN.
Kernel
* grub2-dev: (grub2-dev).                 The GRand Unified Bootloader Dev
* GRUB2: (grub2).                 The GRand Unified Bootloader
* grub2-install: (grub2)Invoking grub2-install.    Install GRUB on your drive
* grub2-mkconfig: (grub2)Invoking grub2-mkconfig.  Generate GRUB configuration
* grub2-mkpasswd-pbkdf2: (grub2)Invoking grub2-mkpasswd-pbkdf2.
* grub2-mkrelpath: (grub2)Invoking grub2-mkrelpath.
* grub2-mkrescue: (grub2)Invoking grub2-mkrescue.  Make a GRUB rescue image
* grub2-mount: (grub2)Invoking grub2-mount.        Mount a file system using GRUB
* grub2-probe: (grub2)Invoking grub2-probe.        Probe device information
* grub2-script-check: (grub2)Invoking grub2-script-check.
Compression
* Gzip: (gzip).                 General (de)compression of files (lzw).
Libraries
* History: (history).       The GNU history library API.
* RLuserman: (rluserman).       The GNU readline library User's Manual.
* libcdio: (libcdio).           GNU Compact Disc Input, Output, and Control Library.
* libgomp: (libgomp).          GNU Offloading and Multi Processing Runtime Library.
* Source-highlight-lib: (source-highlight-lib).  Highlights contents
* Auth-source: (auth).          The Emacs auth-source library.
* CL-Lib: (cl).                 Partial Common Lisp support for Emacs Lisp.
* D-Bus: (dbus).                Using D-Bus in Emacs.
* Emacs MIME: (emacs-mime).     Emacs MIME de/composition library.
* SMTP: (smtpmail).             Emacs library for sending mail via SMTP.
* URL: (url).                   URL loading package.
* Widget: (widget).             The "widget" package used by the Emacs
                                  Customization facility.
Localization
* libchewing: (libchewing).           The libchewing reference manual.
Misc
* Liblouis: (liblouis). A braille translator and back-translator
DOS
* Mtools: (mtools).        Mtools: utilities to access DOS disks in Unix.
Editors
* nano: (nano).                 Small and friendly text editor.
Encryption
* Nettle: (nettle).             A low-level cryptographic library.
System administration
* parted: (parted).                         GNU partitioning software
* Which: (which).                               Show full path of commands.
Texinfo documentation system
* Pinfo: (pinfo).           curses based lynx-style info browser.
Utilities
* Source-highlight: (source-highlight).  Highlights contents
* ZSH: (zsh).                     The Z Shell Manual.
Emacs
* Autotype: (autotype).         Convenient features for text that you enter
                                  frequently in Emacs.
* Bovine: (bovine).             Semantic bovine parser development.
* Calc: (calc).                 Advanced desk calculator and mathematical tool.
* Dired-X: (dired-x).           Dired Extra Features.
* Ebrowse: (ebrowse).           A C++ class browser for Emacs.
* EDE: (ede).                   The Emacs Development Environment.
* Ediff: (ediff).               A visual interface for comparing and
                                  merging programs.
* EDT: (edt).                   An Emacs emulation of the EDT editor.
* Eglot: (eglot).             Language Server Protocol client for Emacs.
* EIEIO: (eieio).               An objects system for Emacs Lisp.
* EasyPG Assistant: (epa).      An Emacs user interface to GNU Privacy Guard.
* ERT: (ert).                   Emacs Lisp regression testing tool.
* Eshell: (eshell).             A command shell implemented in Emacs Lisp.
* EWW: (eww).      Emacs Web Wowser
* Flymake: (flymake).           A universal on-the-fly syntax checker.
* Forms: (forms).               Emacs package for editing data bases
                                  by filling in forms.
* Htmlfontify: (htmlfontify).   Convert source code to html.
* Ido: (ido).                   Interactively do things with buffers and files.
* Modus Themes: (modus-themes). Elegant, highly legible and customizable themes.
* PCL-CVS: (pcl-cvs).           Emacs front-end to CVS.
* RefTeX: (reftex).             Emacs support for LaTeX cross-references
                                  and citations.
* Remember: (remember).         Simple information manager for Emacs.
* Semantic: (semantic).         Source code parser library and utilities.
* SES: (ses).                   Simple Emacs Spreadsheet.
* Speedbar: (speedbar).         File/Tag summarizing utility.
* SRecode: (srecode).           Semantic template code generator.
* Todo Mode: (todo-mode).       Make and maintain todo lists.
* Transient: (transient).       Transient Commands.
* use-package: (use-package). Declarative package configuration for Emacs.
* VIP: (vip).                   An obsolete VI-emulation for Emacs.
* VIPER: (viper).               A VI-emulation mode for Emacs.
* vtable: (vtable).     Variable Pitch Tables.
* Wisent: (wisent).             Semantic Wisent parser development.
* WoMan: (woman).               Browse UN*X Manual Pages "W.O. (without) Man".
* CC Mode: (ccmode).            Emacs mode for editing C, C++, Objective-C,
                                  Java, Pike, AWK, and CORBA IDL code.
* IDLWAVE: (idlwave).           Major mode and shell for IDL files.
* nXML Mode: (nxml-mode).       XML editing mode with RELAX NG support.
* Octave mode: (octave-mode).   Emacs mode for editing GNU Octave files.
* Org Mode: (org).      Outline-based notes management and organizer.
* VHDL Mode: (vhdl-mode).       Emacs mode for editing VHDL code.
* Emacs FAQ: (efaq).            Frequently Asked Questions about Emacs.
* Emacs: (emacs).               The extensible self-documenting text editor.
* Emacs Lisp Intro: (eintr).    A simple introduction to Emacs Lisp programming.
* Elisp: (elisp).               The Emacs Lisp Reference Manual.
* Emacs GnuTLS: (emacs-gnutls). The Emacs GnuTLS integration.
* ERC: (erc).                   Powerful and extensible IRC client for Emacs.
* EUDC: (eudc).                 Emacs client for directory servers (LDAP, BBDB).
* Gnus: (gnus).                 The newsreader Gnus.
* Mairix: (mairix-el).          Emacs interface to the Mairix mail indexer.
* Message: (message).           Mail and news composition mode that
                                  goes with Gnus.
* MH-E: (mh-e).                 Emacs interface to the MH mail system.
* Newsticker: (newsticker).     A feed reader for Emacs.
* PGG: (pgg).                   An obsolete Emacs interface to various
                                  PGP implementations.
* Rcirc: (rcirc).               Internet Relay Chat (IRC) client.
* SASL: (sasl).                 The Emacs SASL library.
* SC: (sc).                     Supercite lets you cite parts of messages
                                  you're replying to, in flexible ways.
* Sieve: (sieve).               Managing Sieve scripts in Emacs.
* Tramp: (tramp).               Transparent Remote Access, Multiple Protocol
                                  Emacs remote file access via ssh and scp.
* Denote Sequence: (denote-sequence).
                                Sequence notes or Folgezettel with Denote.
* Denote: (denote).             Simple notes with an efficient file-naming
                                  scheme.
* Eat: (eat).                   Emulate A Terminal.
* Embark: (embark).             Emacs Mini-Buffer Actions Rooted in Keymaps.
* Forge: (forge).               Access Git Forges from Magit.
* Ghub: (ghub).                 Client library for the Github API.
* Magit: (magit).               Using Git from Emacs with Magit.
* Magit-Section: (magit-section).
                                Use Magit sections in your own packages.
* Orderless: (orderless).       Completion style for matching regexps in any
                                  order.
* Tempel: (tempel).             Simple templates for Emacs.
* With-Editor: (with-editor).   Using the Emacsclient as $EDITOR.

Taking Rich Screenshots with Wezterm

2025-08-22T00:00:00+00:00

Taking Rich Screenshots with Wezterm¶

When working on this site, I’ve often wanted to include a screenshot or two. But since git doesn’t work well with binary files and I don’t want to have to deal with media hosting I’ve tended to shy away from it.

However, a lot of what I want to take a screenshot of can be rendered in a terminal.

Which is suddenly quite interesting as a terminal’s contents can be entirely described in plain(-ish) text. Surely it must be possible to capture the contents of a terminal window and then render it as some kind of image? Perhaps an SVG?

Yes it is!

In this blog post I describe how I arrived at the workflow I used to include the above screenshot on this website.

Taking the screenshot¶

I’ll admit it, this whole idea started out as a solution in search of a problem!

I’ve been using WezTerm as my terminal emulator for a while now, but I wasn’t taking advantage of the fact that it is configured using Lua and so I was looking for an excuse to try something non-trivial.

Before long I found it - pane:get_lines_as_escapes().

The above function returns the current contents of the terminal pane as a string, including all the escape codes that control the formatting! By mashing together a couple of examples from the documentation I soon had a custom command I could invoke through the command palette which would dump the contents of the active pane to a text file.

wezterm.on('augment-command-palette', function(window, pane)
  return {
    {
      brief = 'Take screenshot',
      icon = 'cod_device_camera',

      action = act.PromptInputLine {
        description = 'Screenshot name',
        action = wezterm.action_callback(function(window, pane, line)
          if line then
            local text = pane:get_lines_as_escapes()
            local f = io.open(line .. '.txt', 'w+')

            f:write(text)
            f:flush()
            f:close()
          end
        end),
      },
    },
  }
end)

Flatpak permissions…

I have the Flatpak version of WezTerm installed.

This led to a bizarre issue where the call to io.open() kept failing silently by returning nil. Apparently, in order to get an actual error message you need to wrap the call in an assert().

Eventually, I was able to spot that WezTerm was complaining that the filesystem was read-only. Thankfully, the KDE System Settings app allowed me to grant WezTerm write access to my filesystem without too much hassle.

After running the command and giving it a filename I was left with a plain-ish text file that looked something like the following

\x1b[38:2::255:255:255m\x1b[48:2::49:49:49m|\x1b(B\x1b[0;1m\x1b[38:2::255:255:255m\x1b[40m*gud-test*        x\x1b(B\x1b[0m\x1b['z...

Rendering the Screenshot¶

Now that we have the raw data, there are lots of ways we could approach rendering it, for example you could dump it into xterm.js.

However, while I have nothing against JavaScript it’s always nice not to rely on it unnecessarily. Since this is a static screenshot anyway it makes sense to render it offline and embed the result in the page.

Using the rich Python library, it only takes a few lines of code to convert the screenshot from plain-ish text to an SVG.

import io
import pathlib

from rich.console import Console
from rich.text import Text

content = pathlib.Path('<filepath>').read_text()
text = Text.from_ansi(content)

console = Console(
    file=io.StringIO(),
    force_terminal=True,
    record=True,
    width=None,
    height=None,
)
console.print(text, end="")
svg = console.export_svg(title="Example screenshot")

There is one small issue… passing None for the width and height of the Console object means that it will use the dimensions of the current terminal executing the code rather than the dimensions of the terminal the screenshot was captured from!

Width and Height¶

In order for the captured data to be rendered correctly, rich’s Console object needs to use the same width and height as the original terminal.

This means that when we take the screenshot, we also need to capture the dimensions of the terminal pane. Easy enough with WezTerm’s Lua API, but where to store the values?

Rather than try and invent a file format, why not reuse the asciicast format from asciinema? After all a screenshot is just a single frame screencast!

We don’t need to worry too much about the details of the format, the following snippet should be enough for this use case.

{"version": 3, "term": {"cols": "<width>", "rows": "<height>"}}
[0, "o", "<captured_output>"]

Which we can modify our custom WezTerm command to produce.

wezterm.on('augment-command-palette', function(window, pane)
  return {
    {
      brief = 'Take screenshot',
      icon = 'cod_device_camera',

      action = act.PromptInputLine {
        description = 'Screenshot name',
        action = wezterm.action_callback(function(window, pane, line)
          if line then
            local dimensions = pane:get_dimensions()
            local body = {0, "o", pane:get_lines_as_escapes()}

            local header = {
              version = 3,
              term = {
                cols = dimensions.cols,
                rows = dimensions.viewport_rows,
              },
            }

            local f = io.open(line .. '.cast', 'w+')
            f:write(wezterm.json_encode(header) .. '\n')
            f:write(wezterm.json_encode(body))
            f:flush()
            f:close()
          end
        end),
      },
    },
  }
end)

And update our Python code accordingly

fpath = pathlib.Path('<filepath>')
with fpath.open() as f:
   header = json.loads(f.readline())
   body = json.loads(f.readline())

text = Text.from_ansi(body[-1])

console = Console(
    file=io.StringIO(),
    force_terminal=True,
    record=True,
    width=header["term"]["cols"],
    height=header["term"]["rows"],
)
console.print(text, end="")
svg = console.export_svg(title=self.options.get("title", ""))

At this point you’d think it would be job done, but there is one more detail to consider…

My terminal colours are not being preserved!

SGR Codes¶

I’m not the right person to try and explain how terminal rendering works. I only know enough to say that the story is long and built up using many layers of escape codes…

The codes that matter for this tale are something called SGR codes and they control various formatting properties including colour. Most documentation I’ve found will say that to set the text colour to… red for example, you need to use the following escape sequence:

\x1b[38;2;255;0;0m

However, there are some places that specify an alternative representation:

\x1b[38:2::255:0:0m

It turns out that WezTerm uses this second style, which rich does not recognise, resulting in the colourless image you saw above. To fix this, I tweaked the way rich parsed escape codes to allow for both styles, resulting in the image you see below.

Which is better, but it still isn’t an accurate representation of what I see in my terminal.

Colour Palette¶

Aside from providing specific RGB values, SGR codes also allow for applications to choose from a palette of 16 colours - 8 “normal” and 8 “bright” colours.

The specific colours in this palette are of course dependent on whatever theme you have configured in your terminal emulator, so the final step is to ensure that rich uses the same colour palette as my terminal.

Thankfully the asciicast format supports recording this information alongside the width and height, all we need to do is extend the screenshot command in WezTerm to populate it.

local dimensions = pane:get_dimensions()
local theme = wezterm.get_builtin_color_schemes()[window:effective_config().color_scheme]
local body = {0, "o", pane:get_lines_as_escapes()}

local header = {
  version = 3,
  term = {
    cols = dimensions.cols,
    rows = dimensions.viewport_rows,
    theme = {
      fg = theme.foreground,
      bg = theme.background,
      palette = table.concat(append_tables(theme.ansi, theme.brights), ':'),
    },
  },
}

Expand to see the complete command implementation

wezterm/commands/screenshot.lua

local wezterm = require 'wezterm'
local io = require 'io'
local os = require 'os'
local act = wezterm.action

local M = {}

local function append_tables(...)
 local result = {}
 for _, tbl in ipairs({...}) do
     for i = 1, #tbl do
         result[#result + 1] = tbl[i]
     end
 end
 return result
end

function M.setup()

  wezterm.on('augment-command-palette', function(window, pane)
    return {
      {
        brief = 'Take screenshot',
        icon = 'cod_device_camera',

        action = act.PromptInputLine {
          description = 'Screenshot name',
          -- initial_value = os.date('%Y%m%dT%H%M%S--terminal-screenshot'), -- apparently this requires a nightly build
          action = wezterm.action_callback(function(window, pane, line)
            if line then
              local dimensions = pane:get_dimensions()
              local theme = wezterm.get_builtin_color_schemes()[window:effective_config().color_scheme]
              local body = {0, "o", pane:get_lines_as_escapes()}

              local header = {
                version = 3,
                term = {
                  cols = dimensions.cols,
                  rows = dimensions.viewport_rows,
                  theme = {
                    fg = theme.foreground,
                    bg = theme.background,
                    palette = table.concat(append_tables(theme.ansi, theme.brights), ':'),
                  },
                },
              }

              local f = io.open(line .. '.cast', 'w+')
              f:write(wezterm.json_encode(header) .. '\n')
              f:write(wezterm.json_encode(body))
              f:flush()
              f:close()
            end
          end),
        },
      },
    }
  end)
end

return M

Which of course requires a similar update to the rendering code

term = header["term"]
ansi_colors = term["theme"]["palette"].split(":")
theme = TerminalTheme(
    background=Color.parse(term["theme"]["bg"]).triplet,
    foreground=Color.parse(term["theme"]["fg"]).triplet,
    normal=[Color.parse(col).triplet for col in ansi_colors[:8]],
    bright=[Color.parse(col).triplet for col in ansi_colors[8:]],
)

...

svg = console.export_svg(
    title=self.options.get("title", ""),
    theme=theme,
)

Which, finally, results in the SVG you saw at the start of this blog post!

Sphinx Integration¶

When working on a post for this site, I want to be able to just include the *.cast file produced by WezTerm as I would with any other image

.. termshot:: /images/example.cast
   :title: Example

and have Sphinx perform the conversion to SVG automatically during the build. This can be done by creating a custom directive.

The following class defintion defines the interface for the directive, we want it accept a single argument (/images/example.cast) and have the option to specify a :title:.

class Termshot(SphinxDirective):
    required_arguments = 1
    optional_arguments = 0
    final_argument_whitespace = True

    option_spec = {
        "title": directives.unchanged,
    }

Each directive has a run() method, which should return a list of nodes which are inserted into the document at the location where the directive was used.

Other Sphinx directives like .. figure:: or .. include:: interpret absolute paths as paths relative to the project root. I want the .. termshot:: directive to behave in the same way, so I pass the given filepath through the self.env.relfn2path function which handles that for me

def run(self):
    _, fpath = self.env.relfn2path(self.arguments[0])

Once the svg has been generated, it needs to be included in the final HTML somehow, there are many way to approach this but for simplicity I’ve opted to inline the SVG code using a raw docutils node.

svg = console.export_svg(
    title=self.options.get("title", ""),
    theme=theme,
)

node = nodes.raw("", svg, format="html")
return [node]

Expand to see the complete implementation

from __future__ import annotations

import io
import json
import pathlib
import typing
from os.path import abspath, relpath

from docutils import nodes
from docutils.parsers.rst import directives
from rich.color import Color
from rich.console import Console
from rich.terminal_theme import TerminalTheme
from rich.text import Text
from sphinx.util.docutils import SphinxDirective

if typing.TYPE_CHECKING:
    from sphinx.application import Sphinx


class Termshot(SphinxDirective):
    required_arguments = 1
    optional_arguments = 0
    final_argument_whitespace = True

    option_spec = {
        "title": directives.unchanged,
    }

    def run(self):
        _, fpath = self.env.relfn2path(self.arguments[0])
        with pathlib.Path(fpath).open() as f:
            header = json.loads(f.readline())
            body = json.loads(f.readline())

        term = header["term"]
        ansi_colors = term["theme"]["palette"].split(":")

        text = Text.from_ansi(body[-1])
        theme = TerminalTheme(
            background=Color.parse(term["theme"]["bg"]).triplet,
            foreground=Color.parse(term["theme"]["fg"]).triplet,
            normal=[Color.parse(col).triplet for col in ansi_colors[:8]],
            bright=[Color.parse(col).triplet for col in ansi_colors[8:]],
        )

        console = Console(
            file=io.StringIO(),
            force_terminal=True,
            record=True,
            width=term["cols"],
            height=term["rows"],
        )
        console.print(text, end="")
        svg = console.export_svg(
            title=self.options.get("title", ""),
            theme=theme,
        )

        node = nodes.raw("", svg, format="html")
        return [node]


def setup(app: Sphinx):
    app.add_directive("termshot", Termshot)
    return {"version": "1.0", "parallel_read_safe": True}

Blogging with Sphinx and denote.el

2025-05-17T00:00:00+00:00

Blogging with Sphinx and `denote.el`¶

If you have spent any time following the Emacs community, you will have likely come across the denote.el project. denote.el defines a clever file-naming scheme and provides an associated Emacs package containing utilities for managing files which follow this naming scheme.

In this blog post I outline how I have adopted denote.el to manage the content on this site and how I’ve extended Sphinx to take advantage this.

`denote.el` basics¶

For an in-depth introduction to denote.el be sure to check out this video however, as mentioned in the introduction, the core of denote.el is the file-naming sheme:

<timestamp>--<title>__<tags>

where

<timestamp> captures the date and time the file was created and acts as the file’s unique identifier
<title> a lowercase - separated string which captures your traditional file name
<tags> (called keywords by denote) is a _ separated string of tag names

As an example the filename for this blog post is:

20250217T182726--blogging-with-sphinx-and-denoteel__blog_blogging_denote_emacs_sphinx.rst

The main benefit in naming your files in this way is that you can perform some fairly sophisticated queries across your files with simple text based searches.

reStructuredText and `denote.el`¶

While I don’t use Emacs all the time, it’s safe to say that it has become my default editor when working on personal projects. So it’s nice to be able to make use of the utilities provided by the denote.el package when working on this site.

One of the main features provided by the package is to insert front matter into your notes corresponding to the information encoded in the filename and to keep the two in sync when either one changes.

denote.el has built in support for several markup formats however, the reStructuredText syntax used by Sphinx is not one of them. Of course, being an Emacs package this isn’t something that a few lines of lisp cannot solve!

What about Markdown?

Yes, both denote.el and Sphinx support Markdown however, I simply prefer reStructuredText 😅

(use-package denote
  :ensure t
  :hook ((dired-mode . denote-dired-mode))
  :config

  ;; Add reStructuredText support to denote
  (add-to-list 'denote-file-types `(rst
                                    :extension ".rst"
                                    :date-key-regexp "^:date:"
                                    :date-value-function denote-date-iso-8601
                                    :date-value-reverse-function denote-extract-date-from-front-matter
                                    :front-matter ":title: %s\n:date: %s\n:tags: %s\n:identifier: %s\n\n"
                                    :title-key-regexp "^:title:"
                                    :title-value-function identity
                                    :title-value-reverse-function denote-trim-whitespace
                                    :keywords-key-regexp "^:tags:"
                                    :keywords-value-function ,(lambda (ks) (string-join ks ", "))
                                    :keywords-value-reverse-function denote-extract-keywords-from-front-matter
                                    :identifier-key-regexp "^:identifier:"
                                    :identifier-value-function identity
                                    :identifier-value-reverse-function denote-trim-whitespace
                                    :link ":denote:link:`%2$s <%1$s>`"
                                    :link-in-context-regexp ,(concat ":denote:link:`.*?<\\(?1:" denote-id-regexp "\\)>`"))))

Which gives denote the information it needs to read and write its metadata using reStructuredText’s field list syntax.

:title: Blogging with Sphinx and denote.el
:date: 2025-05-17
:tags: blog, blogging, denote, emacs, sphinx
:identifier: 20250217T182726

It also instructs denote to use a role called denote:link when inserting a link to another note, but we’ll come back to that a bit later on.

Sphinx and `denote.el`¶

Again, denote is primarily a file-naming scheme, so you don’t have to do anything special to get it to work with Sphinx - it will just work^TM. However, by extending Sphinx I get to smooth off some rough edges and take advantage of the metadata in the filename to build some nice features.

Note

I’m only going to elaborate on some aspects of the code underpinning this, so if you want the full details be sure to take a look at the complete implementation of the denote extension.

The Record dataclass captures all of the information encoded in and derived from a denote style filename.
The Denote Sphinx Domain is the main store of Record instances and can be thought of as the “backend” of the extension.
The rest of the extension uses the APIs provided by the two above classes to integrate with Sphinx at various points in the build lifecycle.

Discovering Content¶

Thanks to the source-read event it’s trivial to build an index of all the files in the Sphinx project that have a denote style filename

def discover_records(app: Sphinx, docname: str, content: list[str]):
    """Discover and index records based on their filename"""

    docpath = pathlib.Path(docname)
    if (record := Record.parse(docpath.name)) is None:
        return

    domain: Denote = app.env.domains["denote"]
    domain.add_record(docname, record)

Once Sphinx has parsed the file the doctree-read event is emitted, which we can use to extract additional information from the file’s front matter and content.

def parse_records(app: Sphinx, doctree):
    """Extract additional information from a document's content"""

    docname = app.env.docname
    domain: Denote = app.env.domains["denote"]
    metadata = app.env.metadata.get(docname, {})

    if (record := domain.records.get(docname)) is None:
        return

    if (title := doctree.next_node(condition=nodes.title, descend=True)) is not None:
        record.title = title.astext()

    if (date := metadata.get("date")) is not None:
        record.timestamp = datetime.fromisoformat(date)

        # Assume UTC if no timezone available
        if record.timestamp.tzinfo is None:
            record.timestamp = record.timestamp.replace(tzinfo=UTC)

I primarily use this to change the post date or title without having to change the identifier or title portions of the filename - which would lead to broken links.

Using Tags to Mark Blog Posts¶

There are currently two main sections to this site - blog posts and notes (the code section is broken and we can talk about dotfiles another time! 😅).

The obvious solution to this would be to create a blog/ folder and a notes/ folder and call it a day. However, denote lends itself well to having just a folder full of files and I quite liked the idea of dumping all my files into a content/ folder and using tags to… ahem denote different types of content.

This allows for a nice workflow where something can start as a note but can easily be promoted to a blog post if needed.

Implementing this quite straightforward, choose a name for the tag (blog) and when it is found set the corresponding flag on the Record instance. Note that I also remove the blog tag from the list of tags so that it does not appear in the list of tags in the sidebar.

if (match := FILENAME_PATTERN.match(filename)) is None:
    return None

tags = match.group("tags").split("_")

try:
    tags.remove("blog")
    is_blogpost = True
except ValueError:
    is_blogpost = False

return Record(
    ...,
    tags=tags,
    is_blogpost=is_blogpost,
)

Then make use of the flag when ever it’s relevant, for example when building the index of all blog posts

def add_record(self, docname: str, record: Record):
    """Add a record to the domain"""
    record.docname = docname
    self.records[docname] = record

    if record.is_blogpost:
        self.posts[docname] = record

Pretty URLs¶

You may have noticed that the url to this page is not

https://www.alcarney.me/content/20250217T182726--blogging-with-sphinx-and-denoteel__blog_blogging_emacs_sphinx/

This is because I have created a custom Sphinx builder, derived from the standard DirectoryHTMLBuilder and which implements the following url scheme

Blog posts are written to /blog/<year>/<title> e.g. /blog/2025/blogging-with-sphinx-and-denoteel
Notes are written to /notes/<identifier> e.g. /notes/20250217T182726

Warning

This involves overwriting the get_target_uri and get_output_path methods on the base builder class, which I am 90% sure are not part of Sphinx’s public API. This can and will break on you between Sphinx versions!

Cross References¶

Remember back in the reStructuredText and denote.el section I instructed denote to use the following syntax when inserting links?

:link ":denote:link:`%2$s <%1$s>`"
:link-in-context-regexp ,(concat ":denote:link:`.*?<\\(?1:" denote-id-regexp "\\)>`"

This was so I could define a :denote:link: role as part of the Denote domain

class Denote(Domain):
    """A domain for denote style note taking."""
    name = "denote"
    roles = {
        "link": XRefRole(),
    }

and implement the resolve_xref() method so that links generated by denote-link and related commands link to the correct page when generating the html for this site

def resolve_xref(
    self,
    env: BuildEnvironment,
    fromdocname: str,
    builder: Builder,
    typ: str,
    target: str,
    node: pending_xref,
    contnode: Element,
) -> Element | None:
    """Resolve cross references"""

    if (record := self.records.find(identifier=target)) is None:
        return None

    if record.docname is None:
        return None

    if (linktext := contnode.astext()) == target:
        contnode = nodes.Text(record.title)
    else:
        contnode = nodes.Text(linktext)

    return make_refnode(
        builder, fromdocname, record.docname, None, [contnode], record.title
    )

Example

Here are links to My Emacs Configuration and my neovim config.

See below for how this looks in the source for this page

.. admonition:: Example

   Here are links to :denote:link:`20250216T190756` and my :denote:link:`neovim config <20250216T190621>`.

   See below for how this looks in the source for this page

   .. literalinclude:: ./20250217T182726==1--blogging-with-sphinx-and-denoteel__blog_blogging_denote_emacs_sphinx.rst
      :language: rst
      :start-at: .. admonition:: Example
      :end-at: Generating

Generating Feeds and Archives¶

This is mainly stolen from inspired by the ablog extension (which is a great extension if you want a Sphinx-powered blog by the way)

Using the html-collect-pages event it’s possible to generate entirely new pages during the build process and since all of denote’s metadata is right there in the filename it’s easy to select just the subsets of files you are interested in

Currently I generate

An index of all blog posts

def generate_collections(app: Sphinx):
    """Generate collections of records according to some criteria"""

    domain: Denote = app.env.domains["denote"]

    # Emit an all blog posts page
    context = {"collection": list(domain.posts.all()), "title": "Blog"}
    yield ("blog", context, "blog/collection.html")

An RSS feed of all blog posts

context.update(
    {
        "baseurl": app.config.blog_baseurl,
        "title": app.config.blog_title,
        "now": datetime.now(tz=UTC),
        "relurl": "blog/atom.xml",
        "sphinx_version": "8",
    }
)
yield ("blog/atom", context, "blog/atom.xml")

An index of all blog posts in a given year

by_year = domain.posts.by_year()
for year, collection in by_year.items():
    context = {"collection": collection, "title": f"Posts in {year}"}
    yield (f"blog/{year}", context, "blog/collection.html")

An index of all blog posts and notes with a given tag

by_tag = domain.records.by_tag()
yield ("tag", {"tags": by_tag}, "blog/tags.html")

nodes = []
links = []
all_records = set()
for tag, collection in by_tag.items():
    nodes.append({"id": tag, "kind": "tag"})

    for r in collection:
        links.append({"source": r.identifier, "target": tag})
        all_records.add(r.identifier)

    context = {"collection": collection, "title": f"Tagged with: {tag}"}
    yield (f"tag/{tag}", context, "blog/collection.html")

I still need to make it so that I can pull out a record’s content in the blog/collection.html template - you may notice that the RSS feed only contains post titles at the moment!

But for the HTML at least I was able to cheat and use HTMX to pull the content through when the post scrolls into view.

<div hx-trigger="intersect once"
     hx-get="{{ pathto(post.docname) }}"
     hx-select=".post-teaser"
     hx-swap="innerHTML ignoreTitle:true"
     style="min-width: 0">
</div>

Which of course isn’t ideal, especially when you realise any links in the inserted content will be broken! 😅

Next Steps¶

As you might guess this is still quite immature and there’s plenty that I’d still like to explore or needs fixing!

Fixing Search

The search bar you see in the sidebar is pretty much useless in its current form. Yes, you can type something in and it will take you to a search results page, but all the links on that page will be broken!
Fixing notes

While I could send you a direct link to one of the notes on this site and you could see it, there’s no actual way to browse them at this time.
denote-sequence

There are many pages on this site which form a series of some kind. However, the only way to handle this currently is for me to remember to add relevant links forwards and backwards in the chain!

Building on a part of the denote file-naming scheme I’ve ignored so far - the signature - the denote-sequence package provides a mechanism for encoding hierarchical sequences of related notes which would be perfect for my use case!
Knowledge Graphs

I doubt views like Obsidian’s graph view are that useful in practise - but they are cool to look at! If nothing else it would be a fun excerise to try and build a similar view for this site.

Intstalling Emacs with systemd-sysext

2025-05-02T00:00:00+00:00

Intstalling Emacs with systemd-sysext¶

One issue with building Emacs from source, or more specifically my hack to ensure that the required system libraries are available on Aurora, is that in some situations the system version of Emacs “wins”.

For example, I wanted to experiment with running the Emacs daemon through systemd:

$ systemctl --user start emacs

Which failed to load my config correctly. It was only after a closer look did I realise that despite specifying the --user flag, systemd was defaulting to the emacs.service file provided by the system emacs package in /usr/... and was therefore running Emacs v29 and not my v30 build.

But then I stumbled across systemd-sysext

Enter systemd-sysext¶

I can’t really tell you what a sysext is… I just know that they are yet another potential way to install software on an image based OS. To quote the extensions.fcos.fr website

systemd system extensions (sysexts) are filesystem images that bundle content (binaries, libraries and configuration files) that can be overlayed on demand on image based systems. The content of the sysexts are “merged” with the base content of the OS using an overlayfs mount.

If you want to know more about sysexts, you can watch the Waiter, an OS please, with some sysext sprinkled on top (video) talk at All systems go! Berlin 2024.

I saw that there was an Emacs layer(?), so thought I may as well give it a go!

Initial Setup¶

It looks like a “proper” frontend to these things is still under development so for now we need to do the setup ourselves. Thankfully the page for the emacs sysext provides the steps for us

Create folders to manage sysexts

This step is only necessary when you set up your very first sysext

$ sudo install -d -m 0755 -o 0 -g 0 /var/lib/extensions /var/lib/extensions.d
$ sudo restorecon -RFv /var/lib/extensions /var/lib/extensions.d

Tell systemd about the sysext

First we create a folder under /etc where we can place a *.conf file

$ sudo install -d -m 0755 -o 0 -g 0 /etc/sysupdate.emacs.d
$ sudo restorecon -RFv /etc/sysupdate.emacs.d

I’m not sure why you would create a folder here per sysext, but I’m sure there must be a good reason to. In this folder, we place an emacs.conf file with the following contents

[Transfer]
Verify=false

[Source]
Type=url-file
Path=https://extensions.fcos.fr/extensions/emacs/
MatchPattern=emacs-@v-%w-%a.raw

[Target]
InstancesMax=2
Type=regular-file
Path=/var/lib/extensions.d/
MatchPattern=emacs-@v-%w-%a.raw
CurrentSymlink=/var/lib/extensions/emacs.raw

Again, I don’t really know the details, but the [Source] block appears to be telling systemd where to fetch the emacs sysext from and the [Target] block seems to be telling it where to put it on disk.

Install the sysext¶

We’ve now given systemd enough information to fetch the sysext:

$ sudo /usr/lib/systemd/systemd-sysupdate update --component emacs
Discovering installed instances…
Discovering available instances…
⤵️ Acquiring manifest file https://extensions.fcos.fr/extensions/emacs/SHA256SUMS…
Pulling 'https://extensions.fcos.fr/extensions/emacs/SHA256SUMS'.
Downloading 406B for https://extensions.fcos.fr/extensions/emacs/SHA256SUMS.
Acquired 406B.
Download of https://extensions.fcos.fr/extensions/emacs/SHA256SUMS complete.
Operation completed successfully.
Exiting.
Determining installed update sets…
Determining available update sets…
Selected update '1-29.4-44.fc41' for install.
Making room for 1 updates…
Removed no instances.
⤵️ Acquiring https://extensions.fcos.fr/extensions/emacs/emacs-1-29.4-44.fc41-41-x86-64.raw → /var/lib/extensions.d/emacs-1-29.4-44.fc41-41-x86-64.raw...
Pulling 'https://extensions.fcos.fr/extensions/emacs/emacs-1-29.4-44.fc41-41-x86-64.raw', saving as '/var/lib/extensions.d/.#sysupdateemacs-1-29.4-44.fc41-41-x86-64.raw6b5d6dfbd735d210'.
Downloading 286.2M for https://extensions.fcos.fr/extensions/emacs/emacs-1-29.4-44.fc41-41-x86-64.raw.
Got 1% of https://extensions.fcos.fr/extensions/emacs/emacs-1-29.4-44.fc41-41-x86-64.raw.
...
Got 98% of https://extensions.fcos.fr/extensions/emacs/emacs-1-29.4-44.fc41-41-x86-64.raw. 268ms left at 17.4M/s.
Acquired 286.2M.
Download of https://extensions.fcos.fr/extensions/emacs/emacs-1-29.4-44.fc41-41-x86-64.raw complete.
Operation completed successfully.
Exiting.
Successfully acquired 'https://extensions.fcos.fr/extensions/emacs/emacs-1-29.4-44.fc41-41-x86-64.raw'.
Successfully installed 'https://extensions.fcos.fr/extensions/emacs/emacs-1-29.4-44.fc41-41-x86-64.raw' (url-file) as '/var/lib/extensions.d/emacs-1-29.4-44.fc41-41-x86-64.raw' (regular-file).
Updated symlink '/var/lib/extensions/emacs.raw' → '../extensions.d/emacs-1-29.4-44.fc41-41-x86-64.raw'.
✨ Successfully installed update '1-29.4-44.fc41'.

However, while we can see the sysext on disk…

$  ls /var/lib/extensions -l
lrwxrwxrwx. 1 root root 50 Apr 27 19:44 emacs.raw -> ../extensions.d/emacs-1-29.4-44.fc41-41-x86-64.raw

Emacs itself is still not available

$ emacs
bash: emacs: command not found

Enabling the sysext¶

The sysext still needs to be enabled - or “merged”

$ sudo systemctl restart systemd-sysext.service

Which, in theory, would finally make Emacs available to my system however running systemd-sysext status showed no extensions

$ systemd-sysext status
HIERARCHY EXTENSIONS SINCE
/opt      none       -
/usr      none       -

And there were errors in the systemd-sysext.service

$ systemctl status systemd-sysext.service
● systemd-sysext.service - Merge System Extension Images into /usr/ and /opt/
     Loaded: loaded (/usr/lib/systemd/system/systemd-sysext.service; enabled; preset: enabled)
    Drop-In: /usr/lib/systemd/system/service.d
             └─10-timeout-abort.conf, 50-keep-warm.conf
     Active: active (exited) since Sun 2025-04-27 19:56:39 BST; 1min 0s ago
 Invocation: 3d34eb06a325454b954ff1697193ba9c
       Docs: man:systemd-sysext.service(8)
    Process: 9596 ExecStart=systemd-sysext refresh (code=exited, status=0/SUCCESS)
   Main PID: 9596 (code=exited, status=0/SUCCESS)
   Mem peak: 4.1M
        CPU: 34ms

Apr 27 19:56:38 aurora systemd[1]: Starting systemd-sysext.service - Merge System Extension Images into /usr/ and /opt/...
Apr 27 19:56:39 aurora (sd-merge)[9612]: No suitable extensions found (1 ignored due to incompatible image(s)).
Apr 27 19:56:39 aurora systemd[1]: Finished systemd-sysext.service - Merge System Extension Images into /usr/ and /opt/.

Digging Deeper¶

Without much to go on I started googling around for some more information.

What’s in a sysext?¶

Eventually I found myself on the manpage for systemd-sysext, trying to get a feel for what this *.raw file I downloaded actually looked like.

System extension images may be provided in the following formats:

Plain directories or btrfs subvolumes containing the OS tree

Disk images with a GPT disk label, following the Discoverable Partitions Specification

Disk images lacking a partition table, with a naked Linux file system (e.g. erofs, squashfs or ext4)

Oh, so some kind of file system then?

Digging around in the GitHub Action workflows of the fedora-sysexts project, I found where the *.raw files were actually produced:

${SUDO} mkfs.erofs -z{{compression}} {{name}}-${version}-${version_id}-${arch}.raw rootfs > /dev/null

Seeing this I wondered if I could just mount this file and take a look around:

$ sudo mkdir /run/media/img
$ sudo mount /var/lib/extensions.d/emacs-1-29.4-44.fc41-41-x86-64.raw /run/media/img
$ tree -L 2 /run/media/img
/run/media/img/
└── usr
    ├── bin
    ├── etc
    ├── include
    ├── lib
    ├── lib64
    ├── libexec
    └── share

Sure enough, it was just a folder hierarchy that contained an installation of Emacs. While it was nice to get some more details on what I was dealing with, it didn’t actually help me get any closer to figuring out what was wrong - so I kept reading.

Compatibility Checks¶

There is a simple check performed when merging a sysext in an attempt to ensure it’s compatible with the base system.

A system extension image must carry a /usr/lib/extension-release.d/extension-release.NAME file, which must match its image name, that is compared with the host os-release file: the contained ID= fields have to match unless "_any" is set for the extension.

And here was my issue:

$ grep '^ID=' /etc/os-release
ID=aurora

$ grep '^ID=' /run/media/img/usr/lib/extension-release.d/extension-release.emacs
ID="fedora"

As far as systemd was concerned I am running an incompatible OS, so it refuses to enable the sysext.

Hacking it in¶

Just to check that my assumption was correct, I edited my /etc/os-release file so that the ID field was set to fedora and tried restarting the systemd-sysext.service again:

Apr 27 23:26:46 aurora systemd[1]: Starting systemd-sysext.service - Merge System Extension Images into /usr/ and /opt/...
Apr 27 23:26:46 aurora (sd-merge)[15118]: Using extensions 'emacs'.
Apr 27 23:26:46 aurora (sd-merge)[15118]: Merged extensions into '/usr'.
Apr 27 23:26:46 aurora systemd[1]: Finished systemd-sysext.service - Merge System Extension Images into /usr/ and /opt/.

Which looked promising! I could even see the extension now using the systemd-sysext command:

$ systemd-sysext list
NAME  TYPE PATH                          TIME
emacs raw  /var/lib/extensions/emacs.raw Sat 2025-04-26 14:30:33 BST

And the emacs command was available!

$ emacs --version
GNU Emacs 29.4
Copyright (C) 2024 Free Software Foundation, Inc.
GNU Emacs comes with ABSOLUTELY NO WARRANTY.
You may redistribute copies of GNU Emacs
under the terms of the GNU General Public License.
For more information about these matters, see the file named COPYING.

The only issue now is that it’s still not Emacs v30! While I saw both Emacs v29 and v30 builds were available as sysexts I didn’t think to check which fedora versions they were built for! 🤦‍♂️

emacs-1-29.4-44.fc41-41-aarch64.raw
emacs-1-29.4-44.fc41-41-x86-64.raw
emacs-1-30.1-11.fc42-42-aarch64.raw
emacs-1-30.1-11.fc42-42-x86-64.raw

It turns out Emacs v30 is only available on Fedora 42 - regardless of the installation method used! 😅

TIL: Navigating History with Magit

2025-02-27T00:00:00+00:00

TIL: Navigating History with Magit¶

Building on my previous discovery, today I figured out a few ways of exploring the history of a project with magit. This issue provided a nice motivating example to have in mind - the exceptiongroup Python package was dropped from a requirements.txt file - in which commit did it happen?

From `magit-dispatch`¶

From any file in the repository you can

Invoke magit-dispatch (C-x M-g)
Open the log menu (l)
Limit the log to commits affecting a specific file (--) and enter the filename(s) at the minibuffer prompt (code/requirements-libs.txt) in my case
Generate the log for the current branch (l)

Which resulted in the following buffer:

Commits in develop touching code/requirements-libs.txt
a0b08399 * build(deps): bump websockets from 14.1 to 14.2 in /code
d781cdc0 * code: Update dependencies
f88f8ad5 * build(deps): bump tomli from 2.0.2 to 2.2.1 in /code
83e810c6 * code: Update bundled dependencies
d6462b20 * code: Bundle a basic Sphinx env with the extension

Then with point on one of the commits, hitting SPC will open the corresponding diff in a new window:

commit d781cdc0 limited to file code/requirements-libs.txt
d781cdc087a1846ecc13df36d6ee4b6ad383a3b9
Author:     Alex Carney <alcarneyme@gmail.com>
AuthorDate: Sat Jan 18 20:57:09 2025 +0000
Commit:     Alex Carney <alex.rugby101@gmail.com>
CommitDate: Thu Jan 23 19:56:08 2025 +0000

Parent:     19be4d6b devenv: Use `uv` for dependency resolution
Contained:  develop release
Follows:    esbonio-vscode-extension-v0.96.1 (9)
Precedes:   esbonio-language-server-v1.0.0b10 (32)

code: Update dependencies

1 file changed, 5 insertions(+), 9 deletions(-)
code/requirements-libs.txt | 14 +++++---------

modified   code/requirements-libs.txt
@@ -1,16 +1,12 @@
-#
-# This file is autogenerated by pip-compile with Python 3.9
-# by the following command:
-#
-#    pip-compile --generate-hashes --output-file=requirements-libs.txt requirements-libs.in
-#
+# This file was autogenerated by uv via the following command:
+#    uv pip compile --prerelease=allow --generate-hashes --python-version 3.9 --output-file requirements-libs.txt requirements-libs.in
 aiosqlite==0.20.0 \
     --hash=sha256:36a1deaca0cac40ebe32aac9977a6e2bbc7f5189f23f4a54d5908986729e5bd6 \
     --hash=sha256:6d35c8c256637f4672f843c31021464090805bf925385ac39473fb16eaaca3d7
     # via -r requirements-libs.in
-attrs==24.2.0 \
-    --hash=sha256:5cfb1b9148b5b086569baec03f20d7b6bf3bcacc9a42bebf87ffaaca362f6346 \
-    --hash=sha256:81921eb96de3191c8258c199618104dd27ac608d9366f5e35d011eae1867ede2
+attrs==24.3.0 \
+    --hash=sha256:8f5c07333d543103541ba7be0e2ce16eeee8130cb0b3f9238ab904ce1e85baff \
+    --hash=sha256:ac96cd038792094f438ad1f6ff80837353805ac950cd2aa0e0625ef19850c308
     # via
     #   cattrs
     #   lsprotocol

[back]

However, the magit-log: buffer will still be focused - it turns out navigating up and down the list with n and p will automatically refresh the diff to match the new commit under point!

From `magit-file-dispatch`¶

A perhaps faster way to acheive the same result is to

Navigate to the file of interest code/requirements-libs.txt
Invoke magit-file-dispatch (C-c M-g)
Select log (l)

Navigating history¶

Another nice set of commands in the magit-file-dispatch menu are in the Navigate column:

File actions   Inspect                               Navigate        More actions
   s Stage      D Diff...   L Log...   B Blame...     p Prev blob     c Commit
   u Unstage    d Diff      l Log      b Blame        n Next blob     e Edit line
 , x Untrack                t Trace    m Blame echo   v Goto blob
 , r Rename                                           V Goto file
 , k Delete                                           g Goto status
 , c Checkout                                         G Goto magit

Using n or p you can walk through the history of the current file, or if you know the commit/branch/tag, jump to it directly with v.

Then if you wish you can view the diff for the current revision with d.

Now I wonder if it’s possible to integrate n and p with repeat-mode… 🤔

TIL: Additional magit keybindings

2025-02-21T00:00:00+00:00

TIL: Additional magit keybindings¶

It always helps to spend the time now and again to read the manual for the tools you use! By simply reading the Getting Started section of the magit user manual I found not one but two commands I had never come across before!

Despite using magit on and off for years I had only known about the magit-status command (bound to C-x g by default). However, magit also provides you with the following bindings

magit-dispatch (C-x M-g)

This gives you immediate access to any of the commands you would normally invoke from the magit status buffer!

Transient and dwim commands
 A Apply           i Ignore           r Rebase
 b Branch          I Init             t Tag
 B Bisect          j Display status   T Note
 c Commit          J Display buffer   V Revert
 C Clone           l Log              w Apply patches
 d Diff            L Log (change)     W Format patches
 D Diff (change)   m Merge            X Reset
 e Ediff (dwim)    M Remote           y Show Refs
 E Ediff           N Forge            Y Cherries
 f Fetch           o Submodule        z Stash
 F Pull            O Subtree          Z Worktree
 h Help            P Push             ! Run
                   Q Command

magit-file-dispatch (C-c M-g)

As the name suggests, this opens a transient containing commands that can be applied to the current file

File actions   Inspect                               Navigate        More actions
   s Stage      D Diff...   L Log...   B Blame...     p Prev blob     c Commit
   u Unstage    d Diff      l Log      b Blame        n Next blob     e Edit line
 , x Untrack                t Trace    m Blame echo   v Goto blob
 , r Rename                                           V Goto file
 , k Delete                                           g Goto status
 , c Checkout                                         G Goto magit

Notes for December/January

2025-02-09T00:00:00+00:00

Notes for December/January¶

Somehow it is already Feburary…

So it’s about time I tried another one of these month note things, especially since the last one was in November.

pygls¶

There has been an issue in pygls which has affected esbonio for a while. In December I finally got around to trying to fix it and I opened a pull request introducing a new way of handling request handlers.

Some more testing is needed before you can call it ready, but I’m optimistic it’s going to pan out.

esbonio¶

I also managed to put a fair amount of work into the upcoming v1 of esbonio during January

Added a new ${venv:<path>} config variable, finally making it possible to share pythonCommand settings for projects that use virtual environments directly
Made it possible to customize how often Sphinx builds are triggered
Re-implemented completion suggestions for directive arguments
Fixed quite a few bugs which made their way into the v1.0.0b10 release this week.

TIL: Debugging go programs with gdb

2025-01-03T00:00:00+00:00

TIL: Debugging go programs with `gdb`¶

$ gdb --args podman info
(gdb) run
Starting program: /usr/bin/podman info
[Thread debugging using libthread_db enabled]
...
WARN[0000] Found incomplete layer "6748b9a4e6f2d75763acd0d8351dfb92d4281a0b3546cc68042fb8cb5e1b7c3c", deleting it

Thread 9 "podman" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fbacbfff6c0 (LWP 10496)]
Downloading source file /usr/src/debug/podman-5.3.1-1.fc41.x86_64/vendor/github.com/containers/storage/layers.go
github.com/containers/storage.(*layerStore).load (r=0xc00024c140, lockedForWriting=true, ~r0=<optimized out>, ~r0=<optimized out>, ~r1=..., ~r1=...) at /usr/src/debug/podman-5.3.1-1.fc41.x86_64/vendor/github.com/containers/storage/layers.go:917
917                     if layer.Flags == nil {

<Freeze frame, record scratch>

You might be wondering how I got here…

And no, I wasn’t trying to work on podman!

How I got here¶

Note

This section has nothing to do with using gdb, I just find it amusing how many tangents I let myself take! Feel free to skip this and start with the next section

I was working on a PR for esbonio, and ran into some test failures that I couldn’t easily figure out from the CI logs alone. Trouble was, the tests all relate to my arch nemesis - Windows file paths.

No problem! I’ll use quickemu to fire up a Windows VM

$  quickemu --vm windows-11.conf
ERROR! Requested 'spicy' as viewer, but 'spicy' is not installed.

Oh yes, I haven’t spun up a VM on this machine yet so don’t have a frontend installed. I’ve used remote-viewer in the past, let’s install that… umm… which package provides this again?

It was in this moment he made a grave mistake

Rather than do the sane thing and Google it, I thought this would be a good chance to ask my local AI model for the dnf incantation to search for the package that provides the remote-viewer command. However, I quickly realised none of my containers were working since almost any podman command resulted in a go panic.

$ podman info
WARN[0000] Found incomplete layer "6748b9a4e6f2d75763acd0d8351dfb92d4281a0b3546cc68042fb8cb5e1b7c3c", deleting it
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x100 pc=0x55fab5c85217]

goroutine 1 gp=0xc0000061c0 m=16 mp=0xc00050f508 [running]:
panic({0x55fab6c906e0?, 0x55fab7bac9e0?})
        /usr/lib/golang/src/runtime/panic.go:804 +0x168 fp=0xc00005eda8 sp=0xc00005ecf8 pc=0x55fab529aa88
runtime.panicmem(...)
        /usr/lib/golang/src/runtime/panic.go:262
runtime.sigpanic()

[Many, many more lines omitted...]

After a quick rummage around some issue trackers I found that not only was this a known issue, it has a fix. Unfortunately the fix hasn’t made it into my version of podman yet, so as a workaround, I will have to remove the incomplete layer myself.

<Insert Thanos meme here>

And since I don’t know which file(s)/folder(s) need deleting, I thought the quickest, most fun way to figure this out would be to debug the process.

Which brings us onto the subject of this blog post!

Setup¶

Since I’m using the -dx version of Aurora, I found that I already had gdb installed. From there all you have to do is pass it the program (and any arguments) you want it to run

$ gdb --args podman
GNU gdb (Fedora Linux) 15.2-3.fc41
Reading symbols from podman...

This GDB supports auto-downloading debuginfo from the following URLs:
  <https://debuginfod.fedoraproject.org/>
Enable debuginfod for this session? (y or [n])

You want to say yes to this!

Debuginfod has been enabled.
To make this setting permanent, add 'set debuginfod enabled on' to .gdbinit.
Downloading 70.28 M separate debug info for /usr/bin/podman
Reading symbols from /home/alex/.cache/debuginfod_client/6c1be8511a45f7f8eef6df8bc94c20641d96eb9a/debuginfo...
/var/home/alex/info: No such file or directory.
warning: Missing auto-load script at offset 0 in section .debug_gdb_scripts
of file /var/home/alex/.cache/debuginfod_client/6c1be8511a45f7f8eef6df8bc94c20641d96eb9a/debuginfo.
Use `info auto-load python-scripts [REGEXP]' to list them.

Once enabled, gdb will be able to download any debug symbols it needs automatically, on demand!

(gdb) run
Starting program: /usr/bin/podman
Downloading separate debug info for system-supplied DSO at 0x7f6e30ef1000
Downloading 188.48 K separate debug info for /lib64/libresolv.so.2
Downloading 187.84 K separate debug info for /lib64/libsubid.so.4
Downloading 94.60 K separate debug info for /var/home/alex/.cache/debuginfod_client/05cb3df691d3ecacb634897526293dabb8834d87/debuginfo
Downloading 1.20 M separate debug info for /lib64/libgpgme.so.11
Downloading 2.77 M separate debug info for /var/home/alex/.cache/debuginfod_client/ad0ff4afb064a77890eb5b5dfae388c7598f3180/debuginfo
Downloading 319.12 K separate debug info for /lib64/libseccomp.so.2
Downloading 610.38 K separate debug info for /lib64/libgcc_s.so.1
Downloading 6.22 M separate debug info for /var/home/alex/.cache/debuginfod_client/a120aca96a83fdaa1dbff64ccb67f9b3e46a86fc/debuginfo
Downloading 6.70 M separate debug info for /lib64/libc.so.6
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Downloading 168.53 K separate debug info for /lib64/libaudit.so.1
Downloading 30.00 K separate debug info for /var/home/alex/.cache/debuginfod_client/894985c591daedd80a91b4221c8725fc8aacb96b/debuginfo
Downloading 540.78 K separate debug info for /lib64/libselinux.so.1
Downloading 82.43 K separate debug info for /var/home/alex/.cache/debuginfod_client/a77b40fbe465e7cfd1b53f220a51f41837976591/debuginfo
Downloading 916.87 K separate debug info for /lib64/libsemanage.so.2
Downloading 15.68 K separate debug info for /var/home/alex/.cache/debuginfod_client/92a110d4e6ddc3d7ec8219622107d2ade30338e3/debuginfo
Downloading 129.88 K separate debug info for /lib64/libeconf.so.0
Downloading 5.09 K separate debug info for /var/home/alex/.cache/debuginfod_client/12df41e5399f930a9b1520115cc1f2ed776edfa7/debuginfo
Downloading 392.91 K separate debug info for /lib64/libcrypt.so.2
Downloading 61.04 K separate debug info for /var/home/alex/.cache/debuginfod_client/3b22db61e60f5ba97c4e0bb1cbee6a8107f3f313/debuginfo
Downloading 125.27 K separate debug info for /lib64/libacl.so.1
Downloading separate debug info for /var/home/alex/.cache/debuginfod_client/6e078332ae36e8358da618b723ea4463791c70fc/debuginfo
Downloading separate debug info for /lib64/libattr.so.1
Downloading 7.69 K separate debug info for /var/home/alex/.cache/debuginfod_client/45c3b76fd658b1e86c6ff09d1f06babadf71fa78/debuginfo
Downloading 193.53 K separate debug info for /lib64/libpam.so.0
Downloading separate debug info for /var/home/alex/.cache/debuginfod_client/dc4d461ec91383087257839ec7862ed0f230a765/debuginfo
Downloading 34.12 K separate debug info for /lib64/libpam_misc.so.0
Downloading 299.74 K separate debug info for /lib64/libassuan.so.0
Downloading 555.48 K separate debug info for /lib64/libgpg-error.so.0
Downloading 28.08 K separate debug info for /var/home/alex/.cache/debuginfod_client/1966004cd3bdfb76ca6c281c7ea322057ca67c08/debuginfo
Downloading separate debug info for /lib64/libcap-ng.so.0
Downloading 9.07 K separate debug info for /var/home/alex/.cache/debuginfod_client/32db198012cdf3b3d1fe4db5414ca51ee36474d9/debuginfo
Downloading 1.50 M separate debug info for /lib64/libpcre2-8.so.0
Downloading 77.46 K separate debug info for /var/home/alex/.cache/debuginfod_client/ecf3f9c874cd4aa76ed4bd9e4f4d87d2d2ab9373/debuginfo
Downloading 2.95 M separate debug info for /lib64/libsepol.so.2
Downloading 163.20 K separate debug info for /lib64/libbz2.so.1
Downloading separate debug info for /var/home/alex/.cache/debuginfod_client/e55f754492a153b4068a131340d25bf80918f8cf/debuginfo
Downloading 2.09 M separate debug info for /lib64/libm.so.6

Debugging¶

Setting a Breakpoint¶

It also doesn’t help that the crash has already happened, I want to inspect the state of the program just before the crash. Thankfully, the warning message about the incomplete layer provides the ideal place to set the breakpoint.

After starting a fresh debug session, I can use the break command to set a breakpoint

(gdb) break /usr/src/debug/podman-5.3.1-1.fc41.x86_64/vendor/github.com/containers/storage/layers.go:921
Breakpoint 1 at 0xa63324: file /usr/src/debug/podman-5.3.1-1.fc41.x86_64/vendor/github.com/containers/storage/layers.go, line 921.
(gdb) run
Starting program: /usr/bin/podman info
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
...

Thread 13 "podman" hit Breakpoint 1, github.com/containers/storage.(*layerStore).load (r=0xc000622000, lockedForWriting=true, ~r0=<optimized out>, ~r0=<optimized out>, ~r1=..., ~r1=...) at /usr/src/debug/podman-5.3.1-1.fc41.x86_64/vendor/github.com/containers/storage/layers.go:921
921                             logrus.Warnf("Found incomplete layer %#v, deleting it", layer.ID)
(gdb)

Using the n(ext) and s(tep) commands I was eventually able to step into the function responsible for removing the layer

(gdb) n
921                             logrus.Warnf("Found incomplete layer %#v, deleting it", layer.ID)
(gdb) n
WARN[0014] Found incomplete layer "6748b9a4e6f2d75763acd0d8351dfb92d4281a0b3546cc68042fb8cb5e1b7c3c", deleting it
922                             err := r.deleteInternal(layer.ID)
(gdb) s
github.com/containers/storage.(*layerStore).deleteInternal (r=0xc0004f4000, id=..., ~r0=...) at /usr/src/debug/podman-5.3.1-1.fc41.x86_64/vendor/github.com/containers/storage/layers.go:1888
1888    func (r *layerStore) deleteInternal(id string) error {

Note

It actually took me a few attempts to step into the deleteInternal function.

podman spins up a lot of goroutines and depending on how slow I was, one of the background goroutines would segfault before I could get to it. This would switch the debugger’s context away from the goroutine I was interested in.

I am sure there is a way to switch back but I didn’t try looking it up.

After a few more n commands I arrived at some code that looks like it should be able to tell me where the layers are stored.

1911           os.Remove(r.tspath(id))
1912           os.RemoveAll(r.datadir(id))

And sure enough I was able to locate the directory where the layer was stored

github.com/containers/storage.(*layerStore).tspath (r=0xc000686000, id=...) at /usr/src/debug/podman-5.3.1-1.fc41.x86_64/vendor/github.com/containers/storage/layers.go:1870
1870           return filepath.Join(r.layerdir, id+tarSplitSuffix)
(gdb) p r.layerdir
$7 = 0xc0005a29c0 "/var/home/alex/.local/share/containers/storage/overlay-layers"

Fixing the Panic¶

Turns out the problematic layer (6748b9a4e6f2d75763acd0d8351dfb92d4281a0b3546cc68042fb8cb5e1b7c3c) had been removed, but the record of it still existed in the layers.json metadata file. After manually removing it podman once again worked as expected!

Notes for November 2024

2024-12-04T00:00:00+00:00

Notes for November 2024¶

Doubling my streak of writing notes on the previous month to 2, here are some of the things I worked on in November.

vscode-wasi-pygls¶

This project is a proof of concept demonstrating how a pygls powered language server can be run entirely in the browser by making use of VSCode’s support for WebAssembly.

Thanks to the work I managed to land in pygls in October, this project is now able to use the upstream version of the framework rather than my fork! (PR)
I also pulled in @vscode/wasm-wasi-lsp as a dependency, removing some of the code I had previously taken from the microsoft/vscode-wasm repo (PR)

esbonio¶

esbonio is a language server for your Sphinx documentation projects

I updated esbonio to the latest pygls v2 pre-release

pytest-lsp¶

pytest-lsp is a pytest plugin for writing end-to-end tests for language servers

Updated pytest-lsp to the latest pygls v2 pre-release

lsp-devtools¶

lsp-devtools is an attempt at building some of the developer tools I wished existed when I first started working on esbonio

Dropped support for Python 3.8
Updated lsp-devtools to the latest pygls v2 pre-release

TIL: Inspecting Python Threads

2024-12-01T00:00:00+00:00

TIL: Inspecting Python Threads¶

One of my favourite things about Python is that pretty much everything about the language can be inspected at runtime and today I found out this is true even for threads.

While working on this PR I kept running into issues where the esbonio process would not terminate, despite everything to my knowledge at least, was being shutdown as expected. Having encountered issues like this before, I knew the culprit was likely to be some background thread was stuck on something and keeping the process alive. The problem was I had no idea which thread it could be or what it was doing.

Thanks to this Stack Overflow post, I discovered Python does provide the tools to figure it out.

With a breakpoint set just before the point where the process should be exiting, we can first enumerate all the active threads.

>>> import threading
>>> {t.ident: t.name for t in threading.enumerate()}
{
    140658776004416: 'MainThread',
    140658503321280: 'pydevd.Writer',
    140658492835520: 'pydevd.Reader',
    140658482349760: 'pydevd.CommandThread',
    140658471864000: 'pydevd.CheckAliveThread',
    140658444601024: 'Thread-6',
    140658354423488: 'Thread-7'
}

Ignoring all the threads that appear to be part of the debugger (pydevd.*), the issue is probably in Thread-6 or Thread-7. With the help of the sys module we can get the current stack frame for each thread

>>> import sys
>>> thread_6_stack_frame = sys._current_frames()[140658444601024]
>>> thread_7_stack_frame = sys._current_frames()[140658354423488]

Then using the traceback module we can print the current call stack!

>>> traceback.print_stack(thread_6_stack_frame)
File "/var/home/linuxbrew/.linuxbrew/Cellar/python@3.13/3.13.0_1/lib/python3.13/threading.py", line 1012, in _bootstrap
  self._bootstrap_inner()
File "/var/home/linuxbrew/.linuxbrew/Cellar/python@3.13/3.13.0_1/lib/python3.13/threading.py", line 1041, in _bootstrap_inner
  self.run()
File "/var/home/alex/.vscode/extensions/ms-python.debugpy-2024.12.0-linux-x64/bundled/libs/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_daemon_thread.py", line 53, in run
  self._on_run()
File "/var/home/alex/.vscode/extensions/ms-python.debugpy-2024.12.0-linux-x64/bundled/libs/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_timeout.py", line 43, in _on_run
  self._event.wait(wait_time)
File "/var/home/linuxbrew/.linuxbrew/Cellar/python@3.13/3.13.0_1/lib/python3.13/threading.py", line 656, in wait
  with self._cond:
File "/var/home/linuxbrew/.linuxbrew/Cellar/python@3.13/3.13.0_1/lib/python3.13/threading.py", line 369, in wait
  if not gotit:

Hmm, not so useful.. but then again looking closer at the call stack this thread also seems to be related to the debugger. What about the other thread?

>>> traceback.print_stack(thread_7_stack_frame)
File "/var/home/linuxbrew/.linuxbrew/Cellar/python@3.13/3.13.0_1/lib/python3.13/threading.py", line 1012, in _bootstrap
  self._bootstrap_inner()
File "/var/home/linuxbrew/.linuxbrew/Cellar/python@3.13/3.13.0_1/lib/python3.13/threading.py", line 1041, in _bootstrap_inner
  self.run()
File "/home/alex/.local/share/hatch/env/virtual/esbonio/3pU26gTf/hatch-test.py3.13-sphinx6/lib/python3.13/site-packages/aiosqlite/core.py", line 107, in run
  tx_item = self._tx.get()

Ah! This thread is stuck somewhere in the aiosqlite library, looks like I wasn’t cleaning up all the DB connections like I thought I was!

TIL: Local LLMs with Ollama and gptel

2024-11-16T00:00:00+00:00

TIL: Local LLMs with Ollama and gptel¶

LLMs don’t look like they’re going to go away anytime soon, so I may as well start playing around with them. Since I am also using Emacs more and more it also makes sense to try using it as the interface to any models.

Installing Ollama¶

I’m not terribly interested in messing around with API keys, rate limits and/or billing that seems to come with any of the hosted models. Thankfully Bluefin/Aurorae makes it easy to get Ollama up and running locally.

$ ujust ollama install
$ brew install ollama
$ systemctl --user start ollama

This will make the Ollama API available locally on http://localhost:11434.

Tip

Bluefin/Aurorae also make it easy to spin up an instance of the Open Web UI if you want a Chat GPT style interface

$ ujust ollama install-open-webui
$ systemctl --user start ollama-web

Once everything has spun up, the interface will be available locally on http://localhost:8080

Choosing a Model¶

Installing Ollama by itself however does not give you much, you now have the infrastructure in place to run models and interact with them but it does not come with any models out of the box.

Unfortunately, there are so many models to choose from! I have no idea how to pick between them and most of the benchmark scores are meaningless to me.

However, any model I run will have to run on my CPU so I can let my hardware constraints pick for me. So looking for something small… ah! llama3.2:3b, it’s a new-ish model, 3b parameters and only a 2GB download, seems as good a choice as any. - Any AI is better then no AI… right? 😅

$ ollama pull llama3.2

This was all I needed to do in order for the model to show up in Open Web UI however, as I mentioned in the intro I’m interested in interacting with these models via Emacs.

Installing gptel¶

If you search for packages that involve AI in some way you will quickly discover a lot of them! However, stumbling across this video demo of gptel was enough to sell me on it.

All it takes is a few lines of elisp to point gptel at the local Ollama API.

(package-install 'gptel)
(setq gptel-model 'llama3.2:latest
      gptel-backend (gptel-make-ollama "Ollama"
                      :host "localhost:11434"
                      :stream t
                      :models '(llama3.2:latest)))

And I can now call M-x gptel to open a dedicated chat buffer with the llama3.2 model. Now all I have to do is figure out what these models are actually useful for!

Notes for October

2024-11-03T00:00:00+00:00

Notes for October¶

I want to try and get into the habit of writing more, so taking inspiration from Simon Willison’s weeknotes here’s a quick writeup of what I have been doing in the month of October.

Esbonio¶

esbonio is a language server for your Sphinx documentation projects

Work towards the big 1.0 release continues and in the latest pre-release of the VSCode extension, it should now be possible to preview documentation builds when running on Codespaces. (Issue, PR)

Up until now, Esbonio has required a correctly configured Python environment in order to work at all. However, one of the most common issues I see is people struggling with this initial setup step and Esbonio will refuse to try anything until it is complete.

This isn’t ideal and in reality there is plenty Esbonio could do even if the Python environment is has access to is not complete! So this month I started looking into ways to make the server useful in more situations

The VSCode extension now comes with a default Python enviroment to use when the user does not provide their own (PR)
The language server can now continue to work if one of the required Sphinx extensions is not available (PR)
The language server will fallback to Sphinx’s default alabaster theme if the requested HTML theme is not available (PR)

There will, of course, be many more situations to handle but it’s a start!

pygls¶

pygls is the language server framework I help to maintain and use in Esbonio

Work towards the 2.0 release continues and this month I finally was able to tackle the migration from Python’s low-level to high-level asyncio APIs

I also was able to add TCP and WebSocket support to the language client

Finally, I was able to clean up the library’s support for Pyodide and started thinking about what it would take to embed a simple language client into the docs for demos - disappointingly, it may be harder than I would like!

Setting up a Python WASI Environment with Nix

2023-09-18T00:00:00+00:00

Setting up a Python WASI Environment with Nix¶

In this blog post I look at setting up a local development environment for the WASI build of Python using Nix. You can see the final result here

Why?¶

WASM and Pyodide and WASI, Oh My!

See Brett Cannon’s blog post for a good overview of the different flavours of WebAssembly platforms.

Some time ago now, I stated that I wanted to get the esbonio language server running in vscode.dev. While there has been some progress towards this goal it’s been little more than proof of concepts.

One of the biggest challenges has been sharing the contents of the workspace with the pyodide runtime I was using to host the language server. But now that the VSCode team are turning VSCode into a WASI runtime it might finally be possible to port esbonio to the web!

As a bonus, since WASI is a standard, I can use wasmtime as the host when working on the port locally. Which brings us to the topic of this blog post.

Ok, but why use Nix?¶

I’m sure you could achieve everything that I do here with a bash script but I’ve been playing with Nix a lot recently and I think it’s cool!

If you want Python + WASI without the Nix see this blog post, which I used as the basis for figuring this out.

“Building” Python¶

As an initial step, let’s see if we can get to the point where we can launch a devShell in which we can run the WASI version of Python.

To get the WASM build of Python onto our machine we’ll write a derivation that “builds” it by downloading the release artifact from GitHub and copying it to the output folder

This is “wrong”

To do this the “right” way, I should be using Nix to build the WASI version of Python from source. But since Brett Cannon is publishing builds on GitHub I’m going to leave that as an exercise for the reader.

# In ./nix/python-wasi.nix
{ fetchzip, stdenv }:

stdenv.mkDerivation {
  pname = "python-wasi";
  version = "3.11.4";

  src = fetchzip {
    url = "https://github.com/brettcannon/cpython-wasi-build/releases/download/v3.11.4/python-3.11.4-wasi_sdk-16.zip";
    stripRoot = false;
    sha256 = "sha256-AZGdgpRvHcu6FY/a7capldjDhTpkfhGkqYnm127nAN8=";
  };

  buildCommand = ''
   mkdir $out
   cp -r $src/* $out
  '';
}

In a corresponding flake.nix file we can define a devShell containing both the WASI build of Python and the wasmtime program required to execute it

# In ./flake.nix
utils.lib.eachDefaultSystem(system:
  let
    pkgs = import nixpkgs { inherit system; };
    python-wasi = pkgs.callPackage ./nix/python-wasi.nix {};
  in {

    devShells.default = pkgs.mkShell {
      name = "wasi";
      shellHook = ''
         export PYTHON_WASI=${python-wasi}
      '';
      packages = with pkgs; [ wasmtime ];
    };
  }
);

Which will be enough to give an an environment where we can try to run a simple Python script:

(nix-shell) $ wasmtime run $PYTHON_WASI/python.wasm -- -c "import sys;print(sys.platform)"
Could not find platform independent libraries <prefix>
Could not find platform dependent libraries <exec_prefix>
Python path configuration:
  PYTHONHOME = (not set)
  PYTHONPATH = (not set)
  program name = 'python.wasm'
  isolated = 0
  environment = 1
  user site = 1
  safe_path = 0
  import site = 1
  is in build tree = 0
  stdlib dir = '/usr/local/lib/python3.11'
  sys._base_executable = ''
  sys.base_prefix = '/usr/local'
  sys.base_exec_prefix = '/usr/local'
  sys.platlibdir = 'lib'
  sys.executable = ''
  sys.prefix = '/usr/local'
  sys.exec_prefix = '/usr/local'
  sys.path = [
    '/usr/local/lib/python311.zip',
    '/usr/local/lib/python3.11',
    '/usr/local/lib/python3.11/lib-dynload',
  ]
Fatal Python error: init_fs_encoding: failed to get the Python codec of the filesystem encoding
Python runtime state: core initialized
ModuleNotFoundError: No module named 'encodings'

Current thread 0x00000000 (most recent call first):
  <no Python frame>

Well, that was the plan at least! 😅

Making it work¶

Of course… introducing Nix is going to bring its own set of challenges. Looking at the error message above we can see that Python assumes it has been installed on a traditional Linux operating system and is looking for its standard library under /usr/local. However, looking at the PYTHON_WASI environment variable we can see this is not the case:

(nix-shell) $ echo $PYTHON_WASI
/nix/store/5n0m7jxcnksmnp52maxa4li1q91gwq2v-python-wasi-3.11.4

(nix-shell) $ ls $PYTHON_WASI
lib/  python.wasm*

Thankfully, it’s easy enough to fix with the PYTHONHOME environment variable, we can rename PYTHON_WASI to PYTHONHOME in the shell’s shellHook

shellHook = ''
    export PYTHONHOME=${python-wasi}
''

However, since wasmtime implements a capability security model we also have to grant access to the both the environment variable and the folder it points to:

$ wasmtime run $PYTHONHOME/python.wasm --env PYTHONHOME=$PYTHONHOME --dir $PYTHONHOME -- -c "import sys; print(sys.platform)"
wasi

Success!

Creating a Wrapper¶

While we can now run the WASI Python build, the command to invoke it is rather unwieldy… and we’re not doing anything interesting yet! Thankfully, we can also use Nix to hide some of these details for us.

A common pattern you will see in nixpkgs is to have a <program-name>-unwrapped package whose job it is to build said program into a folder in the nix store. Then a second <program-name> package containing a bash script that handles the details of invoking the program from within the /nix/store.

So let’s do the same here!

# In ./nix/python.nix
{ wasmtime, python-wasi, writeShellScriptBin }:

writeShellScriptBin "python" ''
   ${wasmtime}/bin/wasmtime run ${python-wasi}/python.wasm \
     --env PYTHONHOME=${python-wasi} \
     --dir ${python-wasi} \
     -- "$@"
''

Which allows us to update our devShell definition to the following.

utils.lib.eachDefaultSystem(system:
  let
    pkgs = import nixpkgs { inherit system; };
    python-wasi = pkgs.callPackage ./nix/python-wasi.nix {};
    python = pkgs.callPackage ./nix/python.nix { python-wasi = python-wasi; };
  in {

    devShells.default = pkgs.mkShell {
      name = "wasi";
      packages = [ python ];
    };
  }
 );

Notice how we don’t even need to reference wasmtime here anymore? In fact (assuming the devShell is active), we can call Python as we would normally!:

$ python -c "import sys; print(sys.platform)"
linux

$ nix develop
(nix-shell) $ python -c "import sys; print(sys.platform)"
wasi

🤯

Tip

You can see the contents of the wrapper script we generated by running:

(nix-shell) $ cat $(command -v python)
#!/nix/store/ir0j7zqlw9dc49grmwplppc7gh0s40yf-bash-5.2-p15/bin/bash
/nix/store/kv8y59aqkz8havf9whvvknm7z05by2dk-wasmtime-11.0.1/bin/wasmtime run /nix/store/5n0m7jxcnksmnp52maxa4li1q91gwq2v-python-wasi-3.11.4/python.wasm \
  --env PYTHONHOME=/nix/store/5n0m7jxcnksmnp52maxa4li1q91gwq2v-python-wasi-3.11.4 \
  --dir /nix/store/5n0m7jxcnksmnp52maxa4li1q91gwq2v-python-wasi-3.11.4 \
  -- "$@"

But we’re not done yet! Our Python process does not have access to any files outside of the stdlib

Python 3.11.4 (tags/v3.11.4-dirty:d2340ef, Jun  8 2023, 00:39:32) [Clang 14.0.4 (https://github.com/llvm/llvm-project 29f1039a7285a5c3a9c353d05414 on wasi
Type "help", "copyright", "credits" or "license" for more information.
>>> import os
>>> os.listdir('.')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
PermissionError: [Errno 76] Capabilities insufficient: '.'

But passing an additional --dir . argument to wasmtime will solve that for us. The more challenging issue to solve is passing through third party libraries

Python 3.11.4 (tags/v3.11.4-dirty:d2340ef, Jun  8 2023, 00:39:32) [Clang 14.0.4 (https://github.com/llvm/llvm-project 29f1039a7285a5c3a9c353d05414 on wasi
Type "help", "copyright", "credits" or "license" for more information.
>>> import attrs
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'attrs'

Installing packages¶

Here is where I’m going to take another shortcut, to do this “correctly” I should probably look at overriding the base Python derivation so that it (and everything that depends on it) uses the WASI build of Python. This would trigger Nix to (lazily) rebuild all of the Python packages in nixpkgs against that version of Python.

However, there’s a strong chance that a lot of builds will break and in reality all we really need is a folder full of Python files to import.

What about packages like numpy?

While I think there are some moves towards enabling support for packages like numpy (via the WASM Component Model?), if you want to use packages that contain native code in WebAssembly, pyodide is probably the best bet for the time being, especially for data science libraries.

This however, is beyond the scope of this blog post.

So why not reuse the packages already in nixpkgs?

We can take a list of Python package derivations and construct a string to set as the PYTHONPATH environment variable - making all of the required dependencies available to the interpreter.

Gathering Dependencies¶

The first thing we need to do is to figure out when given a Python package, is what all of its dependencies are. Thankfully, to help us figure it out we can use the Nix REPL allowing us to inspect derivations and evaluate expressions:

$ nix repl
Welcome to Nix 2.11.1. Type :? for help.

nix-repl> :lf .    # Load the flake located in the current directory
Added 9 variables.

First let’s locate a package that we’re interested in, note that the inputs variable corresponds to the inputs attribute set we declared in our flake.nix file:

nix-repl> pkgs = inputs.nixpkgs.legacyPackages.x86_64-linux.python311Packages
nix-repl> pkgs.attrs
«derivation /nix/store/bcgw61xp6ss6vaad5ghwqfbm2m795a2g-python3.11-attrs-22.2.0.drv»

To find out what attributes this derivation has we can type pkgs.attrs. and then hit Tab:

nix-repl> pkgs.attrs.
pkgs.attrs.LANG                         pkgs.attrs.outPath
pkgs.attrs.__ignoreNulls                pkgs.attrs.outputName
pkgs.attrs.__structuredAttrs            pkgs.attrs.outputs
pkgs.attrs.all                          pkgs.attrs.override
pkgs.attrs.args                         pkgs.attrs.overrideAttrs
pkgs.attrs.buildInputs                  pkgs.attrs.overrideDerivation
pkgs.attrs.builder                      pkgs.attrs.overridePythonAttrs
pkgs.attrs.cmakeFlags                   pkgs.attrs.passthru
pkgs.attrs.configureFlags               pkgs.attrs.patches
pkgs.attrs.depsBuildBuild               pkgs.attrs.pname
pkgs.attrs.depsBuildBuildPropagated     pkgs.attrs.postFixup
pkgs.attrs.depsBuildTarget              pkgs.attrs.postInstall
pkgs.attrs.depsBuildTargetPropagated    pkgs.attrs.propagatedBuildInputs
pkgs.attrs.depsHostHost                 pkgs.attrs.propagatedNativeBuildInputs
pkgs.attrs.depsHostHostPropagated       pkgs.attrs.pythonImportsCheck
pkgs.attrs.depsTargetTarget             pkgs.attrs.pythonModule
pkgs.attrs.depsTargetTargetPropagated   pkgs.attrs.pythonPath
pkgs.attrs.disallowedReferences         pkgs.attrs.requiredPythonModules
pkgs.attrs.dist                         pkgs.attrs.src
pkgs.attrs.doCheck                      pkgs.attrs.stdenv
pkgs.attrs.doInstallCheck               pkgs.attrs.strictDeps
pkgs.attrs.drvAttrs                     pkgs.attrs.system
pkgs.attrs.drvPath                      pkgs.attrs.testout
pkgs.attrs.inputDerivation              pkgs.attrs.tests
pkgs.attrs.mesonFlags                   pkgs.attrs.type
pkgs.attrs.meta                         pkgs.attrs.updateScript
pkgs.attrs.name                         pkgs.attrs.userHook
pkgs.attrs.nativeBuildInputs            pkgs.attrs.version
pkgs.attrs.out

Well requiredPythonModules looks interesting:

nix-repl> pkgs.attrs.requiredPythonModules
[ «derivation /nix/store/2dml7fbspshiwb1j896jd3ajrsq81nl5-python3-3.11.4.drv» ]

I guess the only dependencies attrs has is Python itself, that’s good to know as we will have to drop that from the list of dependencies:

nix-repl> lib = inputs.nixpkgs.lib
nix-repl> python = inputs.nixpkgs.legacyPackages.x86_64-linux.python311
nix-repl> python
«derivation /nix/store/2dml7fbspshiwb1j896jd3ajrsq81nl5-python3-3.11.4.drv»

nix-repl> lib.remove python pkgs.attrs.requiredPythonModules
[ ]

Now, how about another package?:

nix-repl> lib.remove python pkgs.pygls.requiredPythonModules
[
  «derivation /nix/store/dsx8aalkf97pi0ja8xs9mlss5lk4bzhv-python3.11-lsprotocol-2023.0.0a2.drv»
  «derivation /nix/store/86933r4czjpmyq3ikz5444f1k1q9rij6-python3.11-typeguard-2.13.3.drv»
  «derivation /nix/store/bcgw61xp6ss6vaad5ghwqfbm2m795a2g-python3.11-attrs-22.2.0.drv»
  «derivation /nix/store/f19ymrh2zv9hzh00wjvjv73v529n3ds4-python3.11-cattrs-23.1.2.drv»
]

Based on some experiments I think requiredPythonModules automatically handles transitive dependencies for us!

Setting `PYTHONPATH`¶

I’ll spare you the trial and error and skip to the following lines of nix code which take a list of pyPackages and converts it into a value we can use with PYTHONPATH

pyDeps = lib.concatMap (pkg: lib.remove pkg.pythonModule pkg.requiredPythonModules) pyPackages;
allPackages = lib.unique (pyPackages ++ pyDeps);
pythonPath = lib.concatMapStringsSep ":" (pkg: "${pkg}/lib/python3.11/site-packages") allPackages;

Don’t forget we will also have to grant the process access to each of the folders we add to the PYTHONPATH

lib.concatMapStringsSep "\\\n  " (pkg: "--dir '${pkg}/lib/python3.11/site-packages' ") allPackages

`mkPythonWASIShell`¶

We now have everything we need to rewrite our ./nix/python.nix file to define a helper function which takes a list of Python packages and creates a devShell containing the WASI build of Python and all of our declared dependencies.

# In ./nix/python.nix

# Dependencies for our function (mostly) coming from `nixpkgs`
{ lib
, python-wasi
, mkShell
, stdenv
, wasmtime
, writeShellScriptBin
}:

# Arguments the user can set when invoking the function
{ pyPackages ? []
}:

# Implementation details
let
  pyDeps = lib.concatMap (pkg: lib.remove pkg.pythonModule pkg.requiredPythonModules) pyPackages;
  allPackages = lib.unique (pyPackages ++ pyDeps);
  pythonPath = lib.concatMapStringsSep ":" (pkg: "${pkg}/lib/python3.11/site-packages") allPackages;

  # This should look familiar...
  python = writeShellScriptBin "python" ''
   ${wasmtime}/bin/wasmtime run ${python-wasi}/python.wasm \
     --env PYTHONHOME=${python-wasi} \
     --env PYTHONPATH='.:${pythonPath}' \
     --dir ${python-wasi} \
     ${lib.concatMapStringsSep "\\\n  " (pkg: "--dir '${pkg}/lib/python3.11/site-packages' ") allPackages} \
     --dir . \
     -- "$@"
  '';

in
# Our function's return value
mkShell {
  name = "python-wasi";
  packages = [ python ];
}

The function only gets a name when we import it into the top-level flake.nix file

pkgs = import nixpkgs { inherit system; };
python-wasi = pkgs.callPackage ./nix/python-wasi.nix {};
mkPythonWASIShell = pkgs.callPackage ./nix/python.nix { python-wasi = python-wasi; };

The callPackage function uses some kind of ✨magic✨ to automatically pass through all the dependencies (like wasmtime) we declared in ./nix/python.nix - except for python-wasi which we pass through by hand.

Defining an environment based on mkPythonWASIShell would then look something like the following

devShells.default = mkPythonWASIShell {
  pyPackages = with pkgs.python311Packages; [
    pygls
  ];
};

Which we can now try to develop with:

$ nix develop
(nix-shell) $ python
Python 3.11.4 (tags/v3.11.4-dirty:d2340ef, Jun  8 2023, 00:39:32) [Clang 14.0.4 (https://github.com/llvm/llvm-project 29f1039a7285a5c3a9c353d05414 on wasi
Type "help", "copyright", "credits" or "license" for more information.
>>> from pygls.server import LanguageServer
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/nix/store/g4p26w5gh74nndclnskypc74ni37jqm6-python3.11-pygls-1.0.1/lib/python3.11/site-packages/pygls/server.py", line 42, in <module>
    from multiprocessing.pool import ThreadPool
ModuleNotFoundError: No module named 'multiprocessing'
>>>

Assuming the packages support the WASI runtime that is! 😅

Next steps¶

If you are interested you can find the final version of the code here. I’m not really sure where it will go from here, as it’s now “good enough” for me to start hacking on the WASI runtime, but there’s certainly plenty that could be improved.

Building Python WASI from source
Automatically choosing the right Python package set to use with mkPythonWASIShell, like python.withPackages
Extending mkPythonWASIShell to allow the user to set the permissions for the Python process
Overriding a base python derivation so that packages are built against this version of Python.
Extending this to work with packages that contain native modules?

Nix: Day to Day Usage

2023-07-27T00:00:00+00:00

Nix: Day to Day Usage¶

This blog post marks a change in my usage of Nix, I’m (just!) past the point of trying to get something to work and now starting to incorporate it into some of my regular workflows. So instead of trying to accomplish some major task, this post is a small collection of things I’ve learned over the past few weeks.

A better `devShell` definition¶

The original issue I’m trying to solve dates back to my first post on using Nix. That is, defining a devShell containing the dependencies of a local Python package doesn’t mean that the local package itself is importable when the devShell is activated.

$ nix develop .#py310
(nix-shell) $ pytest
================================================= test session starts =================================================
platform linux -- Python 3.10.12, pytest-7.2.1, pluggy-1.0.0
rootdir: /var/home/alex/Projects/lsp-devtools/lib/pytest-lsp, configfile: pyproject.toml
plugins: typeguard-3.0.2, asyncio-0.20.3
asyncio: mode=auto
collected 16 items / 1 error

======================================================= ERRORS ========================================================
________________________________________ ERROR collecting tests/test_client.py ________________________________________
ImportError while importing test module '/var/home/alex/Projects/lsp-devtools/lib/pytest-lsp/tests/test_client.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/nix/store/1r6n7v2wam7gkr18gxccpg7p5ywgw551-python3-3.10.12/lib/python3.10/importlib/__init__.py:126: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
tests/test_client.py:9: in <module>
    import pytest_lsp
E   ModuleNotFoundError: No module named 'pytest_lsp'
=============================================== short test summary info ===============================================
ERROR tests/test_client.py
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
================================================== 1 error in 0.16s ===================================================

Initially I tried to solve this by also including the Nix package defined for the local Python package itself in the definition of the devShell but with any tests disabled. While this worked, it wasn’t very useful when trying to do any real development with it.

The problem is that upon activating the shell, Nix will freeze the source as part of the build process. Which means for any edits to take effect, you have to exit the shell and re-enter it to trigger another build to pick up the changes. Not only does this fill your /nix/store with 100s of copies of your project, it gets tedious very quickly!

Since then however, I’ve learned that when installating a Python package, Nix is only adding the /nix/store path for it to the PYTHONPATH environment variable:

(nix-shell) $ echo $PYTHONPATH | tr ':' '\n'
/nix/store/99i2wwkhcgr98kjn5wnr25sb87dk4zkk-python3.10-pygls-1.0.1/lib/python3.10/site-packages
/nix/store/ckmh39zca1gjagq4cmharbvzggcmm4qx-python3.10-lsprotocol-2023.0.0a2/lib/python3.10/site-packages
/nix/store/n80x8k099gfslvbg4s13hpaiiynimsw5-python3.10-attrs-22.2.0/lib/python3.10/site-packages
/nix/store/1r6n7v2wam7gkr18gxccpg7p5ywgw551-python3-3.10.12/lib/python3.10/site-packages
/nix/store/aiabj9kh174a3ybdr00q3zpm7w6vqv99-python3.10-cattrs-22.2.0/lib/python3.10/site-packages
/nix/store/4bv2ic5mbp639xi0r75y5aq3d8yd04qa-python3.10-exceptiongroup-1.1.0/lib/python3.10/site-packages
/nix/store/jxpkywimbcxzmsc604gfgibdvlj8x3ch-python3.10-typeguard-3.0.2/lib/python3.10/site-packages
/nix/store/5vwslcxd6w3ck9dlgf8zw87ha2cnf5zz-python3.10-importlib-metadata-6.0.0/lib/python3.10/site-packages
/nix/store/4s0w0rp502c09f7vngmnwdmxaans4k70-python3.10-toml-0.10.2/lib/python3.10/site-packages
/nix/store/6zrrhy4mv339hd6rhc19immll0qpm9fr-python3.10-zipp-3.15.0/lib/python3.10/site-packages
/nix/store/082nwhxg32ykrc4bcd9wacj1pzgyf7ii-python3.10-typing-extensions-4.5.0/lib/python3.10/site-packages
/nix/store/hzv8xjxk35i72jrljvjhl9y5i00vnsqn-python3.10-pytest-7.2.1/lib/python3.10/site-packages
/nix/store/064q1k7k7g05ls3m7cqdh32nisj51pgw-python3.10-iniconfig-2.0.0/lib/python3.10/site-packages
/nix/store/c5fh1flbs76jpgmvzz96xa26c3fwsq2s-python3.10-packaging-23.0/lib/python3.10/site-packages
/nix/store/0mkyiplpq1iy1y4kvkpj4gwcfism1bkw-python3.10-pluggy-1.0.0/lib/python3.10/site-packages
/nix/store/4k182588zcl6j9n08qmy8395qanxw86r-python3.10-py-1.11.0/lib/python3.10/site-packages
/nix/store/k40s1gy6pkzdzb7l14jhsmfamwjmpgnk-python3.10-tomli-2.0.1/lib/python3.10/site-packages
/nix/store/3k5y2a1my07fpbv1p24a7gplk6nqpnpf-python3.10-pytest-asyncio-0.20.3/lib/python3.10/site-packages

So why not put our local package’s source there as well?

All we need to do is add a shellHook to the devShell’s definiton that adds the working directory to the existing PYTHONPATH:

shellHook = ''
   export PYTHONPATH="./:$PYTHONPATH"
'';

Now the devShell behaves like an editable install of a Python package - no rebuilds required!

Defining a build matrix¶

So far, all my devShell definitions have been making use of a function I wrote called eachPythonVersion (see this section for more details) which would let me define a devShell once, but reuse it across multiple Python versions

devShells = utils.lib.eachDefaultSystemMap (system:
    eachPythonVersion [ "38" "39" "310" "311" ] (pyVersion:
      with pkgs; mkShell {
        name = "py${pyVersion}";

        shellHook = ''
          export PYTHONPATH="./:$PYTHONPATH"
        '';

        packages = with pkgs."python${pyVersion}Packages"; [
          pygls
          pytest
          pytest-asyncio
        ];
      }
  )
);

However, if you look at the implementation of eachPythonVersion

eachPythonVersion = versions: f:
  builtins.listToAttrs (builtins.map (version: {name = "py${version}"; value = f version; }) versions);

only supports parametrising a single version number
only supports producing a single ‘thing’ for each version number
is not easily adapatable to other situations.

Currently I’m working on the next major version of esbonio and need to be able to define multiple devShells per Python version each containing a different version of Sphinx.

So ideally, I’d want to be able to define my build matrix

buildMatrix = {
  py = [ "38" "39" "310" "311" ];
  sphinx = [ "5" "6" "7" ];
};

and then apply it over some function to get definitions for all combinations of supported versions

devShells = utils.lib.eachDefaultSystemMap (system:
  applyMatrix buildMatrix ({ py, sphinx, ...}: {
    "py${py}-esbonio" = pkgs.mkShell { };          # A shell without sphinx avaialable at all
    "py${py}-sphinx${sphinx}" = pkgs.mkShell { };  # A shell containing the given sphinx verison
  })
);

Which expands into a lot of devShells!

$ nix flake show
git+file:///var/home/alex/Projects/esbonio-beta?dir=lib%2fesbonio
├───devShells
│   ├───aarch64-darwin
│   │   ├───py310-esbonio: development environment 'py310-esbonio'
│   │   ├───py310-sphinx5: development environment 'py310-sphinx5'
│   │   ├───py310-sphinx6: development environment 'py310-sphinx6'
│   │   ├───py310-sphinx7: development environment 'py310-sphinx7'
│   │   ├───py311-esbonio: development environment 'py311-esbonio'
│   │   ├───py311-sphinx5: development environment 'py311-sphinx5'
│   │   ├───py311-sphinx6: development environment 'py311-sphinx6'
│   │   ├───py311-sphinx7: development environment 'py311-sphinx7'
│   │   ├───py38-esbonio: development environment 'py38-esbonio'
│   │   ├───py38-sphinx5: development environment 'py38-sphinx5'
│   │   ├───py38-sphinx6: development environment 'py38-sphinx6'
│   │   ├───py38-sphinx7: development environment 'py38-sphinx7'
│   │   ├───py39-esbonio: development environment 'py39-esbonio'
│   │   ├───py39-sphinx5: development environment 'py39-sphinx5'
│   │   ├───py39-sphinx6: development environment 'py39-sphinx6'
│   │   └───py39-sphinx7: development environment 'py39-sphinx7'
│   ├───aarch64-linux
│   │   ├───py310-esbonio: development environment 'py310-esbonio'
│   │   ├───py310-sphinx5: development environment 'py310-sphinx5'
│   │   ├───py310-sphinx6: development environment 'py310-sphinx6'
│   │   ├───py310-sphinx7: development environment 'py310-sphinx7'
│   │   ├───py311-esbonio: development environment 'py311-esbonio'
│   │   ├───py311-sphinx5: development environment 'py311-sphinx5'
│   │   ├───py311-sphinx6: development environment 'py311-sphinx6'
│   │   ├───py311-sphinx7: development environment 'py311-sphinx7'
│   │   ├───py38-esbonio: development environment 'py38-esbonio'
│   │   ├───py38-sphinx5: development environment 'py38-sphinx5'
│   │   ├───py38-sphinx6: development environment 'py38-sphinx6'
│   │   ├───py38-sphinx7: development environment 'py38-sphinx7'
│   │   ├───py39-esbonio: development environment 'py39-esbonio'
│   │   ├───py39-sphinx5: development environment 'py39-sphinx5'
│   │   ├───py39-sphinx6: development environment 'py39-sphinx6'
│   │   └───py39-sphinx7: development environment 'py39-sphinx7'
│   ├───x86_64-darwin
│   │   ├───py310-esbonio: development environment 'py310-esbonio'
│   │   ├───py310-sphinx5: development environment 'py310-sphinx5'
│   │   ├───py310-sphinx6: development environment 'py310-sphinx6'
│   │   ├───py310-sphinx7: development environment 'py310-sphinx7'
│   │   ├───py311-esbonio: development environment 'py311-esbonio'
│   │   ├───py311-sphinx5: development environment 'py311-sphinx5'
│   │   ├───py311-sphinx6: development environment 'py311-sphinx6'
│   │   ├───py311-sphinx7: development environment 'py311-sphinx7'
│   │   ├───py38-esbonio: development environment 'py38-esbonio'
│   │   ├───py38-sphinx5: development environment 'py38-sphinx5'
│   │   ├───py38-sphinx6: development environment 'py38-sphinx6'
│   │   ├───py38-sphinx7: development environment 'py38-sphinx7'
│   │   ├───py39-esbonio: development environment 'py39-esbonio'
│   │   ├───py39-sphinx5: development environment 'py39-sphinx5'
│   │   ├───py39-sphinx6: development environment 'py39-sphinx6'
│   │   └───py39-sphinx7: development environment 'py39-sphinx7'
│   └───x86_64-linux
│       ├───py310-esbonio: development environment 'py310-esbonio'
│       ├───py310-sphinx5: development environment 'py310-sphinx5'
│       ├───py310-sphinx6: development environment 'py310-sphinx6'
│       ├───py310-sphinx7: development environment 'py310-sphinx7'
│       ├───py311-esbonio: development environment 'py311-esbonio'
│       ├───py311-sphinx5: development environment 'py311-sphinx5'
│       ├───py311-sphinx6: development environment 'py311-sphinx6'
│       ├───py311-sphinx7: development environment 'py311-sphinx7'
│       ├───py38-esbonio: development environment 'py38-esbonio'
│       ├───py38-sphinx5: development environment 'py38-sphinx5'
│       ├───py38-sphinx6: development environment 'py38-sphinx6'
│       ├───py38-sphinx7: development environment 'py38-sphinx7'
│       ├───py39-esbonio: development environment 'py39-esbonio'
│       ├───py39-sphinx5: development environment 'py39-sphinx5'
│       ├───py39-sphinx6: development environment 'py39-sphinx6'
│       └───py39-sphinx7: development environment 'py39-sphinx7'

The question is, how do we implement applyMatrix?

Well, fast forwarding through plenty of trial and error and a few “aha!” moments I’m now able to tell you!

First, we need to take the buildMatrix and expand it out into all possible combinations - thankfully nixpkgs provides a function that does exactly that

$ nix repl
> buildMatrix = { py = [ "38" "39" "310" "311" ]; sphinx = [ "5" "6" "7" ]; }
> allCombinations = nixpkgs.lib.cartesianProductOfSets buildMatrix
> :p allCombinations  # ':p' Overrides nix's lazy evaluation to print the
                      # fully expanded version of an object
[
  { py = "38"; sphinx = "5"; }
  { py = "38"; sphinx = "6"; }
  { py = "38"; sphinx = "7"; }
  { py = "39"; sphinx = "5"; }
  { py = "39"; sphinx = "6"; }
  { py = "39"; sphinx = "7"; }
  { py = "310"; sphinx = "5"; }
  { py = "310"; sphinx = "6"; }
  { py = "310"; sphinx = "7"; }
  { py = "311"; sphinx = "5"; }
  { py = "311"; sphinx = "6"; }
  { py = "311"; sphinx = "7"; }
]

Next we need to apply some function over this list to produce the corresponding environment

> shells = builtins.map ({py, sphinx}: {"py${py}-sphinx${sphinx}" = { }; }) allCombinations
> :p shells
[
  { py38-sphinx5 = { }; }
  { py38-sphinx6 = { }; }
  { py38-sphinx7 = { }; }
  { py39-sphinx5 = { }; }
  { py39-sphinx6 = { }; }
  { py39-sphinx7 = { }; }
  { py310-sphinx5 = { }; }
  { py310-sphinx6 = { }; }
  { py310-sphinx7 = { }; }
  { py311-sphinx5 = { }; }
  { py311-sphinx6 = { }; }
  { py311-sphinx7 = { }; }
]

Finally, we need to merge the list of attribute sets down into a single set containing all of the definitions

> result = builtins.foldl' (x: y: x // y) {} shells
> :p result
{
  py310-sphinx5 = { };
  py310-sphinx6 = { };
  py310-sphinx7 = { };
  py311-sphinx5 = { };
  py311-sphinx6 = { };
  py311-sphinx7 = { };
  py38-sphinx5 = { };
  py38-sphinx6 = { };
  py38-sphinx7 = { };
  py39-sphinx5 = { };
  py39-sphinx6 = { };
  py39-sphinx7 = { };
}

Bringing it all together results in a surprisingly compact function definition!

applyMatrix = matrix: f:
  builtins.foldl' (x: y: x // y) {}
    (builtins.map f (nixpkgs.lib.cartesianProductOfSets matrix));

Flakes and Monorepos¶

Previously I tried adding a “top-level” flake.nix to the git repository for the esbonio language server that depended on another flake.nix within a sub directory of the same repository.

It… didn’t work.

I’m still trying to figure out the best way to approach this but I’m currently leaning towards keeping the multiple flake.nix files where

the top-level flake.nix contains “public” entry-points e.g. apps and overlays
“local” flake.nix files for each sub-project containing entry-points that are mainly useful for contributors to the project e.g. devShells
rather than have the top-level flake.nix depend on the “local” flakes, use Nix’s import statement to pull in reusable snippets of Nix code from the subprojects.

🤞 it works out!

Integrating Esbonio with Neovim Using Nix

2023-04-21T00:00:00+00:00

Integrating Esbonio with Neovim Using Nix¶

So far I’ve been learning how to use Nix by trying to package and define development shells for esbonio (see here if you are interested). While useful, the end result is not too dissimilar to what you can get with standard Python tooling. Indeed, the main reason I started looking into Nix was the promise of it being able to manage more than just Python libraries.

Since esbonio is a language server, it would be useful for Nix to create standardised environments where the language server is pre-configured for a given editor - great for debugging and demos!

In this blog post I try to define an environment in which Neovim is installed and configured to use the esbonio language server for reStructuredText files.

Try it yourself!

If I’ve done all my homework right, you should be able to try the result of this blog post for yourself Assuming you have nix installed

$ nix run github:alcarney/esbonio?rev=a077efeed176dcad2ae5e4fd221179d266f88ca1

should be the only command you need. Let me know if you run into any issues!

Defining Applications¶

One of the defined flake outputs is apps.<system>.<name> which as the name suggests allows you to export applications from a flake.

{
  description = "Esbonio";

  inputs = {
    nixpkgs.url = "github:NixOS/nixpkgs/nixpkgs-unstable";
    utils.url = "github:numtide/flake-utils";
  };

  outputs = { self, nixpkgs, utils }:
    utils.lib.eachDefaultSystem (system:
      let
        pkgs = import nixpkgs { inherit system; };
      in {
        apps.default = { type = "app"; program = "${pkgs.neovim}/bin/nvim"; };
      }
    );
}

This simple flake.nix exports neovim as an application which we can launch by running nix run . from the folder containing this flake.

Which works as expected however, it’s also picking up my personal config - not so useful when you’re trying to create a standard, isolated environment.

Isolated Configuration¶

As with most things in Nix, the neovim package definition allows for certain fields to be overridden - including the config. Let’s start by trying provide an empty init.vim file.

utils.lib.eachDefaultSystem (system:
   let
     pkgs = import nixpkgs { inherit system; };
     neovim = pkgs.neovim.override {
       configure = {
         customRC = ''
         '';
       };
     };
   in {
     apps.default = {
       type = "app";
       program = "${neovim}/bin/nvim";
     };
   }
 );

And try nix run . again

Which worked! Sort of… well… not really. 😕

It worked in the sense that it loaded the empty init.vim file we specified (notice that the screenshot above has no line numbers). However, it’s not truly isolated since it went ahead and loaded my plugins anyway due to my user’s home folder being included in the runtimepath

Tip

To get the contents of your runtimepath into a buffer.

In INSERT mode type <c-r>=&rtp and hit enter
Replace all commas with newlines :%s/,/\r/g

So how can we exclude them?

`nvim --clean`¶

Reading through :h ‘runtimepath’ there’s a lot of detail around which paths are searched by default and in what order. But right at the end there’s a little note

With --clean the home directory entries are not included.

Which sounds like just what we need! The question is… how do we start nvim with that flag?

Looking around the nixpkgs repo for a bit I found a set of test cases that made use of a utility for generating a config, along with a wrapper which converts the given config into a shell script. This shell script pulls together various components from /nix/store, before ultimately launching our isolated instance of neovim.

Example wrapper script

Here is an example of a wrapper script generated by nix.

#! /nix/store/0hx32wk55ml88jrb1qxwg5c5yazfm6gf-bash-5.2-p15/bin/bash -e
export NVIM_SYSTEM_RPLUGIN_MANIFEST='/nix/store/jjl5fy7dc5cxvc7mi781vxbk8ag89ih0-neovim-0.8.3-esbonio/rplugin.vim'
export GEM_HOME='/nix/store/4mmkiw8n1nhlfsnh4g2kijzkxnp6fyxb-neovim-ruby-env/lib/ruby/gems/2.7.0'
PATH=${PATH:+':'$PATH':'}
if [[ $PATH != *':''/nix/store/4mmkiw8n1nhlfsnh4g2kijzkxnp6fyxb-neovim-ruby-env/bin'':'* ]]; then
    PATH=$PATH'/nix/store/4mmkiw8n1nhlfsnh4g2kijzkxnp6fyxb-neovim-ruby-env/bin'
fi
PATH=${PATH#':'}
PATH=${PATH%':'}
export PATH
LUA_PATH=${LUA_PATH:+';'$LUA_PATH';'}
LUA_PATH=${LUA_PATH/';''/nix/store/nlmk08cmald0zi7fc6hgpdqrjz7lh8qj-luajit-2.1.0-2022-10-04-env/share/lua/5.1/?/init.lua'';'/';'}
LUA_PATH='/nix/store/nlmk08cmald0zi7fc6hgpdqrjz7lh8qj-luajit-2.1.0-2022-10-04-env/share/lua/5.1/?/init.lua'$LUA_PATH
LUA_PATH=${LUA_PATH#';'}
LUA_PATH=${LUA_PATH%';'}
export LUA_PATH
LUA_PATH=${LUA_PATH:+';'$LUA_PATH';'}
LUA_PATH=${LUA_PATH/';''/nix/store/nlmk08cmald0zi7fc6hgpdqrjz7lh8qj-luajit-2.1.0-2022-10-04-env/share/lua/5.1/?.lua'';'/';'}
LUA_PATH='/nix/store/nlmk08cmald0zi7fc6hgpdqrjz7lh8qj-luajit-2.1.0-2022-10-04-env/share/lua/5.1/?.lua'$LUA_PATH
LUA_PATH=${LUA_PATH#';'}
LUA_PATH=${LUA_PATH%';'}
export LUA_PATH
LUA_CPATH=${LUA_CPATH:+';'$LUA_CPATH';'}
LUA_CPATH=${LUA_CPATH/';''/nix/store/nlmk08cmald0zi7fc6hgpdqrjz7lh8qj-luajit-2.1.0-2022-10-04-env/lib/lua/5.1/?.so'';'/';'}
LUA_CPATH='/nix/store/nlmk08cmald0zi7fc6hgpdqrjz7lh8qj-luajit-2.1.0-2022-10-04-env/lib/lua/5.1/?.so'$LUA_CPATH
LUA_CPATH=${LUA_CPATH#';'}
LUA_CPATH=${LUA_CPATH%';'}
export LUA_CPATH
exec -a "$0" "/nix/store/1czj8mydgi30kyfimq6q4ifh06q131ch-neovim-unwrapped-0.8.3/bin/nvim"  -u /nix/store/fqjv4r08pl8k3vhy6ijxddrn8gpq2h7z-init.vim '--cmd' 'let g:loaded_node_provider=0 | let g:loaded_python_provider=0 | let g:python3_host_prog='\''/nix/store/jjl5fy7dc5cxvc7mi781vxbk8ag89ih0-neovim-0.8.3-esbonio/bin/nvim-python3'\'' | let g:ruby_host_prog='\''/nix/store/jjl5fy7dc5cxvc7mi781vxbk8ag89ih0-neovim-0.8.3-esbonio/bin/nvim-ruby'\''' "$@"

After some trial and error I was able to put together the following

utils.lib.eachDefaultSystem (system:
  let
    pkgs = import nixpkgs { inherit system; };
    nvim-cfg = pkgs.neovimUtils.makeNeovimConfig {
      extraName = "-esbonio";
      customRC = ''
        set number
      '';
    };
    neovim-config = pkgs.lib.attrsets.updateManyAttrsByPath [
      {
        path = ["wrapperArgs"];
        update = old: old ++ [
          "--add-flags" "--clean"
        ];
      }
    ] nvim-cfg;
    neovim = pkgs.wrapNeovimUnstable pkgs.neovim-unwrapped neovim-config;
  in {
    apps.default = {
      type = "app";
      program = "${neovim}/bin/nvim";
    };
  }
);

To summarize

pkgs.neovimUtils.makeNeovimConfig as the name suggests is a utility that generates a neovim “config”. “config” in this case is an attribute set containing all the arguments required to call pkgs.wrapNeovimUnstable.
One of these arguments is called wrapperArgs which contains the list of cli arguments to pass to the wrapped instance of neovim. Well almost.

wrapperArgs aren’t passed through to neovim directly, they are passed to a utility called makeWrapper which is a small program with it’s own set of arguments that allow you to describe how you want to wrap an underlying executable. This is why I’m appending "--add-flags" "--clean" to wrapperArgs and not just --clean.
Finally, the config and base neovim derivation are passed to pkgs.wrapNeovimUnstable to bring it all together.

Unfortunately, after all that I still didn’t end up with the result I was looking for

No plugins, but also note no line numbers 😢¶

Not only does the --clean flag prevent neovim from loading the plugins in my home folder, it also stopped neovim from loading the contents of my customRC - something I would’ve found out if I’d actually read the help text for --clean itself

--clean - Mimics a fresh install of Nvim:

Skips initializations from files and environment variables.

No ‘shada’ file is read or written.

Excludes user directories from ‘runtimepath’

Loads builtin plugins, unlike -u NONE -i NONE.

It should be possible to work around this though by telling neovim to source our init file as well as giving it the --clean flag. Let’s take a look at the exec command nix is currently generating for us in the wrapper script.

exec -a "$0" "/nix/store/1czj8mydgi30kyfimq6q4ifh06q131ch-neovim-unwrapped-0.8.3/bin/nvim" \
     -u /nix/store/fqjv4r08pl8k3vhy6ijxddrn8gpq2h7z-init.vim \
     --cmd '...' \
     --clean \
     "$@"

The -u /nix/store/fqj...-init.vim argument contains the contents of our customRC and I think changing the command to something like

exec -a "$0" "/nix/store/1czj8mydgi30kyfimq6q4ifh06q131ch-neovim-unwrapped-0.8.3/bin/nvim" \
     --clean \
     --cmd 'source /nix/store/fqjv4r08pl8k3vhy6ijxddrn8gpq2h7z-init.vim' \
     "$@"

will result in the behaviour I’m looking for.

Tip

So far I’ve neglected to mention how I’m finding the /nix/store path containing this wrapper script. Using the nix repl you can load your flake and inspect the values it contains.

$ nix repl
Welcome to Nix 2.11.1. Type :? for help.

nix-repl> :lf .        # load the flake located at '.'
warning: Git tree '/var/home/alex/Projects/esbonio-nix' is dirty
Added 9 variables.

nix-repl> outputs.apps.x86_64-linux.default
{ program = "/nix/store/knr1nfdmg9ld0xg813hb7ljl68060jlv-neovim-0.8.3-esbonio/bin/nvim"; type = "app"; }

It’s also useful for figuring out how the many utilities in nixpkgs work

nix-repl> pkgs = import inputs.nixpkgs {system = "x86_64-linux"; }

nix-repl> config = pkgs.neovimUtils.makeNeovimConfig { customRC = "set number"; }

nix-repl> config.wrapperArgs
[ "--inherit-argv0" "--add-flags" "'--cmd' 'let g:loaded_node_provider=0 | let g:loaded_python_provider=0 | let g:python3_host_prog='\\''/1rz4g4znpzjwh1xymhjpm42vipw92pr73vdgl6xs1hycac8kf2n9/bin/nvim-python3'\\'' | let g:ruby_host_prog='\\''/1rz4g4znpzjwh1xymhjpm42vipw92pr73vdgl6xs1hycac8kf2n9/bin/nvim-ruby'\\'''" "--set" "GEM_HOME" "/nix/store/4mmkiw8n1nhlfsnh4g2kijzkxnp6fyxb-neovim-ruby-env/lib/ruby/gems/2.7.0" "--suffix" "PATH" ":" "/nix/store/4mmkiw8n1nhlfsnh4g2kijzkxnp6fyxb-neovim-ruby-env/bin" "--prefix" "LUA_PATH" ";" "/nix/store/nlmk08cmald0zi7fc6hgpdqrjz7lh8qj-luajit-2.1.0-2022-10-04-env/share/lua/5.1/?.lua;/nix/store/nlmk08cmald0zi7fc6hgpdqrjz7lh8qj-luajit-2.1.0-2022-10-04-env/share/lua/5.1/?/init.lua" "--prefix" "LUA_CPATH" ";" "/nix/store/nlmk08cmald0zi7fc6hgpdqrjz7lh8qj-luajit-2.1.0-2022-10-04-env/lib/lua/5.1/?.so" ]

Unfortunately, I could not see an obvious way to rewrite the arguments to exec. The store path for the init.vim file is only generated in the depths of the wrapNeovimUnstable as it is written to disk and trying to manipulate wrapperArgs to extract it isn’t something I’m willing to attempt in Nix just yet!

A New Approach¶

It was at this point I started looking around to see what other people have come up with and before long I found this reddit thread which linked this flake that looked very promising. Not only did it provide a way of creating an isolated config but it also showed how to manage plugins and external binaries!

Following its example I was able to come up with the following definition

utils.lib.eachDefaultSystem (system:
  let
    pkgs = import nixpkgs { inherit system ; };
    initVim = ''
      set number
    '';
    paths = pkgs.lib.makeBinPath [
      pkgs.neovim
    ];
    pluginList = with pkgs.vimPlugins; [
      nvim-lspconfig
    ];
    plugins = pkgs.stdenv.mkDerivation {
      name = "esbonio-nvim-plugins";
      buildCommand = ''
        mkdir -p $out/nvim/site/pack/plugins/start/
        ${pkgs.lib.concatMapStringsSep "\n" (path: "ln -s ${path} $out/nvim/site/pack/plugins/start/")  pluginList }
      '';
    };
    neovim = pkgs.writeShellScriptBin "nvim" ''
      export PATH=${paths}:$PATH
      export XDG_CONFIG_DIRS=
      export XDG_DATA_DIRS=${plugins.outPath}
      nvim --clean --cmd source ${pkgs.writeText "init.vim" initVim} "$@"
    '';
  in {
     apps.default = {
       type = "app";
       program = "${neovim}/bin/nvim";
     };
   }
);

And trying nix run . once more

Success!¶

Not only did I end up with the correct configuration, the runtimepath finally contains just the paths that are necessary!

Integrating Esboino¶

Next we need to make sure the esbonio language server is available in this environment and include the necessary configuration for it in the config.

Including the server should be pretty straightforward as we get to reuse the overlay defined previously.

  inputs = {
    nixpkgs.url = "github:NixOS/nixpkgs/nixpkgs-unstable";
+   esbonio.url = "path:lib/esbonio";
+   esbonio.inputs.nixpkgs.follows = "nixpkgs";
    utils.url = "github:numtide/flake-utils";
  };

- outputs = { self, nixpkgs, utils }:
+ outputs = { self, nixpkgs, esbonio, utils }:

 utils.lib.eachDefaultSystem (system:
   let
-    pkgs = import nixpkgs { inherit system ; };
+    pkgs = import nixpkgs { inherit system ; overlays = [ esbonio.overlays.default ];};
     initVim = ''
       set number
     '';
     paths = pkgs.lib.makeBinPath [
       pkgs.neovim
+      pkgs.python310Packages.esbonio
     ];

Should being the key word here…

$ nix run .
warning: Git tree '/var/home/alex/Projects/esbonio-nix' is dirty
warning: updating lock file '/var/home/alex/Projects/esbonio-nix/flake.lock':
• Added input 'esbonio':
    'path:lib/esbonio?lastModified=1&narHash=sha256-WiFypw4lUZo7P9h82NMudwb5DFV0Nde5cOu1SqDmhVQ=' (1970-01-01)
• Added input 'esbonio/nixpkgs':
    follows 'nixpkgs'
• Added input 'esbonio/pytest-lsp':
    'github:swyddfa/lsp-devtools/6ae80a24b55d2b6943b9d30805cf02440ebbaf5c?dir=lib%2fpytest-lsp' (2023-04-02)
• Added input 'esbonio/pytest-lsp/nixpkgs':
    follows 'esbonio/nixpkgs'
• Added input 'esbonio/pytest-lsp/utils':
    'github:numtide/flake-utils/93a2b84fc4b70d9e089d029deacc3583435c2ed6' (2023-03-15)
• Added input 'esbonio/utils':
    'github:numtide/flake-utils/5aed5285a952e0b949eb3ba02c12fa4fcfef535f' (2022-11-02)
warning: Git tree '/var/home/alex/Projects/esbonio-nix' is dirty
error: undefined variable 'pytest-lsp'

       at /nix/store/96z740kkay7j0cbgmccj2mzbn5z8agvp-source/nix/esbonio-overlay.nix:22:11:

           21|           mock
           22|           pytest-lsp
             |           ^
           23|           pytest-timeout
(use '--show-trace' to show detailed location information)

The overlay exported by the language server’s flake doesn’t include its dependency pytest-lsp which is provided through an overlay of its own. A quick “fix” would be to also pull in the flake for pytest-lsp, but really the language server’s flake should be exporting all of its dependencies.

Composing Overlays¶

Thankfully, nixpkgs provides a function composeManyExtensions that handles this for us. When exporting the overlay from within the language server’s flake we can use it to merge the overlay from pytest-lsp with the overlay containing esbonio.

# In lib/esbonio/flake.nix
overlays.default = self: super: nixpkgs.lib.composeManyExtensions [
  pytest-lsp-overlay
  esbonio-overlay
] self super

However, since flake.lock freezes the language server’s flake as it was before we made this change we need to also update the lock file before trying again

$ nix flake lock --update-input esbonio
warning: Git tree '/var/home/alex/Projects/esbonio-nix' is dirty
warning: updating lock file '/var/home/alex/Projects/esbonio-nix/flake.lock':
• Updated input 'esbonio':
    'path:lib/esbonio?lastModified=1&narHash=sha256-WiFypw4lUZo7P9h82NMudwb5DFV0Nde5cOu1SqDmhVQ=' (1970-01-01)
  → 'path:lib/esbonio?lastModified=1&narHash=sha256-QgSDxOPSrtsaqjeStalef07+bUE3qkzz7pJC4y43ltw=' (1970-01-01)
warning: Git tree '/var/home/alex/Projects/esbonio-nix' is dirty

Now trying nix run . again neovim launches as before, running the command :r !python -m esbonio --help we can verify that the language server is indeed available to the editor.

Almost there!¶

Editor’s Note

Since writing this section and taking the above screenshot, I have been unable to re-produce it! Now :r !python -m esbonio --help results in a esbonio: Module not found error…

When debugging this, I’m not sure how the original ever worked since the flake definition does not include a Python interpreter meaning that python -m esbonio --help is running under the system Python.

The fix then, was to switch from python -m esbonio --help to calling esbonio --help directly, which thankfully, did not require me to change any of the Nix code.

Configuring Neovim¶

Now all that’s left to do is updating our initVim variable to contain the relevant configuration for the language server. Thanks to the example configuration available in the documentation, this can be as straightforward as replacing our hardcoded configuration with a call to Nix (the language’s) builtin readFile function.

 utils.lib.eachDefaultSystem (system:
   let
     pkgs = import nixpkgs { inherit system ; overlays = [ esbonio.overlays.default ];};
-    initVim = ''
-      set number
-    '';
+    initVim = builtins.readFile ./docs/lsp/editors/nvim-lspconfig/init.vim;
     paths = pkgs.lib.makeBinPath [
       pkgs.neovim
       pkgs.python310Packages.esbonio
     ];

And try opening a Sphinx project with it

It’s not pretty, but it works!¶

Wrapping Up¶

The experience as is currently stands is not that inspiring however, with the nix foundations laid it’s now more of a configuring neovim problem rather than a nix one! I am mildly disappointed that this required to dive so deep on the specifics of how neovim is configured, since that probably means you’d have to go to a similar depth to incorporate other editors. That said, once you’ve solved it for a given editor it’s probably solved “forever”.

Next I think I’d be interested in exploring how (or if it’s even possible) to make these Nix definitions more dynamic e.g.

Using the language server from $EDITOR using Python 3.x
Run the language server tests, but with a local checkout of pygls
Edit docs for $PROJECT using Sphinx vX

Obviously, you could achieve a lot of that by just editing the Nix definitions and rebuilding but I wonder if it’s possible to build in support for swapping parts out that can be wrapped up in a Makefile or similar 🤔

Nix Overlays: A follow up

2023-04-02T00:00:00+00:00

Nix Overlays: A follow up¶

It turns out there were a few issues with the setup I put together in my previous post. This time I try and resolve them and get to the point where I have working overlays for both pytest-lsp and esbonio.

`Dependency is not of valid type`¶

At the end of the previous post, I was left scratching my head after encountering a cryptic error message

$ nix develop .#py310
error: Dependency is not of a valid type: element 4 of nativeBuildInputs for py310
(use '--show-trace' to show detailed location information)

Which was coming from the following flake.nix

{
  description = "The Esbonio language server";

  inputs = {
    nixpkgs.url = "github:NixOS/nixpkgs/nixpkgs-unstable";
    pytest-lsp.url = "github:swyddfa/lsp-devtools?dir=lib/pytest-lsp";
    utils.url = "github:numtide/flake-utils";
  };

  outputs = { self, nixpkgs, pytest-lsp, utils }:

    let
      esbonio-overlay = import ./nix/esbonio-overlay.nix;
      eachPythonVersion = versions: f:
        builtins.listToAttrs (builtins.map (version: {name = "py${version}"; value = f version; }) versions);
    in {

    overlays.default = esbonio-overlay;

    devShells = utils.lib.eachDefaultSystemMap (system:
      let
        pkgs = import nixpkgs {
          inherit system;
          overlays = [ pytest-lsp.overlays.pytest-lsp esbonio-overlay ];
        };
      in
        eachPythonVersion [ "38" "39" "310" "311" ] (pyVersion:
          with pkgs; mkShell {
            name = "py${pyVersion}";
            packages = with pkgs."python${pyVersion}Packages"; [
              esbonio
              mock
              pytest
              pytest-lsp
              pytest-timeout
            ];
          }
      )
    );
  };
}

Originally, I thought this was caused by naming conflicts introduced by using pytest-lsp to reference both pytest-lsp the flake, and pytest-lsp the Python package. Indeed, changing the name of the flake input to pytestlsp seemed to at least change the error message I was seeing…

$ nix develop .#py310
error: undefined variable 'pytest-lsp'

       at /nix/store/dihmz79kgwxj1v5mqvxrj0f3ifgvpm9f-source/lib/esbonio/flake.nix:38:15:

           37|               pytest
           38|               pytest-lsp
             |               ^
           39|               pytest-timeout
(use '--show-trace' to show detailed location information)

How can that be?!

Conflicting Overlays¶

It turns out that the overlays pytestlsp.overlays.pytest-lsp and esbonio-overlay conflict with each other! I am not sure what originally led me to try it, but by reversing their order in the array passed to nixpkgs I could produce a similar error for the esbonio package.

$ nix develop .#py310
error: undefined variable 'esbonio'

       at /nix/store/vqff8bn03r11m1fg4f0b7ixnj731g9br-source/lib/esbonio/flake.nix:34:15:

           33|             packages = with pkgs."python${pyVersion}Packages"; [
           34|               esbonio
             |               ^
           35|
(use '--show-trace' to show detailed location information)

But why? 🤔 I thought the whole point of overlays were so that they could be… well, overlayed on an underlying package set without conflicting with each other??

Use the Source Luke¶

To find the answer, I had to remind myself that Nix is not magic (although it can appear to be!) and instead, at it’s core, Nix is a programming language. Which means this concept of “overlays” must be implemented in code somewhere and we can look for ourselves to see how they are handled. Sure enough, after some splunking through the nixpkgs repo I was able to track down the commit introducing the concept.

The majority of that commit appears to be just passing the overlays array through to all the places that require it and updating the documentation. The interesting part is where the overlays are actually applied at the bottom of pkgs/top-level/stage.nix

let
  # The complete chain of package set builders, applied from top to bottom
  toFix = lib.foldl' (lib.flip lib.extends) (self: {}) ([
    stdenvBootstappingAndPlatforms
    stdenvAdapters
    trivialBuilders
    allPackages
    aliases
    stdenvOverrides
    configOverrides
    ] ++ overlays);
in
  # Return the complete set of packages.
  lib.fix toFix

From what I understand

lib.foldl’ applies some combination function - (lib.flip lib.extends) in this case, to a list resulting in a single aggregated value.
lib.flip switches the order of the arguments given to lib.extends
lib.extends is the function we’re actually interested in as it is responsible for applying the overlays.
lib.fix appears to resolve all references to self in toFix to a “proper” value, but I’m not entirely sure how.

Here is the implementation of lib.extends as of the commit introducing overlays

extends = f: rattrs: self: let super = rattrs self; in super // f self super;

As with most things in Nix, I don’t really understand the fine details but it’s interesting to see that it uses the // operator to merge the result of an overlay (f self super) with the current state of the package set (super). One thing that’s interesting to note, when combining attribute sets with the // operator, if both sets contain the same key, then the value from the original set is replaced with the value provided by the second.

$ nix repl
Welcome to Nix 2.11.1. Type :? for help.

nix-repl> x = {a = 1 ; b = 2; c = 3;}

nix-repl> y = {d = 4; c = 5;}

nix-repl> x // y
{ a = 1; b = 2; c = 5; d = 4; }

Foreshadowing…

The Problem¶

Armed with my new found knowledge I had another look at the definitions of the problematic overlays.

# pytest-lsp-overlay.nix
let
  eachPythonVersion = ...
in

self: super:

eachPythonVersion [ "38" "39" "310" "311" ] (pyVersion:
  super."python${pyVersion}".override {
    packageOverrides = pyself: pysuper: {
      pytest-lsp = pysuper.buildPythonPackage { ... };
    };
})

# esbonio-overlay.nix
let
  eachPythonVersion = ...
in

self: super:

eachPythonVersion [ "38" "39" "310" "311" ] (pyVersion:
  super."python${pyVersion}".override {
    packageOverrides = pyself: pysuper: {
      esbonio = pysuper.buildPythonPackage { ... };
    };
})

Well no wonder they conflict with each other, they’re overriding the base pythonXY package directly! Any packageOverrides provided by the first overlay would be wiped out when the second is applied. Surely then there must be a better way to provide your own Python package definitions 🤔

The Solution¶

Somewhat buried on the Python page in the Nixpkgs manual is this handy FAQ question

17.27.3.9. How to override a Python package for all Python versions using extensions?

The following overlay overrides the call to buildPythonPackage for the foo package for all interpreters by appending a Python extension to the pythonPackagesExtensions list of extensions.
final: prev: {
  pythonPackagesExtensions = prev.pythonPackagesExtensions ++ [
    (
       python-final: python-prev: {
         foo = python-prev.foo.overridePythonAttrs (oldAttrs: { ... });
       }
    )
  ];
}

This might be just what we need! Not only do we avoid messing with the base Python package, we also get our packages automatically added to each Python version without the need to roll our own eachPythonVerison helper!

Converting my previous overlay attempts to the above approach results in overlay definitions that are a lot more straight forward. Notice that I was even able to enable tests for them now!

# pytest-lsp-overlay.nix
final: prev: {
  pythonPackagesExtensions = prev.pythonPackagesExtensions ++ [(
    python-final: python-prev: {
      pytest-lsp = python-prev.buildPythonPackage {
        pname = "pytest-lsp";
        version = "0.2.1";

        src = ./..;

        propagatedBuildInputs = with python-prev; [
          pygls
          pytest
          pytest-asyncio
        ];

        doCheck = true;

        nativeCheckInputs = with python-prev; [
          pytestCheckHook
        ];

        pythonImportsCheck = [ "pytest_lsp" ];
      };
    }
  )];
}

# esbonio-overlay.nix
final: prev: {
  pythonPackagesExtensions = prev.pythonPackagesExtensions ++ [(
    python-final: python-prev: {
      esbonio = python-prev.buildPythonPackage {
        pname = "esbonio";
        version = "0.16.1";

        src = ./..;

        propagatedBuildInputs = with python-prev; [
          appdirs
          pygls
          pyspellchecker
          sphinx
          # typing-extensions; only required for Python 3.7
        ];

        doCheck = true;

        nativeCheckInputs = with python-prev; [
          mock
          pytest-lsp
          pytest-timeout
          pytestCheckHook
        ];

        pythonImportsCheck = [ "esbonio.lsp" ];
      };
    }
  )];
}

All that is left to do is to try and enter the devShell for esbonio again

$ nix develop .#py310
error: builder for '/nix/store/027wakjv9wvws6190c66nf5gxc6smc54-python3.10-esbonio-0.16.1.drv' failed with exit code 2;
       last 10 log lines:
       > /nix/store/l69b9xl4pnqqgdx9vp1yg1cbckgcjsfx-python3.10-pytest-7.2.0/lib/python3.10/site-packages/_pytest/assertion/rewrite.py:168: in exec_module
       >     exec(co, module.__dict__)
       > tests/sphinx-default/conftest.py:53: in <module>
       >     ClientServerConfig(
       > E   TypeError: ClientServerConfig.__init__() got an unexpected keyword argument 'client'
       > =========================== short test summary info ============================
       > ERROR  - TypeError: ClientServerConfig.__init__() got an unexpected keyword argument...
       > !!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!
       > =============================== 1 error in 0.60s ===============================
       > /nix/store/3yfs41f4b60jya2gk6xikx4s97zsxjr0-stdenv-linux/setup: line 1573: pop_var_context: head of shell_variables not a function context
For full logs, run 'nix log /nix/store/027wakjv9wvws6190c66nf5gxc6smc54-python3.10-esbonio-0.16.1.drv'.
error: 1 dependencies of derivation '/nix/store/nms3hs5pz1fmyki4k547gfs1281klgl3-py310-env.drv' failed to build

Hey! At least the Nix part is finally working!

Disabling Tests¶

There’s one final detail left to clear up.

Of course, if you are consuming a package (like how esbonio is pulling in pytest-lsp) it’s good to have the tests run so that you can verify everything is working as expected. However, when you are setting up a devShell to work on a package, you don’t really want the tests to run since they will prevent you entering the shell if they fail - as is the case here.

Thankfully, it should just be a case of setting the doCheck flag for esbonio to false when using it within the flake’s devShell definition.

devShells = utils.lib.eachDefaultSystemMap (system:
  let
    pkgs = import nixpkgs {
      inherit system;
      overlays = [ pytest-lsp-overlay esbonio-overlay ];
    };
  in
    eachPythonVersion [ "38" "39" "310" "311" ] (pyVersion:
      pkgs.mkShell {
        name = "py${pyVersion}";

        packages = with pkgs."python${pyVersion}Packages"; [
          esbonio.overridePythonAttrs (_: { doCheck = false; })

          mock
          # Still necessary to avoid a naming conflict with pytest-lsp, the flake
          pkgs."python${pyVersion}Packages".pytest-lsp
          pytest-timeout
        ];
      }
  )
);

And activating the shell as normal.

$ nix develop .#py310
error: Dependency is not of a valid type: element 1 of nativeBuildInputs for py310
(use '--show-trace' to show detailed location information)

No! Not again! 😭

To be honest, I nearly gave up on the whole idea then and there but in a last ditch attempt I moved the overriden esbonio package out into a let binding.

devShells = utils.lib.eachDefaultSystemMap (system:
   let
     pkgs = import nixpkgs {
       inherit system;
       overlays = [ pytest-lsp-overlay esbonio-overlay ];
     };
   in
     eachPythonVersion [ "38" "39" "310" "311" ] (pyVersion:

       let
         esbonio = pkgs."python${pyVersion}Packages".esbonio.overridePythonAttrs (_: { doCheck = false; });
       in

       pkgs.mkShell {
         name = "py${pyVersion}";

         packages = with pkgs."python${pyVersion}Packages"; [
           esbonio

           mock
           pkgs."python${pyVersion}Packages".pytest-lsp
           pytest-timeout
         ];
       }
   )
 );

And tried again

$ nix develop .#py310 -L  # -L = enable verbose logging, useful to actually see what the builds are doing.
python3.10-esbonio> Sourcing python-remove-tests-dir-hook
python3.10-esbonio> Sourcing python-catch-conflicts-hook.sh
python3.10-esbonio> Sourcing python-remove-bin-bytecode-hook.sh
python3.10-esbonio> Sourcing setuptools-build-hook
python3.10-esbonio> Using setuptoolsBuildPhase
python3.10-esbonio> Using setuptoolsShellHook
python3.10-esbonio> Sourcing pip-install-hook
python3.10-esbonio> Using pipInstallPhase
...
python3.10-esbonio> patching script interpreter paths in /nix/store/a9xjjxv1zh3dmhfaxgph8kq0zaxl92g3-python3.10-esbonio-0.16.1-dist
python3.10-esbonio> Rewriting #!/nix/store/sp5x6s8n36gjlwck74xhj1i61p66vcpa-python3-3.10.9/bin/python3.10 to #!/nix/store/sp5x6s8n36gjlwck74xhj1i61p66vcpa-python3-3.10.9
python3.10-esbonio> wrapping `/nix/store/91b7mh7ib0fxwn2kgv47v0sdpl05xqh1-python3.10-esbonio-0.16.1/bin/esbonio'...
python3.10-esbonio> Rewriting #!/nix/store/sp5x6s8n36gjlwck74xhj1i61p66vcpa-python3-3.10.9/bin/python3.10 to #!/nix/store/sp5x6s8n36gjlwck74xhj1i61p66vcpa-python3-3.10.9
python3.10-esbonio> wrapping `/nix/store/91b7mh7ib0fxwn2kgv47v0sdpl05xqh1-python3.10-esbonio-0.16.1/bin/esbonio-sphinx'...
python3.10-esbonio> Executing pythonRemoveTestsDir
python3.10-esbonio> Finished executing pythonRemoveTestsDir
python3.10-esbonio> pythonCatchConflictsPhase
python3.10-esbonio> pythonRemoveBinBytecodePhase
python3.10-esbonio> pythonImportsCheckPhase
python3.10-esbonio> Executing pythonImportsCheckPhase
python3.10-esbonio> Check whether the following modules can be imported: esbonio.lsp

(nix-shell) $

And it actually worked! 🤯

Conclusion¶

I have no idea why Nix needed me to move the overridePythonAttrs call out into a separate let binding, but hey it works!

I’ve finally managed to recreate the setup I had in my original blog post, spinning up devShells in order to test esbonio against a range of Python versions - just with the added flexibility that working with overlays can bring.

If you’re interested you can find the final version of all my *.nix files here (pytest-lsp) and here (esbonio). Hopefully next time we can build on this and finally use Nix for something you can’t get out of standard Python tooling! 😅

My Next Steps with Nix: Overlays

2023-01-25T00:00:00+00:00

My Next Steps with Nix: Overlays¶

Last time, I experimented with writing a flake that defined development environments for the esbonio package spanning multiple Python versions. During that process I also packaged pytest-lsp using an ad-hoc nix expression as part of the esbonio repo.

In this post I look into writing a similar flake for the pytest-lsp package itself, but this time using overlays to override and extend the nixpkgs package set.

If you are interested, you can find the final version of the code here.

Packaging `pytest-lsp`¶

Adapting the flake.nix file and using the package definition from the previous post, it’s easy enough to sketch out a flake that should give us a devShell to work on the pytest-lsp package.

However, trying to activate one we encounter a problem.

$ nix develop -c .#py310
error: builder for '/nix/store/dfd5bixdgkvfcnfa7f9z0ibp4m5zlhkz-python3.10-pytest-lsp-0.2.1.drv' failed with exit code 1;
       last 10 log lines:
       > installing
       > Executing pipInstallPhase
       > /build/pytest-lsp/dist /build/pytest-lsp
       > Processing ./pytest_lsp-0.2.1-py3-none-any.whl
       > Requirement already satisfied: appdirs in /nix/store/yidjmqc5q1j0fz2dk79qgk1fy7dqcliy-python3.10-appdirs-1.4.4/lib/python3.10/site-packages (from pytest-lsp==0.2.1) (1.4.4)
       > Requirement already satisfied: pytest-asyncio in /nix/store/dvz12bivdc0dkn6849zm58754ga06hs6-python3.10-pytest-asyncio-0.20.3/lib/python3.10/site-packages (from pytest-lsp==0.2.1) (0.20.3)
       > Requirement already satisfied: pytest in /nix/store/z5pkmmsdg3bmb35pmsv4rjca1qi7dbnf-python3.10-pytest-7.2.0/lib/python3.10/site-packages (from pytest-lsp==0.2.1) (7.2.0)
       > ERROR: Could not find a version that satisfies the requirement pygls>=1.0.0 (from pytest-lsp) (from versions: none)
       > ERROR: No matching distribution found for pygls>=1.0.0
       >
       For full logs, run 'nix log /nix/store/dfd5bixdgkvfcnfa7f9z0ibp4m5zlhkz-python3.10-pytest-lsp-0.2.1.drv'.
error: 1 dependencies of derivation '/nix/store/kfzlz750xdk71fxwvsgpdbw1w00jbvf9-py310-env.drv' failed to build

In the time between writing the previous blog post and this one, the pytest-lsp package has been migrated to the latest version of pygls. The version available through nixpkgs however, is still the previous release.

While there is an open pull request updating pygls to 1.0, at the time of writing it’s blocked on downstream packages which haven’t migrated yet.

That said, we don’t have to wait for nixpkgs but can instead use an overlay to update it just for this project.

Overriding pygls’ version¶

Overlays can be used to override sections of an existing package definition.

Note

As I mentioned in the previous post, I’m probably not the best person to learn Nix from. Instead, here are some resources you might useful which go into more detail.

The NixOS wiki page on Overlays
Nix Pills: Chapter 14. Override design pattern

As far as I understand it:

Overlays are a useful design pattern, rather than a fundamental concept of the Nix language.
They are “just” a nix function that have access to both the modified version (usually called self or final) of the “thing” they’re modifying, as well as the unmodified version of it (super or prev)
These functions make use of attributes like override or overrideAttrs to make their modifications.
I have no idea how to make something overridable 😅

After reading through the wiki page on overlays a few times, particuarly the sections on overriding a version and python package overlays, I was able to put together an overlay which looked like it should work.

 # In ./nix/pygls-overlay.nix
 self: super: rec {

   python3 = super.python3.override {
     packageOverrides = pyself: pysuper: {

       pygls = pysuper.pygls.overrideAttrs (old: rec {
         version = "1.0.0";
         src = super.fetchFromGitHub {
           owner = "openlawlibrary";
           repo = "pygls";
           rev = "v${version}";
           hash = "sha256-31J4+giK1RDBS52Q/Ia3Y/Zak7fp7gRVTQ7US/eFjtM=";
         };
       });
     };
  };

  python3Packages = python3.pkgs;
}

Using this in the flake is a matter of importing it and passing it to the overlays attribute when importing nixpkgs

# In flake.nix
outputs = { self, nixpkgs, utils }:

 let
   pygls-overlay = import ./nix/pygls-overlay.nix;
   eachPythonVersion = ...
 in {

 devShells = utils.lib.eachDefaultSystemMap (system:
   let
     pkgs = import nixpkgs { inherit system; overlays = [ pygls-overlay ]; };
   in
     eachPythonVersion [ "37" "38" "39" "310" "311" ] (pyVersion:
       let
         pytest-lsp = pkgs.callPackage ./nix/pytest-lsp.nix { pythonPackages = pkgs."python${pyVersion}Packages"; };
       in

With some luck, running nix develop this time should bring in the latest pygls version

$ nix develop -c .#py310
error: builder for '/nix/store/1sha5j0dfyn2g4z82rpk4yqv32awmjfr-python3.10-pytest-lsp-0.2.1.drv' failed with exit code 1;
       ...
       > ERROR: No matching distribution found for pygls>=1.0.0

Huh, same error… 🤔

Let’s take a closer look at where we pull in the pytest-lsp package definition in the flake…

pytest-lsp = pkgs.callPackage ./nix/pytest-lsp.nix {
  pythonPackages = pkgs."python${pyVersion}Packages";
};

Assuming we’re trying to enter the python310 devShell, then we’re passing in the python310Packages package set. But in the overlay, we’re overriding the python3Packages package set, I wonder if we change the overlay to match the flake…

# In ./nix/pygls-overlay.nix
self: super: rec {
  python310 = super.python310.override { ... };
  python310Packages = python310.pkgs;
}

And try again

$ nix develop .#py310
error: builder for '/nix/store/jl23ai588n2b6amaicy5532bdxjiciyy-python3.10-pygls-0.13.0.drv' failed with exit code 1;
       last 10 log lines:
       > removing build/bdist.linux-x86_64/wheel
       > Finished executing setuptoolsBuildPhase
       > installing
       > Executing pipInstallPhase
       > /build/source/dist /build/source
       > Processing ./pygls-0.13.0-py3-none-any.whl
       > Requirement already satisfied: typeguard<3,>=2.10.0 in /nix/store/m4jjcrvbi928pi2d14qh8np1miqfvc0b-python3.10-typeguard-2.13.3/lib/python3.10/site-packages (from pygls==0.13.0) (2.13.3)
       > ERROR: Could not find a version that satisfies the requirement lsprotocol (from pygls) (from versions: none)
       > ERROR: No matching distribution found for lsprotocol
       >
       For full logs, run 'nix log /nix/store/jl23ai588n2b6amaicy5532bdxjiciyy-python3.10-pygls-0.13.0.drv'.
error: 1 dependencies of derivation '/nix/store/f5vasy4x9zpdhcq9jh9rz06qpvriblwp-python3.10-pytest-lsp-0.2.1.drv' failed to build
error: 1 dependencies of derivation '/nix/store/86v8bcxvjq1g9dhpx1wgmckba8bnag7h-py310-env.drv' failed to build

Progress!

Packaging `lsprotocol`¶

pygls is failing to build as the package definition in nixpkgs is missing the new lsprotcol dependency, easy enough to fix - if it was available in nixpkgs. Thankfully, overlays can do more than just override attributes on existing packages, they can be used to extend a package set with entirely new definitions!

We just need to know how to package lsprotocol itself and thanks to the PR linked above we get to cheat a little.

# In ./nix/pygls-overlay.nix
lsprotocol = pysuper.buildPythonPackage rec {
  pname = "lsprotocol";
  version = "2022.0.0a9";
  format = "pyproject";

  src = super.fetchFromGitHub {
    owner = "microsoft";
    repo = pname;
    rev = version;
    hash = "sha256-6XecPKuBhwtkmZrGozzO+VEryI5wwy9hlvWE1oV6ajk=";
  };

  nativeBuildInputs = with super.python310Packages; [
    flit-core
  ];

  propagatedBuildInputs = with super.python310Packages; [
    cattrs
    attrs
  ];

  # Disable tests
  doCheck = false;
};

Note that I’ve cut some corners by disabling any tests, but it allows me to dodge packaging anything else 😅

Then we can also override pygls’ dependencies and reference the newly created lsprotocol package from the modified version of the python310Packages set.

pygls = pysuper.pygls.overrideAttrs (_: rec {
  ...
  propagatedBuildInputs = with self.python310Packages; [
    lsprotocol
    typeguard
  ];
});

With that taken care of, we should be good to go right?

Unlucky `0.13`¶

Attempting to enter the devShell yet again we encounter a familiar error message

error: builder for '/nix/store/s5xp7fr2r9faxgqw7rvs6ffah10f2fz7-python3.10-pytest-lsp-0.2.1.drv' failed with exit code 1;
       last 10 log lines:
       > Finished executing setuptoolsBuildPhase
       > installing
       > Executing pipInstallPhase
       > /build/pytest-lsp/dist /build/pytest-lsp
       > Processing ./pytest_lsp-0.2.1-py3-none-any.whl
       > Requirement already satisfied: pytest-asyncio in /nix/store/dvz12bivdc0dkn6849zm58754ga06hs6-python3.10-pytest-asyncio-0.20.3/lib/python3.10/site-packages (from pytest-lsp==0.2.1) (0.20.3)
       > Requirement already satisfied: pytest in /nix/store/z5pkmmsdg3bmb35pmsv4rjca1qi7dbnf-python3.10-pytest-7.2.0/lib/python3.10/site-packages (from pytest-lsp==0.2.1) (7.2.0)
       > ERROR: Could not find a version that satisfies the requirement pygls>=1.0.0 (from pytest-lsp) (from  versions: none)
       > ERROR: No matching distribution found for pygls>=1.0.0
       >
       For full logs, run 'nix log /nix/store/s5xp7fr2r9faxgqw7rvs6ffah10f2fz7-python3.10-pytest-lsp-0.2.1.drv'.
error: 1 dependencies of derivation '/nix/store/a0smpmj63fw1fzp78i3z53xvd0zsvvhp-py310-env.drv' failed to build

But we just upgraded pygls to 1.0 right? That’s why we had to package lsprotocol in the previous section?

You might have already noticed in the log output above, that despite overriding the version field to 1.0 the Python package was still coming out as 0.13.0 - despite it containing the 1.0 version of the codebase!

> Processing ./pygls-0.13.0-py3-none-any.whl

Plenty of head scratching later, I finally remembered that pygls uses setuptools_scm to automatically derive the version number based on tags in its git repository. But the build is not taking place in a git repo… so nix must be setting that version somehow right?

Yep. A quick trip to the actual file containing pygls’ package definition on nixpkgs (and not just the diff view in the PR!) reveals an additional attribute that needed to be overriden

# In ./nix/pygls-overlay.nix
pygls = pysuper.pygls.overrideAttrs (_: rec {
     version = "1.0.0";
     SETUPTOOLS_SCM_PRETEND_VERSION = version;
     ...
});

Now if we try activating that devShell?

$ nix develop .#py310
(nix-shell) $ pytest
================================== test session starts =================================
platform linux -- Python 3.10.9, pytest-7.2.0, pluggy-1.0.0
rootdir: /var/home/alex/Projects/lsp-devtools/lib/pytest-lsp, configfile: pyproject.toml
plugins: lsp-0.2.1, typeguard-2.13.3, asyncio-0.20.3
asyncio: mode=auto
collected 27 items

tests/test_client.py ...                                    [ 11%]
tests/test_client_methods.py ...................            [ 81%]
tests/test_plugin.py ....                                   [ 96%]
tests/test_server.py .                                      [100%]

================================= 27 passed in 8.57s ==================================

Success!

Note

I’m not 100% sure if I’ve overriden the pygls’ version number correctly, since inspecting the PYTHONPATH the devShell is using shows that the version number of the nix package is still 0.13.0!

(nix-shell) $ echo $PYTHONPATH | tr ':' '\n' | grep pygls
/nix/store/s5jh5s9m5f1163hxzj8768jc5li7cdfg-python3.10-pygls-0.13.0/lib/python3.10/site-packages

But in Python land, everything appears at least, to be consistent, so I’m going with it for now.

Mutliple Python Versions¶

Now that we’ve got it working for Python 3.10, we need to generalise the overlay so that we can use it with any of the Python versions supported by pytest-lsp.

Ideally, what we’d want is to write an expression like the following

# In ./nix/pygls-overlay.nix
self: super:

eachPythonVersion ["37" "38" "39" "310" "311"] (pyVersion:
  super."python${pyVersion}".override {
    packageOverrides = pyself: pysuper: {

       lsprotocol = pysuper.buildPythonPackage rec {
         ...
         nativeBuildInputs = with super."python${pyVersion}Packages"; [
           flit-core
         ];

         propagatedBuildInputs = with super."python${pyVersion}Packages"; [
           cattrs
           attrs
         ];
       };

       pygls = pysuper.pygls.overrideAttrs (_: rec {
         ...
         propagatedBuildInputs = with self."python${pyVersion}Packages"; [
           lsprotocol
           typeguard
         ];
      });
   };
})

And have the eachPythonVersion function handle the details of performing all the overrides.

To start with, let’s define a helper doPythonOverride that eachPythonVersion can use. It should take a version and a function f and use it to perform the override for a single Python version, something like the following pseudo code.

doPythonOverride(version, f) = { "python${version}" = f(version);
                                 "python${version}Packages" = "python${version}".pkgs; }

The only issue is that (as far as I can tell), you can’t use strings as keys in a nix attribute set. However, you can use the builtins.listToAttrs function to build an attribute set from a list of { name = "xxx"; value = 123; } attribute sets, which allows us to define doPythonOverride as follows.

doPythonOverride = version: f:
  let
    overridenPython = f version;
  in
    builtins.listToAttrs [ {name = "python${version}"; value = overridenPython; }
                           {name = "python${version}Packages"; value = overridenPython.pkgs; }];

From there, we can define eachPythonVersion to map the doPythonOverride helper across each of the given Python versions and merge the results into a single attribute set using the foldl’ function.

eachPythonVersion = versions: f: builtins.foldl' (a: b: a // b) {}
  (builtins.map (version: doPythonOverride version f) versions);

Now we should have successfully overriden pygls’ version across all supported Python versions!

Sharing Overlays¶

Up until now, I’ve been mostly focusing on the devShells output of a flake. There are, however, many other items that can be exported from a flake - including overlays. Following the same pattern as the previous section it’s easy enough to convert the pytest-lsp package definition into an overlay

# In ./nix/pytest-lsp-overlay.nix
let
  doPythonOverride = version: f:
    let
      overridenPython = f version;
    in
      builtins.listToAttrs [ {name = "python${version}" ; value = overridenPython ; }
                             {name = "python${version}Packages" ; value = overridenPython.pkgs ; }];

  eachPythonVersion = versions: f: builtins.foldl' (a: b: a // b) {}
    (builtins.map (version: doPythonOverride version f) versions);
in

self: super:

eachPythonVersion [ "37" "38" "39" "310" "311" ] (pyVersion:
  super."python${pyVersion}".override {
    packageOverrides = pyself: pysuper: {

      pytest-lsp = pysuper.buildPythonPackage {
        pname = "pytest-lsp";
        version = "0.2.1";
        src = ./..;
        propagatedBuildInputs = with super."python${pyVersion}Packages"; [
          pygls
          pytest
          pytest-asyncio
        ];
      };
    };
  })

We can then include it in the main flake.nix file just as we did with the pygls overlay, but also assign it to the overlays output to make it available to other projects.

# In flake.nix
outputs = { self, nixpkgs, utils }:

  let
   pygls-overlay = import ./nix/pygls-overlay.nix;
   pytest-lsp-overlay = import ./nix/pytest-lsp-overlay.nix;
 in {

 overlays.pytest-lsp = pytest-lsp-overlay;

 devShells = utils.lib.eachDefaultSystemMap (system:
   let
     pkgs = import nixpkgs { inherit system; overlays = [ pygls-overlay pytest-lsp-overlay ]; };
   in
     ...

In theory, we can update the flake we previously wrote for esbonio to use this overlay to provide the pytest-lsp package definition

# In esbonio/flake.nix

inputs = {
  nixpkgs.url = "github:NixOS/nixpkgs/nixpkgs-unstable";
  pytest-lsp.url = "github:swyddfa/lsp-devtools?dir=lib/pytest-lsp";
  utils.url = "github:numtide/flake-utils";
};

outputs = { self, nixpkgs, pytest-lsp, utils }:

  let
    pygls-overlay = import ./nix/pygls-overlay.nix;
  in {

  devShells = utils.lib.eachDefaultSystemMap (system:
    let
      pkgs = import nixpkgs { inherit system; overlays = [ pygls-overlay pytest-lsp.overlays.pytest-lsp ]; };
    in
      ...

Finally, we should be able to activate a devShell for esbonio as before.

$ nix develop .#py310
error: Dependency is not of a valid type: element 5 of nativeBuildInputs for py310
(use '--show-trace' to show detailed location information)

Ah, well, perhaps that’s a job for another day! 😅

My First Steps with Nix

2022-12-27T00:00:00+00:00

My First Steps with Nix¶

Nix, depending on the context, can refer to a programming language, a package manager or a Linux Distro.

Personally, I’m most insterested in the package manager aspect and the promise of it being able to create declarative, reproducable development environments. With a configuration file and the nix develop command you can activate a kind of “virtual environment” that contains not just your Python packages - but any program defined by the configuration!

I find that idea particuarly exciting when working on a language server like esbonio, since it needs to be able to work against a variety of Python versions, Sphinx versions, as well as various code editors and all their versions! Having the ability to define a particular configuration and have some tool automatically recreate it would be amazing.

But I’m getting ahead of myself, let’s see if I can get to a point where I can easily test esbonio against a range of Python versions.

Intalling Nix¶

What a nightmare! 😭

I should say though, my issues aren’t really Nix’s fault. Trying to install Nix directly on Fedora Kinoite means dealing with issues caused by SELinux (which the Nix installer does not support) and working around the immutable root filesystem.

Basically, don’t do as I do! 😄

If you do find yourself in my situation though, here’s a few things you might find useful

This guide will get you 90% of the way, I was able to piece together the remaining steps from links in the comments.
The Nix installer has been updated since the guide was written to bail if it detects that SELinux has been enabled. You will need to patch out the check_selinux function in the install-multi-user script in the release tarball that the Nix installer downloads.
If you get a cryptic
```
error: could not set permissions on '/nix/var/nix/profiles/per-user' to 755: Operation not permitted
```
message whenever you run a nix command, chances are the nix-daemon is not running. Use systemctl status nix-daemon.service to check its status.
If you see an error in the output of systemctl status nix-daemon.service along the lines of:
```
nix-daemon.service: Failed to locate executable /nix/store/xdlpraypxdimjyfrr4k06narrv8nmfgh-nix-2.11.1/bin/nix-daemon: Permission denied
```
you need to re-apply the SELinux policies defined in the guide linked above by running sudo restorecon -RF /nix

A Simple Flake¶

Note

I’m not the best person to learn how to use Nix from - I’m still trying to figure it out myself! Instead here are a few resources that I’ve found useful which go into more detail.

Nix Flakes: An Introduction, part one of a series of posts.
Jon Ringer’s Youtube Channel

From what I can gather, flakes are a good starting point as they have a well defined structure and seem to be where things are going when it comes to Nix based workflows.

As mentioned in the intro I’d like to get to the point where I can easily test esbonio against a range of Python versions, so let’s start off by writing a flake.nix that provides a devShell containing Python.

{
  description = "The Esbonio language server";

  inputs = {
    nixpkgs.url = "github:NixOS/nixpkgs/nixpkgs-unstable";
    utils.url = "github:numtide/flake-utils";
  };

  outputs = { self, nixpkgs, utils }:
    utils.lib.eachDefaultSystem (system:
      let pkgs = import nixpkgs { inherit system; }; in {

      devShell = with pkgs;
         mkShell {
            packages = [ python3 ];
         };
      }
   );
}

Using nix flake show we can see what outputs are produced by this flake

$ nix flake show
warning: Git tree '/var/home/alex/Projects/esbonio' is dirty
error: getting status of '/nix/store/9s8zs1hrqiingklv86fd18x2mbgsfw0w-source/lib/esbonio/flake.nix': No such file or directory

Oh! I always forget, when working with flakes nix will only see a file if it is tracked by git - we don’t need to commit it, but it needs to at least be staged.

$ git add flake.nix
$ nix flake show
warning: Git tree '/var/home/alex/Projects/esbonio' is dirty
git+file:///var/home/alex/Projects/esbonio?dir=lib%2fesbonio
└───devShell
   ├───aarch64-darwin: development environment 'nix-shell'
   ├───aarch64-linux: development environment 'nix-shell'
   ├───x86_64-darwin: development environment 'nix-shell'
   └───x86_64-linux: development environment 'nix-shell'

This shows that we’ve already defined development environments for MacOS and Linux on both x86 and Arm platforms! To “activate” the correct environment we only need to run nix develop. Nix is smart enough to choose the one compatible with our current system and will proceed to setup all the packages required for that environment.

$ nix develop
(nix-shell) $ command -v python
/nix/store/qc8rlhdcdxaf6dwbvv0v4k50w937fyzj-python3-3.10.8/bin/python

(nix-shell) $ python
Python 3.10.8 (main, Oct 11 2022, 11:35:05) [GCC 11.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>>

Nice!

Tip

See here for details on how I configured my bash prompt to detect if I’m in a nix shell or not.

Adding Python Packages¶

Of course, this environment isn’t that useful at the moment as any of the packages required for esbonio and its test suite are not available

(nix-shell) $ pytest
bash: pytest: command not found

If we’re lucky, the packages we need are already part of nixpkgs and we just need to add them to the devShell’s packages.

devShell = with pkgs;
  mkShell {
    packages = [
      python3

      # esbonio's dependencies
      python3Packages.appdirs
      python3Packages.sphinx
      python3Packages.pygls
      python3Packages.typing-extensions

      # test suite dependencies
      python3Packages.mock
      python3Packages.pytest
      python3Packages.pytest-lsp
      python3Packages.pytest-timeout
    ];
  };

And reactivate the environment

$ nix develop
warning: Git tree '/var/home/alex/Projects/esbonio' is dirty
error: attribute 'pytest-lsp' missing

      at /nix/store/ll2pir6ii65n4cplan9iykxy7cksw6k8-source/lib/esbonio/flake.nix:27:13:

         26|             python3Packages.pytest
         27|             python3Packages.pytest-lsp
           |             ^
         28|             python3Packages.pytest-timeout
(use '--show-trace' to show detailed location information)

Unfortunately, pytest-lsp is not available through nixpkgs but since it’s an unknown library I wrote to help test esbonio I can’t say I’m surprised! 😄 It should however, be relatively straightforward to package it ourselves, especially if we use an example from the nixpkgs repo as a guide.

# In ./nix/pytest-lsp.nix
{ pythonPackages }:

pythonPackages.buildPythonPackage rec {
  pname = "pytest-lsp";
  version = "0.1.3";

  src = pythonPackages.fetchPypi {
    inherit pname version;
    sha256 = "sha256-WxTh9G3tWyGzYx1uHufkwg3hN6jTbRjlGLKJR1eUNtY=";
  };

  buildInputs = [
    pythonPackages.appdirs
    pythonPackages.pygls
    pythonPackages.pytest
  ];

  propagatedBuildInputs = [
    pythonPackages.pytest-asyncio
  ];

  # Disable tests
  doCheck = false;
}

You probably don’t want to use this as an example of packaging a Python package with Nix, as I don’t fully understand what I’m doing and I’ve taken a few shortcuts (like disabling tests), but here’s a few notes.

The { pythonPackages } : syntax at the top of the file is defining a function that accepts pythonPackages as an argument. This is what allows this definition to be used with multiple Python versions later on in this blog post.
As the name implies, the fetchPypi function is used to pull the sources for pytest-lsp straight from PyPi.
propagtedBuildInputs are also available for use at runtime, while buildInputs are “hidden” from the final runtime environment.

Then, to use this package definition in our flake.nix file we use the callPackage function and pass it the correct python package set.

# In ./flake.nix
let
   pkgs = import nixpkgs { inherit system; };
   pytest-lsp = pkgs.callPackage ./nix/pytest-lsp.nix { pythonPackages = pkgs.python3Packages; };
in {
   devShell = with pkgs;
     mkShell {
       packages = [
         # ...
         pytest-lsp
       ];
     };
}

Hopefully, we now have all we need to run the test suite.

(nix-shell) $ pytest
=========================================================================================================== test session starts ============================================================================================================
platform linux -- Python 3.10.8, pytest-7.1.3, pluggy-1.0.0
rootdir: /var/home/alex/Projects/esbonio/lib/esbonio, configfile: pyproject.toml
plugins: typeguard-2.13.3, lsp-0.1.3, asyncio-0.19.0, timeout-2.1.0
asyncio: mode=auto
collected 0 items / 1 error

================================================================================================================== ERRORS ==================================================================================================================
______________________________________________________________________________________________________ ERROR collecting test session _______________________________________________________________________________________________________
/nix/store/qc8rlhdcdxaf6dwbvv0v4k50w937fyzj-python3-3.10.8/lib/python3.10/importlib/__init__.py:126: in import_module
   ...
tests/sphinx-default/conftest.py:12: in <module>
   from esbonio.lsp.sphinx import InitializationOptions
E   ModuleNotFoundError: No module named 'esbonio'
========================================================================================================= short test summary info ==========================================================================================================
ERROR  - ModuleNotFoundError: No module named 'esbonio'

Ah… looks like we have to package esbonio itself, but we already know how to do that, aside from dependencies the only major difference is where we fetch the sources from.

# In ./nix/esbonio.nix

src = ./..

Now we should have everything setup correctly! 🤞

==================================== test session starts =====================================
platform linux -- Python 3.10.8, pytest-7.1.3, pluggy-1.0.0
rootdir: /var/home/alex/Projects/esbonio/lib/esbonio, configfile: pyproject.toml
plugins: typeguard-2.13.3, lsp-0.1.3, asyncio-0.19.0, timeout-2.1.0
asyncio: mode=auto
collected 2487 items

...

=============== 2475 passed, 4 skipped, 8 xfailed in 132.96s (0:02:12) =======================

Success!

Multiple Python Versions¶

Switching to a Nix-ish style of pseudo code for a moment, let’s summarize how our flake is currently defined. We defined a function which takes a system and produces an attribute set (think Python dictionary) with a devShell field

f(system) = { devShell = <devShell for system> }

We then passed that function to the eachDefaultSystem helper from the flake-utils repo. This calls our function with each of the default system architectures before transforming it into a structure compatible with the flake output schema

eachDefaultSystem(f) = applyTransform { aarch64-linux = f(aarch64-linux), ... }
                     = applyTransform { aarch64-linux = { devShell = <devShell for aarch64-linux> }, ... }
                     = { devShell.aarch64-linux.default = <devShell for aarch64-linux>, ... }

Now that we want to support multiple Python versions however, we want to define a function that returns an attribute set with a devShell for each Python version

f(system) = { py37 = <py37 devShell for system>, py38 = <py38 devShell for system>, ... }

Which we can then pass to a mysteryHelper function to perform a similar (but structurally distinct!) transformation on the results of our function f

devShells = mysteryHelper(f)
          = applyTransform { aarch64-linux = f(aarch64-linux), ... }
          = applyTransform {
                             aarch64-linux  = { py37 = <py37 devShell for aarch64-linux>,
                                                py38 = <py38 devShell for aarch64-linux>,
                                                ...
                                              },
                             ...,
                           }
          = {
              aarch64-linux.py37 = <py37 devShell for aarch64-linux>,
              aarch64-linux.py38 = <py38 devShell for aarch64-linux>,
              ...
            }

That’s the idea at least, now to translate it into real Nix code.

Thankfully, finding an implementation for mysteryHelper isn’t too difficult as the flake-utils repo provides eachDefaultSystemMap which does precisely what we want.

outputs = { self, nixpkgs, utils }:
  devShells = utils.lib.eachDefaultSystemMap (system:
    f system;
  );

Now to replace our imaginary function f with an expression that defines our devShells.

Important

Notice that we now assign to devShells?

It turns out that nix the command line tool does a little transformation to turn a devShell entry into a valid devShells entry. Unfortunately, this transformation only works when you define a single shell per system!

Now that we’re defining multiple shells per system, we have to make sure to use devShells - it took me a long time to spot this!

We could simply copy-paste the devShell definition from the previous section a bunch of times and switch out the Python version.

However, since the definitions for each Python version are going to be so similar, a better approach would be to define our own helper that would map a function over a list of versions and have it build the attribute set for us.

It turns out that the implementation of eachDefaultSystemMap is almost identical to what we need, so it was easy enough to adapt it to this use case.

eachPythonVersion = versions: f: builtins.listToAttrs (builtins.map (version: { name = "py${version}"; value = f version; }) versions);

Bringing it all together gives us this final flake definition

outputs = { self, nixpkgs, utils }:

  let
    eachPythonVersion = versions: f: builtins.listToAttrs (builtins.map (version: {name = "py${version}"; value = f version; }) versions);
  in {

  devShells = utils.lib.eachDefaultSystemMap (system:
    let
      pkgs = import nixpkgs { inherit system; };
    in
      eachPythonVersion [ "37" "38" "39" "310" "311" ] (pyVersion:
        let
          pytest-lsp = pkgs.callPackage ./nix/pytest-lsp.nix { pythonPackages = pkgs."python${pyVersion}Packages"; };
          esbonio = pkgs.callPackage ./nix/esbonio.nix { pythonPackages = pkgs."python${pyVersion}Packages"; };
        in

        with pkgs; mkShell {
          name = "py${pyVersion}";

          packages = [
            pkgs."python${pyVersion}"

            esbonio

            # test suite dependencies
            pkgs."python${pyVersion}Packages".mock
            pkgs."python${pyVersion}Packages".pytest
            pytest-lsp
            pkgs."python${pyVersion}Packages".pytest-timeout
          ];
        }
    )
  );
};

With any luck, we should now see a per-python version devShell appear in the output of nix flake show

$ nix flake show
git+file:///var/home/alex/Projects/esbonio?dir=lib%2fesbonio&ref=refs%2fheads%2fnix&rev=4a548327974dff1750099df4d793638a64b663e6
└───devShells
    ├───aarch64-darwin
    │   ├───py310: development environment 'py310'
    │   ├───py311: development environment 'py311'
    │   ├───py37: development environment 'py37'
    │   ├───py38: development environment 'py38'
    │   └───py39: development environment 'py39'
    ├───aarch64-linux
    │   ├───py310: development environment 'py310'
    │   ├───py311: development environment 'py311'
    │   ├───py37: development environment 'py37'
    │   ├───py38: development environment 'py38'
    │   └───py39: development environment 'py39'
    ├───x86_64-darwin
    │   ├───py310: development environment 'py310'
    │   ├───py311: development environment 'py311'
    │   ├───py37: development environment 'py37'
    │   ├───py38: development environment 'py38'
    │   └───py39: development environment 'py39'
    └───x86_64-linux
        ├───py310: development environment 'py310'
        ├───py311: development environment 'py311'
        ├───py37: development environment 'py37'
        ├───py38: development environment 'py38'
        └───py39: development environment 'py39'

To reference a given environment we’d use the .#<envname> syntax when calling nix develop. The --command flag also allows us to run a command within the named environment without having to activate it first!

$ nix develop .#py310 --command pytest
=========================== test session starts ================================
platform linux -- Python 3.10.8, pytest-7.1.3, pluggy-1.0.0
rootdir: /var/home/alex/Projects/esbonio/lib/esbonio, configfile: pyproject.toml
plugins: typeguard-2.13.3, lsp-0.1.3, asyncio-0.19.0, timeout-2.1.0
asyncio: mode=auto
collected 2508 items

...

======== 2496 passed, 4 skipped, 8 xfailed in 344.10s (0:05:27) ================

$ nix develop .#py39 --command pytest
=========================== test session starts ================================
platform linux -- Python 3.9.15, pytest-7.1.3, pluggy-1.0.0
rootdir: /var/home/alex/Projects/esbonio/lib/esbonio, configfile: pyproject.toml
plugins: typeguard-2.13.3, lsp-0.1.3, asyncio-0.19.0, timeout-2.1.0
asyncio: mode=auto
collected 2508 items

...

======== 2496 passed, 4 skipped, 8 xfailed in 344.10s (0:05:44) ================

Achievement unlocked! 🏆

Next Steps¶

This was mainly a “Hello, World” type exercise looking to see if I could get Nix up and running in a real project, but so far I haven’t achieved anything you can’t already do with traditional Python tools like tox. However, this should hopefully serve as a good foundation on which I can explore

Changing the source where dependent libraries are fetched from (e.g. local vs git vs PyPi)
Using overlays (these might help with the previous point?)
Defining environments that contain particular text editor configurations.

If you are interested, you can find the final Nix definitions here.

Bringing Esbonio to the Browser

2021-10-25T00:00:00+00:00

Bringing Esbonio to the Browser¶

For the past year or so I’ve been working on a language server called Esbonio. It’s a language server designed to streamline the process of working with your Sphinx documentation projects. Currently its feature set is quite limited, but I think it does a good job of providing you with completion suggestions for all your roles, directives and cross-references.

Anyway with the recent releases of github.dev and vscode.dev I really want to see if I can bring Esbonio into the browser version of VSCode. There’s only one problem… Esbonio is written in Python! 😬

The Approach¶

I’m sure there will be plenty of problems that need to be solved in order to make this work, but my gut instinct is that it is possible. First of all there is Pyodide which, if you’ve not come across it before, is the Python interpreter, (most of) the standard library and the “scientific stack” compiled down to WebAssembly which can then be executed in a browser.

Also, as part of the vscode.dev announcement stream they demo’d the vscode-pyodide extension which builds on Pyodide and JupyterLite to allow for executable Python notebooks from a web browser! This proves that it’s possible to run Python code via Pyodide from within VSCode running in a web browser.

All that’s left to do is “just” to connect the dots! 😃

Of course, the devil is in the details and there’s going to be all sorts of gotchas on the way to making this work. But I think a rough plan of attack would be to tackle the following steps.

Setup a basic VSCode extension that runs in a browser composed of placeholder language client and server components.
Expand on the language server to include a server implemented with pygls (the library that Esbonio is built on).
Ensure Sphinx works in a browser context. I’m fairly sure Sphinx is a pure Python project so in theory should “just work” but file I/O could prove a challenge.
Bring all of the above together and get the Esbonio language server running.
Ship it!

The Setup¶

Using the lsp-web-extension-sample as a guide, the remainder of this blog post is going to be dedicated to setting up the rough structure of a web-enabled lsp VSCode extension. You can browse the completed extension here.

But before we get too far into it, it’s probably worth zooming out a bit and roughly sketch out all the pieces and how they fit together.

Architecture¶

At the core we have two components, a language client (e.g. our VSCode extension) and a language server (e.g. the esbonio Python package) which communicate by sending messages between each other.

In a web browser context, the client runs as a Web Worker that it managed by VSCode’s Extension Host. The client in turn manages its own Web Worker which hosts the language server. Hopefully, this server web worker will eventually contain an instance of Pyodide running Esbonio managed by a bit of JavaScript glue code.

Writing the Language Client¶

The language client code is pretty straightforward as all the heavy lifting is done by the vscode-languageclient library.

$ npm install --save vscode-languageclient

The client code is pretty much identical to code found in the sample extension I used as reference. All that we have to do it tell the language client which documents we’re interested in

const clientOptions: LanguageClientOptions = {
    documentSelector: [
        { scheme: "file", language: "plaintext" },
    ],
    outputChannelName: "Hello Language Server",
}

and since we’re building this for the web, we need to start a web worker that hosts the language server and pass that to the client also.

const path = Uri.joinPath(context.extensionUri, "dist/server.js")
const worker = new Worker(path.toString())

const client = new LanguageClient("hello-lsp-web", "Hello LSP", clientOptions, worker)
context.subscriptions.push(client.start())

client.onReady().then(() => {
    console.log("hello-lsp-web server is ready")
})

Writing the Language Server¶

The goal of this post is just to get the simplest end-to-end concept working so the “language server” in this case does barely anything except prove that

We can communicate with the client
We can load Pyodide and execute some Python code

Communication with web workers is achieved through sending messages, so to handle incoming messages from the client we create an onmessage event handler and use the postMessage to send our responses.

To prove that the communication works, our “language server” handles the initialize request to establish the session, but ignores everything else.

onmessage = async (event) => {
    console.log(`Client message:`, event.data)

    if (event.data.method === "initialize") {
        postMessage({
            jsonrpc: "2.0",
            id: event.data.id,
            result: {
                serverInfo: { name: "Hello, LSP" },
                capabilities: {}
            }
        })
    }

With the communication taken care of, we can focus on setting up Pyodide. Adapting one of the getting started examples from the Pyodide documentation and using the importScripts function that’s available to web workers it’s relatively straightforward to load Pyodide from a CDN and initialize it.

importScripts("https://cdn.jsdelivr.net/pyodide/v0.18.1/full/pyodide.js")

async function initPyodide() {

    console.log("Initing pyodide.")

    /* @ts-ignore */
    let pyodide = await loadPyodide({
        indexURL: "https://cdn.jsdelivr.net/pyodide/v0.18.1/full/"
    })

    return pyodide
}

const pyodideReady = initPyodide()

Pyodide isn’t exactly a small component to download and especially when we start pulling in packages, will take some time to initialize. Using a global pyodideReady promise we can make any code that depends on pyodide wait until it’s ready to use.

let pyodide = await pyodideReady
console.log(pyodide.runPython("import sys;sys.version"))

Packaging for the Web¶

You may have noticed that the code above was written in TypeScript which needs to be compiled into JavaScript in order to run in the browser. Additionally, due to the way VSCode handles web extensions all the code that comprises the language client (including the dependencies!) needs to be bundled into a single JavaScript file.

To do this we’ll use webpack along with a few other tools.

$ npm install --save-dev webpack webpack-cli typescript ts-loader @types/vscode path-browserify

Like most of this setup, the webpack configuration was based on the lsp-web-extension-sample where we export 2 configurations, one for the client and one for the server.

const clientConfig = {
   target: 'webworker'
   entry: {
      client: './src/client'
   },
   resolve: {
      fallback: {
         path: require.resolve('path-browserify')
      }
      ...
   },
   ...,
   externals: {
      vscode: 'commonjs vscode'
   }
}

const serverConfig = {
   target: 'webworker',
   entry: {
      server: './src/server'
   },
   ...
}

module.exports = [clientConfig, serverConfig]

I’ve omitted most of the more standard configuration fields for brevity but you can look at the code for full details though there are a few things worth mentioning

Both client and server will be running in web workers so we need to make sure we tell webpack to target: webworker
The client depends on the VSCode API but it’s not available at build time, so we use the externals field to tell webpack to translate any import {} from 'vscode' statements into a CommonJS import.
Node libraries like path are not available in the browser, so we use the fallback field to replace calls to the path library, will calls to the path-browserify library which implements the same API, but works within the browser.

Finally to invoke webpack we can add a couple of scripts to our package.json for convenience.

"scripts": {
   "compile": "webpack",
   "watch": "webpack --watch"
}

Trying it out¶

Note

I don’t fully understand how this section works, thankfully the VSCode devs figured out these steps and wrote them up

The last step is to actually try to run this extension in the web version of VSCode and see if it works. Unfortunately testing a web extension is not as straightforward as a desktop one, but with a few npm commands ✨magic happens✨ and the web version of VSCode is able to install our extension from a simple web server running on localhost

First we start by adding a few more scripts to our package.json

"scripts": {
   "serve": "npx serve --cors -l 5000",
   "tunnel": "npx localtunnel -p 5000"
}

Now assuming that we’ve already run npm run watch or npm run compile from the previous section, we run both the serve and tunnel scripts from two separate terminals

$ npm run serve

> hello-lsp-web@ serve blog/code/hello-lsp-web
> npx serve --cors -l 5000

npx: installed 88 in 6.613s

┌──────────────────────────────────────────────────┐
│                                                  │
│   Serving!                                       │
│                                                  │
│   - Local:            http://localhost:5000      │
│   - On Your Network:  http://192.168.0.31:5000   │
│                                                  │
│   Copied local address to clipboard!             │
│                                                  │
└──────────────────────────────────────────────────┘

$ npm run tunnel

> hello-lsp-web@ tunnel blog/code/hello-lsp-web
> npx localtunnel -p 5000

npx: installed 22 in 3.043s
your url is: https://xxxx-yyyy-zzzz.loca.lt

Then to enable the tunnel we have to visit the https://xxxx-yyyy-zzz.loca.lt URL printed by the tunnel script, which takes us to a “Friendly Reminder” screen and we click the Click to Continue button.

Now we can finally open the web version of VSCode itself, open the command palette with F1 and pick the Developer: Install Web Extension... command. When asked for the URL to install from, we paste the URL given to us from the tunnel script above.

With any luck, VSCode should install the extension show it in the installed extensions list in the Extensions tab. All that’s left us to do is actually test the extension!

As you can probably guess from the code we wrote above, there’s actually not much for us to test. However, we can open the dev tools as you would on any other website and create a file test.txt if everything works as expected, we should see the following messages printed to console.

Success!¶

Next Steps¶

That’s it! We have a very simple proof of concept web extension setup where the language server component is able to run Python code and communicate with the client. In the next post I hope to be able to stand up a simple pygls language server and have it work with the online version of VSCode 🤞

Creating a CPython Extension

2021-01-13T00:00:00+00:00

Creating a CPython Extension¶

Previously, as part of my exploration into how programming languages are implemented, I wrote a very simple AST evaluator that knew how to add and multiply floats together. Since constructing these ASTs by hand is quite painful I thought it would be fun to come up with a frontend to my “programming language” which could do it for me.

Now your typical frontend would be some kind of parser built into the compiler/interpreter. However, while I’m definitely interested in parsing I don’t quite feel like tackling that just yet. Instead I’m going to have Python be the frontend and embed my toy language into it via a CPython Extension

Overview¶

Before diving into the detail I think it would be worth keeping in mind what we want the end result to be. By the end of this post, we want to have a CPython extension that allows us to write normal-ish Python code to construct a representation of some expression

>>> import ccalc
>>> expression = (ccalc.Literal(1) + 2) * 3
>>> expression
Multiply<Plus<Literal<1.0>, Literal<2.0>>, Literal<3.0>>

and then be able to pass this expression to the AST evaluator we wrote in the previous post and have it compute the result

>>> ccalc.eval_ast(expression)
9.0

If you’d rather skip all the exposition you can find the final codebase

We can break the implementation down into 3 main steps

Constructing an AST Representation

We need an equivalent Python representation to the AstNode structure, allowing the user to express the expression they want computed.
Setting up the C Extension

Before we can get to the fun part, there’s some setup required to get a CPython extension up and running.
Converting Between Python and C

Finally we need to write the code that converts the Python representation into the C representation, passing it off to the evaluator before converting the result back into Python.

Constructing an AST Representation¶

To represent the AST in Python code we can create a class that captures the same information as our AstNode struct from the C code

class AstNode:

   LITERAL = 0
   PLUS = 1
   MULTIPLY = 2

   def __init__(self, type=None, value=None, left=None, right=None):
      self.type = type
      self.value = value
      self.left = left
      self.right = right

From there it’s simple enough to create some subclasses that help us fill out the correct fields.

class Literal(AstNode):
   def __init__(self, value):
      super().__init__(type=AstNode.LITERAL, value=float(value))

class Plus(AstNode):
   def __init__(self, left, right):
      super().__init__(type=AstNode.PLUS, left=left, right=right)

class Multiply(AstNode):
   def __init__(self, left, right):
      super().__init__(type=AstNode.MULTIPLY, left=left, right=right)

Technically that’s all we need but we haven’t really gained anything in terms of usability, constructing an AST from the classes we have defined so far would be just as painful as it was in C.

Thankfully though, we don’t have to stop here, by taking advantage of being able to define implementations for arithmetic operations on our custom types we can introduce a much nicer method of constructing expressions.

class AstNode:
   ...

   def __init__(self, type=None, value=None, left=None, right=None):

      # Automatically convert python numbers to Literal(x) AST nodes
      if left is not None and not isinstance(left, AstNode):
            left = Literal(left)

      if right is not None and not isinstance(right, AstNode):
            right = Literal(right)

      ...

   def __add__(self, other):
      return Plus(self, other)

   def __radd__(self, other):
      return Plus(other, self)

   def __mul__(self, other):
      return Multiply(self, other)

   def __rmul__(other, self):
      return Multiply(other, self)

Now if we wanted to construct an expression we can do so with fairly straightforward Python code.

>>> import ccalc

>>> (ccalc.Literal(1) + 2) * 3
Multiply<Plus<Literal<1.0>, Literal<2.0>>, Literal<3.0>>

>>> ccalc.Literal(1) + (2 * 3)
Plus<Literal<1.0>, Literal<6.0>>

However, as shown with the second example above we need to be careful when choosing the number to wrap in our ccalc.Literal class if we want to “catch” the expression and construct our AST rather than have Python compute the value directly

>>> 1 + (ccalc.Literal(2) * 3)
Plus<Literal<1.0>, Multiply<Literal<2.0>, Literal<3.0>>

Setting up the C Extension¶

Using this tutorial from Real Python as a guide I was able to get a C Extension up and running surprisingly easily. Be sure to check out the article for details but in short I ended up creating the following directory structure

.
├── ccalc
│   └── __init__.py
├── ccalcmodule.c
└── setup.py

Where the __init__.py file contains the Python code we wrote in the previous section and ccalcmodule.c contains the boilerplate needed to define a Python module

#define PY_SSIZE_T_CLEAN
#include <Python.h>

static struct PyModuleDef ccalcmodule = {
   PyModuleDef_HEAD_INIT,
   "_ccalc",
   "Simple calculator implemented in C",
   -1,
   ccalc_methods
};

PyMODINIT_FUNC
PyInit__ccalc()
{
   return PyModule_Create(&ccalcmodule);
}

The PyModuleDef struct as the name implies, defines some basic information about the module

It’s name _ccalc, specifies what our module is called when we import it in regular Python code
The next argument is the module’s docstring
-1 is something to do with sub-interpreters?
ccalc_methods is an array of structs delcaring all the functions this module exposes to the interpreter.️

Something that caught me out is that the PyInit_<module name> function must match the name we gave the module in PyModuleDef, since the module name is _ccalc I needed an additional _ character in the name so that the module can be registered correctly.

The methods declared in the ccalc_methods array follow a similar pattern

static PyMethodDef ccalc_methods[] = {
   {"hello_world", method_hello_world, METH_VARARGS, "Print 'Hello, World!'."},
   {NULL, NULL, 0, NULL} // I think this is required so that Python knows when
                         // it's reached the end of the array
};

hello_world is the name we want regular Python code to use when calling this method
method_hello_world is the name of the function in our C code
METH_VARARGS tells Python the kinds of arguments our function should be called with. Check out the documentation for more details.
The final parameter sets the docstring for the function

Finally we need the the actual method definition itself.

static PyObject*
method_hello_world(PyObject *self, PyObject *args)
{
   printf("Hello, World!\n");
   Py_RETURN_NONE;
}

Building the Extension¶

To my surprise, this was the easiest step of them all. Rather than worrying about writing a Makefile or providing the right flags to link against my version of Python, it turns out that setuptools takes care of all those details.

All I had to do was write a standard setup.py file, just with some additional information about the extension itself

from setuptools import setup, Extension

ccalcmod = Extension("_ccalc", sources=["ccalcmodule.c"], language="c")

setup(
    name="ccalc",
    version="1.0.0",
    description="A simple calculator implemented in C",
    packages=["ccalc"],
    ext_modules=[ccalcmod],
)

With the packaging defined a python setup.py install was all that was needed to build and install the extension into my virtual environment.

(.env) $ python setup.py install

running install
running bdist_egg
running egg_info
creating ccalc.egg-info
writing ccalc.egg-info/PKG-INFO
writing dependency_links to ccalc.egg-info/dependency_links.txt
writing top-level names to ccalc.egg-info/top_level.txt
writing manifest file 'ccalc.egg-info/SOURCES.txt'
reading manifest file 'ccalc.egg-info/SOURCES.txt'
writing manifest file 'ccalc.egg-info/SOURCES.txt'
installing library code to build/bdist.linux-x86_64/egg
running install_lib
running build_py
creating build
creating build/lib.linux-x86_64-3.8
creating build/lib.linux-x86_64-3.8/ccalc
copying ccalc/__init__.py -> build/lib.linux-x86_64-3.8/ccalc
running build_ext
building '_ccalc' extension
creating build/temp.linux-x86_64-3.8
x86_64-linux-gnu-gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/home/alex/Projects/scratch/.env/include -I/usr/include/python3.8 -c ccalcmodule.c -o build/temp.linux-x86_64-3.8/ccalcmodule.o
x86_64-linux-gnu-gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-Bsymbolic-functions -Wl,-z,relro -g -fwrapv -O2 -Wl,-Bsymbolic-functions -Wl,-z,relro -g -fwrapv -O2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 build/temp.linux-x86_64-3.8/ccalcmodule.o -o build/lib.linux-x86_64-3.8/_ccalc.cpython-38-x86_64-linux-gnu.so
creating build/bdist.linux-x86_64
creating build/bdist.linux-x86_64/egg
creating build/bdist.linux-x86_64/egg/ccalc
copying build/lib.linux-x86_64-3.8/ccalc/__init__.py -> build/bdist.linux-x86_64/egg/ccalc
copying build/lib.linux-x86_64-3.8/_ccalc.cpython-38-x86_64-linux-gnu.so -> build/bdist.linux-x86_64/egg
byte-compiling build/bdist.linux-x86_64/egg/ccalc/__init__.py to __init__.cpython-38.pyc
creating stub loader for _ccalc.cpython-38-x86_64-linux-gnu.so
byte-compiling build/bdist.linux-x86_64/egg/_ccalc.py to _ccalc.cpython-38.pyc
creating build/bdist.linux-x86_64/egg/EGG-INFO
copying ccalc.egg-info/PKG-INFO -> build/bdist.linux-x86_64/egg/EGG-INFO
copying ccalc.egg-info/SOURCES.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying ccalc.egg-info/dependency_links.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying ccalc.egg-info/top_level.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
writing build/bdist.linux-x86_64/egg/EGG-INFO/native_libs.txt
zip_safe flag not set; analyzing archive contents...
__pycache__._ccalc.cpython-38: module references __file__
creating dist
creating 'dist/ccalc-1.0.0-py3.8-linux-x86_64.egg' and adding 'build/bdist.linux-x86_64/egg' to it
removing 'build/bdist.linux-x86_64/egg' (and everything under it)
Processing ccalc-1.0.0-py3.8-linux-x86_64.egg
removing '/home/alex/Projects/scratch/.env/lib/python3.8/site-packages/ccalc-1.0.0-py3.8-linux-x86_64.egg' (and everything under it)
creating /home/alex/Projects/scratch/.env/lib/python3.8/site-packages/ccalc-1.0.0-py3.8-linux-x86_64.egg
Extracting ccalc-1.0.0-py3.8-linux-x86_64.egg to /home/alex/Projects/scratch/.env/lib/python3.8/site-packages
ccalc 1.0.0 is already the active version in easy-install.pth

Installed /home/alex/Projects/scratch/.env/lib/python3.8/site-packages/ccalc-1.0.0-py3.8-linux-x86_64.egg
Processing dependencies for ccalc==1.0.0
Finished processing dependencies for ccalc==1.0.0

With the C code sorted and building, we can import it in Python code and call functions from it just as we would with any other module.

>>> import _ccalc
>>> _ccalc.hello_world()
Hello, World!

Converting Between Python and C¶

Now for the fun part! It’s time to write a method for our extension module method_eval_ast that takes a Python representation of the AST and converts it into our C representation before executing it and passing the result back up to Python.

static PyObject*
method_eval_ast(PyObject *self, PyObject *args)
{
   ...
}

// Be sure to expose the new method to the module
static PyMethodDef ccalc_methods[] = {
   {"eval_ast", method_eval_ast, METH_VARARGS, "Evaluate the given ast."},
   ...
}

This can be broken down into a three step process

Parsing Function Arguments¶

When writing a METH_VARARGS style function, it gets called with 2 parameters self and args. self in this case is a reference to our _ccalc module and args is a reference to a tuple containing the arguments that were passed to our function.

As the contents of this tuple can be arbitrary it’s up to our code to correctly interpret the values that have been given to it. Thankfully Python provides a handy function PyArg_ParseTuple that can take care of this for us.

PyObject *obj = NULL;

if (!PyArg_ParseTuple(args, "O", &obj)) {
   return NULL;
}

This function takes a format string that specifies the number and type of arguments we expect to be given. In this case "O" says that we want to take a single object - our AST. We also need to pass the correct number of pointers into this function so that it can “return” the parsed values to us.

In the case of invalid arguments being given, this function will set the global error indicator for us with the correct error message. So all that would be left for us to do is to return NULL which indicates to the code calling us that there was an error. See the documentation on error handling for more details

Constructing the C AST¶

With a reference to the Python object that (hopefully!) represents a valid AST it’s time to do the conversion into our C representation. To do this we’ll write a function dedicated to handling the conversion and call it from method_eval_ast

AstNode *ast = AstTree_FromPyObject(obj);
if (ast == NULL) {
   return NULL;
};

The first step is to dynamically allocate enough memory to store the C representation. Easy enough to do, assuming that you know the size of the tree…

Allocating Memory¶

It took me a while to realise it, but even though we’re writing C code we are still within the Python interpreter. This means we still have access to all the standard Python functions - we just need to look up their C equivalents. Why not just ask the tree itself how big it is by calling len() on it?

After a quick trip to the documentation I discover that len is “spelt” PyObject_Length in the C API, add some of the required book keeping and we should be able to allocate enough space

static AstNode*
AstTree_FromPyObject(PyObject *obj)
{
   Py_ssize_t num_nodes = PyObject_Length(obj);
   if (num_nodes == -1) {
      return NULL;
   }

   AstNode *ast = malloc(num_nodes * sizeof(AstNode));
   if (ast == NULL) {
      return NULL;
   }

   // ...
}

Something important to note here, up until now we’ve been calling into the Python C API which has been taking care of reporting any errors it encounters. However, the call to malloc is now our code and it’s our responsibility to correctly report any errors we counter.

If we were to leave the code as is and this call to malloc fails, Python would know that an error had occured but not be able to tell the user what was wrong.

>>> import ccalc
>>> import _ccalc
>>> _ccalc.eval_ast(ccalc.Literal(2))
Traceback (most recent call last):
   File "<stdin>", line 1, in <module>
SystemError: <built-in function eval_ast> returned NULL without setting an error

Instead we also need to call PyErr_SetString to raise the appropriate exception that describes our error.

AstNode *ast = malloc(num_nodes * sizeof(AstNode));
if (ast == NULL) {
   PyErr_SetString(PyExc_MemoryError, "Unable to allocate memory for the AST.");
   return NULL;
}

With the information set, Python is able to report a much better error message to the user

>>> import ccalc
>>> import _ccalc
>>> _ccalc.eval_ast(ccalc.Literal(2))
Traceback (most recent call last):
   File "<stdin>", line 1, in <module>
MemoryError: Unable to allocate memory

Finally let’s not forget to jump back to the Python code and implement __len__ on our AstNode class.

class AstNode:
   ...

   def __len__(self):
      left = 0 if self.left is None else len(self.left)
      right = 0 if self.right is None else len(self.right)
      return 1 + left + right

Inspecting Nodes¶

With the memory to hold the tree allocated it’s time to start on the actual conversion. To handle this we’ll write another function AstNode_FromPyObject that we can recursively call whenever we need to descend down a branch. This function will take a reference obj to the Python representation of the node we’re currently converting, another reference ast to the memory we allocated and finally an index into the array that we should write the node to.

static int
AstNode_FromPyObject(PyObject *obj, AstNode *ast, Py_ssize_t index)
{
   AstNode *node = &astindex];
   // ...
}

The first step in the process is to determine the type of node we are converting, which we can do by inspecting the value of the type field.

PyObject *type = PyObject_GetAttrString(obj, "type");
if (type == NULL) {
   return 0;
}

long node_type = PyLong_AsLong(type);
Py_DECREF(type);

The call to PyObject_GetAttrString is equivalent to obj.type in Python and returns a new reference to a generic Python object. In order to get an actual number we need to use PyLong_AsLong to convert it.

In theory type could be a reference to anything, so there’s always the chance that PyLong_AsLong could fail in which case it would return -1. As stated in the documentation we should really be performing extra checks here to determine if the value is actually -1 or if there was an error but I’ve decided to omit those for now.

Something else to note is that type was a new reference and since Python uses Reference Counting internally it’s our responsibility to decrement the count (using Py_DECREF) when we are finished with it - at least that’s what I think should be done based on what I found in the documentation on ownership

Converting the Node¶

Now that we have an integer node_type that corresponds with one of the AstNodeType enum entries we can use a switch statement and start writing the conversion code for each type in turn.

switch(node_type) {
case AST_LITERAL: {

   PyObject *value = PyObject_GetAttrString(obj, "value");
   if (value == NULL) {
      return 0;
   }

   double v = PyFloat_AsDouble(value);
   Py_DECREF(value);

   // ...
}
}

Considering the Literal node types first, we can follow a very similar process to the previous section to extract the value field from the node giving us enough information to fill out our first AstNode instance!

node->type = AST_LITERAL;
node->value = v;
node->left = NULL;
node-> right = NULL;

return 1;

As this node type has no children there’s no further work to do and we can return successfully. However, in the case of AST_PLUS and AST_MULTIPLY things aren’t as straightforward…

Traversing the Tree¶

Handling the other node types starts off easy enough, since they are almost identical we can handle their differences in the switch statement and then use the remainder of the function to handle recursing down both the left and right branches.

case AST_PLUS: {
   node->type = AST_PLUS;
   break;
}
case AST_MULTIPLY: {
   node->type = AST_MULTIPLY;
   break;
}

We can then get references to the child nodes in the same way we’ve been referencing all the other fields so far

PyObject *left = PyObject_GetAttrString(obj, "left");
if (left == NULL) {
   return 0;
}

PyObject *right = PyObject_GetAttrString(obj, "right");
if (right == NULL) {
   return 0;
}

Then “all” that is left to do is set the left and right pointers on the AstNode struct and call AstNode_FromPyObject on each branch - remembering to adjust the index value accordingly.

node->left = &ast[index + 1];
node->right = &ast[index + 2];

if (!AstNode_FromPyObject(left, ast, index + 1)) {
   return 0;
}

if (!AstNode_FromPyObject(right, ast, index + 2)) {
   return 0;
}

return 1;

At least… that’s what I wanted to do initially, unfortunately this solution wouldn’t work in practice,. If left is a reference to anything other than an AST_LITERAL then it’s children would be overwritten when we start processing nodes on the right branch!

This had me scratching my head for quite a while, trying to come up with a way to compute the correct offset for the right branch - without success.

Instead, since this is C I ended up changing the index argument from an actual Py_ssize_t to a pointer to one. This allows recursive calls to increment the index as needed and by the time execution returns to the top level function, the value referenced by the pointer is automatically the correct value.

(*index)++;
node->left = &ast[*index];
if (!AstNode_FromPyObject(left, ast, index)) {
   return 0;
}

(*index)++;
node->right = &ast[*index];
if (!AstNode_FromPyObject(right, ast, index)) {
   return 0;
}

return 1;

I have no idea if this is a terrible idea for real world scenarios, but it seems to work well enough for this at least.

To complete the conversion code, all that remains is to make the initial call to AstNode_FromPyObject from our main AstTree_FromPyObject function

Py_ssize_t index = 0;
if (!AstTree_FromPyObject(obj, ast, &index)) {
   free(ast);
   return NULL;
}

return ast;

Returning the Result¶

Phew! That was a lot of work but we’re almost there. We just need to add a few more lines to method_eval_ast that takes the newly constructed AST and evaluates it, before converting the result into a Python float and returning it.

double result = ast_evaluate(ast);
free(ast);
return PyFloat_FromDouble(result);

That’s the C Extension finished, the only thing we could do is import the _ccalc module from ccalc and expose the methods we want to present a unified interface to users of the module.

# In ccalc/__init__.py
import _ccalc

eval_ast = _ccalc.eval_ast

And that way we can now make the example code from the start of this post actually work!

>>> import ccalc
>>> expression = (ccalc.Literal(1) + 2) * 3
>>> expression
Multiply<Plus<Literal<1.0>, Literal<2.0>>, Literal<3.0>>

>>> ccalc.eval_ast(expression)
9.0

Final Thoughts¶

There was a lot of code flying around in this post, if you want to see the final result in its entirety you can find it

Best Practice?¶

This was my first CPython extension so I might be missing out on some best practices, for example a question I had was around my use of malloc and free. While writing this section I found the documentation on memory management and it appears that the recommendation is to use the PyMem_* family of functions, even for allocations that are not PyObjects. However it looks like there is a learning curve to these as some functions require holding the GIL and some don’t.

Hmm… speaking of the GIL, should this extension be acquiring it at any point? 🤔

Another approach¶

Another note worth mentioning is that it looks like it’s possible to define custom types directly in C. This means it should be possible to extend the AstNode C struct to be an object than can be manipulated directly from Python code - bypassing the need for all the conversion code we had to write.

However, I decided against this approach mainly because the “convert between the representations” approach means we get to evolve the Python and C representations semi independently. Assuming that the Python representation exposes the correct fields then any method of generating the AST from Python is perfectly valid.

Anyway, I think this post has gone on long enough I hope you found it useful and I’ll see you in the next one!

Evaluating a Simple Abstract Syntax Tree

2020-12-17T00:00:00+00:00

Evaluating a Simple Abstract Syntax Tree¶

Programming languages and their implementation is a topic I’ve been interested in for a long time and I thought it would be worth trying to get a bit more hands on and play with some of the ideas in this space. Choosing a topic somewhat at random I’ve chosen to take a look at implementing an Abstract Syntax Tree (AST).

30/12/2025

Edited this post to

Reorganise code snippets so that we can use the current snapshot version of awdur
Add section on printing the AST
Convert mermaid diagrams to SVG

What is an Abstract Syntax Tree?¶

An Abstract Syntax Tree is a way to represent a program’s source code within an interpreter/compiler. Consider an expression like 1 + 2, it could be represented by the following tree.

One of the nice things about ASTs is that they remove some of the ambiguity that can exist in plain text. Consider the expression 1 + 2 * 3, there are two ways to interpret it and depending on which you choose you will get a different result - either (1 + 2) * 3 = 9 or 1 + (2 * 3) = 7 (rules like BODMAS dictate that we should choose the second version but there’s still a choice). This choice gets encoded into the structure of tree itself

Encoding a program in this way allows the interpreter/compiler to focus on the semantic meaning of the program without having to worry about the finer details of how that program happens to be written down.

Representing the AST¶

For no particular reason other than I fancied trying it, I decided to use C to represent my AST. In order to encode the example trees above, we’ll need three node types

simple-ast.c

#include <stdio.h>

typedef enum {
    AST_LITERAL,
    AST_PLUS,
    AST_MULTIPLY,
} AstNodeType;

Where each node is represented by the following struct.

simple-ast.c

typedef struct _ast {
    /* The type of node this represents e.g.
       a literal, an operator etc. */
    AstNodeType type;

    /* In the case of a literal value, this field
       holds the actual value */
    float value;

    /* In the case of an operator, this pointer
       refers to the left child node */
    struct _ast *left;

    /* In the case of an operator, this pointer
       refers to the right child node */
    struct _ast *right;
} AstNode;

Although the typedef will let us refer to this struct using the name AstNode from here on out, this name does not exist within the body of the struct definition itself. In order to have the struct hold pointers to other instances of the same type it’s necessary to also name the struct definition _ast

Printing the AST¶

Keeping things simple, we can recurse over all the nodes and print a single line representation of each node. By indenting each line according to its depth in the tree, it should be possible to visually parse the structure.

simple-ast.c

void
ast_print(AstNode *ast, int level)
{
    for (int i = 0; i < level; i++) {
        printf("  ");
    }

When printing an AST_LITERAL node, it’s enough to just print the contained value

simple-ast.c

switch(ast->type) {
case AST_LITERAL:
    printf("%.2f\n", ast->value);
    break;

For AST_PLUS nodes, we need to print a representation of the operation and then recurse down both branches of the tree.

simple-ast.c

case AST_PLUS:
    printf("+\n");
    ast_print(ast->left, level + 1);
    ast_print(ast->right, level + 1);
    break;

Similarly for AST_MULTIPLY nodes.

simple-ast.c

    case AST_MULTIPLY:
        printf("*\n");
        ast_print(ast->left, level + 1);
        ast_print(ast->right, level + 1);
        break;
    }
}

Evaluating the AST¶

To evaluate an instance of the AST we have defined we can take a similiar approach to printing it.

If the node we are evaluating is an AST_LITERAL then all we have to do is return the value stored in that node

simple-ast.c

float
ast_evaluate(AstNode *ast)
{
    switch(ast->type) {
    case AST_LITERAL:
        return ast->value;

In the case of AST_PLUS, we recursively call ast_evaluate on both the left and right branches of the tree and then add the resulting values together

simple-ast.c

case AST_PLUS: {
    float a = ast_evaluate(ast->left);
    float b = ast_evaluate(ast->right);

    return a + b;
}

Simiarly for AST_MULTIPLY, but returing a * b in this case.

simple-ast.c

    case AST_MULTIPLY: {
        float a = ast_evaluate(ast->left);
        float b = ast_evaluate(ast->right);

        return a * b;
    }
    }
}

Bringing it together¶

We have enough of a toy example together to be able to define, print and evaluate a tree for the two examples in the introduction.

simple-ast.c

int
main()
{
    AstNode one = { .type = AST_LITERAL, .value = 1.0f };
    AstNode two = { .type = AST_LITERAL, .value = 2.0f };
    AstNode three = { .type = AST_LITERAL, .value = 3.0f };

    AstNode plus1 = { .type = AST_PLUS, .left = &one, .right = &two};
    AstNode example1 = { .type = AST_MULTIPLY, .left = &plus1, .right = &three};

    AstNode mul1 = { .type = AST_MULTIPLY, .left = &two, .right = &three};
    AstNode example2 = { .type = AST_PLUS, .left = &one, .right = &mul1};

    float result1 = ast_evaluate(&example1);
    float result2 = ast_evaluate(&example2);

    ast_print(&example1, 0);
    printf("Example 1: %.2f\n", result1);

    ast_print(&example2, 0);
    printf("Example 2: %.2f\n", result2);

    return 0;
}

Which we can execute with the help of a compiler.

$ gcc simple-ast.c -o simple-ast
$ ./simple-ast
*
  +
    1.00
    2.00
  3.00
Example 1: 9.00
+
  1.00
  *
    2.00
    3.00
Example 2: 7.00

And there you have it, a calculator that only knows how to add and multiply floats! If you want to see the entire source file you can find it here.

Passing strings between TinyGo and JavaScript

2020-05-06T00:00:00+00:00

Passing strings between TinyGo and JavaScript¶

After getting a "Hello, World!" WebAssembly application working I thought it would be fun to try and implement a toy programming language in the browser. However before I could even start thinking about parsers, abstract syntax trees and the like I had to be able to pass strings between my WebAssembly module and the surrounding JavaScript.

Turns out that is much trickier than I expected.

Disclaimer

I definitely don’t know what I’m doing, if there’s a better way of doing this I’d love to know about it! 🙂

The Interface¶

The Interface to our ‘interpreter’¶

Our WebAssembly module needs a way to interact with the user in order both collect input and show the output. So to start with I updated the webpage from the previous post to include a number of textarea elements and a “Run” button.

<div>
   <h3>Input</h3>
   <textarea id="input"></textarea>
</div>
<div>
   <button disabled id="run-button">Run</button>
</div>
<div>
   <h3>Output</h3>
   <textarea disabled id="output"></textarea>
</div>
<div class="log-area">
   <textarea disabled id="log"></textarea>
</div>

Now we could just use the println function to do the equivalent of a console.log from our TinyGo code but it could be useful to have a log on screen to provide feedback to the user.

Passing strings from TinyGo to JavaScript¶

To achieve this we first need a JavaScript function which takes some text and appends it to the “log” textarea element on screen.

const log = document.getElementById('log')

function logText(text) {
   log.value += text + '\n'
}

Next we need some way to reference such a function from our go code. If we define a function but not implement it TinyGo will recognise it as an external function whose implementation should be provided by the surrounding JavaScript.

package main

func log(message string)

func main() {
   log("Hello, World!")
}

Note

The Go tools inside of VSCode will complain about the missing implementation for the log function but TinyGo itself will compile this happily.

Lastly all we have to do is adjust our wrapping JavaScript code to pass in the implementation for the log function to the module’s environment. While we’re at we might as well wire up that “Run” button.

...

function onRun(runner, wasm) {
   return (event) => runner.run(wasm)
}

go.importObject["main.go.log"] = logText

WebAssembly.instantiateStreaming(fetch("/js/wisp.wasm"), go.importObject)
   .then(module => {
      let wasm = module.instance

      runButton.disabled = false
      runButton.addEventListener('click', onRun(go, wasm))
   })

That should be everything connected up, time to give it a whirl!

Not exactly what I had in mind…¶

Hmm… 🤔

This result quickly prompted an extended session of searching around for the “right” way to pass values back and forth between my WebAssembly module and the surrounding JavaScript code. Unfortunately I didn’t really come across anything that seemed to work for me.

I did find that Go has a syscall/js module which seems to handle exactly this kind of thing along with a tutorial series that makes use of it. The problem is that it seems to fly in the face of the examples (which I did manage to reproduce) on the TinyGo website where it appears the compiler is handling most of these details.

Unable to find an example to copy I decided it was time for a peek behind the curtain…

Digging Deeper¶

On my travels I did manage to find out some more information about WebAssembly itself

WebAssembly only has basic integer and float types
A module has its own memory and is represented by an ArrayBuffer in JavaScript code.

Before long I had a hunch that the random number that was being displayed in the textarea element was in fact the memory address of the string to be logged. If that was the case, how should the memory in that location be interpreted so that we’re able to extract a string from it?

After some more research I discovered that TinyGo is using the LLVM compiler toolchain and that you can ask it for the intermediate representation which it passes to LLVM in order to generate the WebAssembly code.

$ tinygo build -no-debug -target wasm -printir -o public/js/wisp.wasm main.go > debug.txt

Now, there is a lot going on (20,000+ lines) in this file as it includes not just our simple program but the Go runtime required to execute it. Thankfully with Ctrl-F to the rescue, it’s easy enough to track down where our “Hello, World!” string is defined

...
@"main.go.main$string" = internal unnamed_addr constant [13 x i8] c"Hello, World!"
...

I don’t know the first thing when it comes to LLVM’s IR representation of code but after looking into how it thinks about arrays we see that [13 x i8] indicates that our string is represented by an array of 13, 8-bit integers.

Awesome, we now know how to interpret the values we see in the WebAssembly module’s ArrayBuffer, but how will we know how many values to look for?

Let’s try and find our log function…

declare void @main.go.log(i8*, i32, i8*, i8*)

Ah, just like the log function in our go code it has no implementation since this is to be provided by the JavaScript wrapper. However instead of a single argument it now takes 4! Interesting… let’s track down our main function and see how it is used.

define dso_local void @main.go.main(i8* %context, i8* %parentHandle) unnamed_addr {
entry:
call void @main.go.log(i8* getelementptr inbounds ([13 x i8], [13 x i8]* @"main.go.main$string", i32 0, i32 0), i32 13, i8* undef, i8* undef)
   ret void
}

Wow. Umm… there’s a lot going on here… what if we try “squinting” at this code a bit

define void @main.go.main(...) {
call @main.go.log(i8* getelementptr (... @"main.go.main$string",...), i32 13, ...)
   ret void
}

Ok, so it looks like there’s a call being made to our log function and getelementptr appears to be returning the memory address of our “Hello, World!” string and look! The i32 13 argument appears to be passing in its length! I have no idea what the other arguments are supposed to represent so let’s just ignore those for now! 😃

Instead why don’t we tweak our logText implementation of this function to take a second argument and see what happens

function logText(addr, length) {
   log.value += addr + " " + length + '\n'
}

That looks promising!¶

Extracting the String¶

Assuming our assumptions are correct we should now have all the information we need in order to extract the string from the WebAssembly module’s memory. ArrayBuffer objects in JavaScript don’t seem to be the most intuitive to work with but I was eventually able to come up with this.

function logText(addr, length) {
   let memory = wasm.exports.memory
   let bytes = memory.buffer.slice(addr, addr + length)
   let text = String.fromCharCode.apply(null, new Int8Array(bytes))

   log.value += text + '\n'
}

After getting the reference to the memory object we use the slice method to obtain a copy of only the bytes that represent our string. Unfortunately bytes on their own are meaningless unless you know how to interpret them. To enable this there is a whole collection of views that can be wrapped around a given array of bytes to attach meaning to them. From our explorations above we know that we should use an Int8Array.

From there we map the String.fromCharCode function over the array of ints to convert them to a string. Finally, we should be able to see our “Hello, World!” string output to the log area

Success!¶

This did require a slight tweak to the way we load the module so that we have a global wasm reference that our logText function is able to use to access the module instance directly.

let wasm

...

WebAssembly.instantiateStreaming(fetch("/js/wisp.wasm"), go.importObject)
   .then(module => {
      wasm = module.instance

      runButton.disabled = false
      runButton.addEventListener('click', onRun(go, wasm))
   })

Passing strings from JavaScript to TinyGo¶

Phew, at least we’re halfway there! Now that we’ve figured out how things actually hang together it’s “just” a matter of doing the inverse process to pass a string from our JavaScript code into our WebAssembly module. As a proof of concept let’s create an echo function in TinyGo that will simply log whatever text it receives.

//go:export echo
func echo(message string) {
   log(message)
}

In order to actually pass a string to this function, we need to insert the string into the memory of the WebAssembly module before calling echo with its address and length.

Manipulating Memory¶

The problem is, where exactly in the module’s memory should we place the string? We can’t shove it anywhere as we could corrupt memory required for some other part of the program. It is possible to grow the memory assigned to a module instance which technically would be free for us to use(?) But it doesn’t exactly feel right, having two competing codebases manipulate the same memory layout seems to be asking for trouble…

Thankfully I came across this comment on an issue in the TinyGo project which provides a way we can work around this.

var buf [1024]byte

//go:export getBuffer
func getBuffer() *byte {
   return &buf[0]
}

If we declare an array of bytes in the module, the TinyGo compiler will allocate space and manage it for us. Then by exporting the getBuffer function we provide a way for our wrapping JavaScript to ask for the address to the region of memory we have reserved for it. This region of memory should then be safe for us to write to from JavaScript provided our go code only reads from this array

Now with some reserved memory to use we can write a function that takes a string and inserts it into the module’s memory.

function insertText(text, module) {

   // Get the address of the writable memory.
   let addr = module.exports.getBuffer()
   let buffer = module.exports.memory.buffer

   let mem = new Int8Array(buffer)
   let view = mem.subarray(addr, addr + text.length)

   for (let i = 0; i < text.length; i++) {
      view[i] = text.charCodeAt(i)
   }

   // Return the address we started at.
   return addr
}

As with the earlier case, we need to create an Int8Array in order to attach meaning to the bytes and so that the numbers representing the characters in the string are encoded correctly. Also note that we’re using the subarray method this time so that we’re modifying the original array and not a copy.

Calling the Echo Function¶

With a way for us to insert the string we want into the module’s memory we should now be in a position to call the echo function passing the starting address and length of the string we want it to echo. However instead of hardcoding the string this time why don’t we take it from the input textarea we have on the page.

const input = document.getElementById('input')

function onRun(runner, module) {
   return (event) => {
      // First, we need to run the module in order to define everything.
      runner.run(module)

      let inputText = input.value
      let addr = insertText(inputText, module)

      // Now just pass the memory location to the relevant function.
      module.exports.echo(addr, inputText.length)
   }
}

Wrapping Up¶

That’s about it, if you want to have a look at the final codebase then you can find it here.

This apparently simple task was certainly a lot more work than I expected it to be, but if nothing else it’s forced me to learn a bit more about some of the lower-level details of working with WebAssembly modules.

I don’t think this is necessarily the right approach though.. ok we’re able to pass a (simple!) string back and forth between our TinyGo code and JavaScript. But there are more details that still need to be considered

This solution does not handle Unicode. There is however a TextEncoder API available in the browser that looks like it might go some way towards fixing this.
I thought the fixed buffer size in the go module would be an issue - how would you handle inputs larger than 1024 bytes? However after a quick test with about 5K of text it seemed to not matter? 🤷 Though I’m sure there’s ways to break it.
Finally, what about more complex data structures? Sure we’d probably be able to encode them as JSON and pass them around that way but I’m sure that would introduce unnecessary overhead.

And with all that said isn’t this a problem that the toolchain should be solving? Perhaps I’m just using it wrong 🤔

Saying ‘Hello, World!’ with TinyGo and WebAssembly

2020-04-28T00:00:00+00:00

Saying ‘Hello, World!’ with TinyGo and WebAssembly¶

WebAssembly is something I’ve wanted to play with for quite a while now and I’ve finally got around to taking a look at it. In this post I describe how I managed to use TinyGo to compile a “Hello, World!” Go program into WebAssembly and execute it in the browser. So that I have something to refer back to I also describe setting up my development environment as a container using LXD.

Setting Up¶

Note

What I describe here is definitely not a requirement in order to use TinyGo, but me wanting to play with yet more technologies so feel free to skip this section if you want! 🙂

If you’re looking for details on getting started with TinyGo I suggest taking a look at their documentation

The recent release of Ubuntu 20.04 (and my desire to have a computer that works properly! 😂) has convinced me to switch away from using Arch Linux as my distro of choice. That said I have been spoiled by how easy it is to grab the latest version of some programming language and start playing with it…

Turns out I can have the best of both worlds! Thanks to LXD and the interface it provides around the native container technology built into Linux I was able to spin up an Arch Linux container with all the tools I needed to edit and run my code.

Spinning up Arch Linux¶

Following this`_ article it was easy enough to pull down and launch an Arch Linux image from the community repository

$ snap install lxd
$ lxd init                                 # ... and picking all the defaults
$ lxd launch image:archlinux/amd64 tinygo  # 'tinygo' is the name I've given my container

Once the container has started it’s easy enough to open a bash shell and start installing what we need

$ lxc exec tinygo -- /bin/bash
[root@tinygo] $ pacman -S tinygo llvm lld

Users and Project Data¶

Even in an unprivileged container (which is what I think LXD uses by default?) its probably not the best idea to run programs as root so we should create a normal user account and set a password

[root@tinygo] $ useradd -m alex
[root@tinygo] $ passwd alex

Then its time to share the project folder with the container so that the tools inside are able to work with it.

$ mkdir hello-world
$ lxc config device add tinygo workdir disk source=/host/path/to/hello-world/ path=/home/alex/hello-world/

[root@tinygo] $ ls /home/alex
hello-world

VSCode¶

Attaching VSCode to ‘tinygo’ container.¶

Just because the development tools we’re using are isolated in their own container doesn’t mean we have to give up on all the features of our editor. With VSCode’s Remote Development Extensions we can connect to a remote environment where VSCode will install its server component along with any other extensions we require - all of which will remain isolated in the container.

The remote capabilities of VSCode come in a few flavours and while VSCode has an extension dedicated to containers it appears that it’s hard wired to work with Docker which isn’t much use in this case… but since each LXC container is assigned a local IP address I decided to try the SSH method.

This means we need to enable ssh access from within the container itself.

[root@tinygo] $ pacman -S openssh
[root@tinygo] $ systemctl enable --now sshd.service

Then for all the Go related tooling in VSCode to function I also had to install the main Go implementation.

[root@tinygo] $ pacman -S go

Finally we can discover the IP address of the container using the lxc list command and try to connect VSCode to the container logging in as the alex user we created earlier.

$ lxc list
+--------+---------+---------------------+-----------------------------------------------+-----------+-----------+
|  NAME  |  STATE  |        IPV4         |                     IPV6                      |   TYPE    | SNAPSHOTS |
+--------+---------+---------------------+-----------------------------------------------+-----------+-----------+
| tinygo | RUNNING | 10.68.193.81 (eth0) | fd42:56e0:e39c:b4c1:216:3eff:feaa:3b91 (eth0) | CONTAINER | 0         |
+--------+---------+---------------------+-----------------------------------------------+-----------+-----------+

Access Denied!!¶

Once I started to create the files necessary for the project I very quickly realised that the project folder we passed into the container was mounted as read-only! It turns out there is some extra wizardry required to give the alex@tinygo user permission to write files back to the host…

I really don’t quite understand the details of how users and permissions work with these containers but from what I can gather LXD needs to reuse the ID of my user account on the host as the ID of the user within the container?..

Anyway according to this post the incantations needed are

$ echo "root:$UID:1" | sudo tee -a /etc/subuid /etc/subgid

which is a one time setup allowing my user ID on the host to be reused. Then on a per-container basis we also need to tell LXD to reuse my ID within a given container

$ lxc config set tinygo raw.idmap "both $UID 1000"

Phew! I think, finally, we’re in a position to crack on with the program itself. Was all of that necessary?.. probably not. Was it interesting? I certainly think so! Is it worth the effort? Well, I guess only time will tell…

TinyGo¶

With the setup out of the way time to write our “Hello, World!” program in Go

package main

//go:export main
func main() {
   println("Hello, World!")
}

Notice the //go:export main comment. Using comments like these is how we tell TinyGo which functions we want to be callable from the JavaScript code that will wrap our WebAssembly module. This is then compiled with the following TinyGo command.

$ mkdir public/
$ tinygo build -o public/hello.wasm -target wasm main.go

Why TinyGo?

While the standard Go compiler has support for WebAssembly the binaries it produces are typically quite large.

Whereas TinyGo is an alternate compiler for the Go language that specifically targets constrained environments such as embedded devices and microcontrollers. This means the binaries it produces are typically much smaller - at the expense of missing some features of the language.

WebAssembly¶

With our WebAssembly binary ready to go it’s time to load it into the browser and execute it.

<!DOCTYPE html>
<html lang="en">
<head>
   <meta charset="UTF-8">
   <meta name="viewport" content="width=device-width, initial-scale=1.0">
   <title>WASM Test</title>
</head>
<body>
   <script src="wasm_exec.js"></script>
   <script>
      const go = new Go()

      WebAssembly.instantiateStreaming(fetch("hello.wasm"), go.importObject)
            .then(module => {
               let wasm = module.instance
               go.run(wasm)
            })
   </script>
</body>
</html>

There’s not too much to go into here - I’m surprised at how straightforward this was! 😀 The WebAssembly.instantiateStreaming function is provided by the browser to fetch a wasm executable over the network and compile it. It also takes an object that describes details such as how much memory to allocate to the module and which JavaScript functions should be passed into it.

Thankfully in our case most of these details are handled by the Go() object provided by the wasm_exec.js file from the TinyGo project. All we have to do is make sure that this file is also accessible by the browser. The file itself should be included as part of your TinyGo install, mine was located in /usr/lib/tinygo/targets/wasm_exec.js

More details on using the compiled WebAssembly module can be found in the TinyGo documentation

This is probably my most convoluted ‘Hello, World!’ program to date!¶

All being well, you should be able to open your browser to the webpage we created above and see the message “Hello, World!” written to the console!

Learning Vulkan: Enumerating Physical Devices

2020-01-08T00:00:00+00:00

Learning Vulkan: Enumerating Physical Devices¶

Being an API for talking to GPUs and other compute devices every Vulkan program starts off by looking for an appropriate physical device to use. In this post I write a little C program that initialises the Vulkan API and lists out the available devices in the system.

$ vkdevice
Device Name:            Intel(R) HD Graphics 520 (Skylake GT2)
  Type:                 Integrated GPU
  Vendor ID:            32902
  Device ID:            6422
  API Version:          v1.1.102
  Driver Version:       v19.3.1

Note

This is part of my “Learning Vulkan” series where I try to figure how to use Vulkan to explore various concepts in graphics programming. As mentioned in the Overview I don’t necessarily know what I’m doing!

Setup¶

Unlike Python which has tools like pip and venv to manage dependencies and development environments, C projects (as far as I know) require the environment of your development machine to be “just right”. This means details such as the host operating system and its libraries are more important than normal.

Since this series is a learning exercise, being able to compile and run code across multiple systems is not a huge concern of mine right now. But here is a rough overview of what is required to build this project…

I’m running Arch Linux with gcc and make installed along with the following Vulkan related packages

$ pacman -Q | grep vulkan
vulkan-extra-layers 1.1.130+10614+a70d5d17e-1
vulkan-headers 1:1.1.130-1
vulkan-html-docs 1:1.1.130-1
vulkan-icd-loader 1.1.130-1
vulkan-intel 19.3.1-1
vulkan-man-pages 1:1.0.38-1
vulkan-trace 1.1.130+10614+a70d5d17e-1
vulkan-validation-layers 1.1.130-1

I’m not sure however which of these are essential for the code contained in this post.

Project Structure¶

I’m intending over the course of this series to build up a repository of Vulkan examples/utilities and since the API has a reputation of being verbose I’ll be looking to reuse as much code as I can! So with that in mind I have opted for the following project structure

$ tree -a --dirsfirst
.
├── bin
│   └── vkdevice
├── src
│   ├── vkdevice.c
│   └── vkdevice.o
├── .gitignore
└── Makefile

Nothing ground breaking, a src/ folder to hold the source and intermediate build files, bin/ to hold all the compiled programs and a plain Makefile to orchestrate the whole thing.

The Makefile¶

I know that there are tools such as CMake and Autoconf that are probably preferable to a plain Makefile since they can be used to abstract over some differences between platforms. However since I’m not familiar with them and want to focus on learning Vulkan I’m opting for a simpler approach to start out with.

CC = gcc
CFLAGS = -Wall -Wextra
LDFLAGS = -lvulkan

Here I’m declaring that I’m using the gcc compiler, the CFLAGS variable contains any flags that should passed to compile steps (at the moment this just enables some compiler warnings). Finally LDFLAGS contains any flags related to linking, -lvulkan tells the linker to link our program against the Vulkan SDK.

.PHONY: default clean
default: vkdevice
clean:
   rm src/*.o bin/*

Next I define the default target to be vkdevice so that I can just run make and have it build the project. Then there’s a clean target so that it’s easy to recompile everything from scratch. Declaring both default and clean to be phony targets I think means we’re telling make not to look for matching files on the filesystem.

%.o: %.c
   $(CC) -c $(CFLAGS) $< $@

VKDEVICE = src/vkdevice.o
vkdevice:
   $(CC) $(LDFLAGS) $(VKDEVICE) -o bin/vkdevice

Last but not least we get to the main part of the Makefile. First there is a generic pattern rule that instructs make on how to convert any .c file into an .o file making use of any of the compiler flags defined earlier. This allows the Makefile to be extended to compile each program we write by declaring a target that links the relevant object files (as defined by VKDEVICE) into an executable.

Then we can compile the project by calling make from the same directory as the Makefile

$ make
gcc -c -Wall -Wextra src/vkdevice.c -o src/vkdevice.o
gcc -lvulkan src/vkdevice.o -o bin/vkdevice

Creating an instance¶

With all the housekeeping out of the way, time to dive into the code which starts in a fairly standard way with us including all the header files we need.

#include <stdio.h>
#include <stdlib.h>
#include <vulkan/vulkan.h>

int main() {
   // Insert code...
}

Then we start by filling out the VkApplicationInfo and VkInstanceCreateInfo structs. Unsurprisingly the first is used to provide information about our application such as the version of the API we wish to use. The application name and version fields are arbitrary and can be set to whatever we like.

VkApplicationInfo app_info = {
   .sType = VK_STRUCTURE_TYPE_APPLICATION_INFO,
   .pApplicationName = "vkDevice Info",
   .applicationVersion = VK_MAKE_VERSION(1,0,0),
   .apiVersion = VK_API_VERSION_1_1,
};

VkInstanceCreateInfo vk_info = {
   .sType = VK_STRUCTURE_TYPE_INSTANCE_CREATE_INFO,
   .pApplicationInfo = &app_info,
};

The vk_info struct is then be used to create the instance.

VkInstance vk = NULL;
VkResult res = vkCreateInstance(&vk_info, NULL, &vk);

if (res != VK_SUCCESS) {
   fprintf(stderr, "Unable to create VkInstance!\n");
   return 0;
}

We also have to be sure to destroy the instance once we have finished with it

// Code omitted...

cleanup_instance:
   vkDestroyInstance(vk, NULL);

return 0;

This is me attempting to apply this rule from the Linux kernel style guide to help manage resources through the lifetime of the program.

Listing Devices¶

Now that we have an instance we can start querying the API for the physical devices that are in the system. The issue is however - we don’t know how many devices the system has! To get around this we first have to call vkEnumeratePhysicalDevices with a NULL pointer, it will then mutate the count variable that we give to be equal to the number of available devices.

Note that we skip straight to the cleanup_instance label we defined earlier if this step fails.

uint32_t count = 0;
res = vkEnumeratePhysicalDevices(vk, &count, NULL);
if (res != VK_SUCCESS) {
   fprintf(stderr, "Unable to enumerate physical devices\n");
   goto cleanup_instance;
}

Next we attempt to allocate enough memory to store each of the devices in an array.

VkPhysicalDevice* physical_devices = malloc(count * sizeof(VkPhysicalDevice));
if (physical_devices == NULL) {
   fprintf(stderr, "Unable to enumerate physical devices\n");
   goto cleanup_instance;
}

Finally with the array allocated we can call vkEnumeratePhysicalDevices a second time to populate it. Notice how this time the error path has to jump to the cleanup_devices label so that we can be sure to free the newly allocated memory.

res = vkEnumeratePhysicalDevices(vk, &count, physical_devices);
if (res != VK_SUCCESS) {
   fprintf(stderr, "Unable to enumerate physical devices\n");
   goto cleanup_devices;
}

// Code omitted...

cleanup_devices:
   free(physical_devices);
   physical_devices = NULL;

cleanup_instance:
   vkDestroyInstance(vk, NULL);

Editor’s Note

As I was writing this post I looked up the documentation for vkEnumeratePhysicalDevices and noticed that there is a different way to approach this section. We could’ve instead decided on a fixed size array

uint32_t MAX_DEVICES = 4;
VkPhysicalDevice physical_devices[MAX_DEVICES];
res = vkEnumeratePhysicalDevices(vk, &MAX_DEVICES, physical_devices);

In this situation the function will return up to MAX_DEVICES and if there more devices than can fit in the array then res will be set to VK_INCOMPLETE giving us the option to try again with a larger array.

This creates a dilemma - which approach is better? 🤔

Device Properties¶

It turns out that a VkPhysicalDevice on its own is rather useless since it doesn’t carry any information about itself. In order to find out more about what the device is and what features of the API it supports you need to call additional functions such as

vkGetPhysicalDeviceProperties
vkGetPhysicalDeviceImageFormatProperties
vkGetPhysicalDeviceQueueFamilyProperties
and many more!!

Typically as part of your program’s setup you would call a number of these to gather information about support for features that matter to you to help decide which device is best suited to your use case. However for this toy program we’re only going to call vkGetPhysicalDeviceProperties for each device which will give us information such as its name.

for (uint32_t i = 0; i < count; i++) {
   VkPhysicalDeviceProperties properties = {};
   vkGetPhysicalDeviceProperties(physical_devices[i], &properties);

Version numbers (MAJOR.MINOR.PATCH) in the Vulkan API are encoded into a single 32bit integer as defined in the specification. Thankfully the spec also defines a number of macros that make decoding them nice and easy for us.

uint32_t vk_major = VK_VERSION_MAJOR(properties.apiVersion);
uint32_t vk_minor = VK_VERSION_MINOR(properties.apiVersion);
uint32_t vk_patch = VK_VERSION_PATCH(properties.apiVersion);

uint32_t driver_major = VK_VERSION_MAJOR(properties.driverVersion);
uint32_t driver_minor = VK_VERSION_MINOR(properties.driverVersion);
uint32_t driver_patch = VK_VERSION_PATCH(properties.driverVersion);

All that’s left to do is to print out the information we have gathered.

   printf("Device Name:     \t%s\n", properties.deviceName);
   printf("  Type:          \t%s\n", vkPhysicalDeviceType_as_string(properties.deviceType));
   printf("  Vendor ID:     \t%d\n", properties.vendorID);
   printf("  Device ID:     \t%d\n", properties.deviceID);
   printf("  API Version:   \tv%d.%d.%d\n", vk_major, vk_minor, vk_patch);
   printf("  Driver Version:\tv%d.%d.%d\n", driver_major, driver_minor, driver_patch);
}

I should also note that vkPhysicalDeviceType_as_string is a helper function I defined that converts a member of the VkPhysicalDeviceType enum into a string representation with a switch statement.

Conclusion¶

And that’s that! I’ve taken my first few baby steps with the Vulkan API and I hope that if you’ve read this far then this post was as useful to you as it was to me! If you are interested then you can see the full code listing here and I’ll see you in the next one.

Learning Vulkan: Overview

2020-01-01T00:00:00+00:00

Learning Vulkan: Overview¶

I have always been interested in graphics and have on numerous occasions tried to dip my toe into the world of OpenGL and more recently Vulkan. However I have never been able to get past the “Hello, World” of these technologies - drawing a triangle on screen, I think mostly becuase I never really had a goal in mind once I got that far…

But that’s (hopefully) about to change! What better excuse than a new decade to jump back into this world for the 100th time and try to get to all those interesting ideas I see people playing with all the time!

I should say that I don’t really know what I’m doing and will totally be making it up as I go along. With Python being my favourite language, programming in C (C++ is just too intimidating) will take some getting used to let alone all the concepts specific to graphics programming! However that said I’m not a complete n00b as I have at least played with C on and off over the years - I’m sure I already know enough to be able to shoot myself in the foot! 😁

So the only thing left to do is set some goals so I don’t get lost after getting to the “drawing a triangle on screen” stage again. I imagine this will evolve into some kind of living document that I’ll use to keep track of my progress - if I remember to update it that is!

[ ] Draw a triangle on screen
- [x] Enumerating available VkPhysicalDevices
[ ] OBJ object viewer
[ ] Ray Marching
[ ] Ray Tracing
[ ] Python integrations?

Implementing Click & Drag with Vanilla JS

2019-07-07T00:00:00+00:00

Implementing Click & Drag with Vanilla JS¶

Try clicking and dragging on this circle.

Disclaimer

This post makes use of a number of interactive elements to help illustrate a few concepts. Unfortunately these do not yet work on mobile devices - sorry mobile users!

I have for quite some time now wanted to play around with web development some more, particularly using web technologies to build user interfaces of some kind. However there is just so much out there it’s been impossible for me to really get anywhere past a “Hello, World!” tutorial before I find myself trying out the next new shiny.

So I’ve decided to abandon everything and try a bottom up approach where I see how far I can push the core web technologies - HTML, CSS and JavaScript. Hopefully then by the time I start using one of the gazillion libraries out there I will have a better understanding of why I needed it in the first place.

In this post I will be looking at implementing clicking and dragging functionality using only vanilla JavaScript. Clicking and dragging as a concept can apply to many kinds of interactions so in this instance I’m specifically referring to clicking on an SVG element moving it around on the page as illustrated by the demo above.

Setup¶

All we need to get started is a HTML document that contains some markup which provides someplace we can reference and attach our image element to, as well as loading the code we write.

<div id="main"></div>
<script type="text/javascript" src="/js/click-drag.js"></script>

Then the first step is to create our SVG image element to act as our “canvas” that we can draw on.

click-drag.js

const svgns = "http://www.w3.org/2000/svg"
const main = document.getElementById("main")

const canvas = document.createElementNS(svgns, "svg")
canvas.setAttribute("width", "100%")
canvas.setAttribute("height", "100%")
canvas.style.border = "solid 2px #242930"

main.appendChild(canvas)

A few things to note:

By adding our <svg> element as a child of some <div> element and setting both the width and height to 100% our canvas will be able to scale responsively based on the styles applied to the parent <div>
You might already be familiar with the document.createElement() function for creating HTML elements using JavaScript. However in order to work with SVG elements we need to use the document.createElementNS() function which allows us to use the SVG namespace instead of the HTML default.

The View Box¶

The next step is to construct an appropriate viewBox definition for our canvas. For more information on the viewBox you can refer to the documentation but to briefly summarise. An SVG image exists on an infinite plane and the viewBox is the window we use to view a portion of that space, changing the definition of the viewBox allows you to zoom in and out on particular regions.

For our purposes what’s important is that we construct a viewBox that matches the proportions of the <svg> element as it is displayed in the browser. If these proportions do not match then the element being dragged around will not accurately track the cursor, either racing away from or lagging behind it.

One minor issue is that in our setup we didn’t explicitly set the dimensions of our <svg> element - so how can we know its proportions? Thankfully once the <svg> as been added to the page we can ask the browser for the bounding box around the element.

click-drag.js

let bbox = canvas.getBoundingClientRect()

Among other properties that are outlined on this page we can get the width and height of the rendered image in pixels from which its easy to calculate the aspect ratio.

click-drag.js

const aspectRatio = bbox.width / bbox.height

We’re free to choose whichever scale we want for the vertical height of the viewBox into the <svg> element. I have chosen 100 simply because it feels like a nice round number. Once we’ve decided on a scale for the height, it’s easy enough to calculate the corresponding width from our aspect ratio.

click-drag.js

const height = 100
const width = height * aspectRatio

With the dimensions of the viewBox taken care of all that is left to do is decide on the coordinates to assign to the top left corner of the <svg> element and assign the view box to our canvas.

click-drag.js

const viewBox = {minX: 0, minY: 0, width: width, height: height}

const viewBoxStr = [
  viewBox.minX, viewBox.minY, viewBox.width, viewBox.height
].join(" ")

canvas.setAttribute("viewBox", viewBoxStr)

Something to Click on¶

By this point we have finished preparing our canvas and it’s time to add something for us to interact with. To keep things simple I will stick to a <circle> element, though the method we use here should apply to any SVG element (or any collection of elements under a <g> tag).

click-drag.js

const circle = document.createElementNS(svgns, "circle")
circle.setAttribute("cx", viewBox.width / 2)
circle.setAttribute("cy", viewBox.height / 2)
circle.setAttribute("r", 15)
circle.setAttribute("fill", "#57cc8a")

canvas.appendChild(circle)

Note

Of course the way in which you define the position of your interactive element will depend on the element you have chosen.

Implementing the Drag¶

We will create an event handler for the mousemove event and attach it to our canvas.

canvas.addEventListener("mousemove", (event) => {
  // Do something clever here...
})

The function we write will be called every time the cursor moves regardless of whether the user has clicked or not. This means our event handler has to be able to cope with two situations, the cursor moving when the user has clicked and the cursor moving when the user has not clicked.

To do this we will declare a variable called clicked outside the scope of our function.

click-drag.js

let clicked = false

For the moment we will ignore the details around how this variable is updated (it is covered in the next section), instead let’s focus on what we do while the cursor is moving about the page

Let’s get the simpler case out of the way first

Not Clicked¶

When the mouse is moving but the user has not clicked, then there is nothing for us to do! We can simply check the value of the clicked variable and stop the function if it meets the criteria.

if (!clicked) {
  return
}

Clicked¶

Now for the interesting part! The mouse is moving and the user has clicked on the circle, all we have to do now is update its position to match the cursor’s current position. The only problem is… where is it?

Like all mouse related events the event object passed into the event handler will contain a number of position related properties.

e.client<XY>: Coordinates of the cursor with respect to the current portion of the document visible on the page.
e.offset<XY>: Coordinates of the cursor with respect to the edge of the target element
e.page<XY>: Coordinates of the cursor with respect to the entire HTML page, including any portions of the page not currently visible
e.screen<XY>: Coordinates of the cursor with respect to the user’s display

Reading through those descriptions you would imagine that the e.offset<XY> properties would be the best fit for our use case. However it’s not quite as simple as that.

Below you should see 2 boxes, the bigger one on the left is our canvas. The smaller box on the right contains a smaller circle that represents the calculated position of the cursor based on the offset<XY> properties like so.

const x = event.offsetX
const y = event.offsetY

Try moving the mouse across the canvas and keep an eye on the calculated position.

Cursor Position: Offset

Target:

Position:

Determining the cursor's position using the event.offsetX and event.offsetY properties

Notice the issue when we move across the circle? Why does the calculated position of the cursor suddenly jump whenever we touch it? The answer lies in the description of the offset<XY> property “with respect to the edge of the target element”

When initially trying to implement this I incorrectly assumed that the target element meant the element that we attached the event listener to - the canvas. In fact the target element is whichever element is currently under the cursor

To work around this we can calculate the offset values we need ourselves. In order to do this we will make use of both the bounding box returned from the canvas.getBoundingClientRect() method as well as the client<XY> properties found on the mouse event.

It turns out that the bounding box also returns the coordinates of the top left corner of the canvas relative to the user’s current view of the document - exactly the same coordinate system used by the client<XY> properties! From those two pieces of information it’s easy enough to recover the offset<XY> values ourselves.

bbox = canvas.getBoundingClientRect()

const x = event.clientX - bbox.left
const y = event.clientY - bbox.top

By calculating the coordinates using values that are independent of the element currently underneath the cursor we sidestep any issues that arise from a changing target. Try the same thing again on the canvas below.

Cursor Position: Client

Target:

Position:

Determining the cursor's position using the event.clientX and event.clientY properties

Important

Since the values bbox.top and bbox.left are defined relative to the user’s current view on the document these values are not constant. They will change whenever the user alters their view of the page, i.e. performing actions like resizing the window or scrolling. This is why we ask for an updated bounding box every time our event handler is called.

Now that we can reliably know the position of the cursor, we can focus on the final piece of this puzzle - updating the position of our <circle> element. There is however one further issue to work through. The position of the cursor that we’ve just calculated is using a different coordinate system to the one used to draw our circle!

Although we have managed to correctly calculate the cursor’s position relative to our canvas, that position is measured using pixels which makes it highly dependent on the resolution of the user’s screen. A user who uses a 4K monitor and positions their cursor at the bottom right of the canvas will have a calculated position much larger than a user on a smartphone…

What this means is that if we map this calculated position directly onto the circle it won’t accurately follow the cursor. The only time the circle would follow the cursor correctly is when the pixel based coordinates line up with the coordinate system used in the SVG image. I.e. when the dimensions of the canvas match up exactly with the scale used to define our viewBox which in this case would be 100px tall

Since the mouse circle and the circle use different coordinate systems, we don’t actually care about the exact position we have just calculated. What’s more important is the position of the cursor relative to bounds of the canvas - a percentage. For example, let’s say that the cursor was halfway down the canvas (50%) then we could calculate the corresponding coordinate value in the SVG coordinate system by multiplying the total height by 50%. In our particular case this would mean setting y = 100 * 0.5 = 50.

We can adopt this approach by using the width and height information returned as part of the bounding box to modify our calculation to produce a percentage rather than an absolute value.

const x = (event.clientX - bbox.left) / bbox.width
const y = (event.clientY - bbox.top) / bbox.height

To then convert this percentage into its corresponding value in the SVG coordinate system all we have to do is multiply it by the width and height of that system and assign the result to our circle’s position!

circle.setAttriubte("cx", x * viewBox.width)
circle.setAttribute("cy", y * viewBox.height)

Bringing all that together we end up with the following implementation of our mousemove event handler.

click-drag.js

canvas.addEventListener("mousemove", (event) => {

   if (!clicked) {
      return
   }

   bbox = canvas.getBoundingClientRect()

   const x = (event.clientX - bbox.left) / bbox.width
   const y = (event.clientY - bbox.top) / bbox.height

   circle.setAttribute("cx", x * viewBox.width)
   circle.setAttribute("cy", y * viewBox.height)
})

Nearly there! The only thing left to do is decide on how we want to update the clicked variable.

Click Detection¶

Finally all that’s left is to do is decide how we want to toggle the dragging behaviour. This mostly comes down to how you want the user to interact with the draggable object and will change depending on your use case. To keep things simple I will go with a fairly simple interaction model

If the mouse is over the circle and the user clicks then start dragging
If the user releases the mouse button then stop dragging

click-drag.js

circle.addEventListener("mousedown",  (_) => { clicked = true })
circle.addEventListener("mouseup", (_) => { clicked = false })

Additionally I will impose one final condition

If the mouse leaves the bounds of the canvas then stop dragging.

click-drag.js

canvas.addEventListener("mouseleave", (_) => { clicked = false })

This last point is to work around an issue that arises when the user moves the cursor out of the bounds of the canvas and releases the mouse button. Since the cursor is no longer over the circle the handler for the mouseup event on the circle is never fired so when the user brings their cursor back over the canvas our code still believes the user never released the mouse button and so the circle will appear “stuck” to their cursor until they click again.

Conclusion¶

There you have it, those were some of the basics required to get a working proof of concept of clicking and dragging functionality using only the JavaScript APIs that come with your web browser. However in writing this blog post I realised as with anything that the rabbit hole goes deep and there are many considerations to keep in mind if you wanted to “productionise” this code for any real usage.

Touchscreen Support

If you were reading this post on a mobile device you will already have noticed that none of the interactive demos have worked. This is because touchscreens have their own family of touchXXXX events that are triggered when a user interacts with a webpage. While there is some mouse emulation done by the browsers (e.g. a touchstart event will trigger a mousedown event if not handled), some key events such as mousemove are not emulated and require dedicated support in your code. See these articles for more details
Snap to Center

This is only a minor issue and depending on your use case may not be a problem at all. Currently whenever you pick up the circle its center snaps to the cursor’s position. There may be situations where you would prefer the keep the object’s relative position to the cursor e.g. fine adjustments, the last thing you would want is for the object to jump to the cursor just because the user happened to click on it off center.

A way around this would be to record the original positions of both the mouse cursor and the object on a click, then on each mousemove event calculate the distance moved by the cursor and apply it to the original recorded position of the object.
Multiple Objects

Chances are when using this functionality in a real application you would want the ability to click and drag on multiple objects. Adding support for this would require reworking at least some of the code, having the canvas object handle all mouse movements is probably a good idea but of course updating the circle’s position directly would have to change. It would probably make sense to move the click detection logic onto the canvas also, making use of the event.target property to determine which object would need updating.
Canvas Resizing

While the existing code makes some effort to ensure this works in a responsive manner it is currently only “statically responsive”. What I mean is that on page load all the necessary calculations are performed to ensure that the viewBox has the correct proportions for example. However if the user were to resize the webpage, or rotate their device chances are the proportions of the canvas would change meaning that the circle would no longer follow the cursor correctly.

In order to be truly responsive, we would need to listen for events such as resize and perform all the calculations again.
Pan and Zoom

Something I realised when writing up the part where we map the cursor’s position onto the position of the circle is that I had by chance chosen a special case where the maths is a little easier. The calculations x * viewBox.width and y * viewBox.height only work because our viewBox starts at (0, 0).

Say we had an application that also allowed for panning and zooming of the canvas itself then the chances are our viewBox would not be starting at (0, 0) and we would have to include an offset in our calculations to reflect this. This means for the general case our code should probably look something like this
```
circle.setAttribute("cx", (x * viewBox.width) - viewBox.minX)
circle.setAttribute("cy", (y * viewBox.height) - viewBox.minY)
```

I’m sure there are more edge cases and considerations to think of but this post is long enough already! - Perhaps this is why people use libraries for this kind of thing 🤔…

If you are interested you can find the final version of the code here.

TIL: Python has a cmd module

2019-01-05T00:00:00+00:00

TIL: Python has a cmd module¶

Today I Learned that Python’s standard library has a cmd module and it is awesome!

Interactive program using the cmd module¶

The cmd module contains a single class called Cmd which handles all the details of creating an application similar to Python’s REPL. All you need to do is to provide some command definitions and the Cmd class will handle the rest.

In an attempt to demonstrate why I think this is so cool I’m going to walk through the process of building the application you see in the screencast above.

The example application we’re going to create is a very basic REPL for a passion project of mine called stylo. Stylo is a Python library that allows you to draw images and create animations using code and some mathematics. The application will expose some of the basic shapes available and for the “Print” part of the REPL it will show a preview of your image.

My main focus for this post is the cmd module which means I’m not going to go into any of the specifics of stylo or how to use it. If you want to know more about it I will point you in the direction of the documentation (under construction) and the example gallery

Setup¶

To start with we’re going to create a virtual environment and install stylo into it. This will also install matplotlib which we will be using later on. I’m using Python 3.7 but this application should work on all versions of Python ≥ 3.5.

$ python -m venv env
$ source env/bin/activate
(env) $ pip install stylo

Note: The cmd module is available for even older versions of Python. However we are limited by stylo which only supports Python 3.5+

With the dependencies out of the way we can create a file called stylo-cmd.py and start writing some code!

import cmd

class StyloPrompt(cmd.Cmd):
   pass

if __name__ == '__main__':
   prompt = StyloPrompt()
 prompt.cmdloop()

This is the bare minimum required to get something we can start playing with. If you were to run python stylo-cmd.py you would see the following prompt which comes with a single built-in command help.

(Cmd) help

Documented commands (type help <topic>):
========================================
help

Ctrl-C will exit the application. Obviously this is pretty useless right now so let’s look at adding in some commands of our own.

Adding Commands¶

Any method on our StyloPrompt class with a name of the form do_* is considered a command, with the command name given by whatever is after the underscore. To get ourselves warmed up let’s add two commands reset and save which will allow us to create a fresh image and save it to a file.

from stylo.image import LayeredImage

class StyloPrompt(cmd.Cmd):

   def __init__(self):
      super().__init__()
      self.image = LayeredImage()

   def do_reset(self, args):
      self.image = LayeredImage()

   def do_save(self, args):
      width, height, filename = args.split(" ")

      width = int(width)
      height = int(height)

      self.image(width, height, filename=filename)

As you can see each command receives its arguments as a single string and it is up to the method to handle them - including conversions to appropriate data types as is the case with the width and height arguments. For the sake of being brief proper error handling has been omitted.

Now if we were to fire up the application we would be able to produce an image!

(Cmd) reset
(Cmd) save 1920 1080 image.png

Of course this image is currently empty so next we should add the ability for the user to place shapes on the image. We’ll create two more commands circle and square.

from stylo.color import FillColor
from stylo.shape import Circle, Square

class StyloPrompt(cmd.Cmd):
   ...

   def do_circle(self, args):
      x, y, r, color = args.split(" ")

      circle = Circle(float(x), float(y), float(r), fill=True)
      self.image.add_layer(circle, FillColor(color))

   def do_square(self, args):
      x, y, size, color = args.split(" ")

      square = Square(float(x), float(y), float(size))
      self.image.add_layer(square, FillColor(color))

Now when we use the application we can create something a bit more interesting than a snowman in a blizzard!

Number 3 on a dice¶

(Cmd) square 0 0 1.75 000000
(Cmd) circle 0 0 0.3 ffffff
(Cmd) circle -0.5 0.5 0.3 ffffff
(Cmd) circle 0.5 -0.5 0.3 ffffff
(Cmd) save 1920 1080 image.png

Getting Help¶

Now that we have a few commands available we need to tell users how they can be used. If we were to use the help command we would see something like the following.

(Cmd) help

Documented commands (type help <topic>):
========================================
help

Undocumented commands:
======================
circle reset save square

Not very helpful.

Thankfully the default help system doesn’t require much to get started, all we have to do is add docstrings to our do_* methods!

def do_circle(self, args):
   """usage: circle <x> <y> <r> <color>

   This command will draw a circle centered at the coordinates (<x>, <y>)
   with radius given by <r>. The <color> argument is a 6 digit hex
   representing a color in RGB format.
   """
   ...

Now if we were to run help circle

(Cmd) help circle
circle <x> <y> <r> <color>

      This command will draw a circle centered at the coordinates (<x>, <y>)
      with radius given by <r>. The <color> argument is a 6 digit hex
      representing a color in RGB format.

Much better

Giving Feedback¶

Right now our program is… ok. The user can type in a few commands and they can create some images, but it’s not much of a step up from using the library as they still have to wait until they have saved their image before they can view it. Add in the fact that our program isn’t that flexible they may as well be using the library directly.

If only there was some way we could show the user their image as they build it up a command at a time…

Enter postcmd! This handy method is called each time our program has processed a command - we can use this to redraw the image each time. Then “all” we have to do if find a way to display the current image to the user.

After some searching and head scratching I was able to come up with the following matplotlib incantation to add our image to a figure and display it.

...
import matplotlib.pyplot as plt

class StyloPrompt(cmd.Cmd):

   def __init__(self):
      ...

      self.fig, self.ax = plt.subplots(1)
      self.ax.get_xaxis().set_visible(False)
      self.ax.get_yaxis().set_visible(False)

      self.update_image()
   ...

   def postcmd(self, stop, line):

      if stop:
            return True

      self.update_image()

   def update_image(self):

      # Re-render the image
      self.image(1920, 1080)

      # Update the preview
      self.ax.imshow(image.data)
      self.fig.show()

I won’t go into too much detail here but I will point out a few things.

The stop argument to postcmd indicates whether the previous command wanted to exit the program (by returning True). We have the option of overriding that by not returning True. But in our case we will just pass the message on.
Matplotlib is smart enough to use an existing window when calling show() on a figure so all we have to do is update the plot in the axis object
In the __init__ method we are disabling the scale on the axis so that the user doesn’t see something that looks like a graph.

Finishing Touches¶

With most of the functionality out of the way we can look at tweaking some things to make the overall experience nicer.

Exiting the Program¶

So far we don’t have a clean way to close the program, we can hit Ctrl-C to terminate the script but it results in Python printing a traceback and it looks like an error in our program more than anything.

Instead we can override the default method on our class. This method is called whenever the program doesn’t recogise the user’s input as a valid command and we can use it to look at all of the user’s input (not just the args) and decide what to do with it.

In this case we will say that the program will exit whenever the user types a q or we receive an EOF character (Ctrl-D).

class StyloPrompt(cmd.Cmd):
   ...

   def default(self, line):
      if line == "q" or line == "EOF":
            return True

      return super().default(line)

Changing the Prompt¶

We can change the default prompt (Cmd) by setting the prompt attribute on our class.

class StyloPromt(Cmd):
   prompt = "-> "
   ...

Greeting the User¶

Currently when our program starts it simply shows them the prompt, which if they are using it for the first time they probably won’t know where to start. To help them get started we can set the intro attribute to contain a welcome message.

...
from stylo import __version__

intro_text = """\
Interactive Shell for Stylo v{}
----------------------------------

Type `q` or `Ctrl-D` to quit.
Type `help` or `?` for an overview `help <command>` for more details.
"""

class StyloPrompt(cmd.Cmd):
   intro = intro_text.format(__version__)
   ...

Now when the user starts the program they should have enough information to continue from there.

Interactive Shell for Stylo v0.9.1
----------------------------------

Type `q` or `Ctrl-D` to quit.
Type `help` or `?` for an overview `help <command>` for more details.

->

There are also doc_header, misc_header and undoc_header that you can set to include even more information at different points in your program. You can refer to the docs for more details.

Wrapping Up¶

I can’t believe I only just found out about this module. I hope you found this as useful as I did and I strongly encourage you to take a look at the docs as there are features there that I didn’t get around to mentioning - such as completion!

For those interested the final version of this program (with a few minor tweaks) is available as a Gist on Github. I think what I like most about this module is that it requires very little code before you start seeing real results - Our entire application is only 155 lines of code!

Introducing Stylo Doodles!

2018-10-12T00:00:00+00:00

Introducing Stylo Doodles!¶

A few weeks back at PyConUK I gave my first lighting talk at a conference. During that talk I spoke publically about stylo for the first time. Stylo is a Python library that I have been working on for just over a year and a half and it aims to make the creation of images easier by bringing together ideas from programming and mathematics.

Version 0.6.0 was recently released which included the first feature that wasn’t written by me! It’s very exciting not only to see other people starting to take an interest in the project but taking the time to make a contribution!

Now that stylo seems to be getting to the point that it might me useful to other people wouldn’t it be great if there was a community driven example gallery that people could get inspired by? - Well now there is! And it’s called Stylo Doodles

All the examples are written as a Jupyter Notebook and can be submitted to the gallery by opening a pull request against the stylo-doodles repository. A small python application is then run that builds the website and pushes the update to the live website.

Current Features¶

The gallery website is very new but it currently has the following features

All images are displayed in a grid on the homepage with the order randomly chosen each time the website is built.
Each image has its own page (as shown above) which displays the full resolution image along with information about the author, image and the version of stylo used to generate it.
The source code from the notebook is extracted and is also displayed alongside the image.
You can also play around with any example live in your browser if you follow the binder link in the repository’s README.

Adding Your Own Example¶

If you have an image that you would like to share there are only a couple of things you need to do:

Your image must be stored in a variable called image. The build process will import your notebook as a Python module and look for a variable called image.
You also need to provide some additional information to the build system about your example in the form of a Python dictionary. This dictionary must be called info and it must be in very first cell of the notebook
```
info = {
     "title": "Jack-O-Lantern",
     "author": "Alex Carney",
     "github_username": "alcarney",
     "stylo_version": "0.6.0",
     "dimensions": (1920, 1080)
}
```
The stylo_version field should be set to the value of stylo.__version__ at the time you created your image. The dimensions is tuple of the form (width, height) and will be used by the build system to determine the size of the image (in pixels) when it renders the full size copy for its detail page.
Once your example is ready open a pull request adding your notebook to the notebooks/ folder to the repository.

Be sure to check out the existing examples to use as a guide or drop by the stylo Gitter room if you get stuck we’ll be more than happy to help!

Future Developments¶

Stylo Doodles is far from finished aside from adding examples there are many more things that could be added to the website:

User profiles: A page for every author, which lists the examples they have contributed to the gallery.
Search: As the number of images grow users would probably want to be able to tag their images and be able to narrow down the list of images on the homepage.
Recently Added: Since the order of the homepage is random, as the number of images increases the chance of a new image being buried at the bottom will also increase, it would be good to have a way of sorting the images by date added.
Descriptions: Jupyter Notebooks support more than just code. Cells containing markdown can be placed in between code cells to provide extra context and explanation. It would be great if we could include these on the site as well.

If you are looking for a web based python project to get involved with this would be a great one to get started with and I would be more than happy to have a few contribuitors to work on this (or even stylo itself!) with me.

I’ve Started a Blog… Again!

2018-09-22T00:00:00+00:00

I’ve Started a Blog… Again!¶

Not that you would have known it, but I’ve had a blog since 2014. Well 2015 if you’re feeling generous, the first (and only) post went up in the last few hours of New Year’s Eve. It was a look back on some of the projects I had worked on that year and I announced my intentions to start blogging.

Fast forward nearly 4 years and here I am announcing my intentions to start blogging - again. So I guess you are wondering what happened?

I got lost.

My first attempt at running a blog was using Jekyll and for some mystical reason (It’s been so long I can’t actually remember why), I decided that it was not the static site generator I was looking for. So I promptly set off on a voyage of discovery in search of the ultimate static site generator.

Here is a list of some of the other static site generators I have played with over the years in no particular order:

Metalsmith: Written in Javascript, this one appealed to me with its “everything is a plugin” approach. In theory I should be able to add any feature I wanted simply by finding/writing the right plugin.
Hakyll: I was going through a Haskell phase and I thought it would be a great idea to have my blog powered with it as well. Haskell all the things!
Sphinx: Sphinx is an awesome tool for writing documentation. A big part of that is reStructuredText, add in the ABlog extension and you should have a great setup for a blog.
Pelican & Nikola: However as Sphinx is primarily built for documentation projects, I found that I was fighting it more than anything. That led me to take a look at Pelican and Nikola, both written in Python and have support for reStructuredText and Jupyter.
Vuepress: Having played around a bit with VueJS, the thought of being able to take a dynamic site written in a powerful frontend framework and make a static site out of it seemed appealing. The best of both worlds.
Hugo: Who doesn’t want a static site generator written in Go? :)
Emacs & org-mode: Yes, you can use emacs as a static site generator.

At this point you might be wondering what was wrong with all of the above so that after nearly 4 years of tinkering I still had nothing to show for it?

Nothing. Absolutely nothing.

The problem was with me. I wanted complete control over the output, from the contents of <head> to the CSS styling of links. There would always be a point where I would start fighting against the very abstractions designed to make my life easier! It got to the point where I even tried writing my own static site generator…

A New Perspective¶

After endless hours lost fiddling with scripts and stylesheets I stumbled across a quote on the internet that would snap me out of my spiral of perpetual procrastination.

The technology you use impresses no one.

The experience you create with it is everything. – Sean Gerety

I have found myself saying this again and again, so much so that I think it may have fundamentally altered the way I think about programming. I fell into a trap of getting caught up in the merits of the technology for the sake of the technology itself and lost sight of the experience - the blog itself.

Rejuvenated I’ve gone back to where it all started and have started using Jekyll again. I’m using the Hydeout theme as it’s built in a way that allows me to make a few (minor!) tweaks of my own. Within a few hours I was already working on the draft that became this blog post, a place I never even got to in most of my previous attempts.

It turns out that Markdown is a perfectly acceptable format for a blog. You don’t have to engineer your blog’s theme from the ground up especially when a prebuilt theme exists in the style you were going to build yourself anyway. It doesn’t matter that your blog can seamlessly format a Jupyter Notebook as a regular blog post when you don’t have any notebooks to publish in the first place…

What matters is the content itself, that you have something interesting to say and you have some way of making that available to other people. Everything else is just an implementation detail, which if done right is invisible to the consumers of your content anyway.

It’s strange that I had learn something that is probably obvious to most people the hard way but I’m here now. If you are reading this then things are looking up but I can’t quite declare victory as I’m no further forward than I was 4 years ago, first let’s see if I make it to blog post number #2…

Alex Carney | Blog

Indexing Info Manuals

Indexing Info Manuals¶

dir files¶

build-info-dir.py¶

Building and Using the Index¶

Taking Rich Screenshots with Wezterm

Taking Rich Screenshots with Wezterm¶

Taking the screenshot¶

Rendering the Screenshot¶

Width and Height¶

SGR Codes¶

Colour Palette¶

Sphinx Integration¶

Blogging with Sphinx and denote.el

Blogging with Sphinx and denote.el¶

denote.el basics¶

reStructuredText and denote.el¶

Sphinx and denote.el¶

Discovering Content¶

Using Tags to Mark Blog Posts¶

Pretty URLs¶

Cross References¶

Generating Feeds and Archives¶

Next Steps¶

Intstalling Emacs with systemd-sysext

Intstalling Emacs with systemd-sysext¶

Enter systemd-sysext¶

Initial Setup¶

Install the sysext¶

Enabling the sysext¶

Digging Deeper¶

What’s in a sysext?¶

Compatibility Checks¶

Hacking it in¶

TIL: Navigating History with Magit

TIL: Navigating History with Magit¶

From magit-dispatch¶

From magit-file-dispatch¶

Navigating history¶

TIL: Additional magit keybindings

TIL: Additional magit keybindings¶

Notes for December/January

Notes for December/January¶

pygls¶

esbonio¶

TIL: Debugging go programs with gdb

TIL: Debugging go programs with gdb¶

How I got here¶

Setup¶

Debugging¶

Setting a Breakpoint¶

Fixing the Panic¶

Notes for November 2024

Notes for November 2024¶

vscode-wasi-pygls¶

esbonio¶

pytest-lsp¶

lsp-devtools¶

TIL: Inspecting Python Threads

TIL: Inspecting Python Threads¶

TIL: Local LLMs with Ollama and gptel

TIL: Local LLMs with Ollama and gptel¶

Installing Ollama¶

Choosing a Model¶

Installing gptel¶

Notes for October

Notes for October¶

Esbonio¶

pygls¶

Setting up a Python WASI Environment with Nix

Setting up a Python WASI Environment with Nix¶

Why?¶

Ok, but why use Nix?¶

“Building” Python¶

Making it work¶

Creating a Wrapper¶

Installing packages¶

Gathering Dependencies¶

Setting PYTHONPATH¶

`dir` files¶

`build-info-dir.py`¶

Blogging with Sphinx and `denote.el`¶

`denote.el` basics¶

reStructuredText and `denote.el`¶

Sphinx and `denote.el`¶

From `magit-dispatch`¶

From `magit-file-dispatch`¶

TIL: Debugging go programs with `gdb`¶

Setting `PYTHONPATH`¶

`mkPythonWASIShell`¶

A better `devShell` definition¶

`nvim --clean`¶

`Dependency is not of valid type`¶

Packaging `pytest-lsp`¶

Packaging `lsprotocol`¶

Unlucky `0.13`¶