Integrating Esbonio with Neovim Using Nix

So far I’ve been learning how to use Nix by trying to package and define development shells for esbonio (see here if you are interested). While useful, the end result is not too dissimilar to what you can get with standard Python tooling. Indeed, the main reason I started looking into Nix was the promise of it being able to manage more than just Python libraries.

Since esbonio is a language server, it would be useful for Nix to create standardised environments where the language server is pre-configured for a given editor - great for debugging and demos!

In this blog post I try to define an environment in which Neovim is installed and configured to use the esbonio language server for reStructuredText files.

Try it yourself!

If I’ve done all my homework right, you should be able to try the result of this blog post for yourself Assuming you have nix installed

$ nix run github:alcarney/esbonio?rev=a077efeed176dcad2ae5e4fd221179d266f88ca1

should be the only command you need. Let me know if you run into any issues!

Defining Applications

One of the defined flake outputs is apps.<system>.<name> which as the name suggests allows you to export applications from a flake.

{
  description = "Esbonio";

  inputs = {
    nixpkgs.url = "github:NixOS/nixpkgs/nixpkgs-unstable";
    utils.url = "github:numtide/flake-utils";
  };

  outputs = { self, nixpkgs, utils }:
    utils.lib.eachDefaultSystem (system:
      let
        pkgs = import nixpkgs { inherit system; };
      in {
        apps.default = { type = "app"; program = "${pkgs.neovim}/bin/nvim"; };
      }
    );
}

This simple flake.nix exports neovim as an application which we can launch by running nix run . from the folder containing this flake.

Which works as expected however, it’s also picking up my personal config - not so useful when you’re trying to create a standard, isolated environment.

Isolated Configuration

As with most things in Nix, the neovim package definition allows for certain fields to be overridden - including the config. Let’s start by trying provide an empty init.vim file.

utils.lib.eachDefaultSystem (system:
   let
     pkgs = import nixpkgs { inherit system; };
     neovim = pkgs.neovim.override {
       configure = {
         customRC = ''
         '';
       };
     };
   in {
     apps.default = {
       type = "app";
       program = "${neovim}/bin/nvim";
     };
   }
 );

And try nix run . again

Which worked! Sort of… well… not really. 😕

It worked in the sense that it loaded the empty init.vim file we specified (notice that the screenshot above has no line numbers). However, it’s not truly isolated since it went ahead and loaded my plugins anyway due to my user’s home folder being included in the runtimepath

Tip

To get the contents of your runtimepath into a buffer.

1. In INSERT mode type <c-r>=&rtp and hit enter

2. Replace all commas with newlines :%s/,/\r/g

So how can we exclude them?

nvim --clean

Reading through :h ‘runtimepath’ there’s a lot of detail around which paths are searched by default and in what order. But right at the end there’s a little note

With --clean the home directory entries are not included.

Which sounds like just what we need! The question is… how do we start nvim with that flag?

Looking around the nixpkgs repo for a bit I found a set of test cases that made use of a utility for generating a config, along with a wrapper which converts the given config into a shell script. This shell script pulls together various components from /nix/store, before ultimately launching our isolated instance of neovim.

Example wrapper script

Here is an example of a wrapper script generated by nix.

#! /nix/store/0hx32wk55ml88jrb1qxwg5c5yazfm6gf-bash-5.2-p15/bin/bash -e
export NVIM_SYSTEM_RPLUGIN_MANIFEST='/nix/store/jjl5fy7dc5cxvc7mi781vxbk8ag89ih0-neovim-0.8.3-esbonio/rplugin.vim'
export GEM_HOME='/nix/store/4mmkiw8n1nhlfsnh4g2kijzkxnp6fyxb-neovim-ruby-env/lib/ruby/gems/2.7.0'
PATH=${PATH:+':'$PATH':'}
if [[ $PATH != *':''/nix/store/4mmkiw8n1nhlfsnh4g2kijzkxnp6fyxb-neovim-ruby-env/bin'':'* ]]; then
    PATH=$PATH'/nix/store/4mmkiw8n1nhlfsnh4g2kijzkxnp6fyxb-neovim-ruby-env/bin'
fi
PATH=${PATH#':'}
PATH=${PATH%':'}
export PATH
LUA_PATH=${LUA_PATH:+';'$LUA_PATH';'}
LUA_PATH=${LUA_PATH/';''/nix/store/nlmk08cmald0zi7fc6hgpdqrjz7lh8qj-luajit-2.1.0-2022-10-04-env/share/lua/5.1/?/init.lua'';'/';'}
LUA_PATH='/nix/store/nlmk08cmald0zi7fc6hgpdqrjz7lh8qj-luajit-2.1.0-2022-10-04-env/share/lua/5.1/?/init.lua'$LUA_PATH
LUA_PATH=${LUA_PATH#';'}
LUA_PATH=${LUA_PATH%';'}
export LUA_PATH
LUA_PATH=${LUA_PATH:+';'$LUA_PATH';'}
LUA_PATH=${LUA_PATH/';''/nix/store/nlmk08cmald0zi7fc6hgpdqrjz7lh8qj-luajit-2.1.0-2022-10-04-env/share/lua/5.1/?.lua'';'/';'}
LUA_PATH='/nix/store/nlmk08cmald0zi7fc6hgpdqrjz7lh8qj-luajit-2.1.0-2022-10-04-env/share/lua/5.1/?.lua'$LUA_PATH
LUA_PATH=${LUA_PATH#';'}
LUA_PATH=${LUA_PATH%';'}
export LUA_PATH
LUA_CPATH=${LUA_CPATH:+';'$LUA_CPATH';'}
LUA_CPATH=${LUA_CPATH/';''/nix/store/nlmk08cmald0zi7fc6hgpdqrjz7lh8qj-luajit-2.1.0-2022-10-04-env/lib/lua/5.1/?.so'';'/';'}
LUA_CPATH='/nix/store/nlmk08cmald0zi7fc6hgpdqrjz7lh8qj-luajit-2.1.0-2022-10-04-env/lib/lua/5.1/?.so'$LUA_CPATH
LUA_CPATH=${LUA_CPATH#';'}
LUA_CPATH=${LUA_CPATH%';'}
export LUA_CPATH
exec -a "$0" "/nix/store/1czj8mydgi30kyfimq6q4ifh06q131ch-neovim-unwrapped-0.8.3/bin/nvim"  -u /nix/store/fqjv4r08pl8k3vhy6ijxddrn8gpq2h7z-init.vim '--cmd' 'let g:loaded_node_provider=0 | let g:loaded_python_provider=0 | let g:python3_host_prog='\''/nix/store/jjl5fy7dc5cxvc7mi781vxbk8ag89ih0-neovim-0.8.3-esbonio/bin/nvim-python3'\'' | let g:ruby_host_prog='\''/nix/store/jjl5fy7dc5cxvc7mi781vxbk8ag89ih0-neovim-0.8.3-esbonio/bin/nvim-ruby'\''' "$@"

After some trial and error I was able to put together the following

utils.lib.eachDefaultSystem (system:
  let
    pkgs = import nixpkgs { inherit system; };
    nvim-cfg = pkgs.neovimUtils.makeNeovimConfig {
      extraName = "-esbonio";
      customRC = ''
        set number
      '';
    };
    neovim-config = pkgs.lib.attrsets.updateManyAttrsByPath [
      {
        path = ["wrapperArgs"];
        update = old: old ++ [
          "--add-flags" "--clean"
        ];
      }
    ] nvim-cfg;
    neovim = pkgs.wrapNeovimUnstable pkgs.neovim-unwrapped neovim-config;
  in {
    apps.default = {
      type = "app";
      program = "${neovim}/bin/nvim";
    };
  }
);

To summarize

• pkgs.neovimUtils.makeNeovimConfig as the name suggests is a utility that generates a neovim “config”. “config” in this case is an attribute set containing all the arguments required to call pkgs.wrapNeovimUnstable.

• One of these arguments is called wrapperArgs which contains the list of cli arguments to pass to the wrapped instance of neovim. Well almost.

  wrapperArgs aren’t passed through to neovim directly, they are passed to a utility called makeWrapper which is a small program with it’s own set of arguments that allow you to describe how you want to wrap an underlying executable. This is why I’m appending "--add-flags" "--clean" to wrapperArgs and not just --clean.

• Finally, the config and base neovim derivation are passed to pkgs.wrapNeovimUnstable to bring it all together.

Unfortunately, after all that I still didn’t end up with the result I was looking for

No plugins, but also note no line numbers 😢

Not only does the --clean flag prevent neovim from loading the plugins in my home folder, it also stopped neovim from loading the contents of my customRC - something I would’ve found out if I’d actually read the help text for --clean itself

--clean - Mimics a fresh install of Nvim:

• Skips initializations from files and environment variables.

• No ‘shada’ file is read or written.

• Excludes user directories from ‘runtimepath’

• Loads builtin plugins, unlike -u NONE -i NONE.

It should be possible to work around this though by telling neovim to source our init file as well as giving it the --clean flag. Let’s take a look at the exec command nix is currently generating for us in the wrapper script.

exec -a "$0" "/nix/store/1czj8mydgi30kyfimq6q4ifh06q131ch-neovim-unwrapped-0.8.3/bin/nvim" \
     -u /nix/store/fqjv4r08pl8k3vhy6ijxddrn8gpq2h7z-init.vim \
     --cmd '...' \
     --clean \
     "$@"

The -u /nix/store/fqj...-init.vim argument contains the contents of our customRC and I think changing the command to something like

exec -a "$0" "/nix/store/1czj8mydgi30kyfimq6q4ifh06q131ch-neovim-unwrapped-0.8.3/bin/nvim" \
     --clean \
     --cmd 'source /nix/store/fqjv4r08pl8k3vhy6ijxddrn8gpq2h7z-init.vim' \
     "$@"

will result in the behaviour I’m looking for.

Tip

So far I’ve neglected to mention how I’m finding the /nix/store path containing this wrapper script. Using the nix repl you can load your flake and inspect the values it contains.

$ nix repl
Welcome to Nix 2.11.1. Type :? for help.

nix-repl> :lf .        # load the flake located at '.'
warning: Git tree '/var/home/alex/Projects/esbonio-nix' is dirty
Added 9 variables.

nix-repl> outputs.apps.x86_64-linux.default
{ program = "/nix/store/knr1nfdmg9ld0xg813hb7ljl68060jlv-neovim-0.8.3-esbonio/bin/nvim"; type = "app"; }

It’s also useful for figuring out how the many utilities in nixpkgs work

nix-repl> pkgs = import inputs.nixpkgs {system = "x86_64-linux"; }

nix-repl> config = pkgs.neovimUtils.makeNeovimConfig { customRC = "set number"; }

nix-repl> config.wrapperArgs
[ "--inherit-argv0" "--add-flags" "'--cmd' 'let g:loaded_node_provider=0 | let g:loaded_python_provider=0 | let g:python3_host_prog='\\''/1rz4g4znpzjwh1xymhjpm42vipw92pr73vdgl6xs1hycac8kf2n9/bin/nvim-python3'\\'' | let g:ruby_host_prog='\\''/1rz4g4znpzjwh1xymhjpm42vipw92pr73vdgl6xs1hycac8kf2n9/bin/nvim-ruby'\\'''" "--set" "GEM_HOME" "/nix/store/4mmkiw8n1nhlfsnh4g2kijzkxnp6fyxb-neovim-ruby-env/lib/ruby/gems/2.7.0" "--suffix" "PATH" ":" "/nix/store/4mmkiw8n1nhlfsnh4g2kijzkxnp6fyxb-neovim-ruby-env/bin" "--prefix" "LUA_PATH" ";" "/nix/store/nlmk08cmald0zi7fc6hgpdqrjz7lh8qj-luajit-2.1.0-2022-10-04-env/share/lua/5.1/?.lua;/nix/store/nlmk08cmald0zi7fc6hgpdqrjz7lh8qj-luajit-2.1.0-2022-10-04-env/share/lua/5.1/?/init.lua" "--prefix" "LUA_CPATH" ";" "/nix/store/nlmk08cmald0zi7fc6hgpdqrjz7lh8qj-luajit-2.1.0-2022-10-04-env/lib/lua/5.1/?.so" ]

.

Unfortunately, I could not see an obvious way to rewrite the arguments to exec. The store path for the init.vim file is only generated in the depths of the wrapNeovimUnstable as it is written to disk and trying to manipulate wrapperArgs to extract it isn’t something I’m willing to attempt in Nix just yet!

A New Approach

It was at this point I started looking around to see what other people have come up with and before long I found this reddit thread which linked this flake that looked very promising. Not only did it provide a way of creating an isolated config but it also showed how to manage plugins and external binaries!

Following its example I was able to come up with the following definition

utils.lib.eachDefaultSystem (system:
  let
    pkgs = import nixpkgs { inherit system ; };
    initVim = ''
      set number
    '';
    paths = pkgs.lib.makeBinPath [
      pkgs.neovim
    ];
    pluginList = with pkgs.vimPlugins; [
      nvim-lspconfig
    ];
    plugins = pkgs.stdenv.mkDerivation {
      name = "esbonio-nvim-plugins";
      buildCommand = ''
        mkdir -p $out/nvim/site/pack/plugins/start/
        ${pkgs.lib.concatMapStringsSep "\n" (path: "ln -s ${path} $out/nvim/site/pack/plugins/start/")  pluginList }
      '';
    };
    neovim = pkgs.writeShellScriptBin "nvim" ''
      export PATH=${paths}:$PATH
      export XDG_CONFIG_DIRS=
      export XDG_DATA_DIRS=${plugins.outPath}
      nvim --clean --cmd source ${pkgs.writeText "init.vim" initVim} "$@"
    '';
  in {
     apps.default = {
       type = "app";
       program = "${neovim}/bin/nvim";
     };
   }
);

And trying nix run . once more

Success!

Not only did I end up with the correct configuration, the runtimepath finally contains just the paths that are necessary!

Integrating Esboino

Next we need to make sure the esbonio language server is available in this environment and include the necessary configuration for it in the config.

Including the server should be pretty straightforward as we get to reuse the overlay defined previously.

  inputs = {
    nixpkgs.url = "github:NixOS/nixpkgs/nixpkgs-unstable";
+   esbonio.url = "path:lib/esbonio";
+   esbonio.inputs.nixpkgs.follows = "nixpkgs";
    utils.url = "github:numtide/flake-utils";
  };

- outputs = { self, nixpkgs, utils }:
+ outputs = { self, nixpkgs, esbonio, utils }:

 utils.lib.eachDefaultSystem (system:
   let
-    pkgs = import nixpkgs { inherit system ; };
+    pkgs = import nixpkgs { inherit system ; overlays = [ esbonio.overlays.default ];};
     initVim = ''
       set number
     '';
     paths = pkgs.lib.makeBinPath [
       pkgs.neovim
+      pkgs.python310Packages.esbonio
     ];

Should being the key word here…

$ nix run .
warning: Git tree '/var/home/alex/Projects/esbonio-nix' is dirty
warning: updating lock file '/var/home/alex/Projects/esbonio-nix/flake.lock':
• Added input 'esbonio':
    'path:lib/esbonio?lastModified=1&narHash=sha256-WiFypw4lUZo7P9h82NMudwb5DFV0Nde5cOu1SqDmhVQ=' (1970-01-01)
• Added input 'esbonio/nixpkgs':
    follows 'nixpkgs'
• Added input 'esbonio/pytest-lsp':
    'github:swyddfa/lsp-devtools/6ae80a24b55d2b6943b9d30805cf02440ebbaf5c?dir=lib%2fpytest-lsp' (2023-04-02)
• Added input 'esbonio/pytest-lsp/nixpkgs':
    follows 'esbonio/nixpkgs'
• Added input 'esbonio/pytest-lsp/utils':
    'github:numtide/flake-utils/93a2b84fc4b70d9e089d029deacc3583435c2ed6' (2023-03-15)
• Added input 'esbonio/utils':
    'github:numtide/flake-utils/5aed5285a952e0b949eb3ba02c12fa4fcfef535f' (2022-11-02)
warning: Git tree '/var/home/alex/Projects/esbonio-nix' is dirty
error: undefined variable 'pytest-lsp'

       at /nix/store/96z740kkay7j0cbgmccj2mzbn5z8agvp-source/nix/esbonio-overlay.nix:22:11:

           21|           mock
           22|           pytest-lsp
             |           ^
           23|           pytest-timeout
(use '--show-trace' to show detailed location information)

The overlay exported by the language server’s flake doesn’t include its dependency pytest-lsp which is provided through an overlay of its own. A quick “fix” would be to also pull in the flake for pytest-lsp, but really the language server’s flake should be exporting all of its dependencies.

Composing Overlays

Thankfully, nixpkgs provides a function composeManyExtensions that handles this for us. When exporting the overlay from within the language server’s flake we can use it to merge the overlay from pytest-lsp with the overlay containing esbonio.

# In lib/esbonio/flake.nix
overlays.default = self: super: nixpkgs.lib.composeManyExtensions [
  pytest-lsp-overlay
  esbonio-overlay
] self super

However, since flake.lock freezes the language server’s flake as it was before we made this change we need to also update the lock file before trying again

$ nix flake lock --update-input esbonio
warning: Git tree '/var/home/alex/Projects/esbonio-nix' is dirty
warning: updating lock file '/var/home/alex/Projects/esbonio-nix/flake.lock':
• Updated input 'esbonio':
    'path:lib/esbonio?lastModified=1&narHash=sha256-WiFypw4lUZo7P9h82NMudwb5DFV0Nde5cOu1SqDmhVQ=' (1970-01-01)
  → 'path:lib/esbonio?lastModified=1&narHash=sha256-QgSDxOPSrtsaqjeStalef07+bUE3qkzz7pJC4y43ltw=' (1970-01-01)
warning: Git tree '/var/home/alex/Projects/esbonio-nix' is dirty

Now trying nix run . again neovim launches as before, running the command :r !python -m esbonio --help we can verify that the language server is indeed available to the editor.

Almost there!

Editor’s Note

Since writing this section and taking the above screenshot, I have been unable to re-produce it! Now :r !python -m esbonio --help results in a esbonio: Module not found error…

When debugging this, I’m not sure how the original ever worked since the flake definition does not include a Python interpreter meaning that python -m esbonio --help is running under the system Python.

The fix then, was to switch from python -m esbonio --help to calling esbonio --help directly, which thankfully, did not require me to change any of the Nix code.

Configuring Neovim

Now all that’s left to do is updating our initVim variable to contain the relevant configuration for the language server. Thanks to the example configuration available in the documentation, this can be as straightforward as replacing our hardcoded configuration with a call to Nix (the language’s) builtin readFile function.

 utils.lib.eachDefaultSystem (system:
   let
     pkgs = import nixpkgs { inherit system ; overlays = [ esbonio.overlays.default ];};
-    initVim = ''
-      set number
-    '';
+    initVim = builtins.readFile ./docs/lsp/editors/nvim-lspconfig/init.vim;
     paths = pkgs.lib.makeBinPath [
       pkgs.neovim
       pkgs.python310Packages.esbonio
     ];

And try opening a Sphinx project with it

It’s not pretty, but it works!

Wrapping Up

The experience as is currently stands is not that inspiring however, with the nix foundations laid it’s now more of a configuring neovim problem rather than a nix one! I am mildly disappointed that this required to dive so deep on the specifics of how neovim is configured, since that probably means you’d have to go to a similar depth to incorporate other editors. That said, once you’ve solved it for a given editor it’s probably solved “forever”.

Next I think I’d be interested in exploring how (or if it’s even possible) to make these Nix definitions more dynamic e.g.

• Using the language server from $EDITOR using Python 3.x

• Run the language server tests, but with a local checkout of pygls

• Edit docs for $PROJECT using Sphinx vX

Obviously, you could achieve a lot of that by just editing the Nix definitions and rebuilding but I wonder if it’s possible to build in support for swapping parts out that can be wrapped up in a Makefile or similar 🤔