My Next Steps with Nix: Overlays¶
Packaging pytest-lsp
¶
Adapting the
flake.nix
file and using the
package definition
from the previous post, it’s easy enough to sketch out a flake that should give us a devShell
to work on the pytest-lsp
package.
However, trying to activate one we encounter a problem.
$ nix develop -c .#py310
error: builder for '/nix/store/dfd5bixdgkvfcnfa7f9z0ibp4m5zlhkz-python3.10-pytest-lsp-0.2.1.drv' failed with exit code 1;
last 10 log lines:
> installing
> Executing pipInstallPhase
> /build/pytest-lsp/dist /build/pytest-lsp
> Processing ./pytest_lsp-0.2.1-py3-none-any.whl
> Requirement already satisfied: appdirs in /nix/store/yidjmqc5q1j0fz2dk79qgk1fy7dqcliy-python3.10-appdirs-1.4.4/lib/python3.10/site-packages (from pytest-lsp==0.2.1) (1.4.4)
> Requirement already satisfied: pytest-asyncio in /nix/store/dvz12bivdc0dkn6849zm58754ga06hs6-python3.10-pytest-asyncio-0.20.3/lib/python3.10/site-packages (from pytest-lsp==0.2.1) (0.20.3)
> Requirement already satisfied: pytest in /nix/store/z5pkmmsdg3bmb35pmsv4rjca1qi7dbnf-python3.10-pytest-7.2.0/lib/python3.10/site-packages (from pytest-lsp==0.2.1) (7.2.0)
> ERROR: Could not find a version that satisfies the requirement pygls>=1.0.0 (from pytest-lsp) (from versions: none)
> ERROR: No matching distribution found for pygls>=1.0.0
>
For full logs, run 'nix log /nix/store/dfd5bixdgkvfcnfa7f9z0ibp4m5zlhkz-python3.10-pytest-lsp-0.2.1.drv'.
error: 1 dependencies of derivation '/nix/store/kfzlz750xdk71fxwvsgpdbw1w00jbvf9-py310-env.drv' failed to build
In the time between writing the previous blog post and this one, the pytest-lsp
package has been migrated to the latest version of pygls
.
The version available through nixpkgs however, is still the previous release.
While there is an open pull request updating pygls
to 1.0
, at the time of writing it’s blocked on downstream packages which haven’t migrated yet.
That said, we don’t have to wait for nixpkgs but can instead use an overlay to update it just for this project.
Overriding pygls’ version¶
Overlays can be used to override sections of an existing package definition.
Note
As I mentioned in the previous post, I’m probably not the best person to learn Nix from. Instead, here are some resources you might useful which go into more detail.
The NixOS wiki page on Overlays
Nix Pills: Chapter 14. Override design pattern
As far as I understand it:
Overlays are a useful design pattern, rather than a fundamental concept of the Nix language.
They are “just” a nix function that have access to both the modified version (usually called
self
orfinal
) of the “thing” they’re modifying, as well as the unmodified version of it (super
orprev
)These functions make use of attributes like
override
oroverrideAttrs
to make their modifications.I have no idea how to make something overridable 😅
After reading through the wiki page on overlays a few times, particuarly the sections on overriding a version and python package overlays, I was able to put together an overlay which looked like it should work.
# In ./nix/pygls-overlay.nix
self: super: rec {
python3 = super.python3.override {
packageOverrides = pyself: pysuper: {
pygls = pysuper.pygls.overrideAttrs (old: rec {
version = "1.0.0";
src = super.fetchFromGitHub {
owner = "openlawlibrary";
repo = "pygls";
rev = "v${version}";
hash = "sha256-31J4+giK1RDBS52Q/Ia3Y/Zak7fp7gRVTQ7US/eFjtM=";
};
});
};
};
python3Packages = python3.pkgs;
}
Using this in the flake is a matter of importing it and passing it to the overlays
attribute when importing nixpkgs
# In flake.nix
outputs = { self, nixpkgs, utils }:
let
pygls-overlay = import ./nix/pygls-overlay.nix;
eachPythonVersion = ...
in {
devShells = utils.lib.eachDefaultSystemMap (system:
let
pkgs = import nixpkgs { inherit system; overlays = [ pygls-overlay ]; };
in
eachPythonVersion [ "37" "38" "39" "310" "311" ] (pyVersion:
let
pytest-lsp = pkgs.callPackage ./nix/pytest-lsp.nix { pythonPackages = pkgs."python${pyVersion}Packages"; };
in
With some luck, running nix develop
this time should bring in the latest pygls
version
$ nix develop -c .#py310
error: builder for '/nix/store/1sha5j0dfyn2g4z82rpk4yqv32awmjfr-python3.10-pytest-lsp-0.2.1.drv' failed with exit code 1;
...
> ERROR: No matching distribution found for pygls>=1.0.0
Huh, same error… 🤔
Let’s take a closer look at where we pull in the pytest-lsp
package definition in the flake…
pytest-lsp = pkgs.callPackage ./nix/pytest-lsp.nix {
pythonPackages = pkgs."python${pyVersion}Packages";
};
Assuming we’re trying to enter the python310
devShell, then we’re passing in the python310Packages
package set.
But in the overlay, we’re overriding the python3Packages
package set, I wonder if we change the overlay to match the flake…
# In ./nix/pygls-overlay.nix
self: super: rec {
python310 = super.python310.override { ... };
python310Packages = python310.pkgs;
}
And try again
$ nix develop .#py310
error: builder for '/nix/store/jl23ai588n2b6amaicy5532bdxjiciyy-python3.10-pygls-0.13.0.drv' failed with exit code 1;
last 10 log lines:
> removing build/bdist.linux-x86_64/wheel
> Finished executing setuptoolsBuildPhase
> installing
> Executing pipInstallPhase
> /build/source/dist /build/source
> Processing ./pygls-0.13.0-py3-none-any.whl
> Requirement already satisfied: typeguard<3,>=2.10.0 in /nix/store/m4jjcrvbi928pi2d14qh8np1miqfvc0b-python3.10-typeguard-2.13.3/lib/python3.10/site-packages (from pygls==0.13.0) (2.13.3)
> ERROR: Could not find a version that satisfies the requirement lsprotocol (from pygls) (from versions: none)
> ERROR: No matching distribution found for lsprotocol
>
For full logs, run 'nix log /nix/store/jl23ai588n2b6amaicy5532bdxjiciyy-python3.10-pygls-0.13.0.drv'.
error: 1 dependencies of derivation '/nix/store/f5vasy4x9zpdhcq9jh9rz06qpvriblwp-python3.10-pytest-lsp-0.2.1.drv' failed to build
error: 1 dependencies of derivation '/nix/store/86v8bcxvjq1g9dhpx1wgmckba8bnag7h-py310-env.drv' failed to build
Progress!
Packaging lsprotocol
¶
pygls is failing to build as the package definition in nixpkgs is missing the new lsprotcol
dependency, easy enough to fix - if it was available in nixpkgs.
Thankfully, overlays can do more than just override attributes on existing packages, they can be used to extend a package set with entirely new definitions!
We just need to know how to package lsprotocol
itself and thanks to the PR linked above we get to cheat a little.
# In ./nix/pygls-overlay.nix
lsprotocol = pysuper.buildPythonPackage rec {
pname = "lsprotocol";
version = "2022.0.0a9";
format = "pyproject";
src = super.fetchFromGitHub {
owner = "microsoft";
repo = pname;
rev = version;
hash = "sha256-6XecPKuBhwtkmZrGozzO+VEryI5wwy9hlvWE1oV6ajk=";
};
nativeBuildInputs = with super.python310Packages; [
flit-core
];
propagatedBuildInputs = with super.python310Packages; [
cattrs
attrs
];
# Disable tests
doCheck = false;
};
Note that I’ve cut some corners by disabling any tests, but it allows me to dodge packaging anything else 😅
Then we can also override pygls’ dependencies and reference the newly created lsprotocol
package from the modified version of the python310Packages
set.
pygls = pysuper.pygls.overrideAttrs (_: rec {
...
propagatedBuildInputs = with self.python310Packages; [
lsprotocol
typeguard
];
});
With that taken care of, we should be good to go right?
Unlucky 0.13
¶
Attempting to enter the devShell yet again we encounter a familiar error message
error: builder for '/nix/store/s5xp7fr2r9faxgqw7rvs6ffah10f2fz7-python3.10-pytest-lsp-0.2.1.drv' failed with exit code 1;
last 10 log lines:
> Finished executing setuptoolsBuildPhase
> installing
> Executing pipInstallPhase
> /build/pytest-lsp/dist /build/pytest-lsp
> Processing ./pytest_lsp-0.2.1-py3-none-any.whl
> Requirement already satisfied: pytest-asyncio in /nix/store/dvz12bivdc0dkn6849zm58754ga06hs6-python3.10-pytest-asyncio-0.20.3/lib/python3.10/site-packages (from pytest-lsp==0.2.1) (0.20.3)
> Requirement already satisfied: pytest in /nix/store/z5pkmmsdg3bmb35pmsv4rjca1qi7dbnf-python3.10-pytest-7.2.0/lib/python3.10/site-packages (from pytest-lsp==0.2.1) (7.2.0)
> ERROR: Could not find a version that satisfies the requirement pygls>=1.0.0 (from pytest-lsp) (from versions: none)
> ERROR: No matching distribution found for pygls>=1.0.0
>
For full logs, run 'nix log /nix/store/s5xp7fr2r9faxgqw7rvs6ffah10f2fz7-python3.10-pytest-lsp-0.2.1.drv'.
error: 1 dependencies of derivation '/nix/store/a0smpmj63fw1fzp78i3z53xvd0zsvvhp-py310-env.drv' failed to build
But we just upgraded pygls to 1.0
right? That’s why we had to package lsprotocol
in the previous section?
You might have already noticed in the log output above, that despite overriding the version
field to 1.0
the Python package was still coming out as 0.13.0
- despite it containing the 1.0
version of the codebase!
> Processing ./pygls-0.13.0-py3-none-any.whl
Plenty of head scratching later, I finally remembered that pygls uses setuptools_scm to automatically derive the version number based on tags in its git repository. But the build is not taking place in a git repo… so nix must be setting that version somehow right?
Yep. A quick trip to the actual file containing pygls’ package definition on nixpkgs (and not just the diff view in the PR!) reveals an additional attribute that needed to be overriden
# In ./nix/pygls-overlay.nix
pygls = pysuper.pygls.overrideAttrs (_: rec {
version = "1.0.0";
SETUPTOOLS_SCM_PRETEND_VERSION = version;
...
});
Now if we try activating that devShell?
$ nix develop .#py310
(nix-shell) $ pytest
================================== test session starts =================================
platform linux -- Python 3.10.9, pytest-7.2.0, pluggy-1.0.0
rootdir: /var/home/alex/Projects/lsp-devtools/lib/pytest-lsp, configfile: pyproject.toml
plugins: lsp-0.2.1, typeguard-2.13.3, asyncio-0.20.3
asyncio: mode=auto
collected 27 items
tests/test_client.py ... [ 11%]
tests/test_client_methods.py ................... [ 81%]
tests/test_plugin.py .... [ 96%]
tests/test_server.py . [100%]
================================= 27 passed in 8.57s ==================================
Success!
Note
I’m not 100% sure if I’ve overriden the pygls’ version number correctly, since inspecting the PYTHONPATH
the devShell is using shows that the version number of the nix package is still 0.13.0
!
(nix-shell) $ echo $PYTHONPATH | tr ':' '\n' | grep pygls
/nix/store/s5jh5s9m5f1163hxzj8768jc5li7cdfg-python3.10-pygls-0.13.0/lib/python3.10/site-packages
But in Python land, everything appears at least, to be consistent, so I’m going with it for now.
Mutliple Python Versions¶
Now that we’ve got it working for Python 3.10, we need to generalise the overlay so that we can use it with any of the Python versions supported by pytest-lsp
.
Ideally, what we’d want is to write an expression like the following
# In ./nix/pygls-overlay.nix
self: super:
eachPythonVersion ["37" "38" "39" "310" "311"] (pyVersion:
super."python${pyVersion}".override {
packageOverrides = pyself: pysuper: {
lsprotocol = pysuper.buildPythonPackage rec {
...
nativeBuildInputs = with super."python${pyVersion}Packages"; [
flit-core
];
propagatedBuildInputs = with super."python${pyVersion}Packages"; [
cattrs
attrs
];
};
pygls = pysuper.pygls.overrideAttrs (_: rec {
...
propagatedBuildInputs = with self."python${pyVersion}Packages"; [
lsprotocol
typeguard
];
});
};
})
And have the eachPythonVersion
function handle the details of performing all the overrides.
To start with, let’s define a helper doPythonOverride
that eachPythonVersion
can use.
It should take a version
and a function f
and use it to perform the override for a single Python version, something like the following pseudo code.
doPythonOverride(version, f) = { "python${version}" = f(version);
"python${version}Packages" = "python${version}".pkgs; }
The only issue is that (as far as I can tell), you can’t use strings as keys in a nix attribute set.
However, you can use the
builtins.listToAttrs
function to build an attribute set from a list of { name = "xxx"; value = 123; }
attribute sets, which allows us to define doPythonOverride
as follows.
doPythonOverride = version: f:
let
overridenPython = f version;
in
builtins.listToAttrs [ {name = "python${version}"; value = overridenPython; }
{name = "python${version}Packages"; value = overridenPython.pkgs; }];
From there, we can define eachPythonVersion
to map the doPythonOverride
helper across each of the given Python versions and merge the results into a single attribute set using the
foldl’
function.
eachPythonVersion = versions: f: builtins.foldl' (a: b: a // b) {}
(builtins.map (version: doPythonOverride version f) versions);
Now we should have successfully overriden pygls’ version across all supported Python versions!