Survey
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
Python on Nix tutorial Documentation Release 0.0.1 Frederik Rietdijk Mar 25, 2017 Contents 1 2 Python on Nix 1.1 Introduction to Nix . . . . . . 1.2 Getting started . . . . . . . . 1.3 Using Python . . . . . . . . . 1.4 Developing a Python package 1.5 Organising your packages . . 1.6 Python on Nix internals . . . 1.7 FAQ . . . . . . . . . . . . . . Indices and tables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 3 3 3 10 14 15 15 17 i ii Python on Nix tutorial Documentation, Release 0.0.1 Contents: Contents 1 Python on Nix tutorial Documentation, Release 0.0.1 2 Contents CHAPTER 1 Python on Nix You’re a Python dev, and recently you’ve heard about Nix. Immediately convinced by sane package management you decide you want to start using Nix. But before you do so, there’s still an important question that you want an answer to. How does Nix affect my Python development workflow? In this tutorial we show you how you use and develop with Python on Nix. Introduction to Nix While we assume you’ve already read a bit on Nix before going through this tutorial, we recap a couple of things first. Nix is a purely functional package manager for *Nix systems. Packages are specified using the Nix expression language. NixOS is an operating system built around the Nix package manager. On NixOS not only packages but also services (called modules) are described using the Nix expression language. Nixpkgs is a collection of packages (and NixOS modules). Getting started If you don’t have Nix yet, and you would like to install it on your *Nix system, then head on to the Nix Quick Start. if instead you want to install NixOS, then please continue now with the NixOS installation guide. Using Python Installing software Now that you’ve installed Nix you’re ready to start installing software. With the Nix package manager you can ad hoc install software in your profile using nix-env -iA. e.g. $ nix-env -iA nixpkgs.pkgs.pandoc 3 Python on Nix tutorial Documentation, Release 0.0.1 would install the Pandoc tool in your profile. From then on you will have the possibility to run pandoc from within a shell. If you are running NixOS and you have root/sudo access, then you can specify declaratively which packages need to be installed by appending those packages to environment.systemPackages in the /etc/nixos/ configuration.nix file ... environment.systemPackages = with pkgs; [ busybox chromium pandoc ]; ... A common method on Nix is however not to install all the software you need, but instead using the Nix shell (nix-shell) to open a shell with exactly those packages that you need when you need them. E.g., say you want to convert some files and so you want to use Pandoc, then you could run $ nix-shell -p pandoc which opens a shell from which you can run pandoc [nix-shell:~] pandoc tutorial.md -o tutorial.pdf Installing Python? You might be wondering now why, if this tutorial is about Python, we are using Pandoc as an example and not Python? Well, that’s because with Python you’re generally interested in not just the interpreter, but also Python packages. On Nix most software can be installed in a profile, either ad hoc or declaratively. However, this is not possibly with Python and packages. Actually, to be precise, the tools allow you to install to your profile e.g. Python 3.5 using $ nix-env -iA nixpkgs.pkgs.python35 Most likely, now running $ python35 will actually open the interpreter. Nothing wrong, right? Well, not entirely. There is a problem though with installing Python modules/packages in this way; generally they cannot be accessed from the interpreter. Obviously you do not want this. What’s the solution to that, you might ask? Installing Python declaratively perhaps? How about the following in your /etc/nixos/configuration.nix in case you’re running NixOS? ... environment.systemPackages = with pkgs; [ busybox chromium pandoc python35 python35Packages.numpy python35Packages.toolz 4 Chapter 1. Python on Nix Python on Nix tutorial Documentation, Release 0.0.1 ]; ... Nope, it might work, but likely not always. But why does it not work then? With Nix you install only applications. Because with Nix you can have multiple versions of a library/application at the same time, an application needs to know which exact libraries to use, that is, which exact entries in the Nix store (/nix/store). Libraries that are needed by an application are defined as buildInputs in the Nix expression of the application. When building/installing the application, the libraries are also built/installed. You won’t ever manually install libraries using nix-env -iA or in environment.systemPackages. But now let’s consider Python. You might want to install the interpreter system-wide, along with maybe some Python packages. As user, you realise you want to have some additional packages so you install them using nix-env -iA. Remember, with Nix, you can have multiple versions of libraries because different applications might require different versions. How now, would the interpreter decide which version of say numpy to use, when multiple are installed? The bottomline is that installing Python and packages is not supported. The way to go though is environments... Python using nix-shell Perhaps the easiest way to get a functional Python environment is by using nix-shell. Executing $ nix-shell -p python35Packages.numpy python35Packages.toolz opens a Nix shell which has available the requested packages and dependencies. Now you can launch the Python interpreter (which is itself a dependency) [nix-shell:~] python3 If the packages were not available yet in the Nix store, Nix would download or compile them automatically. A convenient option with nix-shell is the --run option, with which you can execute a command in the nix-shell. Let’s say we want the above environment and directly run the Python interpreter $ nix-shell -p python35Packages.numpy python35Packages.toolz --run "python3" You can also use the --run option to directly execute a script $ nix-shell -p python35Packages.numpy python35Packages.toolz --run "python3 myscript. ˓→py" For this specific case there is another convenient method; you can add a shebang to your script specifying which dependencies Nix shell needs. With the following shebang, you can use nix-shell myscript.py and it will make available all dependencies and run the script in the python3 shell. #! /usr/bin/env nix-shell #! nix-shell -i python3 -p python35Packages.numpy python35Packages.toolz The first line here is a standard shebang. We say we want to use nix-shell. With the Nix shell you are however not limited to only a single line, but you can have multiple. The second line instructs Nix shell to create an environment with the -p packages in it, as we did before, and then run the -i interpreter. Note that the -i option can only be used as part of a shebang. In other cases you will have to use the --run option as shown above. By default all installed applications are still accessible from the Nix shell. If you do not want this, you can use the --pure option. 1.3. Using Python 5 Python on Nix tutorial Documentation, Release 0.0.1 $ nix-env -iA nixpkgs.pkgs.pandoc $ nix-shell -p python35Packages.numpy python35Packages.toolz --pure [nix-shell:~] pandoc The program ‘pandoc’ is currently not installed. You can install it by typing: nix-env -iA nixos.pandoc Likely you do not want to type your dependencies each and every time. What you can do is write a simple Nix expression which sets up an environment for you, requiring you only to type nix-shell. Say we want to have Python 3.5, numpy and toolz, like before, in an environment. With a shell.nix file containing with import <nixpkgs> {}; ( pkgs.python35.buildEnv.override { extraLibs = with pkgs.python35Packages; [ numpy toolz ]; }).env executing nix-shell gives you again a Nix shell from which you can run Python. So what do those lines here mean? Let’s consider line by line: 1. We begin with importing the Nix Packages collections. import <nixpkgs> {} does the actual import and the with statement brings all attributes of nixpkgs in the local scope. Therefore we can now use pkgs. 2. Then we say we want a Python 3.5 environment, so we use the derivation pkgs.python35.buildEnv. Because we want to use it with a custom set of Python packages, we override it. 3. The extraLibs argument of the original buildEnv function can be used to specify which packages you want. We want numpy and toolz. Again, we use the with statement to bring a set of attributes into the local scope. 4. EXPLAIN Declarative environment using myEnvFun Using Nix shell means you either need to add a bunch of arguments to the nix-shell invocation, or executing a specific file with Nix shell. Another option is to instead define your environments declaratively in your user profile. As user you have a ~/.nixpkgs/config.nix file in which you can include overrides specifically for yourself. Here we can add our declarative environments as well. Let’s say we already have the following config.nix. with <nixpkgs> {}; { allowUnfree = true; } This expression imports the Nix packages collections, and says that we allow unfree software. Let’s extend this now with two environments. We add one environment that we use for development, and another for blogging with Pelican. with <nixpkgs> {}; { allowUnfree = true; allowBroken = true; packageOverrides = pkgs: with pkgs; { devEnv = pkgs.myEnvFun { name = "work"; buildInputs = with python34Packages; [ python34 numpy 6 Chapter 1. Python on Nix Python on Nix tutorial Documentation, Release 0.0.1 toolz ]; }; blogEnv = pkgs.myEnvFun { name = "blog"; buildInputs = with python27Packages; [ python27 pelican markdown ]; }; }; } For the first environment we want Python 3.4, and for the second Python 2.7. Note that we have to explicitly include the interpreter when using myEnvFun! We can install these environments with nix-env -i env-<name> and use them by calling load-env-<name>. In both cases <name> is the argument name of the function myEnvFun. $ nix-env -i env-work installing ‘env-work’ $ load-env-work env-work loaded work:[~]$ You can now start the interpreter, python3. Missing Python modules? At this point you might have gone ahead using the Nix shell or myEnvFun to create Python environments, and got some very surprising import errors, unrelated to those explained before. If you haven’t encountered these yet, try running $ nix-shell -p python27 --run python and then >>> import sqlite3 You will notice that you get an ImportError. >>> import sqlite3 Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/nix/store/v9pq6f0s1r5fdybqc7pbv7mkb33lx9yy-python-2.7.10/lib/python2.7/sqlite3/ ˓→__init__.py", line 24, in <module> from dbapi2 import * File "/nix/store/v9pq6f0s1r5fdybqc7pbv7mkb33lx9yy-python-2.7.10/lib/python2.7/sqlite3/ ˓→dbapi2.py", line 28, in <module> from _sqlite3 import * ImportError: No module named _sqlite3 What’s going on here? In Nixpkgs the Python 2.x interpreters were sent on a diet. To reduce dependency bloat some modules were removed from the 2.x interpreters. If you do want some of these modules, then you have to include them explicitly, e.g. 1.3. Using Python 7 Python on Nix tutorial Documentation, Release 0.0.1 $ nix-shell -p python27 python27.modules.sqlite3 --run python to include the sqlite3 module. For convenience, there is also a python27Full package which includes all these modules $ nix-shell -p python27Full python27Packages.numpy python27Packages.toolz --run python How to find Python packages? So far we only considered two python packages, numpy and toolz. At this point you might be wondering how to find packages. You can search for packages with nix-env -qa. The -q stands for query and -a for available derivations. Let’s search for numpy $ nix-env -qa '.*numpy.*' pypy2.6-numpydoc-0.5 python2.7-numpy-1.10.1 python2.7-numpydoc-0.5 python3.4-numpy-1.10.1 python3.4-numpydoc-0.5 python3.5-numpy-1.10.1 python3.5-numpydoc-0.5 A tool that is generally easier to begin with is Nox. You can install Nox with nix-env -i nox or try it with nix-shell -p nox. Let’s search for numpy with Nox. $ nox numpy Refreshing cache 1 pypy2.6-numpydoc-0.5 (nixos.pypyPackages.numpydoc) Sphinx extension to support docstrings in Numpy format 2 python2.7-numpy-1.10.1 (nixos.python27Packages.numpy) Scientific tools for Python 3 python2.7-numpydoc-0.5 (nixos.python27Packages.numpydoc) Sphinx extension to support docstrings in Numpy format 4 python3.4-numpy-1.10.1 (nixos.python34Packages.numpy) Scientific tools for Python 5 python3.4-numpydoc-0.5 (nixos.python34Packages.numpydoc) Sphinx extension to support docstrings in Numpy format 6 python3.5-numpy-1.10.1 (nixos.python35Packages.numpy) Scientific tools for Python 7 python3.5-numpydoc-0.5 (nixos.python35Packages.numpydoc) Sphinx extension to support docstrings in Numpy format Packages to install: Nox provides, among other things, an easier interface to nix-env for querying and installing packages. Nox shows you the name with version of packages, along with the Nix attribute, e.g. nixos.python34Packages.numpy. The first part is the identifier of the channel, in this case nixos, since $ nix-channel --list nixos https://nixos.org/channels/nixos-unstable Another example $ nox pandas 1 pypy2.6-pandas-0.17.0 (nixos.pypyPackages.pandas) Python Data Analysis Library 2 python2.7-pandas-0.17.0 (nixos.python27Packages.pandas) 8 Chapter 1. Python on Nix Python on Nix tutorial Documentation, Release 0.0.1 Python Data Analysis Library 3 python3.4-pandas-0.17.0 (nixos.python34Packages.pandas) Python Data Analysis Library 4 python3.5-pandas-0.17.0 (nixos.python35Packages.pandas) Python Data Analysis Library Packages to install: Alternative interpreters and shells So far we considered only the CPython interpreter, but in the examples shown just before, you could see that packages for the PyPy interpreter also show up. Indeed, you can use the PyPy interpreter on Nix as well $ nix-shell -p pypy --run pypy Python 2.7.9 (295ee98b69288471b0fcf2e0ede82ce5209eb90b, Sep 21 2015, 22:02:02) [PyPy 2.6.0 with GCC 4.9.3] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>>> We can get an environment with PyPy just like we did before. $ nix-shell -p pypyPackages.numpy pypyPackages.toolz but this results in an error error: numpy-1.10.1 not supported for interpreter pypy (use ‘--show-trace’ to show detailed location information) Why is that? As the message explains, numpy is not supported for pypy, just like many other packages that include extension types. This is however not a Nix issue, but a PyPy issue. Even so, you will encounter these kind of errors more often, since also with CPython certain packages are supported on certain versions, but not all. Included in the Nix packages collection are also alternative Python shells, like Jupyter/IPython. Say we want to use numpy and toolz again but now using the IPython interpreter $ nix-shell -p python34Packages.ipython python34Packages.numpy python34Packages.toolz ˓→--run ipython Python 3.4.3 (default, Jan 01 1970, 00:00:01) Type "copyright", "credits" or "license" for more information. IPython 4.0.0 -- An enhanced Interactive Python. ? -> Introduction and overview of IPython's features. %quickref -> Quick reference. help -> Python's own help system. object? -> Details about 'object', use 'object??' for extra details. In [1]: We can also use the Jupyter QtConsole nix-shell -p python34Packages.qtconsole --run "jupyter qtconsole" or the Jupyter Notebook nix-shell -p python34Packages.notebook --run "jupyter notebook" 1.3. Using Python 9 Python on Nix tutorial Documentation, Release 0.0.1 Developing a Python package Now that you know how to get a working Python environment on Nix, it is time to go forward and start actually developing with Python. We will first have a look at how Python packages are packaged on Nix. Then, we will look how you can use development mode with your code. Python packaging on Nix On Nix all packages are built by functions. The main function in Nix for building Python packages is buildPythonPackage. Let’s see how we would build the toolz package. According to python-packages. nix toolz is build using toolz = buildPythonPackage rec{ name = "toolz-${version}"; version = "0.7.4"; src = pkgs.fetchurl{ url = "https://pypi.python.org/packages/source/t/toolz/toolz-${version}.tar.gz"; sha256 = "43c2c9e5e7a16b6c88ba3088a9bfc82f7db8e13378be7c78d6c14a5f8ed05afd"; }; meta = { homepage = "http://github.com/pytoolz/toolz/"; description = "List processing tools and functional utilities"; license = licenses.bsd3; maintainers = with maintainers; [ fridh ]; }; }; What happens here? The function buildPythonPackage is called and as argument it accepts a set. In this case the set is a recursive set (rec). One of the arguments is the name of the package, which consists of a basename (generally following the name on PyPi) and a version. Another argument, src specifies the source, which in this case is fetched from an url. fetchurl not only downloads the target file, but also validates its hash. Furthermore, we specify some (optional) meta information. The output of the function is a derivation, which is an attribute with the name toolz of the set pythonPackages. Actually, sets are created for all interpreter versions, so python27Packages, python34Packages, python35Packages and pypyPackages. The above example works when you’re directly adding or modifying packages to python-packages.nix. Often though, you will want to test a Nix expression outside of the Nixpkgs tree. If you create a shell.nix file with the following contents with import <nixpkgs> {}; pkgs.python35Packages.buildPythonPackage rec { name = "toolz-${version}"; version = "0.7.4"; src = pkgs.fetchurl{ url = "https://pypi.python.org/packages/source/t/toolz/toolz-${version}.tar.gz"; sha256 = "43c2c9e5e7a16b6c88ba3088a9bfc82f7db8e13378be7c78d6c14a5f8ed05afd"; }; meta = { homepage = "http://github.com/pytoolz/toolz/"; 10 Chapter 1. Python on Nix Python on Nix tutorial Documentation, Release 0.0.1 description = "List processing tools and functional utilities"; license = licenses.bsd3; maintainers = with maintainers; [ fridh ]; }; } and then execute nix-shell will result in an environment in which you can use Python 3.5 and the toolz package. As you can see we had to explicitly mention for which Python version we want to build a package. Often though, you will want to use a package in environments together with other packages. If we create a shell. nix file with the following contents with import <nixpkgs> {}; ( let toolz = pkgs.python35Packages.buildPythonPackage rec { name = "toolz-${version}"; version = "0.7.4"; src = pkgs.fetchurl{ url = "https://pypi.python.org/packages/source/t/toolz/toolz-${version}.tar.gz ˓→ "; sha256 = "43c2c9e5e7a16b6c88ba3088a9bfc82f7db8e13378be7c78d6c14a5f8ed05afd"; }; meta = { homepage = "http://github.com/pytoolz/toolz/"; description = "List processing tools and functional utilities"; license = licenses.bsd3; maintainers = with maintainers; [ fridh ]; }; }; in pkgs.python35.buildEnv.override rec { extraLibs = [ pkgs.python35Packages.numpy toolz ]; } ).env and again execute nix-shell, then we get a Python 3.5 environment with our locally defined package as well as numpy which is build according to the definition in Nixpkgs. What did we do here? Well, we took the Nix expression that we used earlier to build a Python environment, and said that we wanted to include our own version of toolz. To introduce our own package in the scope of buildEnv.override we used a let expression. Handling dependencies So far our example, toolz, didn’t have any dependencies on other Python packages or system libraries. According to the manual, the buildPythonPackage uses the arguments buildInputs and propagatedBuildInputs. If something is exclusively a build-time dependency, then the dependency should be included as a buildInput, but if it is (also) a runtime dependency, then it should be added to propagatedBuildInputs. The following example shows which arguments are given to buildPythonPackage in order to be build datashape. datashape = buildPythonPackage rec { name = "datashape-${version}"; version = "0.4.7"; 1.4. Developing a Python package 11 Python on Nix tutorial Documentation, Release 0.0.1 src = pkgs.fetchurl { url = "https://pypi.python.org/packages/source/D/DataShape/${name}.tar.gz"; sha256 = "14b2ef766d4c9652ab813182e866f493475e65e558bed0822e38bf07bba1a278"; }; buildInputs = with self; [ pytest ]; propagatedBuildInputs = with self; [ numpy multipledispatch dateutil ]; meta = { homepage = https://github.com/ContinuumIO/datashape; description = "A data description language"; license = licenses.bsd2; maintainers = with maintainers; [ fridh ]; }; }; We can see several runtime dependencies, numpy, multipledispatch, and dateutil. Furthermore, we have one buildInput, i.e. pytest. pytest is a test runner and is only used during the checkPhase and is therefore not added to propagatedBuildInputs. In the previous case we had only dependencies on other packages to consider. Occasionally you have also system libraries to consider. E.g., lxml provides Python bindings to libxml2 and libxslt. These libraries are only required when building the bindings and are therefore added as buildInputs. lxml = buildPythonPackage rec { name = "lxml-3.4.4"; src = pkgs.fetchurl { url = "http://pypi.python.org/packages/source/l/lxml/${name}.tar.gz"; sha256 = "16a0fa97hym9ysdk3rmqz32xdjqmy4w34ld3rm3jf5viqjx65lxk"; }; buildInputs = with self; [ pkgs.libxml2 pkgs.libxslt ]; meta = { description = "Pythonic binding for the libxml2 and libxslt libraries"; homepage = http://lxml.de; license = licenses.bsd3; maintainers = with maintainers; [ sjourdois ]; }; }; In this example lxml and Nix are able to work out exactly where the relevant files of the dependencies are. This is not always the case. The example below shows bindings to The Fastest Fourier Transform in the West, commonly known as FFTW. On Nix we have separate packages of FFTW for the different types of floats ("single", "double", "long-double"). The bindings need all three types, and therefore we add all three as buildInputs. The bindings don’t expect to find each of them in a different folder, and therefore we have to set LDFLAGS and CFLAGS. pyfftw = buildPythonPackage rec { name = "pyfftw-${version}"; version = "0.9.2"; src = pkgs.fetchurl { url = "https://pypi.python.org/packages/source/p/pyFFTW/pyFFTW-${version}.tar.gz"; sha256 = "f6bbb6afa93085409ab24885a1a3cdb8909f095a142f4d49e346f2bd1b789074"; 12 Chapter 1. Python on Nix Python on Nix tutorial Documentation, Release 0.0.1 }; buildInputs = [ pkgs.fftw pkgs.fftwFloat pkgs.fftwLongDouble]; propagatedBuildInputs = with self; [ numpy scipy ]; # Tests cannot import pyfftw. pyfftw works fine though. doCheck = false; preConfigure = '' export LDFLAGS="-L${pkgs.fftw}/lib -L${pkgs.fftwFloat}/lib -L${pkgs. ˓→fftwLongDouble}/lib" export CFLAGS="-I${pkgs.fftw}/include -I${pkgs.fftwFloat}/include -I${pkgs. ˓→fftwLongDouble}/include" ''; meta = { description = "A pythonic wrapper around FFTW, the FFT library, presenting a ˓→unified interface for all the supported transforms"; homepage = http://hgomersall.github.com/pyFFTW/; license = with licenses; [ bsd2 bsd3 ]; maintainer = with maintainers; [ fridh ]; }; }; Note also the line doCheck = false;, we explicitly disabled running the test-suite. Developing with Nix shell When building a package, Nix sequentially goes through a series of phases. Separate phases exist for unpacking the source code, patching it, building it and installing it. buildPythonPackage modifies these generic phases slightly in order to use the Python infrastructure. If you tested the above examples, then hopefully everything ran without problems. Unfortunately, that is not always the case. Sometimes you encounter problems when building a package. When this happens Nix aborts the build, and removes all build data. This can be annoying, especially when after a long time of compiling just a single test fails and Nix decides to abort (example: scikitlearn). Luckily there is the -K option to meaning sometimes there’s maybe a single test failing, and youE.g., if you’re building a Python package which took a long time to compile (e.g. scikitlearn) In such cases you might want to quickly change Develop local package As a Python developer you’re likely aware of development mode (python setup.py develop); instead of installing the package this command creates a special link to the project code. That way, you can run updated code without having to reinstall after each and every change you make. Development mode is also available on Nix as explained in the Nixpkgs manual. Let’s see how you can use it. In the previous Nix expression the source was fetched from an url. We can also refer to a local source instead using src = ./path/to/source/tree; Now, if we create a shell.nix file which calls buildPythonPackage, and if src is a local source, and if the local source has a setup.py, then development mode is activated. 1.4. Developing a Python package 13 Python on Nix tutorial Documentation, Release 0.0.1 In the following example we create a simple environment that has a Python 3.5 version of our package in it, as well as its dependencies and other packages we like to have in the environment, all specified with propagatedBuildInputs. Indeed, we can just add any package we like to have in our environment to propagatedBuildInputs. with import <nixpkgs>; with pkgs.python35Packages; buildPythonPackage rec { name = "mypackage"; src = ./path/to/package/source; propagatedBuildInputs = [ pytest numpy pkgs.libsndfile ]; }; It is important to note that due to how development mode is implemented on Nix it is not possible to have multiple packages simultaneously in development mode. Organising your packages So far we discussed how you can use Python on Nix, and how you can develop with it. We’ve looked at how you write expressions to package Python packages, and we looked at how you can create environments in which specified packages are available. At some point you’ll likely have multiple packages which you would like to be able to use in different projects. In order to minimise unnecessary duplication we now look at how you can maintain yourself a repository with your own packages. The important functions here are import and callPackage. In previous examples we used import generally in combination with the with statement, thereby introducing the attributes of the imported attribute set into the local scope. We can also simply assign the imported attribute set using a let expression. Including a derivation using callPackage Earlier we created a Python environment using buildEnv, and included the toolz package via a let expression. Let’s split the package definition from the environment definition. We first create a function that builds toolz in ~/path/to/toolz/release.nix { pkgs, buildPythonPackage }: buildPythonPackage rec { name = "toolz-${version}"; version = "0.7.4"; src = pkgs.fetchurl{ url = "https://pypi.python.org/packages/source/t/toolz/toolz-${version}.tar.gz"; sha256 = "43c2c9e5e7a16b6c88ba3088a9bfc82f7db8e13378be7c78d6c14a5f8ed05afd"; }; meta = { homepage = "http://github.com/pytoolz/toolz/"; description = "List processing tools and functional utilities"; license = licenses.bsd3; maintainers = with maintainers; [ fridh ]; 14 Chapter 1. Python on Nix Python on Nix tutorial Documentation, Release 0.0.1 }; }; It takes two arguments, pkgs and buildPythonPackage. We now call this function using callPackage in the definition of our environment with import <nixpkgs> {}; ( let toolz = pkgs.callPackage ~/path/to/toolz/release.nix { pkgs=pkgs; ˓→buildPythonPackage=pkgs.python35Packages.buildPythonPackage; }; in pkgs.python35.buildEnv.override rec { extraLibs = [ pkgs.python35Packages.numpy toolz ]; } ).env Important to remember is that the Python version for which the package is made depends on the python derivation that is passed to buildPythonPackage. Nix tries to automatically pass arguments when possible, which is why generally you don’t explicitly define which python derivation should be used. In the above example we use buildPythonPackage that is part of the set python35Packages, and in this case the python35 interpreter is automatically used. Creating a set of packages With the previously explained method you can split packages from environments. and you can call as many packages as you want from whichever environment. Likely you will have at some point developed several packages with dependencies between each other. In such case it might make sense to make a set of packages which you can then import in your environments, just like we do with the nixpkgs repository. Python on Nix internals Important files and folders • buildPythonPackage is a function to build Python packages. • python-packages.nix is a Nix expression listing most of Python packages that are available in Nix. • Folder with expressions for Python interpreters. • ‘wrapper.nix’ wraps Python binaries. FAQ How can I prevent my packages from being garbage-collected? So you’re using nix-shell and whenever you run nix-collect-garbage -d your packages are gone? With nix-shell... On NixOS the system.extraDependencies option for configuration.nix exists. Packages added here are added to the Nix store, but not made available to users. 1.6. Python on Nix internals 15 Python on Nix tutorial Documentation, Release 0.0.1 ... system.extraDependencies = with pkgs.python35Packages; [ numpy toolz ]; ... Why should I use buildEnv for creating environments instead of buildPythonPackage or mkDerivation? 16 Chapter 1. Python on Nix CHAPTER 2 Indices and tables • genindex • modindex • search 17