[LAD] installing python extensions

[Christopher Arndt]

Am 15.08.22 um 14:30 schrieb Fons Adriaensen:
> I have some mixed python/C++ packages, e.g. zita-audiotools
> and zita-jacktools.
> 
> To install these I expect the following to happen:
> 
> 1. The C++ parts are compiled and combined into a *.so
>     file which is a python extension.
> 2. The *.so and the python parts, any data etc. get
>     installed into the user's /usr/lib/python*.*/site-packages.

The way to install a python package nowadays should be via a wheel 
package (unless installed via a (Linux) distro package).

These wheel files are basically Zip archives with some additional 
meta-data and can contain compiled Python extensions (which makes them 
Python version, OS and architecture-dependent, of course).

The advantages of using wheels are:

- no code from the package is executed at installation time (e.-g. 
setup.py etc.)
- users can choose to install system-wide, under ~/.local or into a 
virtual environment.
- installation is generally faster and wheels can be cached for speedup 
of repeated installations (e.g. in a development or CI environment).


> Originally this used distutils, when that got 'deprecated'
> this changed to setuptools. [...]
> Then I got warnings telling me that calling setup.py directly
> is now  also deprecated, and that I should use 'official tools'
> to build and install. What exactly that means I was unable to
> find out, but the following seems to work:

This guide explains the up-to-date process to define a package, at least 
for pure Python packages:

https://packaging.python.org/en/latest/tutorials/packaging-projects/

If you have Python extension, you need to look into the documentation of 
your chosen build backend (e.g. poetry, flit, etc.).

In short, you write a pyproject.toml file, in which you define which 
build backend to use and then, depending on that backend, you either 
define the package meta data and build instructions also in that file or 
(if you still want to use setuptools as a build backend) in a pair of 
setup.py / setup.cfg files. The goal is to have as little actual python 
code in setup.py as possible and define all static meta data elsewhere.

>> WARNING: Running pip as the 'root' user can result in broken
>> permissions and conflicting behaviour with the system package
>> manager. It is recommended to use a virtual environment instead.

What this basically says is that you shouldn't normally mess with the 
system-wide Python packages in /usr/lib/pythonX.Y/site-packages, because 
this is used by the Python packages installed via the distribution 
packaging system. Programs installed also via distro packages may rely 
on packages installed there and specific versions of them.

If you install a Python package via pip as root into this location, pip 
will resolve dependencies, which may result in some existing packages 
being upgraded to newer versions. As the warning says , this may cause 
conflicts (or subtly break things).

> Now clearly installing things in site-packages requires root,
> so what is then the recommended method ?? And why the virtual
> environment (which is used by build anyway) ??

IMO you should distinguish between these scenarios:

a) A program depends on, say, zita-xyz and is installed via a distro 
package. It should depend on the respective distro-package for zita-xyz. 
This installs into /usr/lib/pythonX.Y/site-packages. You can let the 
distro packages worry about the pip warning, you just provide the 
correct pyproject.toml and other build files.

b) A program depends on zita-xyz and is installed via pip. Then it 
should define this dependency in pyproject.toml and pip will take care 
of installing zita-xyz. The user running pip can decide whether to 
install the program system-wide (not recommended), under ~/.local or 
into a virtual environment (e.g. using pipx [1]). zita-xyz will be 
installed in the same location. If both are installed in a dedicated 
virtual environment, it ensures that there are no conflicts with other 
programs, possibly depending on a different version of zita-xyz.

c) A developer (or "power-user") uses zita-xyz for the implementation of 
a program or some utility scripts. He/she uses pip to install zita-xyz 
(and its dependencies, if any).

If he/she want zita-xyz to be importable from any Python interpreter 
with a matching version started *by that user*, he installs it with the 
--user option into ~/.local.

If he/she wants to make sure that a certain program always has access to 
and uses a certain zita-xyz version it depends on, he/she installs it 
into dedicated virtual environment for that (collection) of program(s) 
and always starts the program (or the Python interpreter) from the bin 
directory of that virtual env.

There are several ways to facilitate / automate the latter, e.g. via 
poetry [2], which I have mostly good experiences with.

And oh, btw, any virtual environment created by the build process is 
ephemeral and not relevant to the actual installation.

One last thing: extension packages come with the additional difficulty 
that they either need to be compiled on installation by the user (then 
requiring the presence of needed headers etc.) or somebody (e.g. the 
project maintainer) needs to provide a host of Python 
version/OS/architecture-dependent variants of wheel packages. For 
Windows and macOS this is relatively easy via CI environments like 
Gitlab CI oder Github actions. But for Linux, for these binary wheels to 
be compatible with a wide variety of Linux distributions, special care 
needs to be taken. For more information, for example, look at [3].


Hth, Chris

[1] https://pypi.org/project/pipx/
[2] https://pypi.org/project/poetry/
[3] https://github.com/pypa/cibuildwheel