15 Aug
2022
15 Aug
'22
4:30 p.m.
Hello all,
I have some mixed python/C++ packages, e.g. zita-audiotools
and zita-jacktools.
To install these I expect the following to happen:
1. The C++ parts are compiled and combined into a *.so
file which is a python extension.
2. The *.so and the python parts, any data etc. get
installed into the user's /usr/lib/python*.*/site-packages.
To make this as easy as possible I provide a setup.py and a
Makefile, so that all that should be required is:
make; sudo make install
Originally this used distutils, when that got 'deprecated'
this changed to setuptools. So until recently the Makefile
was something like:
----
PY = /usr/bin/python3
build:
$(PY) ./setup.py build
install:
$(PY) ./setup.py install
clean:
$(PY) ./setup.py clean
rm -rf build dist zita_jacktools.egg-info
---
Then I got warnings telling me that calling setup.py directly
is now also deprecated, and that I should use 'official tools'
to build and install. What exactly that means I was unable to
find out, but the following seems to work:
----
PY = /usr/bin/python3
build:
$(PY) -m build -w
install:
pip install --force-reinstall dist/*.whl
clean:
rm -rf build dist *.egg-info *~
----
But this still produces a warning:
WARNING: Running pip as the 'root' user can
result in broken
permissions and conflicting behaviour with the system package
manager. It is recommended to use a virtual environment instead.
Now clearly installing things in site-packages requires root,
so what is then the recommended method ?? And why the virtual
environment (which is used by build anyway) ??
If anyone can shed some light on this mess he/she will deserve
my eternal gratitude.
Ciao,
--
FA
15 Aug
15 Aug
6:21 p.m.
Am 15.08.22 um 14:30 schrieb Fons Adriaensen:
I have some mixed python/C++ packages, e.g.
zita-audiotools
and zita-jacktools.
To install these I expect the following to happen:
1. The C++ parts are compiled and combined into a *.so
file which is a python extension.
2. The *.so and the python parts, any data etc. get
installed into the user's /usr/lib/python*.*/site-packages.
The way to install a python package nowadays should be via a wheel
package (unless installed via a (Linux) distro package).
These wheel files are basically Zip archives with some additional
meta-data and can contain compiled Python extensions (which makes them
Python version, OS and architecture-dependent, of course).
The advantages of using wheels are:
- no code from the package is executed at installation time (e.-g.
setup.py etc.)
- users can choose to install system-wide, under ~/.local or into a
virtual environment.
- installation is generally faster and wheels can be cached for speedup
of repeated installations (e.g. in a development or CI environment).
Originally this used distutils, when that got
'deprecated'
this changed to setuptools. [...]
Then I got warnings telling me that calling setup.py directly
is now also deprecated, and that I should use 'official tools'
to build and install. What exactly that means I was unable to
find out, but the following seems to work:
This guide explains the up-to-date process to define a package, at least
for pure Python packages:
https://packaging.python.org/en/latest/tutorials/packaging-projects/
If you have Python extension, you need to look into the documentation of
your chosen build backend (e.g. poetry, flit, etc.).
In short, you write a pyproject.toml file, in which you define which
build backend to use and then, depending on that backend, you either
define the package meta data and build instructions also in that file or
(if you still want to use setuptools as a build backend) in a pair of
setup.py / setup.cfg files. The goal is to have as little actual python
code in setup.py as possible and define all static meta data elsewhere.
> WARNING: Running pip as the 'root' user
can result in broken
> permissions and conflicting behaviour with the system package
> manager. It is recommended to use a virtual environment instead.
What this basically says is that you shouldn't normally mess with the
system-wide Python packages in /usr/lib/pythonX.Y/site-packages, because
this is used by the Python packages installed via the distribution
packaging system. Programs installed also via distro packages may rely
on packages installed there and specific versions of them.
If you install a Python package via pip as root into this location, pip
will resolve dependencies, which may result in some existing packages
being upgraded to newer versions. As the warning says , this may cause
conflicts (or subtly break things).
Now clearly installing things in site-packages
requires root,
so what is then the recommended method ?? And why the virtual
environment (which is used by build anyway) ??
IMO you should distinguish between these scenarios:
a) A program depends on, say, zita-xyz and is installed via a distro
package. It should depend on the respective distro-package for zita-xyz.
This installs into /usr/lib/pythonX.Y/site-packages. You can let the
distro packages worry about the pip warning, you just provide the
correct pyproject.toml and other build files.
b) A program depends on zita-xyz and is installed via pip. Then it
should define this dependency in pyproject.toml and pip will take care
of installing zita-xyz. The user running pip can decide whether to
install the program system-wide (not recommended), under ~/.local or
into a virtual environment (e.g. using pipx [1]). zita-xyz will be
installed in the same location. If both are installed in a dedicated
virtual environment, it ensures that there are no conflicts with other
programs, possibly depending on a different version of zita-xyz.
c) A developer (or "power-user") uses zita-xyz for the implementation of
a program or some utility scripts. He/she uses pip to install zita-xyz
(and its dependencies, if any).
If he/she want zita-xyz to be importable from any Python interpreter
with a matching version started *by that user*, he installs it with the
--user option into ~/.local.
If he/she wants to make sure that a certain program always has access to
and uses a certain zita-xyz version it depends on, he/she installs it
into dedicated virtual environment for that (collection) of program(s)
and always starts the program (or the Python interpreter) from the bin
directory of that virtual env.
There are several ways to facilitate / automate the latter, e.g. via
poetry [2], which I have mostly good experiences with.
And oh, btw, any virtual environment created by the build process is
ephemeral and not relevant to the actual installation.
One last thing: extension packages come with the additional difficulty
that they either need to be compiled on installation by the user (then
requiring the presence of needed headers etc.) or somebody (e.g. the
project maintainer) needs to provide a host of Python
version/OS/architecture-dependent variants of wheel packages. For
Windows and macOS this is relatively easy via CI environments like
Gitlab CI oder Github actions. But for Linux, for these binary wheels to
be compatible with a wide variety of Linux distributions, special care
needs to be taken. For more information, for example, look at [3].
Hth, Chris
[1] https://pypi.org/project/pipx/
[2] https://pypi.org/project/poetry/
[3] https://github.com/pypa/cibuildwheel
6:54 p.m.
Hi Fons,
Christopher Arndt sent a long and detailed answer, here's a shorter one.
Python packages can be installed in user's home or virtual environments.
For example, to install the zita tools in my home environment, I cd to
the zita-audiotools-1.3.0 directory, then used:
pip install .
Installation was successful, so I used the same method for zita-jacktools.
It includes the .so modules:
ls $HOME/.local/lib/python3.*/site-packages/zita_*tools/
For completion I uninstalled the packages in my home:
pip uninstall zita-audiotools zita-jacktools
Then I deleted the makefiles and reinstalled the packages (using same
method).
I also reinstalled them globally using:
sudo pip install .
It worked too:
ls /usr/local/lib/python3.*/dist-packages/zita_*tools/*
There's newer configuration methods, but for now there's no need to
update your packages, apart from removing the makefiles (and update the
documentation).
Marc
Le 2022-08-15 à 08 h 30, Fons Adriaensen a écrit :
Hello all,
I have some mixed python/C++ packages, e.g. zita-audiotools
and zita-jacktools.
To install these I expect the following to happen:
1. The C++ parts are compiled and combined into a *.so
file which is a python extension.
2. The *.so and the python parts, any data etc. get
installed into the user's /usr/lib/python*.*/site-packages.
To make this as easy as possible I provide a setup.py and a
Makefile, so that all that should be required is:
make; sudo make install
Originally this used distutils, when that got 'deprecated'
this changed to setuptools. So until recently the Makefile
was something like:
----
PY = /usr/bin/python3
build:
$(PY) ./setup.py build
install:
$(PY) ./setup.py install
clean:
$(PY) ./setup.py clean
rm -rf build dist zita_jacktools.egg-info
---
Then I got warnings telling me that calling setup.py directly
is now also deprecated, and that I should use 'official tools'
to build and install. What exactly that means I was unable to
find out, but the following seems to work:
----
PY = /usr/bin/python3
build:
$(PY) -m build -w
install:
pip install --force-reinstall dist/*.whl
clean:
rm -rf build dist *.egg-info *~
----
But this still produces a warning:
WARNING: Running pip as the 'root' user
can result in broken
permissions and conflicting behaviour with the system package
manager. It is recommended to use a virtual environment instead.
Now clearly
installing things in site-packages requires root,
so what is then the recommended method ?? And why the virtual
environment (which is used by build anyway) ??
If anyone can shed some light on this mess he/she will deserve
my eternal gratitude.
Ciao,
7:18 p.m.
Le 2022-08-15 à 10 h 54, I wrote :
Python packages can be installed in user's home or
virtual environments.
For example, to install the zita tools in my home environment, I cd to
the zita-audiotools-1.3.0 directory, then used:
pip install .
It's also possible to install the packages using their download urls:
pip install
https://kokkinizita.linuxaudio.org/linuxaudio/downloads/zita-audiotools-1.3…
pip install
https://kokkinizita.linuxaudio.org/linuxaudio/downloads/zita-jacktools-1.6.…
ls $HOME/.local/lib/python3.*/site-packages/zita_*tools/
Marc
11:21 p.m.
On Mon, Aug 15, 2022 at 10:54:06AM -0400, Marc Lavallée wrote:
Christopher Arndt sent a long and detailed answer,
here's a shorter one.
One (metric) unit of Eternal Gratitude to both of you.
So
pip install .
will install to ~/.local/lib
while
sudo pip install .
will install to
/usr/local/lib
I find the destination directory depending on who the user pretends
to be a bit strange, but it works !
Also tested this with a package that includes data files (*.npy)
that should be found by the installed code, and also this works.
The only minor problem is that the sudo version leaves two
directories (build and *.egg-info) that can only be cleaned up
by root. No problem on systems that allow sudo everything, but
I may keep the Makefile just to offer 'sudo make clean',
assuming most system will allow this. Unless there is a
cleaner solution.
Ciao
--
FA
16 Aug
16 Aug
12:16 a.m.
Am 15.08.22 um 21:21 schrieb Fons Adriaensen:
I find the destination directory depending on who the
user pretends
to be a bit strange, but it works !
"pip install" as a normal user used to fail with an (apparently for most
users) incomprehensible error message about not being able to write to
the system-wide site-packages (doh!) and you needed the --user option to
install into ~/.local/lib. That changed a while ago, so it automatically
assumes --user, if not run as root and not told otherwise. I'm on the
fence about whether I agree with this change or not.
Unless there is a cleaner solution.
$ python setup.py build
$ sudo python setup.py install --skip-build
(not sure about the appropriate options for other build backends)
"install" implies build (and build_ext, if necessary) unless you tell it
otherwise, and that step creates the .egg-info dir.
Chris
1:14 a.m.
Le 2022-08-15 à 15 h 21, Fons Adriaensen a écrit :
One (metric) unit of Eternal Gratitude to both of you.
One unit of "Fons Gratitude" is already a lot, but thanks (infinity / 2)
times!
So
And when a virtualenv is activated, packages are installed in the ./lib
sub-directory of the virtualenv directory.
pip install .
will install to ~/.local/lib
while
When installing with the root user, the default
prefix is /usr/local
because /usr is usually managed by the system.
But it's possible to install in /root/.local/lib, using:
sudo pip install --user .
Or as the root user, to install to your home directory, try:
sudo su - fons pip install .
The --user option is (now) the default, except when installing as root.
sudo pip install .
will install to
/usr/local/lib
I find the destination directory depending on who the user pretends
to be a bit strange, but it works ! The only minor problem is that the sudo version leaves
two
directories (build and *.egg-info) that can only be cleaned up
by root. No problem on systems that allow sudo everything, but
I may keep the Makefile just to offer 'sudo make clean',
assuming most system will allow this. Unless there is a
cleaner solution.
That could be something simple like:
PY = env python3
PIP = env pip3
build:
$(PIP) wheel .
install:
$(PIP) install --force-reinstall *.whl
uninstall:
$(PIP) uninstall zita-audiotools
clean:
rm -rf build *.egg-info *.whl
Marc
1:33 a.m.
Am 15.08.22 um 23:14 schrieb Marc Lavallée:
Le 2022-08-15 à 15 h 21, Fons Adriaensen a écrit :
When installing with the root user, the default prefix is /usr/local
because /usr is usually managed by the system.
Actually, it will install to sys.prefix, which is usually /usr, if the
Python version used to run pip is installed via a distro package.
BTW, the recommended way to run pip is actually
python -m pip <options>
because then you can control (via PATH or the command name you use)
which Python interpreter is used.
Chris
sudo pip
install .
will install to /usr/local/lib
1:48 a.m.
Le 2022-08-15 à 17 h 33, Christopher Arndt a écrit :
BTW, the recommended way to run pip is actually
python -m pip <options>
because then you can control (via PATH or the command name you use)
which Python interpreter is used.
So the Makefile could be:
PIP = python3 -m pip
PKG = zita_audiotools
build:
$(PIP) wheel .
install:
$(PIP) install --force-reinstall $(PKG)*.whl
uninstall:
$(PIP) uninstall $(PKG)
clean:
rm -rf build $(PKG)*.egg-info $(PKG)*.whl
Marc
22 Sep
22 Sep
11:56 a.m.
On Mon, Aug 15, 2022 at 05:48:34PM -0400, Marc Lavallée wrote:
So the Makefile could be:
PIP = python3 -m pip
PKG = zita_audiotools
build:
$(PIP) wheel .
install:
$(PIP) install --force-reinstall $(PKG)*.whl
uninstall:
$(PIP) uninstall $(PKG)
clean:
rm -rf build $(PKG)*.egg-info $(PKG)*.whl
This worked perfectly a month ago on my main system. Now I'm in holidays
with only my laptop, and it fails:
python3 -m pip wheel .
Processing /home/fons/python/jackimpfilt
Preparing metadata (setup.py) ... [?25ldone
[?25hBuilding wheels for collected packages: jackimpfilt
Building wheel for jackimpfilt (setup.py) ... [?25lerror
error: subprocess-exited-with-error
× python setup.py bdist_wheel did not run successfully.
│ exit code: 1
╰─> [6 lines of output]
usage: setup.py [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
or: setup.py --help [cmd1 cmd2 ...]
or: setup.py --help-commands
or: setup.py cmd --help
error: invalid command 'bdist_wheel'
[end of output]
python version is 3.10.5
Another difference IIRC is that on my main system pip would set up
a virtual environment, while this doesn't seem to happen here.
Have things changed again ????
--
FA
5:50 p.m.
Hi Fons,
Le 2022-09-22 à 03 h 56, Fons Adriaensen a écrit :
error: invalid command 'bdist_wheel'
It looks like wheel is not installed (locally or globally). Try
installing it with "pip install wheel", or install it on the system
(python3-wheel on Debian); it could be enough to fix the issue.
python version is 3.10.5
My system still use
python version 3.8, so for a test I installed python
version 3.10.5 using pyenv (https://github.com/pyenv/pyenv) :
pyenv install 3.10.5
To create and activate a virtualenv for my test I used:
https://github.com/pyenv/pyenv-virtualenv
pyenv virtualenv 3.10.5 zita
pyenv activate zita
But that was not enough; compiling your extension failed because wheel
is missing in the virtualenv, so I installed it using "python3 -m pip
install wheel", then compiling succeeded. There was one warning about
the version of pip; to avoid it (before compiling): "python3 -m pip
install --upgrade pip"
After my test session, I deactivated the virtualenv using "source
deactivate", then I deleted the virtualenv using "pyenv
virtualenv-delete zita"
Another difference IIRC is that on my main system pip
would set up
a virtual environment, while this doesn't seem to happen here.
To my knowledge,
pip does not install a virtual environment, but a
virtual environment includes pip. Because wheel is not installed by
default, adding a build step (in the makefile) could help: "$(PIP)
install wheel".
Have things changed again ????
Python is
changing faster now, so testing on different versions is a
good idea.
Marc
6:51 p.m.
On Thu, Sep 22, 2022 at 09:49:41AM -0400, Marc Lavallée wrote:
With Archlinux you always (only) get the latest and greatest :-)
Ciao,
--
FA
Le 2022-09-22 à 03 h 56, Fons Adriaensen a écrit :
That was it, many thanks !
To it looked as if pip didn't know the bdist_wheel command, and indeed
pip help-commands didn't include it. No indication at all that something
else was missing...
The relations and dependencies between the various python tools - pip,
setup, wheel, ... remain a mistery to me, and there seems to be little
up to date documentation. Examples all assume you want to make a package
for the PPI and no other use cases...
error: invalid command
'bdist_wheel'
It looks like wheel is not installed (locally or globally).
Try installing
it with "pip install wheel", or install it on the system (python3-wheel on
Debian); it could be enough to fix the issue. Have things
changed again ????
Python is changing faster now, so testing on different versions
is a good
idea.
9:01 p.m.
Le 2022-09-22 à 10 h 51, Fons Adriaensen a écrit :
That was it, many thanks !
Yay!
To it looked as if pip didn't know the bdist_wheel
command, and indeed
pip help-commands didn't include it. No indication at all that something
else was missing...
The relations and dependencies between the various python tools - pip,
setup, wheel, ... remain a mistery to me, and there seems to be little
up to date documentation. Examples all assume you want to make a package
for the PPI and no other use cases...
The wheel package must be installed... it
came after pip and setuptools,
that both added support for wheel (not requiring it):
https://peps.python.org/pep-0427/
I notice that audiotools is used by jacktools, so a requirement could be
added to the setup function (in setup.py):
install_requires=[
"zita-audiotools >= 1.3.0",
],
Also, in order to create a valid source distribution, a MANIFEST.in file
is required.
Here's a starting point:
global-exclude *.py[cod]
include Makefile
graft source
First line is to exclude bytecode files.
Second line is to include your (optional) Makefile (because it works).
Third line is to include all files from the source directory (including
.h files).
More info:
https://docs.python.org/3/distutils/sourcedist.html
There's a newer packaging method that replace setup.py with a
pyproject.toml file:
https://pip.pypa.io/en/stable/reference/build-system/pyproject-toml/
More info on wheels and distribution on pypi.org:
https://realpython.com/python-wheels/
The documentation is all over the place, and I'm also learning...
With Archlinux you always (only) get the latest and
greatest :-)
Latest version of Python is 3.10.7, so 3.10.5 is behind by two patch
versions. ;-)
Marc
826
days inactive
864
days old
linux-audio-dev@lists.linuxaudio.org
12 comments
3 participants
participants (3)
-
Christopher Arndt -
Fons Adriaensen -
Marc Lavallée