Setuptools And Python Wheels
Feb 26th, 2021 - written by Kimserey with .
During the development of a Python application, we interact with tools like pip and libraries like setuptools. Pip allows us to install packages while setuptools is a library built on top of distutils providing necessary tools to package and distribute our own application. Python supports two types of distributions, wheels and source distributions. In today’s post we will look at the usage of both of them and also look at how we can create them oursevles for our own application.
Packaging Python Application
To package a Python application, we can use setuptools. setuptools comes pre-installed in virtual environment, so we start first by creating a virtual environment:
1
2
❯ python3 -m venv venv
❯ source venv/bin/activate
We can then create a setup.py file at the root of our project:
1
2
3
4
5
6
7
8
9
10
from setuptools import setup, find_packages
setup(
name="mypackage",
version="0.0.1",
author="Kimserey Lam",
author_email="[email protected]",
description="Example package",
packages=find_packages(),
)
The setup.py is the build script used by setuptools to build different distributions. Here we use find_packages, a utility provided by setuptools to automatically find packages to ship within our project. Once we have the setup.py file, we can then make sure we have setuptools and wheel installed.
1
❯ pip install --upgrade setuptools wheel
This will install wheel and upgrade setuptools if necesary. wheel package provides the tool necessary to build a wheel. Once installed, we can then package our application in the two standard format; source distribution and wheel:
1
❯ python3 setup.py sdist
This will create a source distribution under /dist. The source distrubtion is a tarball; mypackage-0.0.1.tar.gz. We can list its content:
1
2
3
4
5
6
7
8
9
10
11
12
❯ tar -tf ./dist/mypackage-0.0.1.tar.gz
mypackage-0.0.1/
mypackage-0.0.1/PKG-INFO
mypackage-0.0.1/my_package/
mypackage-0.0.1/my_package/__init__.py
mypackage-0.0.1/my_package/my_package.py
mypackage-0.0.1/mypackage.egg-info/
mypackage-0.0.1/mypackage.egg-info/PKG-INFO
mypackage-0.0.1/mypackage.egg-info/SOURCES.txt
mypackage-0.0.1/mypackage.egg-info/dependency_links.txt
mypackage-0.0.1/mypackage.egg-info/top_level.txt
mypackage-0.0.1/setup.py
We can see that the source distribution is simply a zip containing all the source code and extra .egg-info information (find more about egg_info in the official documentation).
1
python3 setup.py bdist_wheel
This will create a wheel distribution under /dist. The wheel is a whl file which is just a zip; mypackage-0.0.1-py3-none-any.whl. Similarly we can inspect its content:
1
2
3
4
5
6
7
❯ tar -tf ./dist/mypackage-0.0.1-py3-none-any.whl
my_package/__init__.py
my_package/my_package.py
mypackage-0.0.1.dist-info/METADATA
mypackage-0.0.1.dist-info/WHEEL
mypackage-0.0.1.dist-info/top_level.txt
mypackage-0.0.1.dist-info/RECORD
We can see that the wheel distribution contains the Python source code as well as simple Python code can be shipped as code, the only consideration would be the Python version, here py3 as we have build the distribution with Python3. py3 is actually specified in the naming of the wheel file, the format is specified in PEP-491:
1
{distribution}-{version}(-{build tag})?-{python tag}-{abi tag}-{platform tag}.whl
distributionwould bemypackage,versionwould be0.0.1,python tagwould bepy3,abi tagwould benone,platform tagwould beany.
abi tag and platform tag are relevant when the package contains C extension. For example we were to build the wheel for a package containing C extension, we would get a wheel containing compiled extensions only compatible with the machine we built the extension:
1
mypackage-0.0.1-cp38-cp38-macosx_10_9_x86_64.whl
After building a wheel locally, from my computer I would get cp38 as abi tag which is CPython 3.8 and macosx_10_9_x86_64 as platform tag as I am building from MacOS. And if I list the content of the wheel, I will see that it contains the compiled C extension under .so file, compiled specifically for macosx_10_9_x86_64.
This is the main difference between a source distribution and a wheel, the source distribution will zip the source code, and compilation will happen on user machine when the package is installed, while a wheel will have pre-compiled binaries usable directly after installation.
Pypi And Pip
To install packages, we use pip the package manager provided by Python. Just like setuptools, pip comes pre install in the virtual environment. When doing:
1
❯ pip install somepackage
We are pulling packages from Pypi, the Python Package Index.
As we just saw, there are two type of distributions for Python packages, source distribution and wheel, while source distribution distributes the source code to be compiled on installation, wheels distribute pre-compiled binaries. This distinction makes installation of wheel much faster, and hence is the prefered option from pip. When uploading packages to Pypi, it is recommended to upload both source distributions and wheels as if no wheel matches the user machine, pip will revert to using the source distribution and build the wheel locally.
We can see for example uWSGI only comes as a source distribution:
1
uWSGI-2.0.19.1.tar.gz (803.9 kB)
When we pip install it, the source will be extracted and compiled locally before being installed in the virtual environemnt.
As opposed to a regular application like marshmallow:
1
marshmallow-3.10.0-py2.py3-none-any.whl (46.4 kB)
Which would be directly unzipped and installed.
Conclusion
Today we looked into how packaging a Python project is done. We looked at the different mode of distribution, source and wheel, and we looked at what they meant. We also took a quick look at Pipy the Python Package Index and how pip used the different distributions to install packages. Hope you liked this post and I see you on the next one!