Packaging¶

Packaging your code is the process of organizing your project files so that it can be built and installed by other users. As well as your source code, there are a few files that are needed so that your project “pip-installable”, i.e. can be installed with pip and uploaded to PyPI.

While the Python Packaging Authority (PyPA) provides a comprehensive guide on packaging Python projects, this project instead uses Poetry. Poetry is one of many tools that can be used to manage Python packages and environments. It simplifies/automates a lot of small aspects than if we were to use approach described by PyPA.

uv is another popular tool for Python packaging, written in Rust.

Poetry setup¶

As a first-time setup, Poetry needs to be installed. The process is described on this page. I recommend using the pipx method:

Install pipx: https://pipx.pypa.io/stable/installation/
Install Poetry: pipx install poetry

Project structure¶

A typical (minimal) file structure for a Python project to be “pip-installable” looks like so:

project/
|-- __init__.py
|-- code.py
|-- # other files
README.rst     # or .md, .txt, etc
pyproject.toml   # or setup.cfg or setup.py

where:

The folder project contains your source code files and an __init__.py, which could be empty.
README.rst which is not necessary for building the Python, but is often the first file that new users will look into to learn about the project, how to install it, and how to use it. On GitHub and PyPI, it will be rendered as the “homepage” for your project.
pyproject.toml is a configuration file for building the project. More on that below. In the past, Python projects used setup.py for this purpose, but pyproject.toml is becoming more and more the norm. As this project used to use setup.py, more on that can be found below.

Project Management and Packaging with Poetry¶

This page provides an overview of how to use Poetry:

For a new project, you can run poetry new poetry-demo. This will create a new folder with the project structure described above, including an empty tests folder.
For an existing project, you can run poetry init. Through the command line, you will be asked a series of questions to fill in the pyproject.toml file.

The project can then be built (locally) with the command below:

(project_env) poetry install

The project can be imported in your Python script as:

import project

Note

One useful feature of Poetry is that with a single command you can install new dependencies and update the pyproject.toml file. For example, to install pandas and add to pyproject.toml, you can run:

(project_env) poetry add pandas

You can also create groups of dependencies, e.g. for development dependencies, by running:

(project_env) poetry add --group dev sphinx

So that during installation, you can specify which group of dependencies:

(project_env) poetry install --with dev

More on managing dependencies can be found here.

pyproject.toml¶

Using a pyproject.toml file is not unique to Poetry, but is becoming more and more the norm for Python projects. Below is the pyproject.toml file for this project.

pyproject.toml¶

[tool.poetry]
name = "pydevtips"
version = "0.0.4"
description = "Functions and scripts to demonstrate Python development tips."
authors = ["Eric Bezzam <ebezzam@gmail.com>"]
license = "MIT"

# -- manually added --
readme = "README.rst"
package-mode = true   # https://python-poetry.org/docs/basic-usage/#operating-modes
# --------------------

[tool.poetry.dependencies]
python = "^3.10"
numpy = "^2.1.2"
scipy = "^1.14.1"
matplotlib = "^3.9.2"
hydra-core = "^1.3.2"
tqdm = "^4.66.5"


[tool.poetry.group.dev.dependencies]
black = "^24.10.0"
isort = "^5.13.2"
flake8 = "^7.1.1"
pytest = "^8.3.3"
pre-commit = "^4.0.1"
twine = "^5.1.1"

[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

#### -- manually added (below)

[project.urls]
Homepage = "https://github.com/ebezzam/python-dev-tips"
Issues = "https://github.com/ebezzam/python-dev-tips/issues"
Documentation = "https://pydevtips.readthedocs.io"

[tool.isort]
profile = "black"

[tool.black]
line-length = 100
include = '\.pyi?$'
exclude = '''
/(
    \.git
  | build
  | dist
)/
'''

Everything except the sections labeled with “manually added” was automatically generated by Poetry.

More info on configuring the pyproject.toml file can be found at the links below:

General configuration (short and in-depth guides).
Poetry configuration: https://python-poetry.org/docs/pyproject/

Publishing to PyPI with Poetry¶

Poetry also makes building and deploying to PyPI easy. This article sums up the process well:

(First-time) setup:
- Create an account on PyPI: https://pypi.org/account/register/
- Create a token: https://pypi.org/manage/account/
- Add the token to Poetry: poetry config pypi-token.pypi <your-token>
Update the version number in the pyproject.toml file. See Semantic Versioning for recommendations on picking version numbers.
Build the package: poetry build
Upload to PyPI: poetry publish. Check https://pypi.org/project/pydevtips/ 🎉

If there are issues in publishing to PyPI, you can check the logs with:

poetry publish --dry-run

Or if there are issues with rendering the README file, you can check the logs with (twine is required):

twine check dist/pydevtips-X.X.X.tar.gz   # replace X.X.X with version number

Creating a new release on GitHub¶

For a new release, you need to create a new tag on GitHub. This can be done with the commands below:

git tag -a X.X.X -m "version X.X.X"
git push origin X.X.X

You can create a new release with the following steps:

From the tags page, click (the rightmost) “…” dropdown menu.
Select “Create release”.
At the bottom, press “Publish release”.

setup.py (old way)¶

The use of setup.py is an older way to Python projects, e.g, with setuptools. Below is a previously-used setup.py file for this project.

setup.py¶

import setuptools

with open("README.rst", "r", encoding="utf-8") as fh:
    long_description = fh.read()

setuptools.setup(
    name="pydevtips",
    version="0.0.2",
    author="Eric Bezzam",
    author_email="ebezzam@gmail.com",
    description="Functions and scripts to demonstrate Python development tips.",
    long_description=long_description,
    long_description_content_type="text/x-rst",
    url="https://github.com/ebezzam/python-dev-tips",
    packages=setuptools.find_packages(),
    classifiers=[
        "Programming Language :: Python :: 3",
        "Operating System :: OS Independent",
    ],
    python_requires=">=3.8",
    install_requires=[
        "numpy",
        "scipy",
        "matplotlib",
        "hydra-core",
        "tqdm",
    ],
    include_package_data=True,
)

Lines 3-4: use the contents of README.rst to be rendered for the homepage on PyPI. Be sure to set the correct file extension and set Line 13 accordingly.
Line 7: specifies the name of the package / in which folder the source code is located, such that the package can be installed with pip install pydevtips if on PyPI and imported as import pydevtips.
Line 8: sets the package version, which should be (typically) modified before uploading a new version to PyPI (below).
Line 9-10: for your name and contact info.
Line 20-26: specifies the Python version and package dependencies.

The project can then be built (locally) with the command below:

(project_env) pip install -e .

The project can be imported in your Python script as:

import project

For a more in-depth description, check out this article.

Uploading your project to PyPI is traditionally done with the twine library.

# inside virtual environment
(project_env) pip install twine

In the steps below, replace “X.X.X” with the appropriate version number, matching the one in your setup.py file. See Semantic Versioning for recommendations on picking version numbers.

# edit version in `setup.py`
# build package
(project_env) python setup.py sdist bdist_wheel
# -- creates zip in dist folder

# upload to pypi
(project_env) python -m twine upload  dist/pydevtips-X.X.X.tar.gz
# -- X.X.X is the version number in setup.py
# -- enter username and password
# -- check https://pypi.org/project/pydevtips/X.X.X/

# new tag on GitHub
git tag -a X.X.X -m "version X.X.X"
git push origin X.X.X

If you project is hosted on GitHub, you can create a new release by:

Clicking (the rightmost) “…” dropdown menu (from the tags page).
Selecting “Create release”.
At the bottom pressing “Publish release”.