Testing

“If debugging is the process of removing bugs, then programming must be the process of putting them in.”

– Edsger Dijkstra

But there are ways to make it easier to find bugs and prevent the introduction of new ones. This is where testing comes in. There are two types that I often use (discussed below):

We also the discuss the use of GitHub Actions. for continuous integration (CI), namely the software practice of committing code to a shared repository where the project is built and tested.

Assertion tests

Assertion tests are lightweight Boolean checks that you can include in your code to check that certain conditions are met. For example, you can check that the input/output is what you expect. If the condition is not met, the test will fail and code execution will stop.

For example, for pydevtips.fftconvolve.RFFTConvolve we check that the input is indeed real:

def __init__(self, filt, length) -> None:
    assert np.isreal(filt).all(), "Filter must be real."
    ...

Unit tests

Unit testing is a method to test small pieces of code, usually functions. With a large code base, having unit tests can ensure you don’t break core functionality when you make changes.

In Python, there is the pytest package that can be used to write and run unit tests. A common practice is to create a tests folder in the root of your project and write your tests there. The test functions should begin with test_ so that pytest can find them.

For example, for our FFT convolvers – pydevtips.fftconvolve.RFFTConvolve and pydevtips.fftconvolve.FFTConvolve – we can write unit tests to check that they are consistent with numpy.convolve(), as done in this script.

To run the unit tests:

# install in virtual environment (if not done already)
# -- pytest in the dev group
(project_env) poetry install --with dev

# run tests
(project_env) poetry run pytest

# -- if not using Poetry
# (project_env) pip install pytest
# (project_env) pytest

To run a specific test:

# inside virtual environment
(project_env) poetry run pytest tests/test_fftconvolve.py::test_fft

# -- if not using Poetry
# (project_env) pytest tests/test_fftconvolve.py::test_fft

Continuous integration with GitHub Actions

Continuous integration (CI) is the practice of automatically building and testing code whenever a change is made to the codebase. This is useful to ensure that the codebase is always in a working state.

With GitHub Actions, you can set up a workflow that will run, e.g. on every push to the repository. This workflow can build the package, run the unit tests, build the documentions, etc for different versions of Python and operating systems.

Workflows are defined in YAML files in the .github/workflows folder. For example, the workflow for this project is defined in this file, whose code is shown below:

poetry.yml
 1# Poetry GitHub Action: https://github.com/marketplace/actions/python-poetry-action
 2name: pydevtips
 3
 4# on: [push, pull_request]
 5on:
 6  # trigger on pushes and PRs to main
 7  push:
 8    branches:
 9      - main
10  pull_request:
11    branches:
12      - main
13
14jobs:
15  build:
16
17    runs-on: ${{ matrix.os }}
18    strategy:
19      fail-fast: false
20      max-parallel: 12
21      matrix:
22        os: [ubuntu-latest, macos-latest, windows-latest]
23        python-version: ["3.10", "3.11"]
24        poetry-version: ["1.8.4"]
25    steps:
26    - uses: actions/checkout@v4
27    - name: Checkout submodules
28      shell: bash
29      run: |
30        auth_header="$(git config --local --get http.https://github.com/.extraheader)"
31        git submodule sync --recursive
32        git -c "http.extraheader=$auth_header" -c protocol.version=2 submodule update --init --force --recursive --depth=1
33    - name: Set up Python ${{ matrix.python-version }}
34      uses: actions/setup-python@v4
35      with:
36        python-version: ${{ matrix.python-version }}
37    - name: Install poetry
38      uses: abatilo/actions-poetry@v3.0.0
39      with:
40        poetry-version: ${{ matrix.poetry-version }}
41    - name: Install dependencies
42      run: |
43        poetry install --with dev
44    - name: Lint with flake8
45      run: |
46        # stop the build if there are Python syntax errors or undefined names
47        poetry run flake8 pydevtips --count --select=B,C,E,F,W,T4,B9 --show-source --statistics --max-complexity=18 --max-line-length=100 --ignore=E203,E266,E501,W503,F403,F401,C901
48        poetry run flake8 examples --count --select=B,C,E,F,W,T4,B9 --show-source --statistics --max-complexity=18 --max-line-length=100 --ignore=E203,E266,E501,W503,F403,F401,C901
49        poetry run flake8 profile --count --select=B,C,E,F,W,T4,B9 --show-source --statistics --max-complexity=18 --max-line-length=100 --ignore=E203,E266,E501,W503,F403,F401,C901
50        poetry run flake8 tests --count --select=B,C,E,F,W,T4,B9 --show-source --statistics --max-complexity=18 --max-line-length=100 --ignore=E203,E266,E501,W503,F403,F401,C901
51    - name: Format with black
52      run: |
53        poetry run black pydevtips -l 100
54        poetry run black examples -l 100
55        poetry run black profile -l 100
56        poetry run black tests -l 100
57    - name: Test with pytest
58      run: poetry run pytest -v

The workflow performs the following:

  • (Lines 5-12) Triggers on pushes and pull requests to the main branch.

  • (Lines 21-24) Performs the test on all combinations of Python versions (3.10 and 3.11) and operating systems Ubuntu, Windows, and macOS.

  • (Lines 33-43) Installs Python, Poetry, and the package with its dependencies.

  • (Lines 44-56) Checks for code formatting and style and errors if it doesn’t conform. Make sure it matches the code formatting you’ve setup in your project, e.g. via pre-commit hooks.

  • (Lines 57-58) Runs the unit tests.

An older version of the workflow (not using Poetry but rather setup.py with setuptools) can be found below:

setuptools.yml (OLD WAY)
 1name: pydevtips setuptools
 2
 3# HACK to not trigger GitHub action
 4on:
 5  push:
 6    branches:
 7      - non-existant-branch
 8
 9
10jobs:
11  build:
12
13    runs-on: ${{ matrix.os }}
14    strategy:
15      fail-fast: false
16      max-parallel: 12
17      matrix:
18        os: [ubuntu-latest, macos-latest, windows-latest]
19        python-version: [3.8, 3.9, "3.10"]
20    steps:
21    - uses: actions/checkout@v3
22    - name: Checkout submodules
23      shell: bash
24      run: |
25        auth_header="$(git config --local --get http.https://github.com/.extraheader)"
26        git submodule sync --recursive
27        git -c "http.extraheader=$auth_header" -c protocol.version=2 submodule update --init --force --recursive --depth=1
28    - name: Set up Python ${{ matrix.python-version }}
29      uses: actions/setup-python@v4
30      with:
31        python-version: ${{ matrix.python-version }}
32    - name: Install dependencies and build package
33      run: |
34        python -m pip install --upgrade pip
35        python -m pip install -e .
36    - name: Lint with flake8
37      run: |
38        pip install flake8
39        # stop the build if there are Python syntax errors or undefined names
40        flake8 . --count --select=B,C,E,F,W,T4,B9 --show-source --statistics --max-complexity=18 --max-line-length=100 --ignore=E203,E266,E501,W503,F403,F401,C901
41    - name: Format with black
42      run: |
43        pip install black
44        black *.py -l 100
45        black examples/*.py -l 100
46        black profile/*.py -l 100
47        black pydevtips/*.py -l 100
48        black tests/*.py -l 100
49    - name: Test with pytest
50      run: |
51        pip install -U pytest
52        pytest

More information on configuring GitHub Actions can be found in their documentation.