Test an examples directory for a Python project

[wigging]

I have a project for developing a Python package where the structure of the project is similar to the following:

myproject/
├── README.md
├── examples/
│   ├── ex1.py
│   └── ex2.py
├── pyproject.toml
├── src/
│    └── mypackage/
│        ├── __init__.py
│        ├── adder.py
│        └── divider.py
└── tests/
    ├── test_adder.py
    ├── test_divider.py
    └── test_examples.py

The project is for developing a Python package named mypackage which is located in the src directory. The package is uploaded to PyPI where users can pip install it. Tests for the package are run with pytest and are located in the tests directory. Examples of using the package are in the examples directory. The examples are just scripts as shown below for ex1.py

"""
Example 1
"""
from mypackage import adder

x = 2.5
y = 8
a = adder(x, y)

print('a is', a)

The purpose of test_examples.py is to test the example files, its contents are shown below:

from examples import ex1
from examples import ex2

def test_ex1():
    ex1

def test_ex2():
    ex2

When I run pytest in the myproject directory I get the error shown here:

$ cd myproject
$ pytest
platform darwin -- Python 3.10.6, pytest-7.1.2, pluggy-1.0.0
rootdir: /Users/gavinw/Desktop/test-examples
collected 2 items / 1 error

================================================================== ERRORS ==================================================================
_________________________________________________ ERROR collecting tests/test_examples.py __________________________________________________
ImportError while importing test module '/Users/gavinw/Desktop/test-examples/tests/test_examples.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/opt/miniconda3/envs/ztest/lib/python3.10/importlib/__init__.py:126: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
tests/test_examples.py:1: in <module>
    from examples import ex1
E   ModuleNotFoundError: No module named 'examples'
========================================================= short test summary info ==========================================================
ERROR tests/test_examples.py
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
============================================================= 1 error in 0.05s =============================================================

It looks like pytest is not able to run the files in the examples directory because the location does not allow them to be imported. Any suggestions on how I can test the example files? Should I even use pytest for testing examples or is there a different testing tool for this?

[RonnyPfannschmidt]

pytest has no idea that for a examle test something has to added to the pythonpath,
so that on you to fix

the pythonpath config option is the starting point for
and in case you have to support pytest < 7.x https://pypi.org/project/pytest-pythonpath/

[wigging]

Ok, I'll use the pyproject.toml approach. Also, do you have any comments about how I implement the tests for the examples? The only approach I know is to import each example and execute it in a test function such as:

from examples import ex1

def test_ex1():
    ex1

[RonnyPfannschmidt]

well, commonly one would ensure that the examples actually do what one thinks they do

i would consider examples the same as other code - the test should ensure the behavior

depending on the intent of the examples it may be a good idea to put their tests next to them

[wigging]

I just want to make sure the examples will run after changes have been made to the package. Beyond that, I don't have a reason for testing them. Lots of projects provide an examples directory in the project's repository. But I have not come across any Python projects on GitHub that actually test the examples. Do you know of any open source Python projects that test their examples?

[RonnyPfannschmidt]

in pytest itself we typically verify the outputs/examples with the regendoc tools

as for testing examples, when doing it, it tends to be a good idea to do it right instead of doing it "just runs" as just running examples may still turn wrong by accident

[wigging]

Where are the pytest examples? I don't see an examples directory in this project.