Easier way to debug in pytest test ?

[inkhey]

Hello,

I'm playing with pytest test in twister (https://docs.zephyrproject.org/latest/develop/test/pytest.html).

As a python developer, I was a bit bothered that Pytest is wrapped in a bigger process in a way that I don't have the direct output of the test. That understandable for CI, but it's clearly less practical for writing test and/or doing Test Driven Development.

It seems that we can find the command line used (https://youtu.be/6PJ9BnukwoA?si=6wYiy81QolIV6wLs&t=496) and this does work, and that's nice, but I found the way to do it a bit too much complicated.

Is there any easier way to do this ? Having some quick way to run a "one target" build + test with direct pytest process will be really great.

The benefit is that it's became easy and quick to debug pytest based test by putting breakpoint() in test, which mean a smaller feedback loop. Pytest also provide the pytest -pdb option (https://docs.pytest.org/en/stable/how-to/failures.html). Also, it can also permit to receive instant error using pytest plugin like pytest-sugar (https://pypi.org/project/pytest-sugar/)

Also thanks for this really nice software, the pytest integration is great.

[PerMac]

When we started creating this plugin we also had a python developer mindset and we opted for it being a "proper" plugin. However, twister's maintainer had a very strong opinion against it. After some time I started to better understand his view point. The argument is that this "plugin" shouldn't be treated as a real plugin, but as a core functionality of twister, therefore be tightly coupled with it and being part of the repo. The same way as twister is a part of zephyr. You cannot have one without the other nor mix zephyr and twister versions, so no mixing of twister and pytest-plugin versions. You checkout a given zephyr version and all test tooling is aligned without extra effort.
This forced us to come with some counter intuitive (from python/pytest pov) solutions. Up to blocking the plugin being installed as a real plugin (hence withouth the need of PYTHONPATH), unless a user selects a certain option in twister (https://github.com/zephyrproject-rtos/zephyr/blob/main/scripts/pylib/twister/twisterlib/environment.py#L438). The argument was that a user could install some version, and being used to having all synced with zephyr version. forget about updating the plugin and having unsynced plugin with twister.
One thing we are considering is putting all the options that are passed to pytest into some meta file (json/yaml). It should be rather easy to add, but we haven't feel that it would change "that much". I think @gchwier has some conftests for his on desk development that helps a bit, but not that much. You can parametrize some more in pytest.ini, conftest.py and/or pyproject.toml
This is his example:

# Add the directory to PYTHONPATH
zephyr_base = os.getenv("ZEPHYR_BASE")
if zephyr_base:
    sys.path.insert(0, os.path.join(zephyr_base, "scripts", "pylib", "pytest-twister-harness", "src"))
else:
    raise EnvironmentError("ZEPHYR_BASE environment variable is not set")

pytest_plugins = [
    "twister_harness.plugin",
]