Run Test

OpenBox uses pytest as the test framework. Tests are automatically run by GitHub Actions when a commit is pushed (to master branch) or a pull request is opened. Tests can also be run locally.

For pytest

See [tool.pytest.ini_options] in pyproject.toml. Options that are not specified in command line are read from this file.

Pytest documentation: https://docs.pytest.org/

Pytest will run all files of the form test_*.py or *_test.py in testpath directories (test/ in this repo). From those files, collect test items:

• test prefixed test functions or methods outside of class.

• test prefixed test functions or methods inside Test prefixed test classes (without an __init__ method). Methods decorated with @staticmethod and @classmethods are also considered.

Test discovery rules: https://docs.pytest.org/en/8.0.x/explanation/goodpractices.html#test-discovery

Example command:

pytest -rap test
pytest -h  # show help

Use -r[chars] to show extra test summary info as specified by chars:

(f)ailed, (E)rror, (s)kipped, (x)failed, (X)passed,
(p)assed, (P)assed with output, (a)ll except passed
(p/P), or (A)ll. (w)arnings are enabled by default (see
--disable-warnings), 'N' can be used to reset the list.
(default: 'fE').

For GitHub Actions

See .github/workflows/test.yml for the configuration of GitHub Actions.

Documentation: https://docs.github.com/actions

An example repository with GitHub Actions for testing: https://github.com/jhj0411jhj/test_action

Code coverage

codecov: https://app.codecov.io

openbox codecov: https://app.codecov.io/gh/PKU-DAIR/open-box

To manage codecov of OpenBox, you need to be a member of PKU-DAIR organization.

Codecov report is automatically generated and uploaded.

TODO: pylint

TODO: use pylint (and flake8, …) to check code quality.