Overview
Many test frameworks support CPU parallelism, which lets you speed up your test runs by spreading them across multiple CPU cores. pytest is a popular Python testing framework, but it doesn't support CPU parallelism out of the box. This article walks you through how to set it up and keep it under control when running pytest in CircleCI.
Please note that you should not use more cores than available for a specific resource class, you may find resource class specs for Docker executors here: https://circleci.com/docs/reference/configuration-reference/
Solution
As mentioned earlier, in order for pytest to run tests in parallel, you need to install additional package before running the test:
jobs:
pytest-parallel:
docker:
- image: cimg/python:3.12
# Optional: specify the required resource_class
# resource_class: large
working_directory: ~/project
steps:
- checkout
- run:
name: Install pytest xdist library
command: |
python -m pip install --upgrade pip
pip install pytest pytest-xdist
# If you use a lock/requirements file, prefer that instead of bare pytest:
# pip install -r requirements.txt
# pip install -r requirements-dev.txt
- run:
name: Run tests (pytest-xdist)
# -n auto: worker count ~= logical CPUs visible to the container
# Add your normal pytest flags (paths, markers, junit, coverage, etc.)
command: pytest -n auto -qPlease be informed that pytest -n auto will use all detected cores, this may lead to resources overuse and errors.