Pytest with Python — from beginner to expert

If you are looking for an end-to-end guide for how to use pytest in real-world Python codebases, look no further.

Mar 20, 2024

Note: This article was edited on 26 March 2024 to include more reading materials.

If you are looking for an end-to-end guide for how to use pytest in real-world Python codebases, look no further.

Before we get started — pytest’s documentation is great but can be slightly hard to navigate at first. I highly recommend going through the Getting Started page at least. Go ahead, this post will still be here when you get back.

Note: You can find runnable code with versions of all the following examples on GitHub.

· ABCs of testing with Python
· Basic unit tests
· Testing exceptions
· Built-in Fixtures: Capturing logs and output in tests
· Built-in Fixtures: Higher, Further, Faster
· Built-in Fixtures: monkeypatch
∘ Patching configurations
∘ Mocking functions to test error scenarios
∘ Testing exception handling with mocking
· Custom fixtures
∘ Data Fixtures
∘ Fixtures for API responses to be used by tests
∘ FastAPI test clients
∘ Monkeypatched fixtures
· Coverage of your test suite
· Testcontainers for testing real dependencies
· Organizing your test suite
· Further reading
· That’s all, folks!

ABCs of testing with Python

If you’re reading this post, I hope I don’t need to get into why you need to write tests —you already know. What I will say is that I remember the confusion of getting started with Pytest and not knowing what any of the words meant or how to make the leap from manual testing to automated testing.

So in this guide, I will be going over all the features of pytest that I have personally used over the years and found to be incredibly useful in maintaining a stable and reliable codebase, whether you are writing thousands of lines of code each month or maintaining a legacy codebase.

Some of the terminology may be new if you are completely new to testing, but in the interest of keeping this to a reasonable length, I’ve linked to external resources throughout this article that should help you get a better grasp on things. I’ve focused more on code examples here because that’s how I learn best!

My goal is to take you from wherever you currently are to being comfortable around testing in no time flat!

“From zero to hero in no time flat” - Disney’s Hercules

Basic unit tests

These are largely covered in pytest’s own Getting Started documentation, but I’ll cover it here again. Assuming you’ve abstracted out logical code into separate methods, you’re one step ahead — every logical method you have can have its own comprehensive set of unit tests.

Say you have a super simple atomic method to yield a subset of a list:

def chunks(lst, n):
    """Yield successive n-sized chunks from lst."""
    for i in range(0, len(lst), n):
        yield lst[i:i + n]

It’s pretty easy to look at this code and say it’ll work. But what if someone replaces the 0 with 1 tomorrow without anyone knowing? That’s where tests come in. Here’s an example of basic tests with pytest:

def test_chunks_success():
    """Method to test chunking logic"""
    # general case
    arr = range(101)
    res = chunks(list(arr), 10)
    assert isinstance(res, Generator)  # ensure method returns a Generator
    res = list(res) # get all chunks
    assert len(res) == 11  # for 101 elements in groups of 10, there should be 11 chunks
    assert res[-1] == [100]  # last chunk should have only one value with the last element

def test_chunks_empty_input_no_error():
    """Method to test edge case where input is empty"""
    # edge case: empty list input
    res = chunks([], 10)
    assert isinstance(res, Generator)  # ensure method still returns a generator
    res = list(res)  # get all chunks
    assert not res  # should be empty

Now you have covered your bases — the type of returned value, the expected behaviour, and an edge case.

Basic tests use vanilla Python and simple assert statements to ensure your methods work as expected in every scenario. The more of these you have (particularly for edge cases and rare scenarios), the more stable and reliable your codebase will be.

Testing exceptions

Now, if someone were to call this method with a non-iterable object, it would raise an exception. You may update the test suite to add the following test:

# edge case: invalid datatype
res = chunks(12, 10)
with pytest.raises(TypeError) as e:
    res = list(res)
e.match("object of type 'int' has no len()")

(and if you need to, you could always update the method to handle these exceptions and instead, test that this doesn’t raise an exception)

Built-in Fixtures: Capturing logs and output in tests

Pytest has handy built-in fixtures which allow you to trap the logs and output statements generated by your code. This is particularly useful to ensure that your application logs are getting generated as expected in certain scenarios, so you know that when you go into debug a production issue at 2 am on a Saturday, the logs you want are going to be there.

Let us update our function with some logs and print statements:

def chunks(lst, n):
    """Yield successive n-sized chunks from lst."""
    try:
        for i in range(0, len(lst), n):
            yield lst[i:i + n]
    except TypeError as e:
        logging.error(f"Type {type(lst)} not supported")
        print("Error: invalid datatype input to chunks(lst, n)", file=sys.stderr)
        yield None

In our test case, we can verify this log gets generated in the expected scenario. The fixture caplog allows us to check logger statements and capsys lets us inspect stdout/stderr.

Note — just like with any other Pytest fixtures, you don’t need to import anything explicitly for caplog and capsys to be accessible to your test functions. As long as you run the tests with the pytest command, your tests can access the fixtures.

def test_chunks_invalid_datatype_verify_error_logs(caplog, capsys):
    """
    test method to verify that expected logs are printed in case of
    invalid datatype
    """
    # edge case: invalid datatype
    capsys.readouterr()  # clears any stdout/stderr statements so far
    caplog.clear()  # clears all log statements so far
    expected_log = f"Type {type(12)} not supported"

    # caplog.at_level overrides default log levels that your app may have
    with caplog.at_level(logging.INFO):  
        res = chunks(12, 10)
        for i in res:
            assert i is None

        # ensure errors are logged as expected
        error_logs = [record.message for record in caplog.records 
                      if record.levelno == logging.ERROR]
        assert expected_log in error_logs

    # ensure stderr contains the expected log line
    captured = capsys.readouterr()
    assert "Error: invalid datatype input to chunks(lst, n)" in captured.err

Built-in Fixtures: Higher, Further, Faster

You can do so much more with built-in fixtures. In the interest of time, here’s a short list of possibilities, which you can read more about in the pytest documentation.

Use temporary directories, paths, and files for your tests: testdir, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory
Configure the behaviour of your test functions based on command line inputs: request
monkeypatch — which we will get into more next.

Built-in Fixtures: monkeypatch

This is where testing starts to get fun. What do you do when you have external dependencies in the methods you want to test? External APIs that may cost money, external databases that you don’t want to pollute with test data, or simply slow dependencies that you don’t want to slow down your test execution?

At its core, monkeypatching is a way to override functions, attributes, environment variables etc so that you can run your tests exactly the way you want, without any dependencies on external systems. How you can use monkeypatch is limited only by your imagination, but I will share the three most useful examples from my experience.

Consider the below application code snippet:

import requests
import logging

class Settings:
    api_url = "https://randomuser.me/api/"

def make_request():
    """Method to make API call to get information about a random user"""
    try:
        response = requests.get(Settings.api_url, timeout=5)
        return response.json()
    except requests.exceptions.JSONDecodeError as e:
        logging.error("Invalid response format; unable to parse json")
        return None
    except requests.exceptions.Timeout as e:
        logging.error(f"{Settings.api_url} timed out: {e}")
        return None

Patching configurations

Supposing you have an application whose behaviour can be configured with some settings. You want to test what happens if there’s a problem with the settings without changing your application code. The following test demonstrates the usage of monkeypatching for the same —

def test_invalid_json_logs_error(monkeypatch, caplog):
    """Test case for if api gives non-json response"""
    # patch to api which does not give json response
    monkeypatch.setattr(Settings, "api_url", "https://randomuser.me/")

    # make request uses above patched api
    resp = make_request()
    assert resp is None

    errs = [i.message for i in caplog.records if i.levelno == logging.ERROR]
    assert "Invalid response format; unable to parse json" in errs
    caplog.clear()

This test patches the api_url to one that gives a non-JSON response. We then verify that the application shows the expected logs.

Mocking functions to test error scenarios

Alternatively, say you have written some excellent exception handling, but you want to test the exception handling to ensure your app works like it should in case of timeouts, errors, etc.

The following test uses a “mock” method to override the default requests.get method with one that enforces a timeout error. We can then verify that this error is handled appropriately.

def test_api_timeout_logs_error(monkeypatch, caplog):
    """Test case for if api times out"""

    # mock function to simulate timeout from requests.get
    def mock_get_timeout(*args, **kwargs):
        raise requests.exceptions.Timeout("simulated timeout")

    monkeypatch.setattr(requests, "get", mock_get_timeout)

    # this will use the above mock_get_timeout instead of requests.get
    resp = make_request()
    assert resp is None

    # verify that appropriate logs are present
    errs = [i.message for i in caplog.records if i.levelno == logging.ERROR]
    assert f"{Settings.api_url} timed out: simulated timeout" in errs

    caplog.clear()

Testing exception handling with mocking

Consider you have a method that raises certain known exceptions. If you want to verify that these are being raised as expected, we can use a combination of mocking, monkeypatching, and exception testing —

def make_request():
    """Method to make API call to get information about a random user"""
    try:
        response = requests.get(Settings.api_url, timeout=5)
        return response.json()
    except requests.exceptions.Timeout as e:
        logging.error(f"{Settings.api_url} timed out: {e}")
        raise RuntimeError(f"Timeout: {e}") from e

We can verify the appropriate RuntimeError is raised with the following test:

def test_api_timeout_raises_runtimeerror(monkeypatch, caplog):
    """Test case for if api times out"""
    # mock function to simulate timeout from requests.get
    def mock_get_timeout(*args, **kwargs):
        raise requests.exceptions.Timeout("simulated timeout")
    monkeypatch.setattr(requests, "get", mock_get_timeout)

    # this will use the above mock_get_timeout instead of requests.get
    with pytest.raises(RuntimeError) as e:
        resp = make_request()
        assert resp is None
    e.match("Timeout: simulated timeout")

    # verify that appropriate logs are present
    errs = [i.message for i in caplog.records if i.levelno == logging.ERROR]
    assert f"{Settings.api_url} timed out: simulated timeout" in errs

    caplog.clear()

For more examples and info about mocking and monkeypatching, you can refer to this section of the pytest docs. While the examples in this post are trivial, a combination of mocking and monkeypatching can help you write truly great tests, particularly when used alongside testcontainers as described later in this post.

I would, however, suggest using mocking and monkeypatching with caution as tests which diverge from actual production behaviour will end up being redundant. These are good backup tools for when testing with production services is not feasible.

Custom fixtures

Pytest has extensive support for test configurations which is outside the scope of this post. If this interests you, you can see the possibilities in the documentation. I will go over the most useful test configurations I’ve used — custom fixtures.

Custom fixtures can be used for test data, in combination with monkeypatch to yield versions of your application with different settings, and are a super clean way to test FastAPI applications.

Fixtures are defined in a file commonly named conftest.py , typically in the root directory of your tests or the root directory of your application. The pytest.fixture decorator is what makes them accessible in all your tests without any extra imports.

Data Fixtures

You can create a fixture to return fixed data to be used by any test method, such as fixed test IDs or sample data.

@pytest.fixture(scope="session")
def sample_user_id():
    """Pytest fixture to get user id to be used for testing"""
    return "test_u_2b5e3801a8624f2d854127ae019e0714"

Fixtures for API responses to be used by tests

In case your test suite relies on some configurations that are read from an API, you can create an appropriately scoped fixture to abstract out the API call and make the API response accessible to any test:

@pytest.fixture(scope="module")
def versions():
    """Pytest fixture to yield latest version numbers as per API"""
    versions = requests.get("https://get/versions/url", timeout=10).json()
    yield versions

This has the bonus of ensuring the API call happens only once in case of session- or module-scoped fixtures.

FastAPI test clients

If you have APIs developed with FastAPI, having your test client be part of your fixtures is an easy way to avoid reinitialization of the TestClient with each test.

from app.server import fastapi_app

@pytest.fixture(scope="session")
def test_client():
    """Pytest fixture to get test client of FastAPI app"""
    client = TestClient(fastapi_app)
    yield client

Monkeypatched fixtures

If you find yourself repeating the same monkeypatching steps in many tests, it may be worth creating a fixture out of it. Following is an example where the versions used by the FastAPI client are patched to fixed values by mocking the function which returns the versions to be used —

from app.server import fastapi_app
from app import api_helpers

@pytest.fixture(scope="function")
def test_client_patch_versions(monkeypatch):
    """Pytest fixture to get API client that uses fixed version numbers"""
    # monkeypatch fn to simulate version change
    def mock_version_change(channels):
        return {k: 10 for k in channels}

    # replace utilities fn
    monkeypatch.setattr(api_helpers, 'get_latest_versions', mock_version_change)

    client = TestClient(app)
    yield client

Coverage of your test suite

You can use the coverage library to see how much of your code is covered by tests (the results may surprise you).

Many organizations have conditions like the codebase must always have >85%, >90%, or >95% of the application code covered by tests. This may or may not be overkill depending on how much critical code is in your application, but this metric is still good to know.

Even better — it helps you identify parts of your code that you thought may have been tested but aren’t. It could be the difference between one small if condition with an undefined variable breaking production because none of the tests ever reached that code (not that that has ever happened to me, of course).

To check your code coverage, all you need to do is run your tests with coverage run -m pytest . instead of pytest . , and then generate the report with coverage report (or coverage html if you want a clickable HTML report that you can navigate and search for uncovered code — I recommend this one for local development). Any flags you usually pass to pytest can be passed here as well.

Testcontainers for testing real dependencies

This is a super useful open-source project which can help you work with external dependencies like databases, message queues, etc without actually touching your production systems, needing database access, or needing firewall access. The full list of supported packages is listed in the documentation.

Note: As of the writing of this article, the Python documentation for testcontainers is limited and contains only interface definitions. Reader discretion is advised.

For example, if your application code uses a Redis client to save some data during runtime and you want to write unit tests that won’t unnecessarily burden your production systems —

import redis

class Settings:
    REDIS_HOST = "prod.redis.host"
    REDIS_PORT = 6379

def save_to_redis(key: str, value: str) -> None:
    """Method to save specified key-value pair to Redis"""
    client = redis.StrictRedis(host=Settings.REDIS_HOST,
                               port=Settings.REDIS_PORT)
    client.set(key, value)

The following test shows you how easy it is to spin up a Redis instance for your test to use, which gets conveniently destroyed at the end of the test —

from testcontainers.redis import RedisContainer

def test_save_to_redis_success(monkeypatch):
    """Method to test saving to redis with a testcontainer"""
    with RedisContainer() as redis_container:
        host = redis_container.get_container_host_ip()
        port = redis_container.get_exposed_port(redis_container.port)

        # patch host and port configs to the testcontainer host and port
        monkeypatch.setattr(Settings, "REDIS_HOST", host)
        monkeypatch.setattr(Settings, "REDIS_PORT", port)

        # test save_to_redis method
        save_to_redis("test_key", "test_value")

        # verify that data is saved in this redis instance
        client = redis_container.get_client()
        assert client.get("test_key") == b"test_value"

Similarly, you can test your interfaces to MongoDB, MySQL, Kafka, Pub/Sub, or any other supported technologies.

Organizing your test suite

There is some great documentation about test layouts here that I recommend going through. What works best for you depends on the nature and structure of your application and its deployment.

In my personal opinion, having a separate tests/ directory is a good way to keep all the tests together. I also find that for unit testing, simple Python test functions as shown above work best. For integration testing where each method may have multiple associated tests, the class structure is most useful. It keeps each scenario separate while grouping tests related to the same method.

Considering the same example above, this is how I would organize the tests in a class structure:

app/main.py :

import requests
import logging


class Settings:
    api_url = "https://randomuser.me/api/"


def make_request():
    """Method to make get JSON details of a single user from an API"""
    try:
        response = requests.get(Settings.api_url, timeout=5)
        return response.json()
    except requests.exceptions.JSONDecodeError as e:
        logging.error("Invalid response format; unable to parse json")
        return None
    except requests.exceptions.Timeout as e:
        logging.error(f"{Settings.api_url} timed out: {e}")
        raise RuntimeError(f"Timeout: {e}") from e

tests/test_main.py

"""Tests related to app/main.py"""
import pytest


class TestMakeRequest:
    """Tests related to make_request() method"""
    @staticmethod
    def test_make_request():
        # test basic success case
        resp = make_request()
        assert resp["results"]
        
    @staticmethod
    def test_invalid_json_logs_error(monkeypatch, caplog):
        """Test case for if api gives non-json response"""
        # patch to api which does not give json response
        monkeypatch.setattr(Settings, "api_url", "https://randomuser.me/")
        resp = make_request()
        assert resp is None

        errs = [i.message for i in caplog.records if i.levelno == logging.ERROR]
        assert "Invalid response format; unable to parse json" in errs
        caplog.clear()

    @staticmethod
    def test_api_timeout_raises_runtimerror(monkeypatch, caplog):
        """Test case for if api times out"""
        # mock function to simulate timeout from requests.get
        def mock_get_timeout(*args, **kwargs):
            raise requests.exceptions.Timeout("simulated timeout")
        monkeypatch.setattr(requests, "get", mock_get_timeout)

        # this will use the above mock_get_timeout instead of requests.get
        with pytest.raises(RuntimeError) as e:
            resp = make_request()
            assert resp is None
        e.match("Timeout: simulated timeout")

        # verify that appropriate logs are present
        errs = [i.message for i in caplog.records if i.levelno == logging.ERROR]
        assert f"{Settings.api_url} timed out: simulated timeout" in errs

        caplog.clear()

In the case of a large project with multiple folders, following a similar folder structure in the tests/ directory also keeps things quite pleasant to work with.

I have created a GitHub project that you can use as a reference point here.

That’s all, folks!

Every test case I’ve had to write in the last 5+ years has largely used one of the above techniques, with testcontainers being the newest entrant. I will always advocate for high test coverage in production systems because as developers, it makes our own lives easier. My midnight production debugging sessions have drastically decreased in frequency ever since I introduced tests into all my codebases. Adding new features is stress-free and bugs are caught long before they reach production. The extra upfront time required to add tests will save literal days in debugging time over the years — particularly in legacy codebases.

That’s all from me, now go write some tests!

Krupa’s Substack

Discussion about this post

Krupa’s Substack

Pytest with Python — from beginner to expert

If you are looking for an end-to-end guide for how to use pytest in real-world Python codebases, look no further.

Contents:

ABCs of testing with Python

Basic unit tests

Testing exceptions

Built-in Fixtures: Capturing logs and output in tests

Built-in Fixtures: Higher, Further, Faster

Built-in Fixtures: monkeypatch

Patching configurations

Mocking functions to test error scenarios

Testing exception handling with mocking

Custom fixtures

Data Fixtures

Fixtures for API responses to be used by tests

FastAPI test clients

Monkeypatched fixtures

Coverage of your test suite

Testcontainers for testing real dependencies

Organizing your test suite

Further reading

That’s all, folks!

Discussion about this post