Markdown ******** Usage examples ============== Any fenced code block with a recognized Python language tag (e.g., "python", "py") will be collected and executed automatically, if your pytest configuration allows that. Standalone code blocks ---------------------- Note: Note that "name" value has a "test_" prefix. *Filename: README.md* ```python name=test_basic_example import math result = math.pow(3, 2) assert result == 9 ``` ====================================================================== Grouping multiple code blocks ----------------------------- It's possible to split one logical test into multiple blocks. They will be tested under the first "name" specified. Note the "" directive. Note: Note that "continue" directive of the "test_grouping_example_part_2" and "test_grouping_example_part_3" refers to the "test_grouping_example". *Filename: README.md* ```python name=test_grouping_example x = 1 ``` Some intervening text. ```python name=test_grouping_example_part_2 y = x + 1 # Uses x from the first snippet assert y == 2 ``` Some intervening text. ```python name=test_grouping_example_part_3 print(y) # Uses y from the previous snippet ``` The above mentioned three snippets will run as a single test. ====================================================================== Async ----- You can use *top-level await* in your code blocks. The code will be automatically wrapped in an async function. *Filename: README.md* ```python name=test_async_example import asyncio result = await asyncio.sleep(0.1, result=42) assert result == 42 ``` ====================================================================== Adding pytest markers to code blocks ------------------------------------ It's possible to add custom pytest markers to your code blocks. That allows adding custom logic and mocking in your "conftest.py". In the example below, "django_db" marker is added to the code block. Note: Note the "pytestmark" directive "django_db" marker. *Filename: README.md* ```python name=test_django from django.contrib.auth.models import User user = User.objects.first() ``` Requesting pytest fixtures for code blocks ------------------------------------------ It's possible to request existing or custom pytest fixtures for code blocks. That allows adding custom logic and mocking in "conftest.py". In the example below, "tmp_path" fixture is requested for the code block. Note: Note the "pytestfixture" directive "tmp_path" fixture. *Filename: README.md* ```python name=test_path d = tmp_path / "sub" d.mkdir() # Create the directory assert d.is_dir() # Verify it was created and is a directory ``` ====================================================================== Multiple "pytestfixture" directives are supported. Add one on each line. Note: When combining "pytestfixture" and "continue" directives together, request pytest-fixtures only in the first "code-block", as they will automatically become available in all continuing blocks. Custom pytest-fixtures are supported as well. Just define them in your "conftest.py" file. Customisation/hooks =================== Tests can be extended and fine-tuned using pytest's standard hook system. Below is an example workflow: 1. **Add custom pytest markers** to the code blocks ("fakepy", "aws", "openai"). 2. **Implement pytest hooks** in "conftest.py" to react to those markers. Add custom pytest markers ------------------------- Add "fakepy" marker ~~~~~~~~~~~~~~~~~~~ The example code below will generate a PDF file with random text using fake.py library. Note, that a "fakepy" marker is added to the code block. In the Implement pytest hooks section, you will see what can be done with the markers. Note: Note the "pytestmark" directive "fakepy" marker. *Filename: README.md* ```python name=test_create_pdf_file from fake import FAKER FAKER.pdf_file() ``` Add "aws" marker ~~~~~~~~~~~~~~~~ Sample boto3 code to create a bucket on AWS S3. Note: Note the "pytestmark" directive "aws" marker. *Filename: README.md* ```python name=test_create_bucket import boto3 s3 = boto3.client("s3", region_name="us-east-1") s3.create_bucket(Bucket="my-bucket") assert "my-bucket" in [b["Name"] for b in s3.list_buckets()["Buckets"]] ``` Add "openai" marker ~~~~~~~~~~~~~~~~~~~ Sample openai code to ask LLM to tell a joke. Note, that next to a custom "openai" marker, "xfail" marker is used, which allows underlying code to fail, without marking entire test suite as failed. Note: Note the "pytestmark" directive "xfail" and "openai" markers. *Filename: README.md* ```python name=test_tell_me_a_joke from openai import OpenAI client = OpenAI() completion = client.chat.completions.create( model="gpt-4o", messages=[ {"role": "developer", "content": "You are a famous comedian."}, {"role": "user", "content": "Tell me a joke."}, ], ) assert isinstance(completion.choices[0].message.content, str) ``` ====================================================================== Implement pytest hooks ---------------------- In the example below: * moto is used to mock AWS S3 service for all tests marked as "aws". * Environment variable "OPENAI_BASE_URL" is set to "http://localhost:11434/v1" (assuming you have Ollama running) for all tests marked as "openai". * "FILE_REGISTRY.clean_up()" is executed at the end of each test marked as "fakepy". *Filename: conftest.py* import os from contextlib import suppress import pytest from fake import FILE_REGISTRY from moto import mock_aws from pytest_codeblock.constants import CODEBLOCK_MARK # Modify test item during collection def pytest_collection_modifyitems(config, items): for item in items: if item.get_closest_marker(CODEBLOCK_MARK): # All `pytest-codeblock` tests are automatically assigned # a `codeblock` marker, which can be used for customisation. # In the example below we add an additional `documentation` # marker to `pytest-codeblock` tests. item.add_marker(pytest.mark.documentation) if item.get_closest_marker("aws"): # Apply `mock_aws` to all tests marked as `aws` item.obj = mock_aws(item.obj) # Setup before test runs def pytest_runtest_setup(item): if item.get_closest_marker("openai"): # Send all OpenAI requests to locally running Ollama for all # tests marked as `openai`. The tests would x-pass on environments # where Ollama is up and running (assuming, you have created an # alias for gpt-4o using one of the available models) and would # x-fail on environments, where Ollama isn't runnig. os.environ.setdefault("OPENAI_API_KEY", "ollama") os.environ.setdefault("OPENAI_BASE_URL", "http://localhost:11434/v1") # Teardown after the test ends def pytest_runtest_teardown(item, nextitem): # Run file clean up on all tests marked as `fakepy` if item.get_closest_marker("fakepy"): FILE_REGISTRY.clean_up()