pytest-codeblock **************** Test your documentation code blocks. [image: PyPI Version][image][image: Supported Python versions][image][image: Build Status][image][image: Documentation Status][image][image: llms.txt - documentation for LLMs][image][image: MIT][image][image: Coverage][image] pytest-codeblock is a Pytest plugin that discovers Python code examples in your reStructuredText and Markdown documentation files and runs them as part of your test suite. This ensures your docs stay correct and up-to-date. Features ======== * **reStructuredText and Markdown support**: Automatically find and test code blocks in reStructuredText (".rst") and Markdown (".md") files. The only requirement here is that your code blocks shall have a name starting with "test_". Async code snippets are supported as well. * **Grouping**: Split a single example across multiple code blocks; the plugin concatenates them into one test. * **Pytest markers support**: Add existing or custom pytest markers to the code blocks and hook into the tests life-cycle using "conftest.py". * **Pytest fixtures support**: Request existing or custom pytest fixtures for the code blocks. Prerequisites ============= * Python 3.10+ * pytest is the only required dependency (on Python 3.11+; for Python 3.10 tomli is also required). Documentation ============= * Documentation is available on Read the Docs. * For reStructuredText, see a dedicated reStructuredText docs. * For Markdown, see a dedicated Markdown docs. * Both reStructuredText docs and Markdown docs have extensive documentation on pytest markers and corresponding "conftest.py" hooks. * For guidelines on contributing check the Contributor guidelines. Installation ============ Install with pip: pip install pytest-codeblock Or install with uv: uv pip install pytest-codeblock Configuration ============= For most use cases, no configuration needed. All your *.rst* and *.md* files shall be picked automatically. However, if you need to add another file extension or use or another language identifier for python in codeblock, you could configure that. See the following example of *pyproject.toml* configuration: *Filename: pyproject.toml* [tool.pytest-codeblock] rst_user_codeblocks = ["c_py"] rst_user_extensions = [".rst.txt"] md_user_codeblocks = ["c_py"] md_user_extensions = [".md.txt"] See customisation docs for more. Usage ===== reStructruredText usage ----------------------- Any code directive, such as ".. code-block:: python", ".. code:: python", or literal blocks with a preceding ".. codeblock-name: ", will be collected and executed automatically by pytest. "code-block" directive example ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Note: Note that ":name:" value has a "test_" prefix. *Filename: README.rst* .. code-block:: python :name: test_basic_example import math result = math.pow(3, 2) assert result == 9 "literalinclude" directive example ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Note: Note that ":name:" value has a "test_" prefix. *Filename: README.rst* .. literalinclude:: examples/python/basic_example.py :name: test_li_basic_example See a dedicated reStructuredText docs for more. Markdown usage -------------- Any fenced code block with a recognized Python language tag (e.g., "python", "py") will be collected and executed automatically by pytest. Note: Note that "name" value has a "test_" prefix. *Filename: README.md* ```python name=test_basic_example import math result = math.pow(3, 2) assert result == 9 ``` See a dedicated Markdown docs for more. Tests ===== Run the tests with pytest: pytest Troubleshooting =============== If something doesn't work, try to add this to your pyproject.toml: *Filename: pyproject.toml* [tool.pytest.ini_options] testpaths = [ "**/*.rst", "**/*.md", ] Writing documentation ===================== Keep the following hierarchy. ===== title ===== header ====== sub-header ---------- sub-sub-header ~~~~~~~~~~~~~~ sub-sub-sub-header ^^^^^^^^^^^^^^^^^^ sub-sub-sub-sub-header ++++++++++++++++++++++ sub-sub-sub-sub-sub-header ************************** License ======= MIT Support ======= For security issues contact me at the e-mail given in the Author section. For overall issues, go to GitHub. Author ====== Artur Barseghyan ====================================================================== reStructuredText ================ The following directives are supported: * ".. code-block:: python" * ".. code:: python" * ".. codeblock-name: " * ".. literalinclude::" Any code directive, such as ".. code-block:: python", ".. code:: python", ".. literalinclude::" or literal blocks with a preceding ".. codeblock-name: ", will be collected and executed automatically, if your pytest configuration allows that. Usage examples -------------- Standalone code blocks ~~~~~~~~~~~~~~~~~~~~~~ "code-block" directive """""""""""""""""""""" Note: Note that ":name:" value has a "test_" prefix. *Filename: README.rst* .. code-block:: python :name: test_basic_example import math result = math.pow(3, 2) assert result == 9 ====================================================================== "literalinclude" directive """""""""""""""""""""""""" *Filename: README.rst* .. literalinclude:: examples/python/basic_example.py :name: test_li_basic_example ====================================================================== "codeblock-name" directive """""""""""""""""""""""""" You can also use a literal block with a preceding name comment: *Filename: README.rst* .. codeblock-name: test_grouping_example_literal_block This is a literal block:: y = 5 print(y * 2) ====================================================================== Grouping multiple "code-block" directives ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ It's possible to split one logical test into multiple blocks. They will be tested under the first ":name:" specified. Note the ".. continue::" directive. Note: Note that "continue" directive of the "test_grouping_example_part_2" and "test_grouping_example_part_3" refers to the "test_grouping_example". *Filename: README.rst* .. code-block:: python :name: test_grouping_example x = 1 Some intervening text. .. continue: test_grouping_example .. code-block:: python :name: test_grouping_example_part_2 y = x + 1 # Uses x from the first snippet assert y == 2 Some intervening text. .. continue: test_grouping_example .. code-block:: python :name: test_grouping_example_part_3 print(y) # Uses y from the previous snippet The above mentioned three snippets will run as a single test. ====================================================================== Async ~~~~~ You can use *top-level await* in your code blocks. The code will be automatically wrapped in an async function. *Filename: README.rst* .. code-block:: python :name: test_async_example import asyncio result = await asyncio.sleep(0.1, result=42) assert result == 42 ====================================================================== Adding pytest markers to "code-block" and "literalinclude" directives ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ It's possible to add custom pytest markers to your "code-block" or "literalinclude" directives. That allows adding custom logic and mocking in your "conftest.py". In the example below, "django_db" marker is added to the "code-block" directive. Note: Note the "pytestmark" directive "django_db" marker. *Filename: README.rst* .. pytestmark: django_db .. code-block:: python :name: test_django from django.contrib.auth.models import User user = User.objects.first() ====================================================================== In the example below, "django_db" marker is added to the "literalinclude" directive. *Filename: README.rst* .. pytestmark: django_db .. literalinclude:: examples/python/django_example.py :name: test_li_django_example ====================================================================== Requesting pytest fixtures for "code-block" and "literalinclude" directives ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ It's possible to request existing or custom pytest fixtures in "code- block" or "literalinclude" directives. That allows adding custom logic and mocking in "conftest.py". In the example below, "tmp_path" fixture is requested for the "code- block" directive. Note: Note the "pytestfixture" directive "tmp_path" fixture. *Filename: README.rst* .. pytestfixture: tmp_path .. code-block:: python :name: test_path d = tmp_path / "sub" d.mkdir() # Create the directory assert d.is_dir() # Verify it was created and is a directory ====================================================================== In the example below, "tmp_path" fixture is requested for the "literalinclude" directive. *Filename: README.rst* .. pytestfixture: tmp_path .. literalinclude:: examples/python/tmp_path_example.py :name: test_li_tmp_path_example ====================================================================== Multiple "pytestfixture" directives are supported. Add one on each line. Note: When combining "pytestfixture" and "continue" directives together, request pytest-fixtures only in the first "code-block", as they will automatically become available in all continuing blocks. Custom pytest-fixtures are supported as well. Just define them in your "conftest.py" file. Customisation/hooks ------------------- Tests can be extended and fine-tuned using pytest's standard hook system. Below is an example workflow: 1. **Add custom pytest markers** to the "code-block" or "literalinclude" ("fakepy", "aws", "openai"). 2. **Implement pytest hooks** in "conftest.py" to react to those markers. Add custom pytest markers ~~~~~~~~~~~~~~~~~~~~~~~~~ Add "fakepy" marker """"""""""""""""""" The example code below will generate a PDF file with random text using fake.py library. Note, that a "fakepy" marker is added to the "code- block". In the >>`Implement pytest hooks`_<< section, you will see what can be done with the markers. Note: Note the "pytestmark" directive "fakepy" marker. *Filename: README.rst* .. pytestmark: fakepy .. code-block:: python :name: test_create_pdf_file from fake import FAKER FAKER.pdf_file() ====================================================================== In the example code below, a "fakepy" marker is added to the "literalinclude" block. *Filename: README.rst* .. pytestmark: fakepy .. literalinclude:: examples/python/create_pdf_file_example.py :name: test_li_create_pdf_file ====================================================================== Add "aws" marker """""""""""""""" Sample boto3 code to create a bucket on AWS S3. Note: Note the "pytestmark" directive "aws" marker. *Filename: README.rst* .. pytestmark: aws .. code-block:: python :name: test_create_bucket import boto3 s3 = boto3.client("s3", region_name="us-east-1") s3.create_bucket(Bucket="my-bucket") assert "my-bucket" in [b["Name"] for b in s3.list_buckets()["Buckets"]] ====================================================================== Add "openai" marker """"""""""""""""""" Sample openai code to ask LLM to tell a joke. Note, that next to a custom "openai" marker, "xfail" marker is used, which allows underlying code to fail, without marking entire test suite as failed. Note: Note the "pytestmark" directive "xfail" and "openai" markers. *Filename: README.rst* .. pytestmark: xfail .. pytestmark: openai .. code-block:: python :name: test_tell_me_a_joke from openai import OpenAI client = OpenAI() completion = client.chat.completions.create( model="gpt-4o", messages=[ {"role": "developer", "content": "You are a famous comedian."}, {"role": "user", "content": "Tell me a joke."}, ], ) assert isinstance(completion.choices[0].message.content, str) ====================================================================== Implement pytest hooks ~~~~~~~~~~~~~~~~~~~~~~ In the example below: * moto is used to mock AWS S3 service for all tests marked as "aws". * Environment variable "OPENAI_BASE_URL" is set to "http://localhost:11434/v1" (assuming you have Ollama running) for all tests marked as "openai". * "FILE_REGISTRY.clean_up()" is executed at the end of each test marked as "fakepy". *Filename: conftest.py* import os from contextlib import suppress import pytest from fake import FILE_REGISTRY from moto import mock_aws from pytest_codeblock.constants import CODEBLOCK_MARK # Modify test item during collection def pytest_collection_modifyitems(config, items): for item in items: if item.get_closest_marker(CODEBLOCK_MARK): # All `pytest-codeblock` tests are automatically assigned # a `codeblock` marker, which can be used for customisation. # In the example below we add an additional `documentation` # marker to `pytest-codeblock` tests. item.add_marker(pytest.mark.documentation) if item.get_closest_marker("aws"): # Apply `mock_aws` to all tests marked as `aws` item.obj = mock_aws(item.obj) # Setup before test runs def pytest_runtest_setup(item): if item.get_closest_marker("openai"): # Send all OpenAI requests to locally running Ollama for all # tests marked as `openai`. The tests would x-pass on environments # where Ollama is up and running (assuming, you have created an # alias for gpt-4o using one of the available models) and would # x-fail on environments, where Ollama isn't runnig. os.environ.setdefault("OPENAI_API_KEY", "ollama") os.environ.setdefault("OPENAI_BASE_URL", "http://localhost:11434/v1") # Teardown after the test ends def pytest_runtest_teardown(item, nextitem): # Run file clean up on all tests marked as `fakepy` if item.get_closest_marker("fakepy"): FILE_REGISTRY.clean_up() ====================================================================== Markdown ======== Usage examples -------------- Any fenced code block with a recognized Python language tag (e.g., "python", "py") will be collected and executed automatically, if your pytest configuration allows that. Standalone code blocks ~~~~~~~~~~~~~~~~~~~~~~ Note: Note that "name" value has a "test_" prefix. *Filename: README.md* ```python name=test_basic_example import math result = math.pow(3, 2) assert result == 9 ``` ====================================================================== Grouping multiple code blocks ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ It's possible to split one logical test into multiple blocks. They will be tested under the first "name" specified. Note the "" directive. Note: Note that "continue" directive of the "test_grouping_example_part_2" and "test_grouping_example_part_3" refers to the "test_grouping_example". *Filename: README.md* ```python name=test_grouping_example x = 1 ``` Some intervening text. ```python name=test_grouping_example_part_2 y = x + 1 # Uses x from the first snippet assert y == 2 ``` Some intervening text. ```python name=test_grouping_example_part_3 print(y) # Uses y from the previous snippet ``` The above mentioned three snippets will run as a single test. ====================================================================== Async ~~~~~ You can use *top-level await* in your code blocks. The code will be automatically wrapped in an async function. *Filename: README.md* ```python name=test_async_example import asyncio result = await asyncio.sleep(0.1, result=42) assert result == 42 ``` ====================================================================== Adding pytest markers to code blocks ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ It's possible to add custom pytest markers to your code blocks. That allows adding custom logic and mocking in your "conftest.py". In the example below, "django_db" marker is added to the code block. Note: Note the "pytestmark" directive "django_db" marker. *Filename: README.md* ```python name=test_django from django.contrib.auth.models import User user = User.objects.first() ``` Requesting pytest fixtures for code blocks ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ It's possible to request existing or custom pytest fixtures for code blocks. That allows adding custom logic and mocking in "conftest.py". In the example below, "tmp_path" fixture is requested for the code block. Note: Note the "pytestfixture" directive "tmp_path" fixture. *Filename: README.md* ```python name=test_path d = tmp_path / "sub" d.mkdir() # Create the directory assert d.is_dir() # Verify it was created and is a directory ``` ====================================================================== Multiple "pytestfixture" directives are supported. Add one on each line. Note: When combining "pytestfixture" and "continue" directives together, request pytest-fixtures only in the first "code-block", as they will automatically become available in all continuing blocks. Custom pytest-fixtures are supported as well. Just define them in your "conftest.py" file. Customisation/hooks ------------------- Tests can be extended and fine-tuned using pytest's standard hook system. Below is an example workflow: 1. **Add custom pytest markers** to the code blocks ("fakepy", "aws", "openai"). 2. **Implement pytest hooks** in "conftest.py" to react to those markers. Add custom pytest markers ~~~~~~~~~~~~~~~~~~~~~~~~~ Add "fakepy" marker """"""""""""""""""" The example code below will generate a PDF file with random text using fake.py library. Note, that a "fakepy" marker is added to the code block. In the >>`Implement pytest hooks`_<< section, you will see what can be done with the markers. Note: Note the "pytestmark" directive "fakepy" marker. *Filename: README.md* ```python name=test_create_pdf_file from fake import FAKER FAKER.pdf_file() ``` Add "aws" marker """""""""""""""" Sample boto3 code to create a bucket on AWS S3. Note: Note the "pytestmark" directive "aws" marker. *Filename: README.md* ```python name=test_create_bucket import boto3 s3 = boto3.client("s3", region_name="us-east-1") s3.create_bucket(Bucket="my-bucket") assert "my-bucket" in [b["Name"] for b in s3.list_buckets()["Buckets"]] ``` Add "openai" marker """"""""""""""""""" Sample openai code to ask LLM to tell a joke. Note, that next to a custom "openai" marker, "xfail" marker is used, which allows underlying code to fail, without marking entire test suite as failed. Note: Note the "pytestmark" directive "xfail" and "openai" markers. *Filename: README.md* ```python name=test_tell_me_a_joke from openai import OpenAI client = OpenAI() completion = client.chat.completions.create( model="gpt-4o", messages=[ {"role": "developer", "content": "You are a famous comedian."}, {"role": "user", "content": "Tell me a joke."}, ], ) assert isinstance(completion.choices[0].message.content, str) ``` ====================================================================== Implement pytest hooks ~~~~~~~~~~~~~~~~~~~~~~ In the example below: * moto is used to mock AWS S3 service for all tests marked as "aws". * Environment variable "OPENAI_BASE_URL" is set to "http://localhost:11434/v1" (assuming you have Ollama running) for all tests marked as "openai". * "FILE_REGISTRY.clean_up()" is executed at the end of each test marked as "fakepy". *Filename: conftest.py* import os from contextlib import suppress import pytest from fake import FILE_REGISTRY from moto import mock_aws from pytest_codeblock.constants import CODEBLOCK_MARK # Modify test item during collection def pytest_collection_modifyitems(config, items): for item in items: if item.get_closest_marker(CODEBLOCK_MARK): # All `pytest-codeblock` tests are automatically assigned # a `codeblock` marker, which can be used for customisation. # In the example below we add an additional `documentation` # marker to `pytest-codeblock` tests. item.add_marker(pytest.mark.documentation) if item.get_closest_marker("aws"): # Apply `mock_aws` to all tests marked as `aws` item.obj = mock_aws(item.obj) # Setup before test runs def pytest_runtest_setup(item): if item.get_closest_marker("openai"): # Send all OpenAI requests to locally running Ollama for all # tests marked as `openai`. The tests would x-pass on environments # where Ollama is up and running (assuming, you have created an # alias for gpt-4o using one of the available models) and would # x-fail on environments, where Ollama isn't runnig. os.environ.setdefault("OPENAI_API_KEY", "ollama") os.environ.setdefault("OPENAI_BASE_URL", "http://localhost:11434/v1") # Teardown after the test ends def pytest_runtest_teardown(item, nextitem): # Run file clean up on all tests marked as `fakepy` if item.get_closest_marker("fakepy"): FILE_REGISTRY.clean_up() ====================================================================== Customisation ============= It's possible to customise which codeblock languages and file extensions are recognised by the plugin. ====================================================================== Languages --------- By default, the plugin recognises the following codeblock languages: * reStructuredText: *python*, *py*, *python3* * Markdown: *python*, *py*, *python3* reStructruredText ~~~~~~~~~~~~~~~~~ For reStructruredText defaults are configured via *rst_codeblocks* setting in the *[tool.pytest-codeblock]* section of your *pyproject.toml*. [tool.pytest-codeblock] rst_codeblocks = ["python", "py", "python3"] Note: Don't touch the defaults, unless you want to remove certain options. If you only want to add custom codeblock languages, use *rst_user_codeblocks*. The following example adds *c_py* as a custom codeblock language: [tool.pytest-codeblock] rst_user_codeblocks = ["c_py"] Now the following codeblock will be recognised and executed: .. code-block:: c_py :name: test_c_py_example print("This is a custom Python codeblock") Markdown ~~~~~~~~ For Markdown defaults configured via *md_codeblocks* setting in the *[tool.pytest-codeblock]* section of your *pyproject.toml*. [tool.pytest-codeblock] md_codeblocks = ["python", "py", "python3"] Note: Don't touch the defaults, unless you want to remove certain options. If you only want to add custom codeblock languages, use *md_user_codeblocks*. The following example adds *c_py* as a custom codeblock language: [tool.pytest-codeblock] md_user_codeblocks = ["c_py"] Now the following codeblock will be recognised and executed: ```c_py name=test_c_py_example print("This is a custom Python codeblock") ``` ====================================================================== Extensions ---------- Note: If you customise both reStructuredText and Markdown configurations, make sure to avoid overlapping file extensions. reStructruredText ~~~~~~~~~~~~~~~~~ By default, the plugin recognises the following file extensions for reStructuredText files: *.rst* These defaults are configured via *rst_extensions* setting in the *[tool.pytest-codeblock]* section of your *pyproject.toml*. [tool.pytest-codeblock] rst_extensions = [".rst"] Note: Don't touch the defaults, unless you want to remove certain options. If you only want to add custom file extensions, use *rst_user_extensions*. The following example adds *.rst.txt* as a custom reStructuredText file extension: [tool.pytest-codeblock] rst_user_extensions = [".rst.txt"] Now the following file will be recognised and processed: *Filename: example.rst.txt* .. code-block:: python :name: test_custom_rst_extension_example print("Custom .rst.txt extension example executed successfully!") Markdown ~~~~~~~~ By default, the plugin recognises the following file extensions for Markdown files: *.md*, *.markdown* These defaults are configured via *md_extensions* setting in the *[tool.pytest-codeblock]* section of your *pyproject.toml*. [tool.pytest-codeblock] md_extensions = [".md", ".markdown"] Note: Don't touch the defaults, unless you want to remove certain options. If you only want to add custom file extensions, use *md_user_extensions*. The following example adds *.md.txt* as a custom Markdown file extension: [tool.pytest-codeblock] md_user_extensions = [".md.txt"] Now the following file will be recognised and processed: *Filename: example.md.txt* ```python name=test_custom_md_extension_example print("Custom .md.txt extension example executed successfully!") ``` ====================================================================== Package ======= Indices and tables ================== * Index * Module Index * Search Page ====================================================================== Project source-tree =================== Below is the layout of our project (to 10 levels), followed by the contents of each key file. Project directory layout pytest-codeblock/ ├── docs │ ├── _static │ ├── _templates │ ├── _implement_pytest_hooks.rst │ ├── changelog.rst │ ├── code_of_conduct.rst │ ├── conf.py │ ├── conf.py.distrib │ ├── contributor_guidelines.rst │ ├── customisation.rst │ ├── documentation.rst │ ├── index.rst │ ├── index.rst.distrib │ ├── llms.rst │ ├── make.bat │ ├── Makefile │ ├── markdown.rst │ ├── package.rst │ ├── requirements.txt │ ├── restructured_text.rst │ ├── security.rst │ └── source_tree.rst ├── examples │ ├── customisation_example │ │ ├── pyproject.toml │ │ ├── test.md │ │ ├── test.md.txt │ │ ├── test.rst │ │ └── test.rst.txt │ ├── md_example │ │ ├── customisation.md │ │ └── README.md │ ├── python │ │ ├── __init__.py │ │ ├── basic_example.py │ │ ├── create_bucket_example.py │ │ ├── create_pdf_file_example.py │ │ ├── django_example.py │ │ └── tell_me_a_joke_example.py │ └── rst_example │ ├── __pycache__ │ ├── __init__.py │ ├── customisation.rst │ ├── django_settings.py │ └── README.rst ├── scripts │ └── generate_project_source_tree.py ├── src │ └── pytest_codeblock │ ├── __pycache__ │ ├── tests │ │ ├── __pycache__ │ │ ├── __init__.py │ │ ├── conftest.py │ │ ├── test_customisation.py │ │ ├── test_integration.py │ │ ├── test_pytest_codeblock.py │ │ ├── tests.md │ │ └── tests.rst │ ├── __init__.py │ ├── collector.py │ ├── config.py │ ├── constants.py │ ├── md.py │ └── rst.py docs/_implement_pytest_hooks.rst -------------------------------- docs/_implement_pytest_hooks.rst In the example below: - `moto`_ is used to mock AWS S3 service for all tests marked as ``aws``. - Environment variable ``OPENAI_BASE_URL`` is set to ``http://localhost:11434/v1`` (assuming you have `Ollama`_ running) for all tests marked as ``openai``. - ``FILE_REGISTRY.clean_up()`` is executed at the end of each test marked as ``fakepy``. *Filename: conftest.py* .. code-block:: python import os from contextlib import suppress import pytest from fake import FILE_REGISTRY from moto import mock_aws from pytest_codeblock.constants import CODEBLOCK_MARK # Modify test item during collection def pytest_collection_modifyitems(config, items): for item in items: if item.get_closest_marker(CODEBLOCK_MARK): # All `pytest-codeblock` tests are automatically assigned # a `codeblock` marker, which can be used for customisation. # In the example below we add an additional `documentation` # marker to `pytest-codeblock` tests. item.add_marker(pytest.mark.documentation) if item.get_closest_marker("aws"): # Apply `mock_aws` to all tests marked as `aws` item.obj = mock_aws(item.obj) # Setup before test runs def pytest_runtest_setup(item): if item.get_closest_marker("openai"): # Send all OpenAI requests to locally running Ollama for all # tests marked as `openai`. The tests would x-pass on environments # where Ollama is up and running (assuming, you have created an # alias for gpt-4o using one of the available models) and would # x-fail on environments, where Ollama isn't runnig. os.environ.setdefault("OPENAI_API_KEY", "ollama") os.environ.setdefault("OPENAI_BASE_URL", "http://localhost:11434/v1") # Teardown after the test ends def pytest_runtest_teardown(item, nextitem): # Run file clean up on all tests marked as `fakepy` if item.get_closest_marker("fakepy"): FILE_REGISTRY.clean_up() docs/changelog.rst ------------------ docs/changelog.rst .. include:: ../CHANGELOG.rst docs/code_of_conduct.rst ------------------------ docs/code_of_conduct.rst .. include:: ../CODE_OF_CONDUCT.rst docs/conf.py ------------ docs/conf.py # Configuration file for the Sphinx documentation builder. # # For the full list of built-in configuration values, see the documentation: # https://www.sphinx-doc.org/en/master/usage/configuration.html import os import sys sys.path.insert(0, os.path.abspath(os.path.join("..", "src"))) # -- Project information ----------------------------------------------------- # https://www.sphinx-doc.org/en/master/usage/configuration.html#project-information try: import pytest_codeblock version = pytest_codeblock.__version__ project = pytest_codeblock.__title__ copyright = pytest_codeblock.__copyright__ author = pytest_codeblock.__author__ except ImportError: version = "0.1" project = "pytest-codeblock" copyright = "2025, Artur Barseghyan " author = "Artur Barseghyan " # -- General configuration --------------------------------------------------- # https://www.sphinx-doc.org/en/master/usage/configuration.html#general-configuration extensions = [ "sphinx.ext.autodoc", "sphinx.ext.viewcode", "sphinx.ext.todo", "sphinx_no_pragma", "sphinx_llms_txt_link", ] templates_path = ["_templates"] exclude_patterns = ["_build", "Thumbs.db", ".DS_Store"] language = "en" release = version # The suffix of source filenames. source_suffix = { ".rst": "restructuredtext", } pygments_style = "sphinx" # -- Options for HTML output ------------------------------------------------- # https://www.sphinx-doc.org/en/master/usage/configuration.html#options-for-html-output html_theme = "sphinx_rtd_theme" html_static_path = ["_static"] # html_extra_path = ["examples"] prismjs_base = "//cdnjs.cloudflare.com/ajax/libs/prism/1.29.0" html_css_files = [ f"{prismjs_base}/themes/prism.min.css", f"{prismjs_base}/plugins/toolbar/prism-toolbar.min.css", # "https://cdn.jsdelivr.net/gh/barseghyanartur/jsphinx@1.3.4/src/css/sphinx_rtd_theme.css", "https://cdn.jsdelivr.net/gh/barseghyanartur/jsphinx/src/css/sphinx_rtd_theme.css", ] html_js_files = [ f"{prismjs_base}/prism.min.js", f"{prismjs_base}/plugins/autoloader/prism-autoloader.min.js", f"{prismjs_base}/plugins/toolbar/prism-toolbar.min.js", f"{prismjs_base}/plugins/copy-to-clipboard/prism-copy-to-clipboard.min.js", # "https://cdn.jsdelivr.net/gh/barseghyanartur/jsphinx@1.3.4/src/js/download_adapter.js", "https://cdn.jsdelivr.net/gh/barseghyanartur/jsphinx/src/js/download_adapter.js", ] # -- Options for todo extension ---------------------------------------------- # https://www.sphinx-doc.org/en/master/usage/extensions/todo.html#configuration todo_include_todos = True # -- Options for Epub output ---------------------------------------------- epub_title = project epub_author = author epub_publisher = "GitHub" epub_copyright = copyright epub_identifier = "https://github.com/barseghyanartur/pytest-codeblock" # URL or ISBN epub_scheme = "URL" # or "ISBN" epub_uid = "https://github.com/barseghyanartur/pytest-codeblock" docs/contributor_guidelines.rst ------------------------------- docs/contributor_guidelines.rst .. include:: ../CONTRIBUTING.rst docs/documentation.rst ---------------------- docs/documentation.rst Project documentation ===================== Contents: .. contents:: Table of Contents .. toctree:: :maxdepth: 2 index restructured_text markdown customisation security contributor_guidelines code_of_conduct changelog package docs/index.rst -------------- docs/index.rst .. include:: ../README.rst .. include:: documentation.rst docs/llms.rst ------------- docs/llms.rst .. include:: ../README.rst ---- .. include:: restructured_text.rst ---- .. include:: markdown.rst ---- .. include:: customisation.rst ---- .. include:: package.rst ---- .. include:: source_tree.rst docs/markdown.rst ----------------- docs/markdown.rst Markdown ======== .. External references .. _Markdown: https://daringfireball.net/projects/markdown/ .. _pytest: https://docs.pytest.org .. _Django: https://www.djangoproject.com .. _pip: https://pypi.org/project/pip/ .. _uv: https://pypi.org/project/uv/ .. _fake.py: https://github.com/barseghyanartur/fake.py .. _boto3: https://github.com/boto/boto3 .. _moto: https://github.com/getmoto/moto .. _openai: https://github.com/openai/openai-python .. _Ollama: https://github.com/ollama/ollama Usage examples -------------- Any fenced code block with a recognized Python language tag (e.g., ``python``, ``py``) will be collected and executed automatically, if your `pytest`_ :ref:`configuration ` allows that. Standalone code blocks ~~~~~~~~~~~~~~~~~~~~~~ .. note:: Note that ``name`` value has a ``test_`` prefix. *Filename: README.md* .. code-block:: markdown ```python name=test_basic_example import math result = math.pow(3, 2) assert result == 9 ``` ---- Grouping multiple code blocks ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ It's possible to split one logical test into multiple blocks. They will be tested under the first ``name`` specified. Note the ```` directive. .. note:: Note that ``continue`` directive of the ``test_grouping_example_part_2`` and ``test_grouping_example_part_3`` refers to the ``test_grouping_example``. *Filename: README.md* .. code-block:: markdown ```python name=test_grouping_example x = 1 ``` Some intervening text. ```python name=test_grouping_example_part_2 y = x + 1 # Uses x from the first snippet assert y == 2 ``` Some intervening text. ```python name=test_grouping_example_part_3 print(y) # Uses y from the previous snippet ``` The above mentioned three snippets will run as a single test. ---- Async ~~~~~ You can use `top-level await` in your code blocks. The code will be automatically wrapped in an async function. *Filename: README.md* .. code-block:: markdown ```python name=test_async_example import asyncio result = await asyncio.sleep(0.1, result=42) assert result == 42 ``` ---- Adding pytest markers to code blocks ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ It's possible to add custom pytest markers to your code blocks. That allows adding custom logic and mocking in your ``conftest.py``. In the example below, ``django_db`` marker is added to the code block. .. note:: Note the ``pytestmark`` directive ``django_db`` marker. *Filename: README.md* .. code-block:: markdown ```python name=test_django from django.contrib.auth.models import User user = User.objects.first() ``` Requesting pytest fixtures for code blocks ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ It's possible to request existing or custom pytest fixtures for code blocks. That allows adding custom logic and mocking in ``conftest.py``. In the example below, ``tmp_path`` fixture is requested for the code block. .. note:: Note the ``pytestfixture`` directive ``tmp_path`` fixture. *Filename: README.md* .. code-block:: markdown ```python name=test_path d = tmp_path / "sub" d.mkdir() # Create the directory assert d.is_dir() # Verify it was created and is a directory ``` ---- Multiple ``pytestfixture`` directives are supported. Add one on each line. .. note:: When combining ``pytestfixture`` and ``continue`` directives together, request pytest-fixtures only in the first ``code-block``, as they will automatically become available in all continuing blocks. Custom pytest-fixtures are supported as well. Just define them in your ``conftest.py`` file. Customisation/hooks ------------------- Tests can be extended and fine-tuned using `pytest`_'s standard hook system. Below is an example workflow: 1. **Add custom pytest markers** to the code blocks (``fakepy``, ``aws``, ``openai``). 2. **Implement pytest hooks** in ``conftest.py`` to react to those markers. Add custom pytest markers ~~~~~~~~~~~~~~~~~~~~~~~~~ Add ``fakepy`` marker ^^^^^^^^^^^^^^^^^^^^^ The example code below will generate a PDF file with random text using `fake.py`_ library. Note, that a ``fakepy`` marker is added to the code block. In the `Implement pytest hooks`_ section, you will see what can be done with the markers. .. note:: Note the ``pytestmark`` directive ``fakepy`` marker. *Filename: README.md* .. code-block:: markdown ```python name=test_create_pdf_file from fake import FAKER FAKER.pdf_file() ``` Add ``aws`` marker ^^^^^^^^^^^^^^^^^^ Sample `boto3`_ code to create a bucket on AWS S3. .. note:: Note the ``pytestmark`` directive ``aws`` marker. *Filename: README.md* .. code-block:: markdown ```python name=test_create_bucket import boto3 s3 = boto3.client("s3", region_name="us-east-1") s3.create_bucket(Bucket="my-bucket") assert "my-bucket" in [b["Name"] for b in s3.list_buckets()["Buckets"]] ``` Add ``openai`` marker ^^^^^^^^^^^^^^^^^^^^^ Sample `openai`_ code to ask LLM to tell a joke. Note, that next to a custom ``openai`` marker, ``xfail`` marker is used, which allows underlying code to fail, without marking entire test suite as failed. .. note:: Note the ``pytestmark`` directive ``xfail`` and ``openai`` markers. *Filename: README.md* .. code-block:: markdown ```python name=test_tell_me_a_joke from openai import OpenAI client = OpenAI() completion = client.chat.completions.create( model="gpt-4o", messages=[ {"role": "developer", "content": "You are a famous comedian."}, {"role": "user", "content": "Tell me a joke."}, ], ) assert isinstance(completion.choices[0].message.content, str) ``` ---- Implement pytest hooks ~~~~~~~~~~~~~~~~~~~~~~ .. include:: _implement_pytest_hooks.rst docs/package.rst ---------------- docs/package.rst Package ======= .. toctree:: :maxdepth: 20 fake Indices and tables ================== * :ref:`genindex` * :ref:`modindex` * :ref:`search` docs/restructured_text.rst -------------------------- docs/restructured_text.rst reStructuredText ================ .. External references .. _reStructuredText: https://docutils.sourceforge.io/rst.html .. _pytest: https://docs.pytest.org .. _Django: https://www.djangoproject.com .. _pip: https://pypi.org/project/pip/ .. _uv: https://pypi.org/project/uv/ .. _fake.py: https://github.com/barseghyanartur/fake.py .. _boto3: https://github.com/boto/boto3 .. _moto: https://github.com/getmoto/moto .. _openai: https://github.com/openai/openai-python .. _Ollama: https://github.com/ollama/ollama The following directives are supported: - ``.. code-block:: python`` - ``.. code:: python`` - ``.. codeblock-name: `` - ``.. literalinclude::`` Any code directive, such as ``.. code-block:: python``, ``.. code:: python``, ``.. literalinclude::`` or literal blocks with a preceding ``.. codeblock-name: ``, will be collected and executed automatically, if your `pytest`_ :ref:`configuration ` allows that. Usage examples -------------- Standalone code blocks ~~~~~~~~~~~~~~~~~~~~~~ ``code-block`` directive ^^^^^^^^^^^^^^^^^^^^^^^^ .. note:: Note that ``:name:`` value has a ``test_`` prefix. *Filename: README.rst* .. code-block:: rst .. code-block:: python :name: test_basic_example import math result = math.pow(3, 2) assert result == 9 ---- ``literalinclude`` directive ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ *Filename: README.rst* .. code-block:: rst .. literalinclude:: examples/python/basic_example.py :name: test_li_basic_example ---- ``codeblock-name`` directive ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ You can also use a literal block with a preceding name comment: *Filename: README.rst* .. code-block:: rst .. codeblock-name: test_grouping_example_literal_block This is a literal block:: y = 5 print(y * 2) ---- Grouping multiple ``code-block`` directives ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ It's possible to split one logical test into multiple blocks. They will be tested under the first ``:name:`` specified. Note the ``.. continue::`` directive. .. note:: Note that ``continue`` directive of the ``test_grouping_example_part_2`` and ``test_grouping_example_part_3`` refers to the ``test_grouping_example``. *Filename: README.rst* .. code-block:: rst .. code-block:: python :name: test_grouping_example x = 1 Some intervening text. .. continue: test_grouping_example .. code-block:: python :name: test_grouping_example_part_2 y = x + 1 # Uses x from the first snippet assert y == 2 Some intervening text. .. continue: test_grouping_example .. code-block:: python :name: test_grouping_example_part_3 print(y) # Uses y from the previous snippet The above mentioned three snippets will run as a single test. ---- Async ~~~~~ You can use `top-level await` in your code blocks. The code will be automatically wrapped in an async function. *Filename: README.rst* .. code-block:: rst .. code-block:: python :name: test_async_example import asyncio result = await asyncio.sleep(0.1, result=42) assert result == 42 ---- Adding pytest markers to ``code-block`` and ``literalinclude`` directives ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ It's possible to add custom pytest markers to your ``code-block`` or ``literalinclude`` directives. That allows adding custom logic and mocking in your ``conftest.py``. In the example below, ``django_db`` marker is added to the ``code-block`` directive. .. note:: Note the ``pytestmark`` directive ``django_db`` marker. *Filename: README.rst* .. code-block:: rst .. pytestmark: django_db .. code-block:: python :name: test_django from django.contrib.auth.models import User user = User.objects.first() ---- In the example below, ``django_db`` marker is added to the ``literalinclude`` directive. *Filename: README.rst* .. code-block:: rst .. pytestmark: django_db .. literalinclude:: examples/python/django_example.py :name: test_li_django_example ---- Requesting pytest fixtures for ``code-block`` and ``literalinclude`` directives ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ It's possible to request existing or custom pytest fixtures in ``code-block`` or ``literalinclude`` directives. That allows adding custom logic and mocking in ``conftest.py``. In the example below, ``tmp_path`` fixture is requested for the ``code-block`` directive. .. note:: Note the ``pytestfixture`` directive ``tmp_path`` fixture. *Filename: README.rst* .. code-block:: rst .. pytestfixture: tmp_path .. code-block:: python :name: test_path d = tmp_path / "sub" d.mkdir() # Create the directory assert d.is_dir() # Verify it was created and is a directory ---- In the example below, ``tmp_path`` fixture is requested for the ``literalinclude`` directive. *Filename: README.rst* .. code-block:: rst .. pytestfixture: tmp_path .. literalinclude:: examples/python/tmp_path_example.py :name: test_li_tmp_path_example ---- Multiple ``pytestfixture`` directives are supported. Add one on each line. .. note:: When combining ``pytestfixture`` and ``continue`` directives together, request pytest-fixtures only in the first ``code-block``, as they will automatically become available in all continuing blocks. Custom pytest-fixtures are supported as well. Just define them in your ``conftest.py`` file. Customisation/hooks ------------------- Tests can be extended and fine-tuned using `pytest`_'s standard hook system. Below is an example workflow: 1. **Add custom pytest markers** to the ``code-block`` or ``literalinclude`` (``fakepy``, ``aws``, ``openai``). 2. **Implement pytest hooks** in ``conftest.py`` to react to those markers. Add custom pytest markers ~~~~~~~~~~~~~~~~~~~~~~~~~ Add ``fakepy`` marker ^^^^^^^^^^^^^^^^^^^^^ The example code below will generate a PDF file with random text using `fake.py`_ library. Note, that a ``fakepy`` marker is added to the ``code-block``. In the `Implement pytest hooks`_ section, you will see what can be done with the markers. .. note:: Note the ``pytestmark`` directive ``fakepy`` marker. *Filename: README.rst* .. code-block:: rst .. pytestmark: fakepy .. code-block:: python :name: test_create_pdf_file from fake import FAKER FAKER.pdf_file() ---- In the example code below, a ``fakepy`` marker is added to the ``literalinclude`` block. *Filename: README.rst* .. code-block:: rst .. pytestmark: fakepy .. literalinclude:: examples/python/create_pdf_file_example.py :name: test_li_create_pdf_file ---- Add ``aws`` marker ^^^^^^^^^^^^^^^^^^ Sample `boto3`_ code to create a bucket on AWS S3. .. note:: Note the ``pytestmark`` directive ``aws`` marker. *Filename: README.rst* .. code-block:: rst .. pytestmark: aws .. code-block:: python :name: test_create_bucket import boto3 s3 = boto3.client("s3", region_name="us-east-1") s3.create_bucket(Bucket="my-bucket") assert "my-bucket" in [b["Name"] for b in s3.list_buckets()["Buckets"]] ---- Add ``openai`` marker ^^^^^^^^^^^^^^^^^^^^^ Sample `openai`_ code to ask LLM to tell a joke. Note, that next to a custom ``openai`` marker, ``xfail`` marker is used, which allows underlying code to fail, without marking entire test suite as failed. .. note:: Note the ``pytestmark`` directive ``xfail`` and ``openai`` markers. *Filename: README.rst* .. code-block:: rst .. pytestmark: xfail .. pytestmark: openai .. code-block:: python :name: test_tell_me_a_joke from openai import OpenAI client = OpenAI() completion = client.chat.completions.create( model="gpt-4o", messages=[ {"role": "developer", "content": "You are a famous comedian."}, {"role": "user", "content": "Tell me a joke."}, ], ) assert isinstance(completion.choices[0].message.content, str) ---- Implement pytest hooks ~~~~~~~~~~~~~~~~~~~~~~ .. include:: _implement_pytest_hooks.rst docs/security.rst ----------------- docs/security.rst .. include:: ../SECURITY.rst docs/source_tree.rst -------------------- docs/source_tree.rst Project source-tree =================== Below is the layout of our project (to 10 levels), followed by the contents of each key file. .. code-block:: bash :caption: Project directory layout pytest-codeblock/ ├── docs │ ├── _static │ ├── _templates │ ├── _implement_pytest_hooks.rst │ ├── changelog.rst │ ├── code_of_conduct.rst │ ├── conf.py │ ├── conf.py.distrib │ ├── contributor_guidelines.rst │ ├── customisation.rst │ ├── documentation.rst │ ├── index.rst │ ├── index.rst.distrib │ ├── llms.rst │ ├── make.bat │ ├── Makefile │ ├── markdown.rst │ ├── package.rst │ ├── requirements.txt │ ├── restructured_text.rst │ ├── security.rst │ └── source_tree.rst ├── examples │ ├── customisation_example │ │ ├── pyproject.toml │ │ ├── test.md │ │ ├── test.md.txt │ │ ├── test.rst │ │ └── test.rst.txt │ ├── md_example │ │ ├── customisation.md │ │ └── README.md │ ├── python │ │ ├── __init__.py │ │ ├── basic_example.py │ │ ├── create_bucket_example.py │ │ ├── create_pdf_file_example.py │ │ ├── django_example.py │ │ └── tell_me_a_joke_example.py │ └── rst_example │ ├── __pycache__ │ ├── __init__.py │ ├── customisation.rst │ ├── django_settings.py │ └── README.rst ├── scripts │ └── generate_project_source_tree.py ├── src │ └── pytest_codeblock │ ├── __pycache__ │ ├── tests │ │ ├── __pycache__ │ │ ├── __init__.py │ │ ├── conftest.py │ │ ├── test_customisation.py │ │ ├── test_integration.py │ │ ├── test_pytest_codeblock.py │ │ ├── tests.md │ │ └── tests.rst │ ├── __init__.py │ ├── collector.py │ ├── config.py │ ├── constants.py │ ├── md.py │ └── rst.py docs/_implement_pytest_hooks.rst -------------------------------- .. literalinclude:: _implement_pytest_hooks.rst :language: rst :caption: docs/_implement_pytest_hooks.rst docs/changelog.rst ------------------ .. literalinclude:: changelog.rst :language: rst :caption: docs/changelog.rst docs/code_of_conduct.rst ------------------------ .. literalinclude:: code_of_conduct.rst :language: rst :caption: docs/code_of_conduct.rst docs/conf.py ------------ .. literalinclude:: conf.py :language: python :caption: docs/conf.py docs/contributor_guidelines.rst ------------------------------- .. literalinclude:: contributor_guidelines.rst :language: rst :caption: docs/contributor_guidelines.rst docs/documentation.rst ---------------------- .. literalinclude:: documentation.rst :language: rst :caption: docs/documentation.rst docs/index.rst -------------- .. literalinclude:: index.rst :language: rst :caption: docs/index.rst docs/llms.rst ------------- .. literalinclude:: llms.rst :language: rst :caption: docs/llms.rst docs/markdown.rst ----------------- .. literalinclude:: markdown.rst :language: rst :caption: docs/markdown.rst docs/package.rst ---------------- .. literalinclude:: package.rst :language: rst :caption: docs/package.rst docs/restructured_text.rst -------------------------- .. literalinclude:: restructured_text.rst :language: rst :caption: docs/restructured_text.rst docs/security.rst ----------------- .. literalinclude:: security.rst :language: rst :caption: docs/security.rst docs/source_tree.rst -------------------- .. literalinclude:: source_tree.rst :language: rst :caption: docs/source_tree.rst examples/md_example/README.md ----------------------------- .. literalinclude:: ../examples/md_example/README.md :language: markdown :caption: examples/md_example/README.md examples/md_example/customisation.md ------------------------------------ .. literalinclude:: ../examples/md_example/customisation.md :language: markdown :caption: examples/md_example/customisation.md examples/python/__init__.py --------------------------- .. literalinclude:: ../examples/python/__init__.py :language: python :caption: examples/python/__init__.py examples/python/basic_example.py -------------------------------- .. literalinclude:: ../examples/python/basic_example.py :language: python :caption: examples/python/basic_example.py examples/python/create_bucket_example.py ---------------------------------------- .. literalinclude:: ../examples/python/create_bucket_example.py :language: python :caption: examples/python/create_bucket_example.py examples/python/create_pdf_file_example.py ------------------------------------------ .. literalinclude:: ../examples/python/create_pdf_file_example.py :language: python :caption: examples/python/create_pdf_file_example.py examples/python/django_example.py --------------------------------- .. literalinclude:: ../examples/python/django_example.py :language: python :caption: examples/python/django_example.py examples/python/tell_me_a_joke_example.py ----------------------------------------- .. literalinclude:: ../examples/python/tell_me_a_joke_example.py :language: python :caption: examples/python/tell_me_a_joke_example.py examples/rst_example/README.rst ------------------------------- .. literalinclude:: ../examples/rst_example/README.rst :language: rst :caption: examples/rst_example/README.rst examples/rst_example/__init__.py -------------------------------- .. literalinclude:: ../examples/rst_example/__init__.py :language: python :caption: examples/rst_example/__init__.py examples/rst_example/customisation.rst -------------------------------------- .. literalinclude:: ../examples/rst_example/customisation.rst :language: rst :caption: examples/rst_example/customisation.rst examples/rst_example/django_settings.py --------------------------------------- .. literalinclude:: ../examples/rst_example/django_settings.py :language: python :caption: examples/rst_example/django_settings.py scripts/generate_project_source_tree.py --------------------------------------- .. literalinclude:: ../scripts/generate_project_source_tree.py :language: python :caption: scripts/generate_project_source_tree.py src/pytest_codeblock/__init__.py -------------------------------- .. literalinclude:: ../src/pytest_codeblock/__init__.py :language: python :caption: src/pytest_codeblock/__init__.py src/pytest_codeblock/collector.py --------------------------------- .. literalinclude:: ../src/pytest_codeblock/collector.py :language: python :caption: src/pytest_codeblock/collector.py src/pytest_codeblock/constants.py --------------------------------- .. literalinclude:: ../src/pytest_codeblock/constants.py :language: python :caption: src/pytest_codeblock/constants.py src/pytest_codeblock/md.py -------------------------- .. literalinclude:: ../src/pytest_codeblock/md.py :language: python :caption: src/pytest_codeblock/md.py src/pytest_codeblock/rst.py --------------------------- .. literalinclude:: ../src/pytest_codeblock/rst.py :language: python :caption: src/pytest_codeblock/rst.py src/pytest_codeblock/tests/__init__.py -------------------------------------- .. literalinclude:: ../src/pytest_codeblock/tests/__init__.py :language: python :caption: src/pytest_codeblock/tests/__init__.py src/pytest_codeblock/tests/test_pytest_codeblock.py --------------------------------------------------- .. literalinclude:: ../src/pytest_codeblock/tests/test_pytest_codeblock.py :language: python :caption: src/pytest_codeblock/tests/test_pytest_codeblock.py src/pytest_codeblock/tests/tests.rst ------------------------------------ .. literalinclude:: ../src/pytest_codeblock/tests/tests.rst :language: rst :caption: src/pytest_codeblock/tests/tests.rst examples/md_example/README.md ----------------------------- examples/md_example/README.md # Markdown example project This is a minimal example showing how pytest-codeblock will discover and run only Python snippets whose `name` starts with test_. ## Simple assertion ```python name=test_simple_assert # A trivial test that always passes assert 2 + 2 == 4 ``` ## Multi-part example It's possible to split one logical test into multiple blocks. All of them share the same ``name``: ```python name=test_compute_square import math ``` Some intervening text. ```python name=test_compute_square result = math.pow(3, 2) assert result == 9 ``` Some intervening text. ```python name=test_compute_square print(result) ``` ## Ignored snippets Blocks without a `name` or without the `test_` prefix are **not** collected: ```python name=example_not_test # Name does not start with `test_`, so this is ignored ``` ## Non-Python blocks are also ignored ```bash name=test_should_be_ignored echo "Not Python → skipped" ``` ## Custom pytest marks ```python name=test_django from django.contrib.auth.models import User user = User.objects.first() ``` examples/md_example/customisation.md ------------------------------------ examples/md_example/customisation.md # Customisation Customisation examples. ## External references - [openai](https://github.com/openai/openai-python) - [moto](https://docs.getmoto.org) - [fake.py](https://github.com/barseghyanartur/fake.py) ## `fake.py` example ```python name=test_create_pdf_file from fake import FAKER FAKER.pdf_file() ``` ## `moto` example ```python name=test_create_bucket import boto3 s3 = boto3.client("s3", region_name="us-east-1") s3.create_bucket(Bucket="my-bucket") assert "my-bucket" in [b["Name"] for b in s3.list_buckets()["Buckets"]] ``` ## `openai` example ```python name=test_tell_me_a_joke from openai import OpenAI client = OpenAI() completion = client.chat.completions.create( model="gpt-4o", messages=[ {"role": "developer", "content": "You are a famous comedian."}, {"role": "user", "content": "Tell me a joke."}, ], ) assert isinstance(completion.choices[0].message.content, str) ``` examples/python/__init__.py --------------------------- examples/python/__init__.py examples/python/basic_example.py -------------------------------- examples/python/basic_example.py import math result = math.pow(3, 2) assert result == 9 examples/python/create_bucket_example.py ---------------------------------------- examples/python/create_bucket_example.py import boto3 s3 = boto3.client("s3", region_name="us-east-1") s3.create_bucket(Bucket="my-bucket") assert "my-bucket" in [b["Name"] for b in s3.list_buckets()["Buckets"]] examples/python/create_pdf_file_example.py ------------------------------------------ examples/python/create_pdf_file_example.py from fake import FAKER file = FAKER.pdf_file() assert file.data["storage"].exists(str(file)) examples/python/django_example.py --------------------------------- examples/python/django_example.py from django.contrib.auth.models import User user = User.objects.first() examples/python/tell_me_a_joke_example.py ----------------------------------------- examples/python/tell_me_a_joke_example.py from openai import OpenAI client = OpenAI() completion = client.chat.completions.create( model="gpt-4o", messages=[ {"role": "developer", "content": "You are a famous comedian."}, {"role": "user", "content": "Tell me a joke."}, ], ) assert isinstance(completion.choices[0].message.content, str) examples/rst_example/README.rst ------------------------------- examples/rst_example/README.rst reStructuredText example project ================================ This is a minimal example showing how `pytest-codeblock` will discover and run only Python snippets whose `:name:` starts with `test_`. Simple assertion ---------------- .. code-block:: python :name: test_simple_assert # A trivial test that always passes assert 2 + 2 == 4 Multi-part example ------------------ It's possible to split one logical test into multiple blocks. They will be tested under the first ``:name:`` specified. Note the ``.. continue::`` directive. .. code-block:: python :name: test_compute_square import math Some intervening text. .. continue: test_compute_square .. code-block:: python :name: test_compute_square_part_2 result = math.pow(3, 2) assert result == 9 Some intervening text. .. continue: test_compute_square .. code-block:: python :name: test_compute_square_part_3 print(result) Ignored snippets ---------------- Blocks without a `:name:` or without the `test_` prefix are **not** collected: .. code-block:: python # No :name:, so this is ignored .. code-block:: python :name: example_not_test # Name does not start with `test_`, so this is ignored Non-Python blocks are also ignored ---------------------------------- .. code-block:: bash :name: test_should_be_ignored echo "Not Python → skipped" Custom pytest marks ------------------- .. pytestmark: django_db .. code-block:: python :name: test_django from django.contrib.auth.models import User user = User.objects.first() examples/rst_example/__init__.py -------------------------------- examples/rst_example/__init__.py examples/rst_example/customisation.rst -------------------------------------- examples/rst_example/customisation.rst Customisation ============= Customisation examples. .. External references .. _openai: https://github.com/openai/openai-python .. _moto: https://docs.getmoto.org .. _fake.py: https://github.com/barseghyanartur/fake.py `fake.py`_ example for `.. code-block::` directive ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. code-block:: rst .. pytestmark: fakepy .. code-block:: python :name: test_create_pdf_file from fake import FAKER file = FAKER.pdf_file() assert file.data["storage"].exists(str(file)) `moto`_ example for `.. code-block::` directive ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. code-block:: rst .. pytestmark: aws .. code-block:: python :name: test_create_bucket import boto3 s3 = boto3.client("s3", region_name="us-east-1") s3.create_bucket(Bucket="my-bucket") assert "my-bucket" in [b["Name"] for b in s3.list_buckets()["Buckets"]] `openai`_ example for `.. code-block::` directive ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. code-block:: rst .. pytestmark: xfail .. pytestmark: openai .. code-block:: python :name: test_tell_me_a_joke from openai import OpenAI client = OpenAI() completion = client.chat.completions.create( model="gpt-4o", messages=[ {"role": "developer", "content": "You are a famous comedian."}, {"role": "user", "content": "Tell me a joke."}, ], ) assert isinstance(completion.choices[0].message.content, str) `fake.py`_ example for `.. literalinclude::` directive ------------------------------------------------------ .. code-block:: rst .. pytestmark: fakepy .. literalinclude:: examples/python/create_pdf_file_example.py :name: test_li_create_pdf_file `moto`_ example for `.. literalinclude::` directive --------------------------------------------------- .. code-block:: rst .. pytestmark: aws .. literalinclude:: examples/python/create_bucket_example.py :name: test_li_create_bucket `openai`_ example for `.. literalinclude::` directive ----------------------------------------------------- .. code-block:: rst .. pytestmark: xfail .. pytestmark: aws .. literalinclude:: examples/python/tell_me_a_joke_example.py :name: test_li_tell_me_a_joke examples/rst_example/django_settings.py --------------------------------------- examples/rst_example/django_settings.py from pathlib import Path from django import conf, http, urls from django.core.handlers import asgi # Build paths inside the project like this: BASE_DIR / 'subdir'. BASE_DIR = Path(__file__).resolve().parent # SECURITY WARNING: don't run with debug turned on in production! DEBUG = True INSTALLED_APPS = [ "django.contrib.admin", "django.contrib.auth", "django.contrib.contenttypes", "django.contrib.sessions", "django.contrib.messages", "django.contrib.staticfiles", ] DATABASES = { "default": { "ENGINE": "django.db.backends.sqlite3", "NAME": BASE_DIR / "django_db.sqlite3", } } conf.settings.configure( ALLOWED_HOSTS="*", ROOT_URLCONF=__name__, INSTALLED_APPS=INSTALLED_APPS, DATABASES=DATABASES, ) app = asgi.ASGIHandler() async def root(_request: http.HttpRequest) -> http.JsonResponse: return http.JsonResponse({"message": "OK"}) urlpatterns = [urls.path("", root)] scripts/generate_project_source_tree.py --------------------------------------- scripts/generate_project_source_tree.py #!/usr/bin/env python3 import argparse import fnmatch import os from pathlib import Path def build_tree( path: Path, max_depth: int, ignore_patterns: list, whitelist_dirs: list, include_all: bool, root: Path, prefix: str = "", ) -> str: """ Recursively build an ASCII tree up to max_depth, applying whitelist and ignore rules. """ if max_depth < 0: return "" entries = sorted( path.iterdir(), key=lambda p: (p.is_file(), p.name.lower()) ) lines = [] for i, entry in enumerate(entries): rel_path = entry.relative_to(root).as_posix() # Skip ignored patterns if any(fnmatch.fnmatch(rel_path, pat) for pat in ignore_patterns): continue # Enforce whitelist if not including all if not include_all and whitelist_dirs and not any( rel_path.startswith(w.rstrip("/")) for w in whitelist_dirs ): continue connector = "└── " if i == len(entries) - 1 else "├── " lines.append(f"{prefix}{connector}{entry.name}") # Recurse into directories if entry.is_dir(): extension = " " if i == len(entries) - 1 else "│ " subtree = build_tree( entry, max_depth - 1, ignore_patterns, whitelist_dirs, include_all, root, prefix + extension, ) if subtree: lines += subtree.splitlines() return "\n".join(lines) def detect_language(path: Path) -> str: """Map file suffix to Sphinx language.""" mapping = { ".py": "python", ".js": "javascript", ".java": "java", ".md": "markdown", ".yaml": "yaml", ".yml": "yaml", ".json": "json", ".sh": "bash", ".rst": "rst", } return mapping.get(path.suffix, "") def main(): p = argparse.ArgumentParser( description="Auto-generate a .rst with tree + literalinclude blocks" ) p.add_argument( "-p", "--project-root", type=Path, default=Path("."), help="Path to your project directory", ) p.add_argument( "-d", "--depth", type=int, default=10, help="How many levels deep to print in the tree", ) p.add_argument( "-o", "--output", type=Path, default=Path("docs/source_tree.rst"), help="Where to write the generated .rst", ) p.add_argument( "-e", "--ext", nargs="+", default=[".py", ".md", ".js", ".rst"], help="Which file extensions to include via literalinclude", ) p.add_argument( "-i", "--ignore", nargs="+", default=["__pycache__", "*.pyc", "*.py,cover"], help="Ignore files or dirs matching these glob patterns (relative to " "project root)", ) p.add_argument( "-w", "--whitelist", nargs="+", default=["src", "docs", "examples", "scripts"], help="Directories (relative to project root) to include " "unless --include-all is given", ) p.add_argument( "--include-all", action="store_true", help="Include all files regardless of whitelist", ) args = p.parse_args() root = args.project_root.resolve() ignore_patterns = args.ignore whitelist_dirs = args.whitelist include_all = args.include_all output = args.output.resolve() output_dir = output.parent.resolve() # Header + tree header = f"""Project source-tree =================== Below is the layout of our project (to {args.depth} levels), followed by the contents of each key file. .. code-block:: bash :caption: Project directory layout {root.name}/ """ tree = build_tree( root, args.depth, ignore_patterns, whitelist_dirs, include_all, root, prefix=" ", ) out = [header, tree, ""] # Walk and collect files for filepath in sorted(root.rglob("*")): if not filepath.is_file() or filepath.suffix not in args.ext: continue rel_path = filepath.relative_to(root).as_posix() # Skip ignored if any(fnmatch.fnmatch(rel_path, pat) for pat in ignore_patterns): continue # Enforce whitelist if ( not include_all and whitelist_dirs and not any( rel_path.startswith(w.rstrip("/")) for w in whitelist_dirs) ): continue # Compute include path relative to output_dir include_path = os.path.relpath(filepath, output_dir).replace( os.sep, "/" ) title = rel_path underline = "-" * len(title) lang = detect_language(filepath) out += [ title, underline, "", f".. literalinclude:: {include_path}", f" :language: {lang}" if lang else "", f" :caption: {rel_path}", # " :linenos:", "", ] # Write output args.output.write_text("\n".join(line for line in out if line is not None)) print(f"Wrote {args.output}") if __name__ == "__main__": main() src/pytest_codeblock/__init__.py -------------------------------- src/pytest_codeblock/__init__.py from pathlib import Path from .config import get_config from .md import MarkdownFile from .rst import RSTFile __title__ = "pytest-codeblock" __version__ = "0.4" __author__ = "Artur Barseghyan " __copyright__ = "2025-2026 Artur Barseghyan" __license__ = "MIT" __all__ = ( "pytest_collect_file", ) def pytest_collect_file(parent, path): """Collect .md and .rst files for codeblock tests.""" config = get_config() # Determine file extension (works for py.path or pathlib.Path) file_name = str(path).lower() if any(file_name.endswith(ext) for ext in config.all_md_extensions): # Use the MarkdownFile collector for Markdown files return MarkdownFile.from_parent(parent=parent, path=Path(path)) if any(file_name.endswith(ext) for ext in config.all_rst_extensions): # Use the RSTFile collector for reStructuredText files return RSTFile.from_parent(parent=parent, path=Path(path)) return None src/pytest_codeblock/collector.py --------------------------------- src/pytest_codeblock/collector.py from dataclasses import dataclass, field from typing import Optional __author__ = "Artur Barseghyan " __copyright__ = "2025-2026 Artur Barseghyan" __license__ = "MIT" __all__ = ( "CodeSnippet", "group_snippets", ) @dataclass class CodeSnippet: """Data container for an extracted code snippet.""" code: str # The code content line: int # Starting line number in the source name: Optional[str] = None # Identifier for grouping (None if anonymous) marks: list[str] = field(default_factory=list) # Collected pytest marks (e.g. ['django_db']), parsed from doc comments fixtures: list[str] = field(default_factory=list) # Collected pytest fixtures (e.g. ['tmp_path']), parsed from doc comments def group_snippets(snippets: list[CodeSnippet]) -> list[CodeSnippet]: """ Merge snippets with the same name into one CodeSnippet, concatenating their code and accumulating marks. Unnamed snippets get unique auto-names. """ combined: list[CodeSnippet] = [] seen: dict[str, CodeSnippet] = {} anon_count = 0 for sn in snippets: key = sn.name if not key: anon_count += 1 key = f"codeblock{anon_count}" if key in seen: seen_sn = seen[key] seen_sn.code += "\n" + sn.code seen_sn.marks.extend(sn.marks) seen_sn.fixtures.extend(sn.fixtures) else: sn.marks = list(sn.marks) # copy sn.fixtures = list(sn.fixtures) # copy seen[key] = sn combined.append(sn) return combined src/pytest_codeblock/constants.py --------------------------------- src/pytest_codeblock/constants.py __author__ = "Artur Barseghyan " __copyright__ = "2025-2026 Artur Barseghyan" __license__ = "MIT" __all__ = ( "CODEBLOCK_MARK", "DJANGO_DB_MARKS", "TEST_PREFIX", ) DJANGO_DB_MARKS = { "django_db", "db", "transactional_db", } TEST_PREFIX = "test_" CODEBLOCK_MARK = "codeblock" src/pytest_codeblock/md.py -------------------------- src/pytest_codeblock/md.py import asyncio import inspect import re import textwrap import traceback from typing import Optional import pytest from .collector import CodeSnippet, group_snippets from .config import get_config from .constants import CODEBLOCK_MARK, DJANGO_DB_MARKS, TEST_PREFIX from .helpers import contains_top_level_await, wrap_async_code __author__ = "Artur Barseghyan " __copyright__ = "2025-2026 Artur Barseghyan" __license__ = "MIT" __all__ = ( "MarkdownFile", "parse_markdown", ) def parse_markdown(text: str) -> list[CodeSnippet]: """ Parse Markdown text and extract Python code snippets as CodeSnippet objects. Supports: - comments immediately before a code fence - comments for naming - comments for grouping with a named snippet - Fenced code blocks with ```python (and optional name= in the info string) Captures each snippet's name, code, starting line, and any pytest marks. """ config = get_config() snippets: list[CodeSnippet] = [] lines = text.splitlines() pending_name: Optional[str] = None pending_continue: Optional[str] = None pending_marks: list[str] = [CODEBLOCK_MARK] pending_fixtures: list[str] = [] in_block = False fence = "" block_indent = 0 code_buffer: list[str] = [] snippet_name: Optional[str] = None start_line = 0 for idx, line in enumerate(lines, start=1): stripped = line.strip() if not in_block: # Check for pytest mark comment if stripped.startswith("", stripped) if m: pending_marks.append(m.group(1)) continue # Check for pytest fixture comment if stripped.startswith("", stripped) if m: pending_fixtures.append(m.group(1)) continue # Check for continue comment if stripped.startswith("", stripped) if m: pending_continue = m.group(1) continue # Check for name comment if stripped.startswith("", stripped ) if m: pending_name = m.group(1) continue # Start of fenced code block? if line.lstrip().startswith("```"): indent = len(line) - len(line.lstrip()) m = re.match(r"^`{3,}", line.lstrip()) if not m: continue fence = m.group(0) info = line.lstrip()[len(fence):].strip() parts = info.split(None, 1) lang = parts[0].lower() if parts else "" extra = parts[1] if len(parts) > 1 else "" if lang in config.all_md_codeblocks: in_block = True block_indent = indent start_line = idx + 1 code_buffer = [] # determine name from info string or pending comment snippet_name = None for token in extra.split(): if ( token.startswith("name=") or token.startswith("name:") ): snippet_name = ( token.split("=", 1)[-1] if "=" in token else token.split(":", 1)[-1] ) break if snippet_name is None: snippet_name = pending_name # reset pending_name; marks stay until block closes pending_name = None continue else: # inside a fenced code block if line.lstrip().startswith(fence): # end of block in_block = False code_text = "\n".join(code_buffer) # continue overrides snippet_name for grouping if pending_continue: final_name = pending_continue pending_continue = None else: final_name = snippet_name snippets.append(CodeSnippet( name=final_name, code=code_text, line=start_line, marks=pending_marks.copy(), fixtures=pending_fixtures.copy(), )) # reset pending marks after collecting pending_marks = [CODEBLOCK_MARK] # Reset to default snippet_name = None pending_fixtures.clear() # Clear pending fixtures else: # collect code lines (dedent by block_indent) if line.strip() == "": code_buffer.append("") else: if len(line) >= block_indent: code_buffer.append(line[block_indent:]) else: code_buffer.append(line.lstrip()) continue return snippets class MarkdownFile(pytest.File): """ Collector for Markdown files, extracting only `test_`-prefixed code snippets. """ def collect(self): text = self.path.read_text(encoding="utf-8") raw = parse_markdown(text) # keep only snippets named test_* tests = [ sn for sn in raw if sn.name and sn.name.startswith(TEST_PREFIX) ] combined = group_snippets(tests) for sn in combined: # Bind the values we need so we don't close over `sn` itself _sn_name = sn.name _fpath = str(self.path) # Build list of fixture names requested by this snippet _fixture_names: list[str] = list(sn.fixtures) # If snippet is marked as needing DB, also request the `db` # fixture, unless user already added it explicitly. if ( DJANGO_DB_MARKS.intersection(sn.marks) and "db" not in _fixture_names ): _fixture_names.append("db") # Generate a real pytest Function so fixtures work def make_func( code, sn_name=_sn_name, fpath=_fpath, fixture_names=_fixture_names, ): # This inner function *actually* has a **fixtures signature, # but we override __signature__ so pytest passes the right # fixtures and names. def test_block(**fixtures): # Auto-wrap async code ex_code = code if contains_top_level_await(code): ex_code = wrap_async_code(code) try: compiled = compile(ex_code, fpath, "exec") except SyntaxError as err: raise SyntaxError( f"Syntax error in " f"codeblock `{sn_name}` in {fpath}:\n" f"\n{textwrap.indent(ex_code, prefix=' ')}\n\n" f"{traceback.format_exc()}" ) from err try: # Make fixtures available as top-level names # inside the executed snippet. exec(compiled, {"asyncio": asyncio, **dict(fixtures)}) except Exception as err: raise Exception( f"Error in " f"codeblock `{sn_name}` in {fpath}:\n" f"\n{textwrap.indent(ex_code, prefix=' ')}\n\n" f"{traceback.format_exc()}" ) from err # Tell pytest which fixture arguments this test has: test_block.__signature__ = inspect.Signature( [ inspect.Parameter( name, inspect.Parameter.POSITIONAL_OR_KEYWORD, ) for name in fixture_names ] ) return test_block callobj = make_func(sn.code) fn = pytest.Function.from_parent( parent=self, name=sn.name, callobj=callobj, ) # apply any marks (e.g. django_db) for m in sn.marks: fn.add_marker(getattr(pytest.mark, m)) yield fn src/pytest_codeblock/rst.py --------------------------- src/pytest_codeblock/rst.py import asyncio import inspect import re import textwrap import traceback from pathlib import Path from typing import Optional, Union import pytest from .collector import CodeSnippet, group_snippets from .config import get_config from .constants import CODEBLOCK_MARK, DJANGO_DB_MARKS, TEST_PREFIX from .helpers import contains_top_level_await, wrap_async_code __author__ = "Artur Barseghyan " __copyright__ = "2025-2026 Artur Barseghyan" __license__ = "MIT" __all__ = ( "RSTFile", "parse_rst", "resolve_literalinclude_path", "get_literalinclude_content", ) def resolve_literalinclude_path( base_dir: Union[str, Path], include_path: str, ) -> Optional[str]: """ Resolve the full path for a literalinclude directive. Returns None if the file doesn't exist. """ _include_path = Path(include_path) # If `include_path` is already absolute or relative and exists, done if _include_path.exists(): return str(_include_path.resolve()) # If base_path is a file, switch to its parent directory _base_path = Path(base_dir) if _base_path.is_file(): _base_path = _base_path.parent try: full_path = _base_path / include_path if full_path.exists(): return str(full_path.resolve()) except Exception: pass return None def get_literalinclude_content(path): try: with open(path) as f: return f.read() except Exception as e: raise RuntimeError( f"Failed to read literalinclude file {path}: {e}" ) from e def parse_rst(text: str, base_dir: Path) -> list[CodeSnippet]: """ Parse an RST document into CodeSnippet objects, capturing: - .. pytestmark: - .. continue: - .. codeblock-name: - .. code-block:: python """ config = get_config() snippets: list[CodeSnippet] = [] lines = text.splitlines() n = len(lines) pending_name: Optional[str] = None pending_marks: list[str] = [CODEBLOCK_MARK] pending_fixtures: list[str] = [] pending_continue: Optional[str] = None i = 0 while i < n: line = lines[i] # -------------------------------------------------------------------- # Collect `.. pytestmark: xyz` # -------------------------------------------------------------------- m = re.match(r"^\s*\.\.\s*pytestmark:\s*(\w+)\s*$", line) if m: pending_marks.append(m.group(1)) i += 1 continue # -------------------------------------------------------------------- # Collect `.. pytestfixture: foo` # -------------------------------------------------------------------- m = re.match(r"^\s*\.\.\s*pytestfixture:\s*(\w+)\s*$", line) if m: pending_fixtures.append(m.group(1)) i += 1 continue # -------------------------------------------------------------------- # The `.. literalinclude` directive # -------------------------------------------------------------------- if line.strip().startswith(".. literalinclude::"): path = line.split(".. literalinclude::", 1)[1].strip() name = None # Look ahead for name j = i + 1 while j < len(lines) and lines[j].strip(): if ":name:" in lines[j]: name = lines[j].split(":name:", 1)[1].strip() break j += 1 if name and name.startswith("test_"): full_path = resolve_literalinclude_path(base_dir, path) if full_path: snippet = CodeSnippet( code=get_literalinclude_content(full_path), line=i + 1, name=name, marks=pending_marks.copy(), fixtures=pending_fixtures.copy(), ) snippets.append(snippet) # TODO: Is this needed? # pending_marks.clear() # pending_fixtures.clear() i = j + 1 continue # -------------------------------------------------------------------- # Collect `.. continue: foo` # -------------------------------------------------------------------- m = re.match(r"^\s*\.\.\s*continue:\s*(\S+)\s*$", line) if m: pending_continue = m.group(1) i += 1 continue # -------------------------------------------------------------------- # Collect `.. codeblock-name: foo` # -------------------------------------------------------------------- m = re.match(r"^\s*\.\.\s*codeblock-name:\s*(\S+)\s*$", line) if m: pending_name = m.group(1) i += 1 continue # -------------------------------------------------------------------- # The `.. code-block` directive # -------------------------------------------------------------------- m = re.match(r"^(\s*)\.\. (?:code-block|code)::\s*(\w+)", line) if m: base_indent = len(m.group(1)) lang = m.group(2).lower() if lang in config.all_rst_codeblocks: # Parse :name: option name_val: Optional[str] = None j = i + 1 while j < n: ln = lines[j] if not ln.strip(): j += 1 continue indent = len(ln) - len(ln.lstrip()) if ln.lstrip().startswith(":") and indent > base_indent: opt = ln.lstrip() if opt.lower().startswith(":name:"): name_val = opt.split(":", 2)[2].strip().split()[0] j += 1 continue break # The j is first code line if j >= n: i = j continue first = lines[j] content_indent = len(first) - len(first.lstrip()) if content_indent <= base_indent: i = j continue # Collect code buf: list[str] = [] k = j while k < n: ln = lines[k] if not ln.strip(): buf.append("") k += 1 continue ind = len(ln) - len(ln.lstrip()) if ind >= content_indent: buf.append(ln[content_indent:]) k += 1 else: break # Decide snippet name: continue overrides name_val/pending_name if pending_continue: sn_name = pending_continue pending_continue = None else: sn_name = name_val or pending_name sn_marks = pending_marks.copy() sn_fixtures = pending_fixtures.copy() pending_name = None pending_marks.clear() pending_fixtures.clear() snippets.append(CodeSnippet( name=sn_name, code="\n".join(buf), line=j + 1, marks=sn_marks, fixtures=sn_fixtures, )) i = k continue else: i += 1 continue # -------------------------------------------------------------------- # The literal-block via "::" # -------------------------------------------------------------------- if line.rstrip().endswith("::") and pending_name: # Similar override logic if pending_continue: sn_name = pending_continue pending_continue = None else: sn_name = pending_name sn_marks = pending_marks.copy() sn_fixtures = pending_fixtures.copy() pending_name = None pending_marks.clear() pending_fixtures.clear() j = i + 1 if j < n and not lines[j].strip(): j += 1 if j >= n: i = j continue first = lines[j] content_indent = len(first) - len(first.lstrip()) buf: list[str] = [] k = j while k < n: ln = lines[k] if not ln.strip(): buf.append("") k += 1 continue ind = len(ln) - len(ln.lstrip()) if ind >= content_indent: buf.append(ln[content_indent:]) k += 1 else: break snippets.append(CodeSnippet( name=sn_name, code="\n".join(buf), line=j + 1, marks=sn_marks, fixtures=sn_fixtures, )) i = k continue i += 1 return snippets class RSTFile(pytest.File): """Collect RST code-block tests as real test functions.""" def collect(self): text = self.path.read_text(encoding="utf-8") raw = parse_rst(text, self.path) # Only keep test_* snippets tests = [ sn for sn in raw if sn.name and sn.name.startswith(TEST_PREFIX) ] combined = group_snippets(tests) for sn in combined: # Bind the values we need so we don't close over `sn` itself _sn_name = sn.name _fpath = str(self.path) # Build list of fixture names requested by this snippet _fixture_names: list[str] = list(sn.fixtures) # If snippet is marked as needing DB, also request the `db` # fixture, unless user already added it explicitly. if ( DJANGO_DB_MARKS.intersection(sn.marks) and "db" not in _fixture_names ): _fixture_names.append("db") def make_func( code, sn_name=_sn_name, fpath=_fpath, fixture_names=_fixture_names, ): # This inner function *actually* has a **fixtures signature, # but we override __signature__ so pytest passes the right # fixtures and names. def test_block(**fixtures): ex_code = code if contains_top_level_await(code): ex_code = wrap_async_code(code) try: compiled = compile(ex_code, fpath, "exec") except SyntaxError as err: raise SyntaxError( f"Syntax error in " f"codeblock `{sn_name}` in {fpath}:\n" f"\n{textwrap.indent(ex_code, prefix=' ')}\n\n" f"{traceback.format_exc()}" ) from err try: # Make fixtures available as top-level names # inside the executed snippet. exec(compiled, {"asyncio": asyncio, **dict(fixtures)}) except Exception as err: raise Exception( f"Error in " f"codeblock `{sn_name}` in {fpath}:\n" f"\n{textwrap.indent(ex_code, prefix=' ')}\n\n" f"{traceback.format_exc()}" ) from err # Tell pytest which fixture arguments this test has: test_block.__signature__ = inspect.Signature( [ inspect.Parameter( name, inspect.Parameter.POSITIONAL_OR_KEYWORD, ) for name in fixture_names ] ) return test_block callobj = make_func(sn.code) fn = pytest.Function.from_parent( parent=self, name=sn.name, callobj=callobj ) # Re-apply any pytest.mark. markers for m in sn.marks: fn.add_marker(getattr(pytest.mark, m)) yield fn src/pytest_codeblock/tests/__init__.py -------------------------------------- src/pytest_codeblock/tests/__init__.py src/pytest_codeblock/tests/test_pytest_codeblock.py --------------------------------------------------- src/pytest_codeblock/tests/test_pytest_codeblock.py from ..collector import CodeSnippet, group_snippets from ..helpers import contains_top_level_await, wrap_async_code from ..md import parse_markdown from ..rst import ( get_literalinclude_content, parse_rst, resolve_literalinclude_path, ) __author__ = "Artur Barseghyan " __copyright__ = "2025-2026 Artur Barseghyan" __license__ = "MIT" __all__ = ( "test_group_snippets_different_names", "test_group_snippets_merges_named", "test_parse_markdown_simple", "test_parse_markdown_with_pytestmark", "test_parse_rst_literalinclude", "test_parse_rst_simple", "test_resolve_literalinclude_and_content", ) def test_group_snippets_merges_named(): # Two snippets with the same name should be combined sn1 = CodeSnippet(name="foo", code="a=1", line=1, marks=["codeblock"]) sn2 = CodeSnippet(name="foo", code="b=2", line=2, marks=["codeblock", "m"]) combined = group_snippets([sn1, sn2]) assert len(combined) == 1 cs = combined[0] assert cs.name == "foo" # Both code parts should appear assert "a=1" in cs.code assert "b=2" in cs.code # Marks should accumulate assert "m" in cs.marks def test_group_snippets_different_names(): # Snippets with different names are not grouped sn1 = CodeSnippet(name="foo", code="x=1", line=1) sn2 = CodeSnippet(name="bar", code="y=2", line=2) combined = group_snippets([sn1, sn2]) assert len(combined) == 2 assert combined[0].name.startswith("foo") assert combined[1].name.startswith("bar") def test_parse_markdown_simple(): text = """ ```python name=test_example x=1 assert x==1 ```""" snippets = parse_markdown(text) assert len(snippets) == 1 sn = snippets[0] assert sn.name == "test_example" assert "x=1" in sn.code def test_parse_markdown_with_pytestmark(): text = """ ```python name=test_db from django.db import models ```""" snippets = parse_markdown(text) assert len(snippets) == 1 sn = snippets[0] # Should include both default and django_db marks assert "django_db" in sn.marks assert "codeblock" in sn.marks def test_resolve_literalinclude_and_content(tmp_path): base = tmp_path / "dir" base.mkdir() file = base / "a.py" file.write_text("print('hello')") # Absolute path resolution abs_path = resolve_literalinclude_path(base, str(file)) assert abs_path == str(file.resolve()) # Relative path resolution rel_path = resolve_literalinclude_path(base, "a.py") assert rel_path == str(file.resolve()) # Content read content = get_literalinclude_content(str(file)) assert content == "print('hello')" def test_parse_rst_simple(tmp_path): # Basic code-block directive rst = """ .. code-block:: python :name: test_simple a=2 assert a==2 """ snippets = parse_rst(rst, tmp_path) assert len(snippets) == 1 sn = snippets[0] assert sn.name == "test_simple" assert "a=2" in sn.code def test_parse_rst_literalinclude(tmp_path): # Create an external file to include include_dir = tmp_path / "inc" include_dir.mkdir() target = include_dir / "foo.py" target.write_text("z=3\nassert z==3") rst = f""" .. literalinclude:: {target.name} :name: test_li """ snippets = parse_rst(rst, include_dir) assert len(snippets) == 1 sn = snippets[0] assert sn.name == "test_li" assert "z=3" in sn.code def test_contains_top_level_await_positive(): """Verify detection of various async constructs.""" # Direct await assert contains_top_level_await("await asyncio.sleep(0)") is True # Async function definition assert contains_top_level_await("async def foo(): pass") is True # Async with assert contains_top_level_await("async with lock: pass") is True # Async for assert contains_top_level_await("async for i in range(1): pass") is True def test_contains_top_level_await_negative(): """Verify that sync code or strings containing keywords are ignored.""" # Standard sync code assert contains_top_level_await("import time; time.sleep(1)") is False # Keywords inside strings assert contains_top_level_await("print('this is an await')") is False # Comments should be ignored assert contains_top_level_await("# await inside comment") is False def test_contains_top_level_await_invalid_syntax(): """Verify that invalid syntax returns False rather than crashing.""" assert contains_top_level_await("def main(:") is False def test_wrap_async_code_structure(): """Verify the transformation logic and indentation.""" code = "await asyncio.sleep(1)\nreturn 42" wrapped = wrap_async_code(code) # Check for the boilerplate components assert "async def __async_main__():" in wrapped assert "asyncio.run(__async_main__())" in wrapped # Check that the original code is indented correctly (4 spaces) assert " await asyncio.sleep(1)" in wrapped assert " return 42" in wrapped def test_wrap_async_code_execution_integrity(): """ Verify that the wrapped code is still valid Python and can be compiled. This ensures wrap_async_code doesn't break the AST. """ code = "val = 1 + 1" wrapped = wrap_async_code(code) # If compile fails, the test fails assert compile(wrapped, "", "exec") src/pytest_codeblock/tests/tests.rst ------------------------------------ src/pytest_codeblock/tests/tests.rst Tests ===== test_group_snippets_merges_named -------------------------------- .. code-block:: python :name: test_group_snippets_merges_named import pytest from pathlib import Path from pytest_codeblock.collector import CodeSnippet, group_snippets from pytest_codeblock.md import parse_markdown from pytest_codeblock.rst import ( parse_rst, resolve_literalinclude_path, get_literalinclude_content, ) # Two snippets with the same name should be combined sn1 = CodeSnippet(name="foo", code="a=1", line=1, marks=["codeblock"]) sn2 = CodeSnippet(name="foo", code="b=2", line=2, marks=["codeblock", "m"]) combined = group_snippets([sn1, sn2]) assert len(combined) == 1 cs = combined[0] assert cs.name == "foo" # Both code parts should appear assert "a=1" in cs.code assert "b=2" in cs.code # Marks should accumulate assert "m" in cs.marks ---- test_group_snippets_different_names ----------------------------------- .. code-block:: python :name: test_group_snippets_different_names import pytest from pathlib import Path from pytest_codeblock.collector import CodeSnippet, group_snippets from pytest_codeblock.md import parse_markdown from pytest_codeblock.rst import ( parse_rst, resolve_literalinclude_path, get_literalinclude_content, ) # Snippets with different names are not grouped sn1 = CodeSnippet(name="foo", code="x=1", line=1) sn2 = CodeSnippet(name="bar", code="y=2", line=2) combined = group_snippets([sn1, sn2]) assert len(combined) == 2 assert combined[0].name.startswith("foo") assert combined[1].name.startswith("bar") ---- test_parse_markdown_simple -------------------------- .. code-block:: python :name: test_parse_markdown_simple import pytest from pathlib import Path from pytest_codeblock.collector import CodeSnippet, group_snippets from pytest_codeblock.md import parse_markdown from pytest_codeblock.rst import ( parse_rst, resolve_literalinclude_path, get_literalinclude_content, ) text = """ ```python name=test_example x=1 assert x==1 ```""" snippets = parse_markdown(text) assert len(snippets) == 1 sn = snippets[0] assert sn.name == "test_example" assert "x=1" in sn.code ---- test_parse_markdown_with_pytestmark ----------------------------------- .. code-block:: python :name: test_parse_markdown_with_pytestmark import pytest from pathlib import Path from pytest_codeblock.collector import CodeSnippet, group_snippets from pytest_codeblock.md import parse_markdown from pytest_codeblock.rst import ( parse_rst, resolve_literalinclude_path, get_literalinclude_content, ) text = """ ```python name=test_db from django.db import models ```""" snippets = parse_markdown(text) assert len(snippets) == 1 sn = snippets[0] # Should include both default and django_db marks assert "django_db" in sn.marks assert "codeblock" in sn.marks ---- test_pytest_fixtures -------------------- .. pytestfixture: tmp_path .. pytestfixture: http_request .. code-block:: python :name: test_pytest_fixtures_1 d = tmp_path / "sub" d.mkdir() # Create the directory assert d.is_dir() # Verify it was created and is a directory assert isinstance(http_request.GET, dict) ---- .. pytestfixture: tmp_path .. pytestfixture: http_request .. code-block:: python :name: test_pytest_fixtures_2 d = tmp_path / "sub" d.mkdir() # Create the directory assert d.is_dir() # Verify it was created and is a directory assert isinstance(http_request.GET, dict) ---- .. pytestfixture: tmp_path .. pytestfixture: http_request .. code-block:: python :name: test_pytest_fixtures_3 d = tmp_path / "sub" d.mkdir() # Create the directory assert d.is_dir() # Verify it was created and is a directory assert isinstance(http_request.GET, dict) ---- test_async_example ------------------ .. code-block:: python :name: test_async_example import asyncio result = await asyncio.sleep(0.1, result=42) assert result == 42 ---- test_group_snippets ------------------- .. code-block:: python :name: test_group_snippets text_2 = "Jude" Something in between .. continue: test_group_snippets .. code-block:: python :name: test_group_snippets_part_2 assert text_2 print(text_2)