testdir runs are affecting coverage, can I tweak to prevent this?

[joshua-badger]

#8868
While working on the issue I'm asking about in the above link, I am running into an issue with code coverage.

Specifically, when I add a test to my test suite (which uses testdir, specifically testdir.runpytest_subprocess(), all of a sudden my coverage drops on my django settings files.

Is there something I can do to prevent my test from reverting this coverage?

The added test file:

import inspect
from os import pathsep

import pytest

from tests import pytest_common_hooks


def test_nplusone_catches_marked_test(testdir, monkeypatch):
    test_contents = """
import logging
import pytest

log = logging.getLogger("nplusone")

def test_single_nplusone_catches():
    # Expect this test to pass but err
    log.warning("Potential n+1 query detected on `Model.column`")

def test_multiple_nplusone_queries_catch():
    # Expect this test to pass but err
    log.warning("Potential n+1 query detected on `Model.column`")
    log.warning("Potential n+1 query detected on `Model.column`")

@pytest.mark.allow_nplusone(column="Model.column", count=3)
def test_marked_allowed_nplusone_queries_pass():
    # Expect no error here, we marked it sufficiently
    for _ in range(3):
        log.warning("Potential n+1 query detected on `Model.column`")

@pytest.mark.allow_nplusone(column="Model.column2", count=1)
@pytest.mark.allow_nplusone(column="Model.column", count=3)
def test_multiple_marks_allowed_nplusone_queries_pass():
    # Expect no error here, we marked it sufficiently
    for _ in range(3):
        log.warning("Potential n+1 query detected on `Model.column`")
    log.warning("Potential n+1 query detected on `Model.column2`")

@pytest.mark.allow_nplusone(column="Model.column2", count=1)
@pytest.mark.allow_nplusone(column="Model.column", count=3)
def test_multiple_marks_allowed_nplusone_queries_still_err_if_not_sufficiently_accounted_for():
    # Expect this test to pass but err; we didn't sufficiently catch it
    for _ in range(3):
        log.warning("Potential n+1 query detected on `Model.column`")
    for _ in range(3):
        log.warning("Potential n+1 query detected on `Model.column2`")

@pytest.mark.allow_nplusone(column="Model.column", count=3)
def test_multiple_marks_allowed_nplusone_queries_still_err_if_an_extra_nplusone_happens():
    # Expect this test to pass but err; we didn't sufficiently catch it
    for _ in range(3):
        log.warning("Potential n+1 query detected on `Model.column`")
    for _ in range(3):
        log.warning("Potential n+1 query detected on `Model.column2`")
    """

    testdir.makepyfile(test_things_work=test_contents)

    testdir.makeconftest(inspect.getsource(pytest_common_hooks))
    monkeypatch.setenv("PYTHONPATH", "/<redacted>", prepend=pathsep)

    result = testdir.runpytest_subprocess()

    result.assert_outcomes(passed=6, errors=4)  # all the tests pass, but only 4 err

The coverage report output:

----------- coverage: platform linux, python 3.8.2-final-0 -----------
Name                          Stmts   Miss Branch BrPart  Cover   Missing
-------------------------------------------------------------------------
<redacted>/settings/base.py     144      3      4      2    97%   212->217, 415, 593-594
<redacted>/settings/test.py      20      0      2      1    95%   37->41
<redacted>/urls.py               11     11      2      0     0%   1-30
<redacted>/wsgi.py                4      4      0      0     0%   10-16
-------------------------------------------------------------------------
TOTAL                          5878     18   1086      3    99%

134 files skipped due to complete coverage.
Coverage HTML written to dir /test-output/coverage
Coverage XML written to file /test-output/coverage.xml

FAIL Required test coverage of 100% not reached. Total coverage: 99.67%

coveragerc file:

[run]
omit =
    # These files run in a configuration not present on the test suite.
    <redacted>/celery.py
    <redacted>/settings/*
    <redacted>/urls.py
    <redacted>/wsgi.py
    **/tests/**
    **/migrations/**

[portyu9]

Hi @joshua-badger

For the pytest-cov behavior used at the time of this thread, the child pytest invocation can explicitly disable pytest-cov:

result = testdir.runpytest_subprocess("--no-cov")

--no-cov is still available for disabling pytest-cov in that invocation.

There is an important current-version difference, though: modern pytest-cov no longer implements its old subprocess coverage mechanism itself. pytest-cov 7 delegates subprocess measurement to coverage.py, typically using coverage's:

[run]
patch = subprocess

If that coverage.py subprocess patch is enabled globally, --no-cov on the nested pytest process may not by itself prevent coverage.py from starting automatically in the subprocess. In that case you also need to disable/change that coverage configuration/environment for the subprocess.

Also, current pytest documentation generally calls the fixture pytester; testdir is the older interface/name.

Please mark this answer as accepted if it helped, thank you.