Skip to content

Fail CI when a public API object is not documented - #79

Open
ocelotl wants to merge 10 commits into
mainfrom
docs_coverage_gate
Open

ocelotl wants to merge 10 commits into
mainfrom
docs_coverage_gate

Conversation

@ocelotl

@ocelotl ocelotl commented Oct 31, 2026 •

Copy link
Copy Markdown
Owner

Adds a check that fails when a public API object is not properly documented, and wires it into CI.

The check is absolute, not a diff. It walks the whole public API and reports every gap it finds, whether the gap arrived with the branch under test or has been in the repository for years. So a pull request that adds an undocumented object fails, and a pull request that changes nothing while an existing object stays undocumented also fails.

This check does not pass on main today. It reports 1369 gaps at 3462000f3, and that is the point of opening it. The repository has never had a gate on documentation, which is how a large part of the public API came to carry no docstring, carry one that documents none of its parameters, or return a value it never describes. The number is the measurement; what to do about it is the decision this pull request asks for.

What the check requires

Three rules, applied to the public API only:

  • missing-docstring: a public module, class, function or method with no docstring.
  • undocumented-parameter: a parameter of a public function or method with no entry in an Args: section. self and cls are never required.
  • missing-returns: a public function or method whose return annotation is present and is not None, with no Returns: and no Yields: section. A function with no return annotation is not asked for one, so the check does not demand documentation from code that carries no annotations at all.

Constructor parameters are accepted on the class docstring rather than on __init__. docs/conf.py does not set special-members, so an __init__ docstring is never rendered, and the class docstring is what the API reference shows.

What it reports today

Measured at the head of this branch, which changes no package source:

rule count
missing-docstring 987
undocumented-parameter 242
missing-returns 140
total 1369

By top-level name: sdk 445, exporter 272, configuration 185, test 132, metrics 71, trace 59, codegen 58, _logs 39, shim 35, propagators 29, context 14, util 13, baggage 9, the rest fewer than 5 each.

Each gap prints on its own line, so the report is a work list:

opentelemetry.util.re.parse_env_headers: missing-returns
opentelemetry.util.re.parse_env_headers: undocumented-parameter (liberal)
opentelemetry.util.re.parse_env_headers: undocumented-parameter (s)
opentelemetry.util.re.parse_headers: missing-docstring
opentelemetry.util.types: missing-docstring
opentelemetry.version: missing-docstring

987 missing-docstring
140 missing-returns
242 undocumented-parameter
1369 documentation gaps in total.

Exit status 1. A gap is identified by the fully qualified name griffe reports as obj.path, so moving a function to a different line does not change its identity.

How to use it while the number is large

tox -e docstring-check runs the absolute check, which is the gate.

tox -e docstring-check -- --against main subtracts the gaps that main already has, leaving only the gaps the working tree adds. That is how a contributor finds their own share of a 1369-line report. It is a reading aid; it is not what CI runs.

The CI job carries the label Approve Docstring check as its escape hatch, rather than sharing Approve Public API check, so that approving a deliberate public API change does not also waive the docstring gate.

Why griffe rather than ruff

ruff's D1 rules cover missing-docstring and nothing else. The DOC rules that would cover the other two are all in preview as of ruff 0.16.1, the version pinned at dev-requirements.txt:23, and none of them reports a parameter that the signature has and the docstring does not. griffe is already a pinned dependency at dev-requirements.txt:2 and already drives scripts/griffe_check.py in the public-symbols-check environment, so this adds no new dependency and follows a pattern the repository already has.

What is not checked

opentelemetry.proto, opentelemetry.proto_json and opentelemetry.semconv are skipped, because they are generated. Private objects are skipped, which is what griffe.Object.is_public decides. Attributes are skipped. Dunder methods other than __init__ are skipped.

Raises: sections are not checked. Deciding which exceptions a function is expected to document needs judgment a static walk cannot supply. If maintainers want that too, it is a fourth rule rather than a change to these three.

Verification

  • tox -e docstring-check at the head of this branch: exit 1, 1369 gaps, under a second.
  • tox -e docstring-check -- --against main: exit 0, because this branch changes no package source.
  • The same command against a branch with four undocumented objects appended to opentelemetry-sdk/src/opentelemetry/sdk/resources/__init__.py: exit 1 naming all five gaps, one per rule plus two for the object missing a docstring entirely.
  • tox -e test-scripts: 18 passed, of which 14 are the new tests in scripts/tests/test_docstring_checker.py. They cover each rule, every case that must not be reported, the constructor convention, and the skip list for generated trees.
  • tox -e ruff: all five pre-commit hooks pass.
  • tox -e lint-license-header-check, tox -e spellcheck, tox -e public-symbols-check: pass.
  • tox -e generate-workflows: leaves no diff, so the generate-workflows job will not report .github/workflows/misc.yml out of date.

The decision this asks for

Merging this as it stands makes CI red until the 1369 gaps are closed, which is not something to do by accident. Three ways to land it, and I will do whichever maintainers prefer:

  1. Merge as is, and close the gaps package by package. Every pull request carries Approve Docstring check until the count reaches zero.
  2. Merge the script and the tests now, and leave the CI job out until the count is low enough to enforce.
  3. Make the CI job run --against main, so it fails only on new gaps, and flip it to absolute once the existing ones are closed. This is what the first revision of this pull request did; the commits are still in the history if that is the preferred shape.

Collects every public API documentation gap at the working tree and at a
base git ref with griffe, and reports only the gaps the branch introduces.
Three rules: a public module, class, function or method with no docstring,
a parameter with no entry in an Args: section, and a function whose return
annotation is not None with no Returns: or Yields: section.

Comparing rather than measuring against a threshold lets the repository
adopt the check while gaps remain, and still refuses a pull request that
adds a new one.
Covers each of the three rules, the cases that must not be reported
(private members, dunder methods other than __init__, an unannotated
return, self and cls), the convention that constructor parameters are
documented on the class docstring, and the skip list for generated trees.
The job needs the same git setup as public-symbols-check, because the
checker loads the base ref: full history and a local main. It carries its
own label, Approve Docstring check, so that approving a public API change
does not also waive the docstring gate.
The check now reports every documentation gap in the public API rather
than the set difference against a base ref, so existing undocumented code
fails it too. There are 1369 gaps at 3462000: 987 missing-docstring,
242 undocumented-parameter, 140 missing-returns.

--against <ref> still subtracts the gaps of a ref, which narrows a long
report to the gaps one working tree adds. It is a reading aid rather than
the gate.
So that tox -e docstring-check -- --against main works.
The absolute check reads no base ref, so the job needs neither the full
history nor a local main. Dropping the pull_request condition lets it run
on merge groups and pushes as well, which an absolute check should.
@ocelotl ocelotl changed the title Fail a pull request that adds an undocumented public API object Fail CI when a public API object is not documented Oct 31, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant