Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion docs/using-diffci.md
Original file line number Diff line number Diff line change
Expand Up @@ -693,7 +693,7 @@ _Module resource and dependency impact contributed by [@FredApps](https://github

### Python

These appear only in Python projects (a root `pyproject.toml`, `setup.py`, `setup.cfg`, `pytest.ini`, `.pytest.ini`, `tox.ini` or `conftest.py`). A changed project file such as `pyproject.toml` or `requirements.txt` falls back under `Configuration file(s) changed`.
These appear only in Python projects (a root `pyproject.toml`, `setup.py`, `setup.cfg`, `pytest.toml`, `.pytest.toml`, `pytest.ini`, `.pytest.ini`, `tox.ini` or `conftest.py`). A changed project file such as `pyproject.toml` or `requirements.txt` falls back under `Configuration file(s) changed`.

| Reason | Example change | Narrow |
| --- | --- | --- |
Expand All @@ -702,6 +702,7 @@ These appear only in Python projects (a root `pyproject.toml`, `setup.py`, `setu
| `Unresolved relative import in …: ……` | `from .gone import x` after deleting `gone.py` | No |
| `Unresolved first-party Python import in …: …` | `import shop.generated` where `shop/generated.py` is created by a build step | No |
| `pytest configuration DiffCI cannot read: …` | `python_files` set to something other than a string or list in `pyproject.toml` | Fix: use a literal string or list |
| `pytest configuration DiffCI cannot read: …` | A `pytest.toml` or `[tool.pytest]` key that isn't a list of strings, both `[tool.pytest]` and `[tool.pytest.ini_options]` in one `pyproject.toml`, or options above the `[pytest]` table in `pytest.toml`; a quoted or dotted key, or a table header with spaces or quotes inside the brackets; or a `pytest.toml`, `.pytest.toml` or `[tool.pytest]` next to another pytest config that pytest before 9 would read instead | Fix: use a list of strings, keep one table, and put options under `[pytest]` (pytest 9 rejects all three); write bare keys and `[pytest]` / `[tool.pytest]` headers; keep a single pytest config file |
| `Python analysis found no Python files` | Project files with every `.py` file inside a virtual environment | No |

_Reading pytest configuration the way pytest does was improved by [@sarankumar1325](https://github.com/sarankumar1325): `pytest.ini` wins even when empty, and `.pytest.ini` is read ([#314](https://github.com/DiffCI/core/pull/314)); `pyproject.toml` lists are read as TOML strings ([#315](https://github.com/DiffCI/core/pull/315)); `tox.ini` and `setup.cfg` settings are covered by tests ([#316](https://github.com/DiffCI/core/pull/316))._
Expand Down
1 change: 1 addition & 0 deletions release-files.json
Original file line number Diff line number Diff line change
Expand Up @@ -284,6 +284,7 @@
"tests/repo/lockfile-free-dependencies.test.ts",
"tests/repo/lockfiles.test.ts",
"tests/repo/python-addopts-plugins.test.ts",
"tests/repo/python-pytest-toml.test.ts",
"tests/repo/mocharc-config-fallback.test.ts",
"tests/repo/python-testpaths-globs.test.ts",
"tests/repo/orphan-tool-config.test.ts",
Expand Down
2 changes: 1 addition & 1 deletion src/git/git-diff.ts
Original file line number Diff line number Diff line change
Expand Up @@ -235,7 +235,7 @@ export const CONFIG_PATTERNS = [
// Python projects (2026-11-06): what pytest, the interpreter and the installed packages are.
/^pyproject\.toml$/,
/^setup\.(?:py|cfg)$/,
/^\.?pytest\.ini$/,
/^\.?pytest\.(?:ini|toml)$/,
/^tox\.ini$/,
/^noxfile\.py$/,
/^requirements[^/]*\.(?:txt|in)$/,
Expand Down
100 changes: 86 additions & 14 deletions src/repo/adapters/python.ts
Original file line number Diff line number Diff line change
Expand Up @@ -65,7 +65,7 @@ export type PythonImport = [module: string, level: number, names: string[]];
export interface PythonFileImports { imports?: PythonImport[]; dynamic?: boolean; error?: string }

const EXCLUDED_DIRS = new Set([".venv", "venv", "env", ".env", ".tox", ".nox", "site-packages", "__pycache__", ".eggs", ".mypy_cache", ".pytest_cache", ".ruff_cache"]);
const PROJECT_FILES = ["pyproject.toml", "setup.py", "setup.cfg", "pytest.ini", ".pytest.ini", "tox.ini", "conftest.py"];
const PROJECT_FILES = ["pyproject.toml", "setup.py", "setup.cfg", "pytest.toml", ".pytest.toml", "pytest.ini", ".pytest.ini", "tox.ini", "conftest.py"];

export function isPythonSource(path: string): boolean {
return path.endsWith(".py") && !path.split("/").some((part) => EXCLUDED_DIRS.has(part) || part.endsWith(".egg-info"));
Expand Down Expand Up @@ -163,35 +163,107 @@ export function readPytestSettings(read: (path: string) => string | undefined):
if (plugins.length) settings.plugins = plugins;
} else settings[key] = values;
};
// pytest's search order. `pytest.ini` and `.pytest.ini` match even when empty or without a `[pytest]`
// section, and then pytest uses its defaults; the other files only match with their section. The first
// match wins and options are never merged.
for (const [file, header] of [["pytest.ini", "pytest"], [".pytest.ini", "pytest"], ["pyproject.toml", "tool.pytest.ini_options"], ["tox.ini", "pytest"], ["setup.cfg", "tool:pytest"]] as const) {
const escape = (text: string) => text.replace(/[\\^$.*+?()[\]{}|]/g, "\\$&");
const isToml = (file: string) => file.endsWith(".toml");
// A table header may be indented and followed by a comment (`[pytest] # selection`). TOML also allows
// spaces inside the brackets and quoted keys (`[ pytest ]`, `[tool."pytest"]`), which this reader doesn't
// match: a header that only matches with spaces and quotes removed makes the file unreadable instead.
const section = (file: string, text: string, header: string): string | undefined => {
const comment = isToml(file) ? "#" : "#;";
return new RegExp(`^[ \\t]*\\[${escape(header)}\\][ \\t]*(?:[${comment}].*)?$([\\s\\S]*?)(?=^[ \\t]*\\[|(?![\\s\\S]))`, "m").exec(text)?.[1];
};
const looseHeader = (file: string, text: string, header: string): boolean =>
isToml(file) && [...text.matchAll(/^[ \t]*\[([^\]\n]*)\]/gm)].some((match) =>
match[1]!.replace(/[\s"']/g, "") === header && section(file, match[0]!.trim(), header) === undefined);
const hasEntries = (body: string) => /^[ \t]*[^\s#]/m.test(body);
const keyAt = (key: string) => `^[ \\t]*${key}[ \\t]*=`;
// A quoted or dotted key (`"testpaths" = …`, `ini_options.testpaths = …`) can't be told apart from the
// keys selection reads, so a TOML table with one is unreadable.
const unclassified = (body: string) => /^[ \t]*(?:"[^"\n]*"|'[^'\n]*'|[A-Za-z0-9_-]+)[ \t]*\./m.test(body) ||
/^[ \t]*(?:"[^"\n]*"|'[^'\n]*')[ \t]*=/m.test(body);
// pytest 9's native TOML mode (`pytest.toml`, `[tool.pytest]`): every key selection reads is a list of
// strings, and each `addopts` item is one argument. Any other type is an error in pytest, so it can't be read.
const readNative = (file: string, body: string) => {
if (unclassified(body)) { settings.unreadable = `${file} key DiffCI can't classify`; return settings; }
for (const key of keys) {
const array = new RegExp(`${keyAt(key)}[ \\t]*(?=\\[)`, "m").exec(body);
if (array) {
const values = readTomlStringArray(body, array.index + array[0].length);
if (values) apply(key, values);
else settings.unreadable = `${file} ${key}`;
} else if (new RegExp(keyAt(key), "m").test(body)) settings.unreadable = `${file} ${key}`;
}
return settings;
};
const order = [["pytest.toml", "pytest"], [".pytest.toml", "pytest"], ["pytest.ini", "pytest"], [".pytest.ini", "pytest"], ["pyproject.toml", "tool.pytest.ini_options"], ["tox.ini", "pytest"], ["setup.cfg", "tool:pytest"]] as const;
// pytest before 9 ignores `pytest.toml`, `.pytest.toml` and `[tool.pytest]`, and uses the first of the
// other files that matches. When a native config wins and such a file also exists, the answer depends on
// the installed pytest, so it can't be read.
const olderPytestFile = (from: number): string | undefined => {
for (const [file, header] of order.slice(from)) {
const text = read(file);
if (text === undefined || file === ".pytest.toml") continue;
if (file === "pytest.ini" || file === ".pytest.ini" || section(file, text, header) !== undefined) return file;
}
return undefined;
};
const native = (file: string, body: string | undefined, index: number) => {
const older = olderPytestFile(index);
if (older) { settings.unreadable = `${file} is ignored before pytest 9, which reads ${older}`; return settings; }
return body === undefined ? settings : readNative(file, body);
};
// pytest's search order. `pytest.toml`, `.pytest.toml`, `pytest.ini` and `.pytest.ini` match even when
// empty or without their `[pytest]` table, and then pytest uses its defaults; the other files only match
// with their section. The first match wins and options are never merged.
for (const [index, [file, header]] of order.entries()) {
const text = read(file);
if (text === undefined) continue;
const section = new RegExp(`^\\[${header.replace(/[.:]/g, "\\$&")}\\]\\s*$([\\s\\S]*?)(?=^\\[|(?![\\s\\S]))`, "m").exec(text);
if (!section) {
if (looseHeader(file, text, "pytest") || looseHeader(file, text, "tool.pytest") || looseHeader(file, text, "tool.pytest.ini_options")) {
settings.unreadable = `${file} pytest table header DiffCI can't read`;
return settings;
}
if (file === "pytest.toml" || file === ".pytest.toml") {
const body = section(file, text, "pytest");
// pytest rejects options written above any table.
if (body === undefined && hasEntries(text.split(/^[ \t]*\[/m, 1)[0]!)) { settings.unreadable = `${file} options outside [pytest]`; return settings; }
return native(file, body, index + 1);
}
if (file === "pyproject.toml") {
// `[tool.pytest]` holds native TOML values, `[tool.pytest.ini_options]` INI-style ones. pytest refuses a
// file that sets both, and an `ini_options` key inside `[tool.pytest]` is a form this reader doesn't parse.
const nativeBody = section(file, text, "tool.pytest");
const ini = section(file, text, header);
if (nativeBody !== undefined && new RegExp(`^[ \\t]*ini_options[ \\t]*[.=]`, "m").test(nativeBody)) { settings.unreadable = `${file} [tool.pytest] ini_options`; return settings; }
if (nativeBody !== undefined && hasEntries(nativeBody)) {
if (ini !== undefined && hasEntries(ini)) { settings.unreadable = `${file} [tool.pytest] and [tool.pytest.ini_options]`; return settings; }
// An empty [tool.pytest.ini_options] beside it is what pytest 8 reads.
if (ini !== undefined) { settings.unreadable = `${file} [tool.pytest] is ignored before pytest 9, which reads [tool.pytest.ini_options]`; return settings; }
return native(file, nativeBody, index + 1);
}
}
const body = section(file, text, header);
if (body === undefined) {
if (file === "pytest.ini" || file === ".pytest.ini") return settings;
continue;
}
const body = section[1]!;
if (file === "pyproject.toml" && unclassified(body)) { settings.unreadable = `${file} key DiffCI can't classify`; return settings; }
for (const key of keys) {
if (file === "pyproject.toml") {
const match = new RegExp(`^${key}\\s*=\\s*("[^"]*"|'[^']*')`, "m").exec(body);
const array = new RegExp(`^${key}\\s*=\\s*(?=\\[)`, "m").exec(body);
const match = new RegExp(`${keyAt(key)}[ \\t]*("[^"]*"|'[^']*')`, "m").exec(body);
const array = new RegExp(`${keyAt(key)}[ \\t]*(?=\\[)`, "m").exec(body);
// A multi-line string would otherwise match as an empty one. Its contents are read as one string;
// a basic string's line-ending backslash only joins lines. An unterminated one cannot be read.
const multiline = new RegExp(`^${key}\\s*=\\s*("""|'{3})([\\s\\S]*?)\\1`, "m").exec(body);
const multiline = new RegExp(`${keyAt(key)}[ \\t]*("""|'{3})([\\s\\S]*?)\\1`, "m").exec(body);
if (multiline) apply(key, splitList(multiline[1] === '"""' ? multiline[2]!.replace(/\\\r?\n/g, " ") : multiline[2]!));
else if (new RegExp(`^${key}\\s*=\\s*(?:"""|'{3})`, "m").test(body)) settings.unreadable = `${file} ${key}`;
else if (new RegExp(`${keyAt(key)}[ \\t]*(?:"""|'{3})`, "m").test(body)) settings.unreadable = `${file} ${key}`;
else if (array) {
const values = readTomlStringArray(body, array.index + array[0].length);
if (values) apply(key, key === "addopts" ? values.flatMap(splitList) : values);
else settings.unreadable = `${file} ${key}`;
} else if (match) apply(key, splitList(match[1]!));
else if (new RegExp(`^${key}\\s*=`, "m").test(body)) settings.unreadable = `${file} ${key}`;
else if (new RegExp(keyAt(key), "m").test(body)) settings.unreadable = `${file} ${key}`;
} else {
// INI values continue on indented lines.
// INI values continue on indented lines, so a key starts at the beginning of a line.
const match = new RegExp(`^${key}\\s*=\\s*(.*(?:\\r?\\n[ \\t]+.*)*)`, "m").exec(body);
if (match) apply(key, splitList(match[1]!));
}
Expand Down
Loading
Loading