Skip to content

[FEATURE] Let test case classes opt in to sharing test instances - #740

Open
bmack wants to merge 1 commit into
TYPO3:mainfrom
bmack:layer0-share-test-instances
Open

bmack wants to merge 1 commit into
TYPO3:mainfrom
bmack:layer0-share-test-instances

Conversation

@bmack

@bmack bmack commented Aug 10, 2026 •

Copy link
Copy Markdown
Member

Every functional test case class gets its own test instance: its own directory, database
and schema, and its own compiled dependency injection container. Setting up an instance
takes about a second, a further test in it a fraction of that. For test case classes with
few tests - typical for extensions - setting up instances dominates the run.

This lets test case classes opt in to sharing an instance with every other class
configured the same way:

#[ShareInstance]
abstract class AbstractFunctionalTestCase extends FunctionalTestCase
{
    protected array $testExtensionsToLoad = ['vendor/my-extension'];
}

The attribute works directly on a class or through a parent class, so an extension can
opt in its whole suite with its base class.

Nothing changes without the attribute

  • getInstanceIdentifier() and getInstancePath() stay protected static with unchanged
    signatures, and return the same values as before (sha1(static::class)), so instance
    paths and database names stay the same.
  • No existing method changes its signature. Everything added to FunctionalTestCase is
    private, so nothing can collide with methods or properties of test case classes in
    extensions. DatabaseSnapshot::initialize() is unchanged, the new naming goes through a
    separate initializeWithSnapshotIdentifier().
  • Nothing version specific, so it works with TYPO3 v14 and v15 alike.

How sharing works

A class opting in is identified by its instance configuration instead of its class name:
the default values of $coreExtensionsToLoad, $testExtensionsToLoad,
$pathsToLinkInTestInstance, $pathsToProvideInTestInstance,
$configurationToUseInTestInstance, $additionalFoldersToCreate and
$initializeDatabase, plus the classes implementing initializeTestDatabase() and
initializeTestDatabaseAndTruncateTables(). A class overriding those relies on setting up
its own instance, so it gets one.

The instance is identified before setUp() of the class runs, so the configuration has to
be declared as property defaults. A class opting in and assigning one of these properties
at runtime fails with a message saying so, instead of silently getting an instance set up
for a different configuration.

When a shared instance is handed over to another test case class, the state a class may
leave behind is reset:

  • the core cache group is dropped, so a TCA schema or backend module registry built by
    one class is not seen by the next. The compiled container in di is kept, it only
    depends on the active packages, and keeping it is what makes sharing worth it
  • typo3conf/system/ is restored to its state after setup: tests write to settings.php,
    configuration:set does for instance, and add files such as file-storages.yaml
  • database snapshots are named after the test case class, so classes sharing an instance
    do not restore each other's snapshot
  • database connections are closed when switching between instances, since ConnectionPool
    keeps them in a static property before TYPO3 v15

Measured on TYPO3 Core

Full functional suite, SQLite, 4 chunks as in CI, with the attribute added to every
functional test class except six that assign their configuration in setUp() and one
that loads its extension list from a file:

time test instances tests / assertions / skips
testing framework without this change 1845 s 1019 baseline
with this change, no attribute 1639 s 1019 identical to baseline
with this change, attribute added 1173 s 408 identical to baseline

The first two arms run the same code path, so the difference between them is noise from
the machine. Sharing is about 28 per cent faster than the same code without the
attribute. Extensions gain more, since their classes typically have fewer tests:
dbdoctor's own version of this idea measured a factor of 3 to 6.

Lint (PHP 8.2), CGL and unit tests are green. A unit test covers how instances are
identified, with and without the attribute.

Every functional test case class gets its own test instance: directory,
database, schema and compiled dependency injection container. Setting
up an instance takes about a second, a further test in it a fraction
of that, so for test case classes with few tests, which is typical for
extensions, setting up instances dominates the run.

Test case classes carrying the new ShareInstance attribute, directly
or through a parent class, are identified by their instance
configuration instead of their class name. Classes configured the same
way share one instance, which is set up once per process.

The configuration is read from the default values of the properties
that shape an instance - extensions, paths to link and provide,
configuration, additional folders, database initialization - plus the
classes implementing the database setup and reset, since a class
overriding those relies on setting up its own instance. Assigning one
of these properties at runtime fails with a clear message, because
the instance has to be identified before setUp() of the class runs.

When a shared instance is handed over to another test case class, the
state a class may leave behind is reset:

* The "core" cache group is dropped, so a TCA schema or backend module
  registry built by one class is not seen by the next. The compiled
  container in "di" is kept, it only depends on the active packages.
* typo3conf/system/ is restored from a copy taken at setup, since tests
  write to settings.php - "configuration:set" does, for instance - and
  add files such as file-storages.yaml.
* Database snapshots are named after the test case class, so classes
  sharing an instance do not restore each other's snapshot.
* Database connections are closed when switching between instances,
  as ConnectionPool keeps them in a static property before TYPO3 v15.

Test case classes without the attribute behave exactly as before: same
instance identifier, same paths, same setup steps. No method changes
its signature, getInstanceIdentifier() and getInstancePath() stay
static.
@bmack
bmack force-pushed the layer0-share-test-instances branch from c5369a3 to 84c2c3a Compare November 14, 2026 19:05
@bmack bmack changed the title [!!!][FEATURE] Share test instances between identically configured test cases [FEATURE] Let test case classes opt in to sharing test instances Nov 14, 2026

@CybotTM CybotTM left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Requesting changes: without the attribute nothing changes (measured), and sharing itself works, but a failed instance setup breaks every later class sharing it, parallel processes destroy each other's shared instance, and no test covers the sharing logic in setUp().

Measured with probe test classes in TYPO3 core main (8ef5361) on top of this PR (testing-framework main 480a74d), SQLite and MariaDB:

  1. A failed setup poisons all later classes. $sharedInstancesSetUp is set before the instance is built (inline comment). With a missing test extension, the first sharing class reports Test extension path ... not found, the second fails with LogicException: ... ConnectionPool can not be used from failsafe container plus a dangling error handler. Without the attribute both classes report the real cause.
  2. Parallel processes break the shared instance. Two runTests.sh processes at the same time, one sharing class each (same configuration), 3 rounds: at least one process failed in every round (Could not determine the current working directory, no such table: pages, missing test_<id>.empty.sqlite). The same classes without the attribute: 6/6 green. This affects paratest or any parallel run on one typo3temp, and the ShareInstance docblock does not mention it. A process token in the shared identifier, or at least a documented requirement, would cover it.
  3. The sharing logic has no test. Replacing the reuse decision with $this->isFirstTest = true; (never reuse) leaves all 132 unit tests green; the new tests cover how instances are identified only. A second review run mutated the hand-over reset, the per-class snapshot name, the runtime-assignment check and the connection close, all green as well. The repository has no functional suite, so either the decision in setUp() needs to become unit-testable or a functional fixture with two sharing classes is needed.
Smaller points
  • No way out below an opted-in base class: sharesInstance() walks up the parents, so a child that assigns its configuration in setUp() cannot opt out, and the exception tells it to "remove the ShareInstance attribute", which sits on the base class (inline comment).
  • SHARED_INSTANCE_RESET_CACHE_GROUP = 'core' and the docblock speak of the "core" cache group; core is a cache of group system. extbase in the same group survives a hand-over. I know of no test this affects.
  • On MySQL and PostgreSQL a hand-over truncates tables only, so tables or columns created at runtime reach the next class; on SQLite the empty database file is copied. Worth saying in the docblock next to "Database rows".
  • Files a class leaves in fileadmin/ are visible in the next class (measured); the docblock states this as a requirement.
Checked and fine
  • Without the attribute, 29 core functional test files (1194 tests, 25793 assertions) give identical results per file with main and with this PR.
  • Two sharing classes use the same instance; a non-sharing class in between gets its own and the switch back works (SQLite, MariaDB).
  • Reset between classes: DB rows, extra files in typo3conf/system/, settings.php values; typo3conf/sites is already removed by tearDown().
  • withDatabaseSnapshot() in two sharing classes restores each class's own snapshot (SQLite, MariaDB).
  • Assigning $coreExtensionsToLoad in setUp() fails with exception 1794738201 and a clear message.
  • The identity covers all configuration properties and both database methods setUp() uses.
  • Gates on PHP 8.2 green (132 unit tests). The timing table was not reproduced.

$this->assertInstanceConfigurationIsDeclaredStatically();
// Shared instances are set up once per process, not once per test case class.
$this->isFirstTest = !isset(self::$sharedInstancesSetUp[$this->identifier]);
self::$sharedInstancesSetUp[$this->identifier] = true;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This marks the shared instance as set up before the instance is built further down. If building it throws (reproduced with a missing test extension), every later class with the same configuration takes the reuse path against a half-built instance and fails with an unrelated error (ConnectionPool can not be used from failsafe container, dangling error handler) instead of the real cause. Setting the flag only after the instance is complete, e.g. after loadExtensionTables(), would keep the original error visible for every class.

$class = new \ReflectionClass(static::class);
do {
$sharesInstance = $class->getAttributes(ShareInstance::class) !== [];
} while (!$sharesInstance && ($class = $class->getParentClass()) !== false);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Because the lookup walks up the parents, a class below an opted-in base class cannot opt out again, e.g. one that has to assign its configuration in setUp(). The exception it then gets says to remove the attribute, which sits on the base class. An opt-out (for example #[ShareInstance(false)] evaluated from the most derived class) or a message covering the inherited case would help the base-class adoption the PR describes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants