Skip to content

Establish reliable runtime regression tests for the alpha #383

Open

Description

@jderochervlk

Part of #382 — v1.0.0 alpha readiness.

Problem

The review at ebe9cca601fb85e6064c786367e546ded9b07d05 found 42 ReScript test fixtures, but the runner executes only seven modules. Only Classify and SafeFetch contain assertions; the other five runtime modules are mostly no-throw/logging smoke tests. The remaining 35 fixtures are compile-only, not browser tests.

The generated-output synchronization check cannot detect drift: it runs git ls-files --modified against generated JavaScript that is ignored and untracked.

Sources: tests/index.js:14-L29, tests/index.js:65-L80, .gitignore:20-L22.

Scope

  • Explicitly distinguish compile-only fixtures, executable runtime tests, and documentation examples; make missing runtime-test registration detectable.
  • Retain useful compile coverage and existing SafeFetch assertions.
  • Replace logging-only success checks with assertions on values and behavior for Headers, Request, Response, URLSearchParams, and URL.
  • Remove the ineffective generated-output guard, or replace it with intentionally maintained golden baselines if generated JavaScript is a supported contract.
  • Add a small real-browser runner for regression tests contributed by the binding-fix issues. Chromium coverage is sufficient for this alpha target; a full browser-engine matrix belongs to later hardening.
  • Add focused expected-failure consumer fixtures for unsafe type widening and nullable/type-contract changes. Match the intended diagnostic, not any compilation failure.

Acceptance criteria

  • A deliberately failing runtime assertion causes the documented test command and CI to fail.
  • Runtime test discovery/registration has a check; a new runtime fixture cannot silently become compile-only.
  • Existing Node-compatible smoke tests assert meaningful results.
  • The misleading Git synchronization check is removed or replaced by a working regression mechanism.
  • A real-browser smoke test runs in CI, and the linked binding fixes can add native API regressions to it.
  • Compile-only fixtures remain supported; do not execute all existing examples indiscriminately. For example, the current Fetch example supplies a body with the default GET method.
  • Test/contributor documentation describes the actual commands and categories.

Boundaries

No requirement to adopt a particular test framework, achieve an arbitrary coverage percentage, or certify every browser API before alpha. Node Fetch tests and DOM emulation alone must not be described as real-browser validation.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions