Skip to content

Failing test: unsupported SQL table functions make the testResults endpoint return 500 - #1736

Draft
suarezrominajulieta wants to merge 2 commits into
masterfrom
feature/sql-table-functions-bug
Draft

Failing test: unsupported SQL table functions make the testResults endpoint return 500#1736
suarezrominajulieta wants to merge 2 commits into
masterfrom
feature/sql-table-functions-bug

Conversation

@suarezrominajulieta

@suarezrominajulieta suarezrominajulieta commented Sep 5, 2026

Copy link
Copy Markdown
Collaborator

As requested, this PR adds only tests — no code is touched — to show the
failure mode we discussed: an SQL construct the heuristics cannot parse escapes as an
exception and takes down the endpoint that reports coverage.

⚠️ CI is red on purpose. One of the two tests is meant to fail: that failure is the
report.

What breaks

SqlNameContext has six deliberate stubs for FROM items it does not support. One of them is

@Override
public void visit(TableFunction valuesList) {
    throw new UnsupportedOperationException();
}

The jasper case study hits it with CROSS JOIN LATERAL jsonb_array_elements_text(...)
(PostgreSQL 16). Only 2 distinct queries trigger it.

The missing support is not the problem. The problem is the asymmetry in
SutController#computeSQLHeuristics:

sqlHandler.handle(...) is wrapped in a try/catch (logged as FAILED TO HANDLE SQL COMMAND)
sqlHandler.getSqlDistances(...), a few lines below, is not

Since the SQL heuristics are serialized in the same HTTP response as the coverage
(EMController#getTestResults), the escaping exception becomes a 500. EvoMaster then
discards the individual and restarts the SUT; on restart the SUT renumbers its targets, so
the same logical target gets counted twice.

Server-side stack trace from the failing test:


ERROR - Thrown exception: null
  at SqlNameContext$AliasVisitor.visit(SqlNameContext.java:252)   <- the stub
  at SqlHandler.computeDistance(SqlHandler.java:272)
  at SqlHandler.getSqlDistances(SqlHandler.java:232)              <- no try/catch
  
The null in the message is not a bug in the test: the stub throws
new UnsupportedOperationException() with no message, and EMController reports it as
{"data":null,"error":"Thrown exception: null"}.

The two tests
client-java/controller/src/test/java/org/evomaster/client/java/controller/internal/db/sql/h2/SqlTableFunctionHeuristicsInH2DBTest.java

Both run the real driver (Jetty + Jersey via InstrumentedSutStarter) and issue a real
GET /controller/api/testResults. H2 has no jsonb_array_elements_text, but UNNEST
produces the exact same parse treea CROSS JOIN whose right item is a TableFunctionso no PostgreSQL/Testcontainers is needed and both tests run in seconds.

testTableFunctionDoesNotBreakTestResultsEndpointFAILS (this is the bug).
A control call in the same test, same action, same settings, with a query the heuristics
do understand, returns 200. Swapping in the table function returns 500. That query is
the only variable that changes.

testTableFunctionWithAdvancedHeuristicsDoesNotBreakTestResultsEndpointPASSES.
Identical scenario with advancedHeuristics=true (heuristicsForSQLAdvanced). That path
does not support table functions eitherSqlHeuristicsCalculator throws
"Must implement TableFunction for computing heuristics"but it catches its own
exception and falls back to a MAX_VALUE distance flagged as an evaluation failure.
The test asserts exactly that, so the 200 cannot be explained by the query being skipped.

The contrast between the two is the whole point: the same unsupported construct is harmless
when the exception is contained, and fatal when it is not.

@suarezrominajulieta suarezrominajulieta changed the title Pr analizing sql table functions issue Failing test: unsupported SQL table functions make the testResults endpoint return 500 Sep 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant