Issue #8138 : Do not send a path option when writing with file_format=jdbc

[vbhanuchander-lang]

Fixes #8138

The defect

SparkFileIoSupport.writeDataset always called the one-argument DataFrameWriter.save(path):

writer.save(path);   // SparkFileIoSupport.java:98

That overload stores its argument as the path writer option before saving. Spark's JDBC provider forwards every option it does not recognise to the driver as a connection property, so a JDBC sink — which has no path — receives one anyway.

Postgres and MySQL ignore connection properties they do not know, so this has been invisible against them. Teradata validates them and fails the write:

[Teradata JDBC Driver] [Error 1536] [SQLState HY000] Invalid connection parameter name path

SparkFileOutputHandler compounded it: it rejected the transform outright when no file path was configured, so the user had to invent a path for a sink that has none — and that invented value is precisely what got sent to the driver.

The fix

isPathless(format) names the formats whose sink is addressed entirely through options rather than a path. Writing one of those now calls the no-argument save(), so nothing enters the writer options:

if (pathless) {
  writer.save();
} else {
  writer.save(path);
}

Only jdbc is in that set today. It is a named set rather than an equals check so other option-addressed sinks can join it, but I deliberately did not add kafka/console/noop speculatively — jdbc is the one with a reported failure.

Also: the error message for a pathless format no longer quotes a path that played no part in the write, and SparkFileOutput no longer demands a file path when the format is pathless.

SparkLakeTableSupport shares writeDataset but only ever passes delta or iceberg, both path-based, so it is unaffected.

Tests

The obvious test — write to an embedded database and see whether it complains — does not work, and it is worth saying why. I tried it with H2 first and it passed with the fix reverted: H2, like Postgres and MySQL, silently ignores connection properties it does not recognise. Only a strict driver such as Teradata's surfaces this, and depending on one in a unit test is not an option.

So the tests assert the thing that is actually wrong — what Spark hands the driver. CapturingJdbcDriver records the Properties of every connection and delegates to H2, and the test asserts path is not among them. Reverting the fix fails it with exactly the right message:

Spark sent 'path' as a connection property: [path] ==> expected: <false> but was: <true>

CapturingJdbcDriver is top-level rather than nested on purpose: Spark's DriverRegistry matches a registered driver by canonical name, which for a nested class does not equal the binary name it is loaded by, and the lookup fails with an internal error.

H2 is added in test scope only, with the version imported from hop-libs-jdbc the same way plugins/actions/waitforsql does it.

Verified locally on JDK 21: hop-engines-spark 189 tests, 0 failures. spotless:check and apache-rat:check clean.

Note on scope

I could not reproduce against Teradata itself — no access to that driver or a Vantage instance. What is verified here is that Hop no longer sends path, which is the cause identified in the report; @tmortada, confirmation against your Teradata setup would be worth having before this is merged.

[tmortada]

@vbhanuchander-lang

Confirmed against a real Teradata Vantage Express instance 20.00.28.81 Thanks!

Testing method

Rather than a full build, I patched just the two changed files ( SparkFileIoSupport. java, SparkFileOutputHandler. java ), compiled them with javac against the classpath already present in a apache/hop: 2.19.0 -based container, and injected the two resulting .class files into the running hop-engines-spark-2.19.0. jar with jar -u

I verified with:

• <file_path>/tmp/hop-bug-repro/category_sales_summary_pg</file_path>
• <file_path />
• Removed

The 3 scenarios work

Log Information

2026/08/29 18:34:02 - General - Handled Spark Lake Table Input : Read Category Sales Summary from Iceberg format=iceberg mode=TABLE table=nessie.retail.category_sales_summary timeTravel=NONE columns=[product_category, total_revenue, transaction_count, avg_sale_value, total_units_sold]
2026/08/29 18:34:04 - General - Handled Spark File Output : Write JDBC Direct to Teradata format=jdbc mode=Overwrite path=/tmp/hop-bug-repro/category_sales_summary
2026/08/29 18:34:05 - bug_repro_iceberg_to_teradata_direct - Spark pipeline finished, result row count=0
HopRun exit.