[dbsql] InsertMany Row Batching

[onelapahead]

When doing:

INSERT INTO transfers (col1, col2, ..., col17)
VALUES ($1, $2, ..., $17), ($18, ..., $34), ...

Your parameter count in SQL is equal to row * columns.

With bulk inserts for 1000s of rows, with 10+ columns, its easy to end up exceeding the PostgreSQL limit on parameters which is uint16's max - 65535.

So, drawing inspiration from LFDT-Paladin/paladin#644, we also add "batching" of multi-row inserts based on a MaxPlaceholders feature, which will chunk the rows across multiple INSERT's to ensure we're underneath the limit. Not as efficient as PostgreSQL's UNNEST from a performance perspective, but still helps avoid ff-common users from hitting DB limits when inserting in bulk.

[onelapahead]

Noting - if trace logs are enabled, this log line is ridiculously massive when you're inserting in bulk.

Not even useful/readable for mortals or AI:

const maxTraceArgs = 100

func limitTraceArgs(args []interface{}) string {
	if len(args) <= maxTraceArgs {
		return fmt.Sprintf("%+v", args)
	}
	return fmt.Sprintf("%+v ...and %d more", args[:maxTraceArgs], len(args)-maxTraceArgs)
}

func limitTraceSeqs(seqs []int64) string {
	if len(seqs) <= maxTraceArgs {
		return fmt.Sprintf("%v", seqs)
	}
	return fmt.Sprintf("%v ...and %d more", seqs[:maxTraceArgs], len(seqs)-maxTraceArgs)
}

func (s *Database) InsertTxRows(ctx context.Context, table string, tx *TXWrapper, q sq.InsertBuilder, postCommit func(), sequences []int64, requestConflictEmptyResult bool) error {
	l := log.L(ctx)
    // ...
	l.Tracef(`SQL-> insert query: %s (args: %s)`, sqlQuery, limitTraceArgs(args))
// ...
}

could be an approach we should take ? Just whats the limit ?

[onelapahead]

We'll tackle this another time

[peterbroadhurst]

Thanks @onelapahead - I approve of the latest commit to remove the config setting.

Think it's unclear that a config setting of that shape makes sense, given this is a fundamental characteristic we're talking about with the max inserts (rather than a tuning option).

Provider implementations (which FF common doesn't really come with right now) for PSQL and others can have their own config settings.

Also has the benefit of not affecting the builds of consumers, including FF core, FFTM etc.