Limit simultaneous SFTP source connections when using wildcard streams

[akkce]

Sling version:

1.5.1

What is the Operating System?

Linux

Do you have a CLI Pro/Platform subscription?

No

Description of the issue

I'm trying to build a sling Replication that will connect to our vendor's SFTP server and download all the files located there. There should be at most 150-ish files, the largest set of 7 are 21000 KB in size, but the vast majority of the files are much smaller (2400 KB or less).

It appears that sling opening too many SFTP connections for the server - after we get to about 15+ connections established (determined via ss -tn | grep), I get the following error (which I'm guessing is the server killing all the open connections)

Some google-ing led me to think that I could set a max_connections variable in the stream but if it appears to be a no-op.

How can I go about performing this task without swamping the SFTP server in connections?

Replication Configuration

source: SFTP
target: MARIADB

defaults:
  mode: full-refresh
  object: 'slingETL.{stream_file_path}'
  columns:
    "*": string
  source_options:
    format: csv
    delimiter: "\t"

streams:
  "/Upload/*.txt":
    source_options:
      limit: 100
      max_connections: 2

Log Output

2026-01-08 12:35:20 DBG opened "sftp" connection (conn-sftp-56e)
fatal:
--- proc.go:283 main ---
--- sling_cli.go:525 main ---
--- sling_cli.go:561 cliInit ---
--- cli.go:287 CliProcess ---
--- sling_run.go:275 processRun ---
--- sling_run..go:87 RunReplication ---
--- sling_run.go:529 replicationRun ---
--- replication.go:1175 Compile ---
--- replication.go:1356 StreamToTaskConfig ---
~ failure running replication (see docs @ https://docs.slingdata.io/sling-cli)
--- config.go:634 Prepare ---
~ Error compiling replication config
--- config.go:822 FormatTargetObjectName ---
~ could not prepare stream config: /Upload/FILE.txt
--- config.go:1041 GetFormatMap ---
~ could not prepare stream task: /Upload/FILE.txt
--- connection.go:362 AsFile ---
~ could not format target object name
--- connection.go:385 AsFileContext ---
~ could not get formatting variables
--- fs.go:162 NewFileSysClientFromURLContext ---
~ could not init source conn as file
--- fs.go:130 NewFileSysClientContext ---
~ Error initiating File Sys Client
--- fs_sftp.go:30 Init ---
~ unable to connect to ssh server
--- fs_sftp.go:115 Connect ---
~ unable to connect to ssh server sftp1.myvendor.com:22
--- ssh.go:142 Connect ---
ssh: handshake failed: read tcp SLING_SERVER_IP:50328->SFTP_SERVER_IP:22: read: connection reset by peer

[flarco]

hey @akkce, thanks for reporting. Can you try with the latest dev build? See here.
There was a bug opening redundant connections in the Compile phase.
Let me know how that works.

[akkce]

Hi @flarco,

Thanks for the prompt rely. I tried the same replication with the latest dev build (1.5.5.dev (2026-01-09)) and unfortunately, I'm still seeing the same condition. Although this time, the connections that are successful do end up writing the data into the database.

I wrote up a quick bash script to keep track of SFTP connections and sling starts reporting failures right under 20 (which would make sense, most of the SFTP server implementations I'm familiar with limit concurrent connections around that number).

#!/bin/bash

while true; do
    ss -tn |grep "SFTP_SERVER_IP" |grep ESTAB | wc -l
    sleep 1
done

I poked through the source code a bit last night and I couldn't find any mention of the max_connection stream variable. I'm beginning to wonder if that particular configuration suggestion is just an AI-mirage. I didn't find a mention of it in the documentation either.

Is it supposed to be a "passthrough" configuration to the system's SSH client? Or does have some effect in the stream_processor.go code that I couldn't find?

I'm happy to help troubleshoot this, if you point me in the right direction.

Thank you.

[flarco]

Hey @akkce , one thing to try: set env var CONCURRENCY_LIMIT=1 or something higher.
Code for that is here. I think this should help by limiting the number of concurrent readers.

There is also the concurrency property. You can set that at the connection level:

connections:
  MY_SFTP:
    type: sftp
    host: <host>
    user: <user>
    port: <port>
    concurrency: 3

Those are mostly undocumented, included a while back, I'll need to look it over to improve and document better.

[akkce]

Unfortunately, I'm still seeing the same behavior, although with concurrency to set 3 (and 1... I tried both), it makes much longer for the SFTP server to kick me.

It's like there needs to be a counter that keep tracks of how many SFTP connections a stream has opened and once it hits a configurable limit it pauses to let some complete before opening more and/or the connections need to explicitly closed once that individual file is downloaded. I'm not sure how difficult that'd be to implement.

The workaround, at least for my case, would be restructure how my upstream vendor outputs the files to reduce the total count, use some kind environmental variable for a datestamp in the filename or just setup explicit streams for every file. It's a small enough set that at ~ 30 jobs wouldn't be too painful to click through to reconfigure but annoying nevertheless.

In an ideal world, I could just add or remove a data export job and sling would pick it up from the right folder on SFTP server and write it into my database without any extra configuration from me. Then all the post-processing / transforms stuff can be done in SQL or in a separate sling replication.

That being said, I imagine that would not be a tenable solution at scale.

Curious to see what your thoughts are?

[akkce]

This appears to be resolved in the latest version (1.5.13). The per-connection concurrency setting is respected by the CSV writers however the more general environmental variable CONCURRENCY_LIMIT did not seem to have an effect.