Steps to reproduce
On a fleet whose instance holds more than one job at a time (blocks set, so jobs are co-located by design):
- Create a backend fleet with blocks, e.g.
nodes: 1, blocks: auto, 4 vCPU.
- Apply a dev environment pinned to that fleet requesting 1 block, and wait for
running.
dstack attach <dev-env>, then ssh <dev-env> - works.
- Apply any short task pinned to the same fleet, also 1 block, and let it run to completion.
ssh <dev-env> again.
Actual behaviour
Step 5 fails with Permission denied (publickey).
The shim appends cfg.HostSshKeys to the host user's authorized_keys when a task starts, and in a deferred call removes those same keys when that task ends:
|
if len(cfg.HostSshKeys) > 0 { |
|
ak := AuthorizedKeys{user: cfg.HostSshUser, lookup: user.Lookup} |
|
if err := ak.AppendPublicKeys(cfg.HostSshKeys); err != nil { |
|
errMessage := fmt.Sprintf("ak.AppendPublicKeys error: %s", err.Error()) |
|
log.Error(ctx, errMessage) |
|
task.SetStatusTerminated(string(types.TerminationReasonExecutorError), errMessage) |
|
return fmt.Errorf("append public keys: %w", err) |
|
} |
|
defer func(cfg TaskConfig) { |
|
err := ak.RemovePublicKeys(cfg.HostSshKeys) |
|
if err != nil { |
|
log.Error(ctx, "Error RemovePublicKeys", "err", err) |
|
} |
|
}(cfg) |
|
} |
RemovePublicKeys drops every matching line unconditionally:
|
func RemovePublicKeys(fileKeys []string, keysToRemove []string) []string { |
|
newKeys := slices.DeleteFunc(fileKeys, func(fileKey string) bool { |
|
delete := slices.ContainsFunc(keysToRemove, func(removeKey string) bool { |
|
return IsPublicKeysEqual(fileKey, removeKey) |
|
}) |
|
return delete |
|
}) |
|
return newKeys |
|
} |
All jobs of a project share the same user key, so the teardown of one job removes the key the other jobs on that instance are still relying on. There is no refcounting and no check for remaining tasks.
Expected behaviour
A job finishing should not affect SSH access to other jobs still running on the same instance. The key should only leave authorized_keys when the last task using it goes away - e.g. refcount the appended keys, or reconcile against the set of currently running tasks instead of removing per task.
dstack version
Reproduced on master at 2517069.
Steps to reproduce
On a fleet whose instance holds more than one job at a time (
blocksset, so jobs are co-located by design):nodes: 1,blocks: auto, 4 vCPU.running.dstack attach <dev-env>, thenssh <dev-env>- works.ssh <dev-env>again.Actual behaviour
Step 5 fails with
Permission denied (publickey).The shim appends
cfg.HostSshKeysto the host user'sauthorized_keyswhen a task starts, and in a deferred call removes those same keys when that task ends:dstack/runner/internal/shim/docker.go
Lines 422 to 436 in 2517069
RemovePublicKeysdrops every matching line unconditionally:dstack/runner/internal/shim/authorized_keys.go
Lines 38 to 46 in 2517069
All jobs of a project share the same user key, so the teardown of one job removes the key the other jobs on that instance are still relying on. There is no refcounting and no check for remaining tasks.
Expected behaviour
A job finishing should not affect SSH access to other jobs still running on the same instance. The key should only leave
authorized_keyswhen the last task using it goes away - e.g. refcount the appended keys, or reconcile against the set of currently running tasks instead of removing per task.dstack version
Reproduced on master at 2517069.