Skip to content

[Bug]: A finishing job revokes host SSH access for still-running co-located jobs #4174

Description

@r4victor

Steps to reproduce

On a fleet whose instance holds more than one job at a time (blocks set, so jobs are co-located by design):

  1. Create a backend fleet with blocks, e.g. nodes: 1, blocks: auto, 4 vCPU.
  2. Apply a dev environment pinned to that fleet requesting 1 block, and wait for running.
  3. dstack attach <dev-env>, then ssh <dev-env> - works.
  4. Apply any short task pinned to the same fleet, also 1 block, and let it run to completion.
  5. ssh <dev-env> again.

Actual behaviour

Step 5 fails with Permission denied (publickey).

The shim appends cfg.HostSshKeys to the host user's authorized_keys when a task starts, and in a deferred call removes those same keys when that task ends:

if len(cfg.HostSshKeys) > 0 {
ak := AuthorizedKeys{user: cfg.HostSshUser, lookup: user.Lookup}
if err := ak.AppendPublicKeys(cfg.HostSshKeys); err != nil {
errMessage := fmt.Sprintf("ak.AppendPublicKeys error: %s", err.Error())
log.Error(ctx, errMessage)
task.SetStatusTerminated(string(types.TerminationReasonExecutorError), errMessage)
return fmt.Errorf("append public keys: %w", err)
}
defer func(cfg TaskConfig) {
err := ak.RemovePublicKeys(cfg.HostSshKeys)
if err != nil {
log.Error(ctx, "Error RemovePublicKeys", "err", err)
}
}(cfg)
}

RemovePublicKeys drops every matching line unconditionally:

func RemovePublicKeys(fileKeys []string, keysToRemove []string) []string {
newKeys := slices.DeleteFunc(fileKeys, func(fileKey string) bool {
delete := slices.ContainsFunc(keysToRemove, func(removeKey string) bool {
return IsPublicKeysEqual(fileKey, removeKey)
})
return delete
})
return newKeys
}

All jobs of a project share the same user key, so the teardown of one job removes the key the other jobs on that instance are still relying on. There is no refcounting and no check for remaining tasks.

Expected behaviour

A job finishing should not affect SSH access to other jobs still running on the same instance. The key should only leave authorized_keys when the last task using it goes away - e.g. refcount the appended keys, or reconcile against the set of currently running tasks instead of removing per task.

dstack version

Reproduced on master at 2517069.

Metadata

Metadata

Assignees

Labels

bugSomething isn't working

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions