SSH key manager

[krokicki]

This PR adds an SSH key manager to the profile menu in the upper right. This lets you create SSH keys and authorize them so that you can seamlessly move between cluster nodes. The main purpose for this is to allow easy on boarding for our Seqera Platform instance.

I tried to be careful about making backups of files in ~/.ssh before changing them, but would welcome feedback especially if there are any edge cases I didn't think of. If you are testing this, I would recommend making a backup of your ~/.ssh directory first, just in case.

@JaneliaSciComp/fileglancer @StephanPreibisch

[mkitti]

I think we should consider treating .ssh as read-only as much as possible. Keys generated by fileglancer could be stored elsewhere.

The main file that may need to be written to here might be .ssh/authorized_keys . We should consider if .ssh/authorized_keys2 could be used or if the sshd configuration could be modified to recognize an authorized_keys file exclusively managed by fileglancer.

Here is the documentation for OpenSSH sshd:
https://man.openbsd.org/sshd_config#AuthorizedKeysFile

[krokicki]

I added a VITE_ENABLE_SSH_KEYS feature toggle, and enabled both this and VITE_ENABLE_TASKS in the CI build. I also modified the implementation to do as little as possible with ~/.ssh.

[krokicki]

@mkitti As discussed, we agree about "treating .ssh as read-only as much as possible". To minimize complexity and because for now this feature has a very small user base, I've modified the GUI to do the most minimal thing possible to ~/.ssh, namely: if id_ed25519 does not exist, this tool can create one for you and add it to authorized_keys using cat. I've removed functionality around managing multiple keys, and all functionality that deletes keys. This minimizes risks to user's existing SSH keys and setups.

[mkitti]

In a few places we still read the private key into resident memory of the Python process as an immutable Python string. This is not good. Core dumps of the Python process may contain the contents of the private key.

Rather than loading file contents into an immutable string, we should read the contents into a mutable bytearray and then explicitly overwrite the contents of the bytearray when the contents are no longer needed.

import os

key_path = "private_key.pem"
file_size = os.path.getsize(key_path)

# 1. Pre-allocate mutable memory
sensitive_data = bytearray(file_size)

# 2. Read directly into the pre-allocated buffer
with open(key_path, "rb") as f:
    f.readinto(sensitive_data)

# Use the bytes as needed

# 3. Securely overwrite the memory with zeros
for i in range(len(sensitive_data)):
    sensitive_data[i] = 0

# 4. Now safe to remove the reference
del sensitive_data

References:

1. https://stackoverflow.com/questions/728164/securely-erasing-password-in-memory-python#:~:text=The%20correct%20solution%20is%20to,bytearray%22%20instead%20of%20python%20strings.
2. https://softwareengineering.stackexchange.com/questions/270113/using-a-bytearray-rather-than-a-string-to-store-password-in-memory#:~:text=Using%20a%20bytearray%20datatype%20to,in%20python%203.4%20using%20bytearray?

[mkitti]

Consider using a bytearray and perhaps a context manager to minimize time that a private key spends in resident memory of the Python process.

Ref: https://softwareengineering.stackexchange.com/a/270159

[mkitti]

Consider making the private key contents a mutable bytearray that we can explicitly overwrite to clear from memory. Avoid converting the bytearray to an immutable string.

[mkitti]

Perhaps we could make SSHKeyContent follow the context manager protocol so we can use a with statement. On __exit__ the context manager would then make sure to shred the key contents in memory by zeroing out the resident bytearray. You could use contextlib to help with this.

I also suggest centralizing the the potentially sensitive reading implementation here.

1. The class would only store a string or Path containing the location of the key file.
2. The context manager __enter__ would read the file contents into a bytearray via readinto and then return that bytearray
3. The context manager __exit__ would then explicitly overwrite the bytearray.

This will minimize the time the any sensitive secret spends in resident memory.

[mkitti]

Consider making the passphrase a mutable bytearray that we can explicitly overwrite to clear from memory. Avoid converting the bytearray to an immutable string.

[mkitti]

Use readinto to read into a mutable bytearray.

[mkitti]

Shred the key contents in a bytearray after usage. Ensure the contents get shreded even if an error occurs. Consider using a context manager here.

[mkitti]

Could we add a restrict keyword here? Does Seqera need an interactive terminal or port forwarding capabilities?

https://manpages.debian.org/experimental/openssh-server/authorized_keys.5.en.html#restrict

[mkitti]

Perhaps we could also restrict what domain names or IP addresses this key can be used from?
https://manpages.debian.org/experimental/openssh-server/authorized_keys.5.en.html#from=_pattern-list_

[mkitti]

At the end could we add a comment as the last space delimited field that says this was added by fileglancer for seqera?

[mkitti]

If I understand correctly, the exclusive user of the private key should be Seqera alone. The private key does not need to be used by the user for any other purpose. Why do we need to store the private or public key to disk at all other than adding it to the authorized_keys file?

Rather we only need to transiently show the private key to the user via the web interface to copy into Seqera. If they fail to do so, then we just generate another private key anew the next time.

That process would work more like how Github personal access tokens work. Take a look at how the user interface works for tokens.

If you push the "Generate a new token" button and fill out the form, it will show you the token once in this example (I deleted the key before posting the screenshot):

It warns that you will not be able to see the personal access token again. Similarly, we can issue the same warning when showing the user the private key for Seqera. If they lose the key, we can just generate another private key.

In summary, there is no need to write the private key to ~/.ssh/id_ed25519 or the public key to ~/.ssh/id_ed25519.pub. Rather we can generate the private key and the public key at some temporary file location or just just store it transiently in memory. All we need to do is add the public key to the ~/.ssh/authorized_keys file, preferably with restrictions and a comment, and show the private key to the user once via the web interface. The private key never needs to be seen again.

[krokicki]

I implemented the bytearray clearing. The rest are also good ideas but will be deferred to a future release.

[mkitti]

Decoding this to a string does defeat the point of using a bytearray. If we could avoid that somehow, that would be better. The problem is that we do not know when the string will be garbage collected and we cannot control when the string will be overwritten.

If elimanating the use of string is not possible, minimizing the number of references to the content would be helpful. In this case, just return directly rather than adding a named reference in result.

Suggested change

	result = sensitive_buffer.decode('utf-8')
	return result
	return sensitive_buffer.decode('utf-8')

[mkitti]

Suggested change

	key_content = _read_file_secure(private_key_path)
	return SSHKeyContent(key=key_content)
	return SSHKeyContent(
	key=_read_file_secure(private_key_path)
	)

Remove extra references to the string.

[mkitti]

Decoding the bytearray to a string largely defeats the point of using the bytearray. The problem is that we cannot control when the string will be cleared from memory. At minimum, we could try to reduce references to the string to increase the chances that it can be immediately garbage collected.

Gemini suggests that it might be possible to keep using a bytearray and avoid the string by using a custom Response class.

from fastapi import FastAPI, Response
import gc

app = FastAPI()

class SecureResponse(Response):
    def __init__(self, content: bytearray, *args, **kwargs):
        super().__init__(content=content, *args, **kwargs)
        # Store a reference to the mutable bytearray to clear it later
        self._sensitive_content = content

    async def __call__(self, scope, receive, send):
        # 1. Standard response logic sends the data to the client
        await super().__call__(scope, receive, send)
        
        # 2. Immediately zero out the sensitive data after sending
        for i in range(len(self._sensitive_content)):
            self._sensitive_content[i] = 0
            
        # 3. Explicitly clear the reference and suggest garbage collection
        del self._sensitive_content
        gc.collect()

@app.get("/secure-data")
async def get_secure_data():
    # sensitive_data must be a mutable bytearray
    sensitive_data = bytearray(b"SENSITIVE_SECRET_2026")
    
    return SecureResponse(
        content=sensitive_data, 
        media_type="application/octet-stream"
    )

I would consider a try-finally around the call to super().__call__.

try:
    await super().__call__(scope, receive, send)
finally:
    # clean-up     

https://share.google/aimode/d54nOXW54RdNTA5aT

[krokicki]

@mkitti @clements @allison-truhlar
This PR is ready to be reviewed, thanks!

[mkitti]

It seems we may have narrowed the scope. If so we might be able to tighten the restrictions on the usage of the keys. For example, we could restrict the valid hosts by adding a "from" field to the authorized_keys entry. If this is only for Seqera, then there should only be one host from which this key is used.

I added pty capability to the key, but pseudo terminal access may also not be necessary if the intended use of the key is limited.

[neomorphic]

There are a few UI tweaks that would make dark mode better, but other than that it looks good. I was able to generate 4 keys, all of which were added to my authorized_keys file.

[allison-truhlar]

I looked at this with Jody earlier and don't have anything to add beyond his comments.

I tried out the UI and was able to add a key to my authorized_keys. When I manually deleted it from authorized_keys, it no longer was visible in the UI, as expected.