Embed the Linux agents in the CLI instead of downloading them

.github/workflows/release.yaml

@@ -155,6 +155,12 @@ jobs:
         with:
           go-version: 1.21.8
 
+      # The CLI embeds the Linux agents, which must have the same version as the CLI
+      - name: Build Linux agents
+        shell: bash
+        run: |
+          LDFLAGS="-s -w -X github.com/loft-sh/devpod/pkg/version.version=v${{ needs.create-release.outputs.original_package_version }} -X github.com/loft-sh/devpod/pkg/devcontainer/crane.craneSigningKey=${{ secrets.CRANE_PRIVATE_KEY }}" ./hack/build-agents.sh
+
       - name: Build Sidecar CLI
         if: matrix.settings.host != 'windows-latest'
         run: |

AGENTS.md

@@ -29,6 +29,7 @@ go build ./...
 go vet ./...
 ./hack/unit-tests.sh          # what CI runs; `go test ./pkg/... ./cmd/...` is the quick version
 make build                    # local CLI + desktop build via hack/rebuild.sh
+./hack/build-agents.sh        # the Linux agents the CLI embeds; run before `go build` of a CLI you want to use
 ```
 
 Lint, the same way CI does (golangci-lint v1.60.1 cannot be built with the newest Go):
@@ -64,15 +65,16 @@ cd src-tauri && cargo +stable check
   deep link scheme, the Flatpak id and existing installs.
 - **Dependencies are vendored.** Builds use `-mod=vendor`. After changing imports or `go.mod`, run
   `GOFLAGS=-mod=mod go mod tidy && go mod vendor` and commit `vendor/`.
-- **The same binary runs on both sides.** The CLI injects itself as the agent into the machine/container (or the
-  agent downloads it from `curityio/devpod` releases). Structs serialized between them (`CLIOptions`,
-  `AgentWorkspaceInfo`, `ContainerWorkspaceInfo`, etc.) are a wire format between CLI and agent.
-- **Dev builds can run a stale agent.** Unless the host is Linux with the container's arch (then the CLI injects
-  itself), the CLI injects `devpod-linux-<arch>` from `devpod-cache/` under Go's `os.TempDir()` (`$TMPDIR` on
-  macOS) and never refreshes a file that is already there. `hack/build-e2e.sh` and `hack/rebuild.sh` (when building
-  linux) copy fresh binaries there; if you build some other way, copy them yourself. Also, all dev builds report
-  version `v0.0.0`, so a container that already has an agent keeps it: recreate the workspace to test agent
-  changes.
+- **The agent is embedded in the CLI.** `hack/build-agents.sh` builds the Linux agents (amd64 and arm64, with the
+  `devpod_agent` build tag) into `pkg/agent/agentbin/bin/`, and the CLI embeds them; it never downloads an agent.
+  The agent injects itself from the machine into the container. Without that script, `go build` makes a CLI with
+  no agents, which can't create workspaces (unless it's built for Linux, where it uses itself). DevPod replaces an
+  agent of another version, but state an older agent wrote on a machine or in a container (e.g. the serialized
+  `AgentWorkspaceInfo` and `ContainerWorkspaceInfo`) is still read by the new one.
+- **Builds without a version keep a container's agent.** DevPod only replaces an agent whose version differs.
+  `hack/rebuild.sh` and `hack/build-e2e.sh` give each build its own version (`v0.0.0-dev.<UTC time>`, or `VERSION`),
+  shared by the CLI and its agents. A plain `hack/build-agents.sh` and `go build` both use `v0.0.0`, so a container
+  that already has a `v0.0.0` agent keeps it.
 - **Don't import `k8s.io/client-go/kubernetes`** (or its typed clients, or anything that uses them, such as
   go-containerregistry's `authn/kubernetes`). Its scheme links every Kubernetes API group and adds over 20 MB to
   the binary, which also runs as the agent. Use `pkg/kubeclient`, which only registers core/v1 and rbac/v1; add a

CHANGELOG.md

@@ -35,6 +35,14 @@ Add new entries to the top of the "Unreleased" section.
 
 ### Changed
 
+- The DevPod CLI embeds the Linux agent for amd64 and arm64 ([#19](https://github.com/curityio/devpod/pull/19)),
+  whatever platform it runs on, and copies it into machines and dev containers itself. It no longer downloads the
+  agent from GitHub releases, so the agent always has the same version as the CLI, and workspaces can be created
+  without access to GitHub. The `AGENT_URL` context option, the `DEVPOD_AGENT_URL` environment variable, the
+  provider's `agent.downloadURL` and the desktop app's "Agent URL" setting are removed.
+- When DevPod replaces the agent of an existing dev container that a different DevPod version set up, it warns that
+  you may want to re-create the workspace with `devpod up --recreate`
+  ([#19](https://github.com/curityio/devpod/pull/19)).
 - The `devpod` binary is about 40% smaller ([#21](https://github.com/curityio/devpod/pull/21)): the stripped
   linux/amd64 build went from 68 MB to 41 MB (20 MB to 14 MB gzipped). The Kubernetes driver and the in-cluster
   registry keychain now use a small client for the core/v1 and rbac/v1 APIs instead of client-go's clientset,

CONTRIBUTING.md

@@ -3,16 +3,12 @@
 ## Development Setup
 
 1. Clone the repository locally
-2. If you want to change something in DevPod agent code:
-   1. Exchange the URL in [DefaultAgentDownloadURL](./pkg/agent/agent.go) with a
-      custom public repository release you have created.
-   2. Build devpod via: `./hack/rebuild.sh`
-   3. Upload `test/devpod-linux-amd64` and `test/devpod-linux-arm64` to the public
-      repository release assets.
-3. Build devpod via: `./hack/rebuild.sh` (asking for sudo password)
-4. Add docker provider via `devpod provider add docker`
-5. Configure docker provider via `devpod use provider docker`
-6. Start devpod in vscode with `devpod up examples/simple`
+2. Build devpod via: `./hack/rebuild.sh` (asking for sudo password). It first builds the Linux agents with
+   `./hack/build-agents.sh`, which the CLI embeds and copies into machines and dev containers. Each build gets its
+   own version (`v0.0.0-dev.<UTC time>`), so DevPod replaces the agent in existing workspaces.
+3. Add docker provider via `devpod provider add docker`
+4. Configure docker provider via `devpod use provider docker`
+5. Start devpod in vscode with `devpod up examples/simple`
 
 ## Build from source
 
@@ -21,7 +17,12 @@ Prerequisites CLI:
 - [Go 1.20](https://go.dev/doc/install)
 
 Once installed, run
-`CGO_ENABLED=0 go build -ldflags "-s -w" -o devpod-cli`
+`./hack/build-agents.sh && CGO_ENABLED=0 go build -ldflags "-s -w" -o devpod-cli`
+
+This build has version `v0.0.0`, so DevPod won't replace a `v0.0.0` agent that an earlier build put into a
+workspace. Without `./hack/build-agents.sh`, the CLI has no agent to copy into machines and dev containers (only a CLI built for
+Linux can then use itself as the agent, for its own architecture). Pass the same `-ldflags` to both (as `LDFLAGS`
+for the script), so that the agent has the same version as the CLI.
 
 Prerequisites GUI:
 

cmd/agent/workspace/logs.go

@@ -49,7 +49,7 @@ func (cmd *LogsCmd) Run(ctx context.Context) error {
 	logger := log.Default.ErrorStreamOnly()
 
 	// create new runner
-	runner, err := devcontainer.NewRunner(agent.ContainerDevPodHelperLocation, agent.DefaultAgentDownloadURL(), workspaceInfo, logger)
+	runner, err := devcontainer.NewRunner(agent.ContainerDevPodHelperLocation, workspaceInfo, logger)
 	if err != nil {
 		return fmt.Errorf("create runner: %w", err)
 	}

cmd/agent/workspace/up.go

@@ -137,7 +137,7 @@ func (cmd *UpCmd) devPodUp(ctx context.Context, workspaceInfo *provider2.AgentWo
 }
 
 func CreateRunner(workspaceInfo *provider2.AgentWorkspaceInfo, log log.Logger) (devcontainer.Runner, error) {
-	return devcontainer.NewRunner(agent.ContainerDevPodHelperLocation, agent.DefaultAgentDownloadURL(), workspaceInfo, log)
+	return devcontainer.NewRunner(agent.ContainerDevPodHelperLocation, workspaceInfo, log)
 }
 
 func InitContentFolder(workspaceInfo *provider2.AgentWorkspaceInfo, log log.Logger) (bool, error) {

cmd/build.go

@@ -208,8 +208,6 @@ func buildAgentClient(ctx context.Context, workspaceClient client.WorkspaceClien
 			},
 			workspaceClient.AgentLocal(),
 			workspaceClient.AgentPath(),
-			workspaceClient.AgentURL(),
-			true,
 			command,
 			stdinReader,
 			stdoutWriter,

cmd/logs.go

@@ -94,8 +94,6 @@ func (cmd *LogsCmd) Run(ctx context.Context, args []string) error {
 			},
 			client.AgentLocal(),
 			client.AgentPath(),
-			client.AgentURL(),
-			true,
 			sshServerCmd,
 			stdinReader,
 			stdoutWriter,

cmd/machine/ssh.go

@@ -87,8 +87,6 @@ func (cmd *SSHCmd) Run(ctx context.Context, args []string) error {
 			},
 				machineClient.AgentLocal(),
 				machineClient.AgentPath(),
-				machineClient.AgentURL(),
-				true,
 				command,
 				stdin,
 				stdout,

cmd/up.go

@@ -408,8 +408,6 @@ func (cmd *UpCmd) devPodUpMachine(
 			},
 			client.AgentLocal(),
 			client.AgentPath(),
-			client.AgentURL(),
-			true,
 			sshCmd,
 			sshTunnelStdinReader,
 			sshTunnelStdoutWriter,

desktop/src/types.ts

@@ -200,10 +200,7 @@ export type TSupportedIDE = (typeof SUPPORTED_IDES)[number]
 //#region Context
 export type TContextOptions = Record<TContextOptionName, TContextOption>
 // See pkg/config/context.go
-export type TContextOptionName =
-  | "AGENT_URL"
-  | "SSH_INJECT_DOCKER_CREDENTIALS"
-  | "SSH_INJECT_GIT_CREDENTIALS"
+export type TContextOptionName = "SSH_INJECT_DOCKER_CREDENTIALS" | "SSH_INJECT_GIT_CREDENTIALS"
 export type TContextOption = Readonly<{
   name: TContextOptionName
   description: string | null | undefined

desktop/src/views/Settings/Settings.tsx

@@ -39,7 +39,6 @@ import {
 } from "../../lib"
 import { useWelcomeModal } from "../../useWelcomeModal"
 import {
-  useAgentURLOption,
   useDockerCredentialsForwardingOption,
   useGitCredentialsForwardingOption,
 } from "./useContextOptions"
@@ -88,7 +87,6 @@ export function Settings() {
 function GeneralSettings() {
   const { settings, set } = useChangeSettings()
   const { modal: welcomeModal, show: showWelcomeModal } = useWelcomeModal()
-  const { input: agentURLInput, helpText: agentURLHelpText } = useAgentURLOption()
   const { input: proxyInput, helpText: proxyHelpText } = useProxyOptions()
   const {
     badge: installCLIBadge,
@@ -127,10 +125,6 @@ function GeneralSettings() {
         </Button>
       </SettingSection>
 
-      <SettingSection title="Agent URL" description={agentURLHelpText}>
-        {agentURLInput}
-      </SettingSection>
-
       <SettingSection title="Proxy Configuration" description={proxyHelpText}>
         {proxyInput}
       </SettingSection>

desktop/src/views/Settings/useContextOptions.tsx

@@ -1,12 +1,9 @@
-import { Code, Switch } from "@chakra-ui/react"
+import { Switch } from "@chakra-ui/react"
 import { useMutation, useQuery, useQueryClient } from "@tanstack/react-query"
-import { useCallback, useMemo } from "react"
+import { useMemo } from "react"
 import { client } from "../../client"
 import { QueryKeys } from "../../queryKeys"
 import { TContextOptionName } from "../../types"
-import { ClearableInput } from "./ClearableInput"
-
-const DEFAULT_DEVPOD_AGENT_URL = "https://github.com/curityio/devpod/releases/latest/download/"
 
 export function useContextOptions() {
   const queryClient = useQueryClient()
@@ -32,41 +29,6 @@ export function useContextOptions() {
   )
 }
 
-export function useAgentURLOption() {
-  const { options, updateOption } = useContextOptions()
-
-  const handleChanged = useCallback(
-    (newValue: string) => {
-      const value = newValue.trim()
-      updateOption({ option: "AGENT_URL", value })
-    },
-    [updateOption]
-  )
-
-  const input = useMemo(
-    () => (
-      <ClearableInput
-        placeholder="Override Agent URL"
-        defaultValue={options?.AGENT_URL.value ?? ""}
-        onChange={handleChanged}
-      />
-    ),
-    [handleChanged, options?.AGENT_URL.value]
-  )
-
-  const helpText = useMemo(
-    () => (
-      <>
-        Set the Agent URL. If you leave this empty, it will be pulled from{" "}
-        <Code>{DEFAULT_DEVPOD_AGENT_URL}</Code>
-      </>
-    ),
-    []
-  )
-
-  return { input, helpText }
-}
-
 export function useDockerCredentialsForwardingOption() {
   const { options, updateOption } = useContextOptions()
 

e2e/e2e_suite_test.go

@@ -1,17 +1,12 @@
 package e2e
 
 import (
-	"os"
-	"runtime"
 	"testing"
-	"time"
 
 	"github.com/onsi/ginkgo/v2"
 
 	"github.com/onsi/gomega"
 
-	"github.com/loft-sh/devpod/e2e/framework"
-
 	// Register tests
 	_ "github.com/loft-sh/devpod/e2e/tests/build"
 	_ "github.com/loft-sh/devpod/e2e/tests/context"
@@ -30,17 +25,6 @@ import (
 // generated in this directory, and cluster logs will also be saved.
 // This function is called on each Ginkgo node in parallel mode.
 func TestRunE2ETests(t *testing.T) {
-	if runtime.GOOS != "linux" {
-		go framework.ServeAgent()
-
-		// wait for http server to be up and running
-		for {
-			time.Sleep(time.Second)
-			if os.Getenv("DEVPOD_AGENT_URL") != "" {
-				break
-			}
-		}
-	}
 	gomega.RegisterFailHandler(ginkgo.Fail)
 	ginkgo.RunSpecs(t, "DevPod e2e suite")
 }

hack/build-agents.sh

@@ -0,0 +1,18 @@
+#!/usr/bin/env bash
+
+# Builds the Linux agent binaries (amd64 and arm64) that get embedded into the devpod CLI, see pkg/agent/agentbin.
+# Run it before building the CLI, with the same LDFLAGS (the version especially), so that the agent matches the CLI.
+# Without LDFLAGS, the agents have version v0.0.0, like a plain `go build` of the CLI.
+
+set -e
+
+SRC_DIR="${SRCDIR:=.}"
+LDFLAGS="${LDFLAGS:=-s -w}"
+OUT_DIR="$SRC_DIR/pkg/agent/agentbin/bin"
+
+for arch in amd64 arm64; do
+    echo "[INFO] Building agent for linux/$arch"
+    CGO_ENABLED=0 GOOS=linux GOARCH=$arch go build -tags devpod_agent -ldflags "$LDFLAGS" -o "$OUT_DIR/devpod-linux-$arch" "$SRC_DIR"
+    gzip -9 -n -f "$OUT_DIR/devpod-linux-$arch"
+done
+echo "[INFO] Built agents in $OUT_DIR"