L'Atelier

L'Atelier: What v1 of My Agent Platform's Security Got Wrong

An open agent port, SSH host keys shipped in a public image, one user's GitHub token lent to another. How a one-user design held until a team used it, and the fix for each hole.

By 21 min readUpdated
Threat ModelingKubernetesNetworkPolicySSHGitHub OAuthSelf-HostingAI AgentsRust

Until early November 2026, any pod on the sandbox network could send POST /exec to any other sandbox’s agent and run a root command there. The in-pod agent listened on 0.0.0.0:9998 with no authentication, and the namespace had no NetworkPolicy. The commit that closed it (c6b9ed84, 2026-11-05) opens with that sentence. I wrote that commit, and the code it fixed.

L’Atelier is my self-hosted orchestrator for AI coding agents: each sandbox is a Kubernetes pod inside a Kata VM, and a Rust agent inside it takes commands from the server (why I built it, the Kubernetes move). I designed it for one user, me. Then my team at Frak started using it, and in November a second company, Circles, deployed its own instance. I went through the code asking one question per boundary: what happens when the person on the other side is not me?

Each hole below comes with the date it appeared and the date it was fixed, what an attacker could actually do given how the cluster is wired, and where I could not tell. The product side of the same shift is in a sibling article: team use, the review Action, the MCP rework, the features I deleted. This one is only security.

I have no evidence that any of these was exploited, and I have not looked for any beyond the commit-time checks quoted below. Every hole in the table below is closed on main. All the fixes are in the v3.0.2 tag (2026-11-11) except two: service accounts (eca96762) and the revoked-token sign-out (334cc7be), which came after it. Other gaps are still open and listed at the end.

The Assumption Behind Each Hole

Every weakness below was a reasonable shortcut under one of three assumptions: there is one user, the network inside the cluster is friendly, and there is one GitHub identity. A table of when each appeared and left:

WeaknessIntroducedFixed
Agent API open to the pod network2026-03-02 (af79edac, vsock to TCP)2026-11-05 (c6b9ed84)
Host keys baked into the image2026-06-07 (25df6f91)2026-09-23 (25f8d52c), pin required 2026-10-13 (f42184b8)
Any SSH key into any sandboxin-server SSH gateway, 2026-07-16 (bb5d1ab7), possibly earlier2026-10-12, PR #21 by aramalipoor
No sandbox access rule on the API/v1 shell, 2026-07-022026-10-12 (3a5b67c0, PR #22)
/api/secrets without authorization5e59ebdf, 2026-07-022026-10-12 (PR #23)
Another user’s GitHub token as fallback70323995, 2026-03-262026-10-13 (143c1403)
Tool URLs checked against nothing, then against “signed in”forward-auth annotations existed before d210268e2026-10-16, then per sandbox on 2026-11-05 (33a79917)

The dates come from git log -S on the relevant lines, so “introduced” is the commit that added the code. The dates run from March to July, and until a team used the tool nobody but me had an account, so these sat latent for months.

After the fixes, the same boundaries look like this:

The Agent Port Was Open to Every Pod

In the Firecracker version, the agent talked to the host over vsock (a host-to-guest socket), which is unreachable from anywhere else. On 2026-03-02 I moved it to a TCP listener, because pods have IPs and Kubernetes has no vsock. The commit message mentions the transport. It says nothing about who could now connect.

The agent exposes POST /exec (a login shell as root in the pod), file writes, process control and a terminal. The pod already received a per-sandbox AGENT_PASSWORD in its environment. The Rust agent never checked it. Under k3s, with flannel and no policy, every pod can reach every pod.

The attacker I worry about is a sibling. Pod IPs live on the cluster’s overlay network, so the likely source is another sandbox. That sandbox runs an AI agent that reads issues, web pages and dependencies. One prompt injection is enough to send it probing its neighbours, as the header comment of the NetworkPolicy file says: “a prompt-injected agent in sandbox A could otherwise reach sandbox B’s agent”. From B’s agent it could read the git credentials and resolved secrets written into B, or push as B’s owner.

I fixed it with two layers, deliberately redundant. The first is the network:

# infra/k8s/85-network-policy.yaml (abridged)
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-ingress
  namespace: atelier-v2-sandboxes
spec:
  podSelector: {}
  policyTypes:
    - Ingress
---
# ... metadata: allow-from-server
  ingress:
    - from:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: atelier-v2-system
          podSelector:
            matchLabels:
              app.kubernetes.io/name: atelier-v2
              app.kubernetes.io/component: server
---
# ... metadata: allow-from-ingress-controller
  ingress:
    - from:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: kube-system
          podSelector:
            matchLabels:
              app.kubernetes.io/name: traefik

The Traefik rule allows any port, because tool ports are declared per spec and unknown to the policy.

The second is the token. Every route except GET /health needs the bearer token, and an agent with no token configured serves nothing else, instead of degrading to open:

// apps/agent/src/auth.rs
fn check(&self, presented: Option<&str>) -> Result<(), Denied> {
    let Some(expected) = &self.token else {
        return Err(Denied::NotConfigured);
    };
    match presented {
        Some(p) if constant_time_eq(p.as_bytes(), expected.as_bytes()) => Ok(()),
        _ => Err(Denied::Unauthorized),
    }
}

Both WebSocket listeners check it too, from the header or a ?token= query, because the server’s relays dial bare URLs. That query parameter is an open item (see the end). The same commit hardened exec itself. The shell now leads its own process group, so a timeout or an output overrun kills the whole tree (npm test, oras) instead of the shell alone. File writes go to a temp file created with its final mode and owner before any content lands, then rename into place, because those files carry resolved secrets and used to be briefly readable at the default mode.

The commit is flagged as breaking: the server, the agent image and the policy have to roll together, because a new agent refuses a server that sends no token, and an old agent ignores it. deploy.sh applies the policy now (52b04b96, the same day).

In production the three policies exist, created on 2026-11-05 at 09:04 UTC. I have not run the probe the policy header describes, one sandbox calling another’s :9998, against production. Two things remain open. Egress from sandboxes is unrestricted, since they need to clone and install. And AGENT_PASSWORD is set as a literal value in the pod spec instead of a secret reference, so anything that can read pods in that namespace can read it.

Host Keys in a Public Image

The dev-base Dockerfile got RUN ... && ssh-keygen -A on 2026-06-07. The reason was boot time: docs/sandbox-boot-optimization.md measured 100 to 300 ms of key generation on every spawn. The comment I left next to it:

# ssh-keygen -A bakes host keys into the image so they are not regenerated on
# every boot. Sandboxes share host keys, which is safe here: SSH only enters via
# sshpiper, which sets ignore_hostkey: true (no client ever pins a sandbox key).

That reasoning holds while nobody verifies host keys and nobody else can read the image. On 2026-07-21, commit 15c0797e added a base-images job to release.yml that builds dev-base and pushes it to ghcr.io/frak-id/atelier-dev-base, so a new operator can start a sandbox without an in-cluster builder. The package is public: an anonymous token request for repository:frak-id/atelier-dev-base:pull succeeds. The ssh-keygen -A is in the Dockerfile at v3.0.0, and every nightly build until the fix ran the same line. So from that day, the private halves of the host keys every sandbox shared sat in an image anyone could pull. I did not pull a pre-fix layer to check. The Dockerfile and the workflow are my evidence.

The keys alone log nobody in. A host key proves the server’s identity to the client. Here the client is the gateway, which dials the pod and authenticates with its own key. The gateway’s key is what opens sshd, and it lives in a Kubernetes Secret, not in the image. An attacker holding the host private key cannot log in to anything. They could impersonate a sandbox to the gateway, which needs a network position between the gateway and the pod IP, and which the flat pod network of the previous section made less remote than I would like. The fake upstream would receive whatever the user typed or forwarded through that SSH session, including VS Code Remote traffic. SSH public-key authentication signs the session identifier of that one connection, so a fake server cannot replay the gateway’s signature elsewhere. I did not try the attack.

In practice the check did not exist. Both gateways ignored upstream host keys: the sshpiper Pipe had ignore_hostkey, and the in-server proxy had hostVerifier: () => true. A leaked key mattered only once something started checking, and it would have made the check worthless.

The fix on 2026-09-23 has three parts. The image ships no host keys, the Dockerfile removes /etc/ssh/ssh_host_*, and sandbox-boot.sh runs ssh-keygen -A on every boot, before sshd and the agent start. The agent gets GET /ssh/host-keys, which returns the public halves over the authenticated agent channel, so there is no trust-on-first-use over SSH. The server fetches them at boot and resume and pins them: the in-server proxy compares the upstream key against the stored one, and the sshpiper Pipe got known_hosts_data through a JSON merge-patch.

That first version had a fallback for old images: if the agent returned 404, SSH stayed unpinned. I wrote it for compatibility and it lived 20 days. On 2026-10-13 f42184b8 removed it, because every sandbox now ran an agent that serves the endpoint and the fallback also swallowed transient fetch errors. A fetch that fails after retries now fails the boot, and the proxy fails closed:

// apps/server/src/ssh/proxy.ts (abridged)
const pinned = hostKeyLines?.length ? parseAuthorizedKeys(hostKeyLines) : [];
if (pinned.length === 0) {
  log.warn({ sandboxId }, "no pinned ssh host key for sandbox; refusing the upstream");
  return () => false;
}
return (keyBuf: Buffer) =>
  pinned.some((k) => keyEquals(k.getPublicSSH(), keyBuf));

The commit message records a staging check: every sandbox had recorded host keys. The keys change on every boot, so every boot and resume re-pins.

The leaked keys cannot be recalled. The pre-fix image I checked no longer pulls, though not by design: the 3.0.0 tag is still listed on GHCR, and a pull answers manifest unknown. A daily prune job deletes untagged package versions older than a week. As far as I can tell it counts the per-architecture images inside each multi-arch tag as untagged, so it has been deleting the image under every tag it was meant to keep, including the server and console releases, which is a separate bug to fix. For the keys, it means anyone who pulled a pre-fix image between 2026-07-21 and the day the job deleted it keeps them. I do not know exactly when that was.

The fix also has a gap I did not close. ssh-keygen -A only creates missing keys, and the pod uses imagePullPolicy: IfNotPresent. A node that cached a pre-fix dev-base, or any image, toolbox or prebuild built FROM one, still boots with the shared keys, and the server then pins the leaked key. I have not audited node caches or old prebuilds for it. Nothing in CI checks that a built image has no ssh_host_* files either, though a one-line docker run ... ls step would.

One Access Rule, and a 404

The first multi-user pass left the API in a state I only noticed by reading it as a second user.

SSH was the clearest case. The gateway took the sandbox id as the SSH username and admitted any registered public key, the whole table. So a colleague with a key could log in to my sandbox. An outside contributor, aramalipoor, saw this while building the Steery integration in PR #21 and made the SSH gateway admit a key only when its owner created the sandbox or belongs to its org. His PR description says why: “/v1 has no per-user sandbox rule today, so this is the narrowest one the records support”.

That sentence was right, and PR #22, merged right after it on 2026-10-12, generalised it. The same rule now covers /v1/sandboxes/:id*, the agent and terminal routes, both WebSocket upgrades, the MCP tools, toolbox capture and snapshot sources, so a pause snapshot of my sandbox cannot seed someone else’s. It lives in one pure function and one guard per router:

// apps/server/src/control/sandbox-access.ts
export function canAccessSandbox(deps, userId, labels): boolean {
  const ownerId = sandboxOwnerId(labels);
  // Before the bypass: a service account never reaches a person's sandbox,
  // local mode included.
  if (deps.isServiceAccount(userId)) return ownerId === userId;
  if (deps.isAuthBypassed()) return true;
  if (ownerId === userId) return true;
  if (isOwnerOrOrgMember(deps, userId, labels)) return true;
  return (
    ownerId !== undefined &&
    deps.isServiceAccount(ownerId) &&
    deps.isServerAdmin(userId)
  );
}
// apps/server/src/api/sandbox-guard.ts
const labels = runtime.getLabels(sandboxId);
if (!labels || !control.canAccessSandbox(userId, labels)) {
  throw new NotFoundError("Sandbox", sandboxId);
}

A sandbox you cannot reach answers 404, the same as an unknown id, so the response does not tell you which ids exist. Server-wide admin actions got the same treatment: /v1/images*, config and toolset publishing need a server admin, meaning an owner of the default org. Callers can no longer set atelier.dev/owner-id or the org annotation on create. Those labels are the rule’s input. Job results also stopped carrying the generated block. The jobs feed was readable by every authenticated user, and that block held the per-sandbox agent password, the pod IP and the SSH host keys. The leak mattered more once the agent started checking its token, 24 days later.

Two smaller findings from the same pass. SSH keys with a CR or LF are rejected, because a newline in a pasted key would add authorized_keys lines. And a sandbox with neither an owner nor an org label used to stay open to every authenticated caller (143c1403). Owner ids have been recorded since 2026-07-09, so such a sandbox could only be a pre-ownership leftover, and the commit message records a staging check that found none. It is now reachable by nobody.

Revocation had a gap of its own. With the old sshpiper gateway, SSH keys were fixed when a sandbox was created, so a revoked key kept working on existing sandboxes. The in-server gateway checks keys on every connection. sshpiper left the code on 2026-11-05, and nothing of it remains in the production cluster.

The Secrets API

/api/secrets had existed since the API shell on 2026-07-02, with no authorization at all. Any signed-in user could list, set and delete the secrets of any org, and the global (org-less) ones too. Listing returns names only (SecretService.list is documented that way), so nobody could read a value through it. Writing was the problem. A spec holds {"$secret": "NAME"} references resolved at boot, and overwriting a secret changes what every sandbox referencing it receives on its next boot. A global one reaches every org’s sandboxes.

PR #23, merged 2026-10-12, set the rule:

SecretListSet / delete
Org secretany memberowner or admin
Global (org-less)only when auth is bypassed (local mode)same

The admin role went away on 2026-11-05 (ea7f7f84), and since then only the owner writes org secrets.

In production, global secrets are now managed by the server alone: the API refuses them, even for an org owner. Deleting another org’s secret answers 404 for the same reason as the sandboxes. The console had a related trap. Settings > Secrets defaulted to “Personal (no org)”, which was the global scope shared by every user, so a person who thought they were saving a private secret saved a global one. It now defaults to the personal org, and secrets saved under the old option are global and disappear from the console in production. They still apply to sandboxes created without an org. On 2026-11-05 a read-only query of production found exactly one global secret, an SSH host key.

Never Lend a Token Across Users

resolveGitHubToken(userId?) had this fallback, added on 2026-03-26 when tokens moved onto the users table:

// apps/server/src/control/modules/user/user.service.ts (before 143c1403)
if (userId) {
  const token = this.getById(userId)?.githubAccessToken;
  if (token) return token;
}
return this.userRepository.findFirstWithToken()?.githubAccessToken;

When the caller had no token of their own, the server took the first user in the table that had one and wrote it into the new sandbox’s git credentials. With one user that is a convenience. With two, a person without GitHub OAuth got another person’s token in their sandbox, and an agent in it could push as them. That is the OAuth token with repo scope and no expiry, as the service-account commit message describes it.

How likely was it? Login requires membership of the allowed GitHub org and mints a token, so a user without one is unusual. On 2026-09-24 I had already stopped the resume path from falling back for a sandbox with no recorded owner. On 2026-10-13 I removed the function. userId is required, only that user’s token comes back, and findFirstWithToken no longer exists. The commit message records a staging check that found no user without a token. I did not repeat it on production.

Two related changes took the token out of places it had no business being. Until 2026-11-05 (905c5d00), the spec stored with a sandbox was the enriched one: secret values and the owner’s git credential file were in sandboxes.spec. Specs are now stored unresolved. resolveBootSpec runs at every boot, resolving $secret references and rebuilding the git files from the owner’s current token, so a rotated secret applies on the next boot and nothing secret is persisted in that column. The same commit stopped GET /v1/sandboxes/:id from returning generated credentials. And in the same release the user’s GitHub token in the database became encrypted at rest (ea7f7f84, AES-256-GCM with the secrets key).

A session can also outlive the credential behind it. A person’s console session is a 7-day JWT, and GitHub revokes tokens for several reasons (app revoked, a year unused, or the oldest of more than ten for the same user and scopes). The dashboard stayed signed in on a dead token, and running sandboxes kept the token of the login that booted them. 334cc7be (2026-11-12) ties the session to the stored token and checks it against GitHub (GET /user, at most every 5 minutes per person); only a 401 counts as a revocation.

Agents Need Their Own Identity

An agent shared by several people had to use one person’s API key, so it acted as that person. The commit message for service accounts (eca96762, 2026-11-11) lists the consequences: its sandboxes were the person’s, boot wrote the person’s GitHub token into every sandbox it drove, and the key reached /mcp/admin and every admin route when that person was a server admin.

A service account is a users row of kind service, with id sa_… and username <name>[bot] (brackets cannot appear in a GitHub login, so the name never collides with a person who signs in later). Server admins manage them, and the security properties are all negatives:

API keys also got a scope. An mcp key works on /mcp and gets a 403 on /mcp/admin and the HTTP API. Keys of a service account default to mcp.

The limit is in the docs: an account in an org receives every secret of that org by name, so an agent belongs in an org of its own that holds only what it needs. And anything the agent produces is untrusted input for people. A person who boots a service account’s prebuild runs agent-chosen code with their own token. docs/service-accounts.md says to take an agent’s work through a branch and a pull request. That gap is open.

Tool URLs

Per-sandbox tool URLs (code-server, the browser, and pi-web, the coding agent’s web UI) can carry a forward-auth annotation, which makes Traefik ask the server before each request. Before d210268e (2026-10-16), the annotations pointed at an endpoint that did not exist and staging configured none: “anyone with a pi-web link could drive the coding agent”. GET /auth/forward now lets a signed-in session through, sends a browser page load through the GitHub login (return_to limited to the base domain, to avoid an open redirect), and answers 401 to the rest.

The first version of the rule was “signed in to Atelier”, which is the wrong granularity: any member of the GitHub org could open any sandbox’s code-server. 33a79917 (2026-11-05) added the sandbox access rule, so the endpoint answers 403 without access. The host id is 12 random characters, so a link had to be shared or leaked to be used.

Prebuild dev-server previews stay public so links can go to people without an account, and GET /auth/wake resumes a paused sandbox for anyone holding one of its preview URLs.

What Is Still Open

The roadmap lists the known gaps, and these are the ones I would act on first:

Rules I Apply Now

These are the checks I would run on any self-hosted agent platform before a second user touches it.

  1. Treat the sandbox next door as hostile. A sandbox runs an agent that reads untrusted text, so the network policy is the first line and a per-sandbox credential the second. Default-deny ingress, then name the two callers.
  2. Never generate a secret at image build time. Anything made with ssh-keygen, openssl or a token mint in a Dockerfile is in every copy of the image, and public images get copied. Generate at boot, and make the build fail if a key file is in the final layer.
  3. One access function, bound once per router, with denial and absence returning the same 404. Every new route has to go through that guard, which beats auditing each handler.
  4. A compatibility fallback needs an expiry date when I write it. The unpinned-SSH fallback lasted 20 days. The token fallback lasted six and a half months.
  5. Resolve credentials at use and store references. A spec that holds resolved secrets is a second secret store with weaker access control.
  6. Give every actor its own identity. An agent that borrows a person’s key inherits their sandboxes, their token and their admin rights, and no log can tell the two apart.
  7. A default that grants everything has to fail startup. With no auth.defaultOrg, every user is a server admin. Production only logged a warning for that until 52b04b96 (2026-11-05) made it refuse to start.
  8. Keep the known gaps in the repo where the next contributor reads them, as docs/roadmap.md does.

The next piece of work is the one from rule 2 that I did not do: a CI step that fails when a built base image contains host keys.

100%