An AI agent that edits a repository needs somewhere to run commands and keep the results between turns. At training scale, thousands of those environments may have to start together, then wait while a model decides what to do next. DeepSeek’s September 19 technical report on DeepSeek Elastic Compute, or DSec, describes the sandbox system it says handles that workload.

The authors describe the request path, image storage and what happens when a training job is interrupted. Their performance and deployment figures come from their own measurements. The expanded report is an arXiv submission; the abstract says an earlier, two-page extended abstract received first-round conference review.

The big change

  • What changed: DeepSeek has documented DSec as a shared platform for agent training and evaluation, with function calls, containers, microVMs and full VMs available through one internal client library.
  • Why it matters: The report connects agent training to the infrastructure that keeps many task environments running at once. Requests pass through authorization, placement and local admission; layers are composed at creation and image data fetched as needed. Training can pause stateful sandboxes when GPU jobs are preempted.
  • What to watch: Callers still choose the backend. The paper's production workload measurements cover containers and microVMs, which use different storage paths and incur different resource costs. Its scale figures describe one DSec unit.

One request, four kinds of sandbox

In the paper’s account, DeepSeek’s training frameworks, evaluation frameworks and data pipelines call a Python library named libdsec. A typical creation request selects the backend and environment artifact, sets CPU and memory limits, lifetime and network rules, and supplies an initial user context. Once the sandbox is ready, the caller can run commands or tool calls, collect output and return status, and stop the session. The paper’s sample session uses a container, a memory limit, an idle timeout and network rules that allow PyPI while denying NPM. It documents an interface used inside DeepSeek’s platform, not an external access path.

The four backends cover different tasks. FnCall runs short, stateless work in reusable, precreated containers, avoiding a new sandbox for each invocation. Containers serve repository work and general tool use; they start quickly and pack densely, but share a kernel with other containers in their host VM. Firecracker microVMs provide a VM boundary for tasks needing stronger separation, with more startup and memory overhead. Full VMs handle operating systems or graphics workloads that need capabilities the lighter backends do not provide. The authors say containers and microVMs account for most production instances and resource use. These are design choices described by the paper, not a measured security comparison.

Behind the client, DSec authenticates a management request, selects a node using a periodically refreshed view of health and load, and sends the request to that node’s edge service. The edge checks local capacity before creating the sandbox; it can reject a placement made from stale cluster information. Running container and VM sandboxes use a proxy called aether and shell-session processes called chronus for commands, file operations and streaming output. FnCall takes a separate route through a precreated container. This distinction matters because the single client entry point does not erase differences in execution or failure handling.

Building an environment without copying it whole

The paper identifies three parts of a typical agent environment: a base image, a task workspace and a toolkit that may change independently. If every combination were baked into one image, a toolkit update would force many images to be rebuilt. DSec instead stacks read-only layers with a writable layer on top. For containers, a modified Docker runtime composes these layers with overlayfs. MicroVMs use read-only EROFS layers alongside writable disks, with a different block-storage path where filesystem compatibility requires it.

The authors report that one production week involved 11,266 container base images and 102,171 container workspaces. Such variety limits the value of keeping complete images on each node. DSec stores read-only image data in DeepSeek’s 3FS distributed filesystem, keeps writes on local storage and fetches image content when a sandbox reads it. Container image metadata is copied locally so routine path lookups do not require remote reads. The microVM path uses OverlayBD, ublk and a local cache to handle block reads and incremental snapshots.

In a separate 10-node evaluation, the authors started 8,192 containers under an agent evaluation workload. Their on-demand EROFS path completed the tasks in about 35 minutes, compared with more than 60 minutes for cold, eager image pulling; a fully cached baseline also finished in about 35 minutes. The reported disk writes were about 700 GB per node for on-demand loading versus more than 1,600 GB for eager pulling. These figures compare the configurations in the authors’ test. They do not establish the same gain for another image collection or storage system.

Keeping idle sessions and interrupted rollouts usable

An agent sandbox can wait between commands while retaining files, processes and memory. The authors’ week-long sample found that about 90% of container and microVM sandboxes used no more than 5% of their requested CPU capacity on average. DSec therefore packs many live sessions onto nodes, while trying to control memory waste and contention. For microVMs, the paper describes sharing read-only file cache through virtio-pmem with DAX, and reclaiming cold guest pages with DAMON and balloon free-page reporting. It also separates latency-sensitive tasks from best-effort work through Linux scheduling controls. The paper reports benefits for these mechanisms in its own evaluation, alongside trade-offs such as higher transient CPU use with virtio-pmem.

Training interruptions create another problem: a rollout may still have useful state when its GPU job is preempted. The authors say that, starting with DeepSeek-V4.1, DSec runs the agent loop outside the preemptible GPU pool, in a worker container and an agent sandbox. The training job can reconnect to that state. When training pauses, the framework can request that DSec pause the associated sandboxes and reclaim memory. Containers are frozen and reclaimed; microVMs save execution state in a snapshot before their Firecracker process stops. A later operation resumes the sandbox. This is the paper’s account of DeepSeek’s training integration, without a general recovery guarantee.

What the reported scale does and does not show

DeepSeek says one DSec scale unit has nearly 160 CPU nodes, about 30,000 cores and roughly 250 TB of DRAM. It reports about three million sandbox instances on a typical day, peak concurrency around 380,000 and a creation rate above 5,000 instances per second for that unit. These are author-reported production figures for one scale unit, not independently audited totals for DeepSeek’s whole fleet. The paper’s evaluation experiments ran on a separate 10-node cluster.

The report also describes failure boundaries. Its authors recount agents seeking answers through unintended channels and ordinary commands that crashed a kernel or filled storage with output. They describe AppArmor file and socket controls and per-sandbox network rules as mitigations, while explicitly saying the controls do not prevent every harmful behavior. The paper gives no public DSec service endpoint, external SDK distribution, access terms or price. It documents the system design and the conditions of the authors' tests, but supplies no external access path for running its sample code.

Sources & further reading