Architecture

This document explains how Milieu is architected and how it fits an existing estate. It describes the system as built: what runs, what it stores, how it reaches other systems, and where the boundaries of trust and accountability sit.

For the vocabulary, read CONCEPTS first. For the commands that drive any of this, see OPERATIONS and the REFERENCE.

Contents

• 1. System overview
• 2. Deployment view
• 3. Control surfaces
• 4. Execution model
• 5. Coordination between agents
• 6. Information architecture
• 7. Capabilities and tools
• 8. Integration with enterprise systems
• 9. Identity and trust boundaries
• 10. Records and audit
• 11. State and the filesystem
• 12. Availability and recovery
• 13. Technology choices

1. System overview

Milieu is a single Go binary operating on a single directory of state. There is no application server, no message bus, and no external database to provision. An agent is a directory of configuration and content. It runs only for the duration of one piece of work, inside a sandbox, and then exits.

Three properties follow from that shape:

• The unit of deployment is small. One binary, one state directory, one configuration file. It runs on a virtual machine (VM), in a container, or on Kubernetes.
• The unit of recovery is one path. Everything the installation knows lives under $MILIEU_HOME. Backup, restore, migration, and cloning an environment are all operations on that one path.
• The system is inspectable at rest. Configuration, briefs, knowledge, memory, transcripts, and the audit log are files you can read, diff, and put under version control.

2. Deployment view

Only seven processes are long-running, and an installation can run as few as one of them.

The gateway is the one part with a deployment constraint that matters. It exists to keep credentials off the host where model-directed code runs, so putting it on the same host removes most of its value. Everything else can be co-located.

Two topologies cover most installations:

• Single host. Daemon, API, and portal on one VM or container. The gateway runs on a second, smaller one. This is the shape the container image and the systemd units in DEPLOYMENT assume.
• Kubernetes. The same processes as separate workloads over one persistent volume, with the gateway in its own namespace and network policy.

Scheduled work is the coupling to know about when more than one host shares state: schedules fire only where a daemon runs, and a claim is taken under a file lock so two daemons cannot both fire one slot.

3. Control surfaces

There are three ways to change an installation, all of them calling the same services underneath.

• milieu setup is the guided first run. It asks a short series of questions, writes milieu.yaml, and applies it.
• milieu <command> is the imperative CLI, for exploration, one-off work, and day-to-day operations.
• milieu apply -f milieu.yaml is the declarative control plane. It takes a multi-document YAML manifest and reconciles agents, projects, knowledge, schedules, the organisation’s charter, the domain model, the gateway, the API, and the portal to match it.

The manifest shape (apiVersion, kind, metadata, spec) mirrors Kubernetes, so an installation is described the way the rest of a modern estate is described, and the same documents can map onto custom resources later. Reconcile is upsert without prune: declared resources are set to match the manifest, and resources absent from it are left alone. Secrets never appear in the manifest. apply mints API keys and reports them once. Each declaration it changes is digested with the files it names, snapshotted under var/applied/, and sealed into the audit log with its old and new digest.

milieu export is the inverse. It reads a live installation back out as a manifest directory, with secrets and mutable state omitted. Use it to turn an installation built by hand into one described in version control.

4. Execution model

One run

A run is one skill executed as one agent: input in, reply out, transcript written, audit rows appended.

Every entry point converges here. The CLI, the REST API’s run routes, the A2A endpoint, and the daemon’s wake cycle all execute a turn through the same code, so a piece of work is scoped, run, and recorded identically whichever entry point the request arrived by. A message carries no project and no session, so a turn the wake cycle runs is always unscoped and one-shot.

The loop is bounded. A skill declares how many passes its work needs, a skill that declares none gets ten, and whatever is declared is capped by an installation-wide ceiling. A run that ends at the cap, at the model’s token limit, or on a malformed tool call returns the partial reply and states why: the reason appears on the result, the audit row, the transcript, and the session turn, so a REST client and an A2A peer both see that the reply was truncated.

When the model asks for several tools in one turn and every call only reads (knowledge, transcripts, a skill’s bundled files, or memory’s ls, get and search), the calls run at once, up to four, and the results go back matched to their calls. A turn with anything else in it (a command, the gateway, an MCP tool, consult, session, graph, code, or a memory write) runs its calls in order, because a later call may depend on an earlier one’s effect.

Work that starts on its own

Everything in the preceding section is arrival-driven. A schedule is the only way work begins without a request: a declaration naming an agent, a skill, an input, and a cron expression, fired from the daemon’s tick.

Scheduling is off by default. An environment variable on the daemon decides whether any schedule fires, so an installation can run agents that only answer requests until you enable schedules. Declaring, linting, and applying schedules works either way, so a manifest can be prepared on a host that will never run it.

Four properties follow:

• Declared in the manifest. Schedules are manifest resources, so recurring behaviour is part of the installation’s declaration and is captured by a backup of the state directory. Nothing recurring lives in a host crontab, where no other part of the system could see it.
• A cause in the log. The firing writes its own audit row ahead of the work it produces. Every other entry point binds a principal at a trust boundary. A firing names the declaration instead, so the log still records who asked for the work.
• At-most-once catch-up. A daemon that was down for a week fires each daily schedule once on return. Missed slots are not replayed individually.
• No session. A scheduled run has no client, so it opens no session and is checked against no session quota.

The declaration and the clock are separate files, so applying a manifest cannot reset a running schedule’s history and a firing cannot rewrite what you declared.

Measuring a change

Provenance records what ran: a declared version, a digest of the exact build, a snapshot of every build that has ever run, and a rollback command. It doesn’t record whether a change was an improvement. milieu eval grades a skill against recorded cases and exits non-zero when a case fails, so a suite can gate a release.

Cases live outside every agent’s writable area, so an agent cannot edit the tests it is measured by. They are declared in the manifest like everything else, because the state directory is derived data you should be able to delete and rebuild.

A graded run is isolated

Nothing is recalled into a graded run’s prompt, no transcript is written, and no topic is classified. A run’s transcript is normally recalled into the next run on the same topic, so a suite that recorded one would be graded the second time on a prompt containing its own previous answer. The audit row is still written, because an evaluation spends real money.

Assertions favour behaviour over prose

Text assertions are cheap and brittle, and a model that rewords a correct answer breaks them. Milieu records every tool a run consulted, so assertions about which tools were used, which were forbidden, and how many calls were made are far more stable. A ceiling on tool calls catches a regression that text assertions miss: a skill that still answers correctly but now takes eleven steps to do it.

A model can also be asked to grade a reply’s substance. That costs a call per case and is not deterministic, so it runs only on request and its tally is reported apart from the pass rate.

5. Coordination between agents

Agents collaborate only by message. Each agent has an inbox and an outbox, held as two small embedded databases in its own directory, and three verbs move a message between them:

1. send writes a row to the sender’s outbox.
2. route copies undelivered outbox rows into recipients’ inboxes.
3. wake drains an inbox, runs a skill for each unread message, and writes the reply back to the outbox.

The daemon runs the last two continuously.

Two mechanisms keep this from misbehaving. A message is marked consumed only after its reply has been sent, so a crashed wake leaves the message to be processed again. Every message also carries a reply depth that increments on reply and stops at five, so an exchange between two agents cannot continue indefinitely.

milieu ask is the synchronous form over the same machinery: it sends a message with a correlation id and waits for the matching reply. A reserved subject family keeps the daemon from auto-replying to a reply.

Attachments travel by reference. The bytes go once into a content-addressed store, keyed by their SHA-256 digest, and the message carries the digest and media type. The recipient’s skill receives them as image or document content.

6. Information architecture

Knowledge and memory are each scoped in three layers:

A read applies two rules. Union: every applicable layer is merged, and search merges results across them. Shadow: on a name clash the more specific layer wins, in the order agent, then project, then global, and a direct fetch returns the most specific match. No content is copied between layers. A global policy article is the house default, a project can override it with its own, an agent can override that again, and deleting the override restores the default.

The two stores differ in who writes them, and that difference is a governance boundary:

• Knowledge is authored by people. It’s Markdown, filed by how stable it is (axioms, articles, news, events), and optionally carrying Open Knowledge Format metadata: type, tags, status, an expiry, who generated it, who verified it, and what it supersedes. Two of those fields keep a base usable as it ages. An expiry marks time-bound items, so a finished outage notice stops outranking live policy, and stale or non-stable items are labelled in every result, including the ones the model sees. Listing superseded ids makes “what is still in force” answerable by a query.
• Memory is written by the agent. One file per fact, with a description used to judge relevance at recall time. An index of every memory is prepended to each run, and the bodies are fetched on demand. Writing memory is a granted permission, because an agent that writes memory is editing its own future prompts.

Each layer carries its own search index: a lexical full-text index, plus a semantic vector index when an embedding backend is configured, with results fused when both are present. Reindexing is per layer and explicit. Editing a file does not update the index. Embedding runs out of process against an OpenAI-compatible or Voyage endpoint, so a locally hosted model is a supported choice for organisations that will not send content to a vendor.

7. Capabilities and tools

A skill follows Anthropic’s Agent Skills format: a Markdown file, or a directory with a SKILL.md and bundled resources beside it. Frontmatter declares the name, the description, and optionally a model, a tool list, a version, a licence, and an iteration budget. The body is the system prompt.

Skills load progressively. The catalogue an agent sees holds names and descriptions, a run loads one body, and bundled resources are read only when the skill asks for them. A model can be allowed to choose its own skill from the catalogue, and the choice is audited.

What a run may call is the intersection of two lists: the tools the skill declares and the permissions the agent has been granted. A skill that asks for a command the agent does not hold does not get it, so a skill author cannot widen an agent’s authority, and an operator’s grant does not oblige a skill to use it.

Tools come in two kinds. Most are real command-line programs, executed inside a sandbox. A handful are built in and run in the host process: reading knowledge, reading a skill’s own resources, recalling past transcripts, writing memory, declaring that a reply is a question, making a brokered call through the gateway, and running a short script.

The script tool runs a Starlark program. Starlark was chosen because it has no standard library: a program reaches only the functions Milieu injects, and which ones those are is derived from that agent’s own grants. Running a command from inside a script goes through the same runner and writes its own audit record. The tool exists to save model round trips: a task that visits fifty records costs fifty round trips as individual tool calls, and one as a script.

Every build of every skill that has run is snapshotted, and every run records both the declared version and a digest covering the skill and all its bundled resources. An author can forget to update the version, but the digest identifies the skill that answered.

8. Integration with enterprise systems

An agent reaches a system of record in only one way: through a route an operator declared on the gateway. There is no ambient credential in the sandbox and no path from a skill to an arbitrary endpoint.

Three of the route kinds need more detail.

SQL routes take a named query and parameters, never raw SQL and never the connection string. The set of statements an agent can cause to run is a fixed list an operator wrote and can review.

MCP works in both directions, and the two directions are kept apart. As a client, the gateway connects to a remote server, so its tools reach the model with their own schemas while its credential stays in the trusted zone. You declare which of the server’s tools this installation exposes, and a tool the server advertises isn’t available to the model until you declare it. A route may also pin a fingerprint of the tool set it was approved against, so a server that later rewrites a tool’s description is refused.

As a server, milieu mcp serve offers each agent to an MCP host as one tool, ask_<agent>, the way the A2A server offers it to a peer: a call is a turn in a session, taken through the same entry point. It never re-serves a brokered tool. A route adds the agent’s identity, its grant, and the audit entry around the call. Passing the tool through would hand out the credential’s reach without any of them.

A2A works in both directions, with a different scope in each. Milieu serves the Agent2Agent protocol in full, with a public agent card per agent and streaming, so a partner’s agent can hold a conversation with yours over the same session record as any other channel. It consumes A2A one exchange at a time through a gateway route, which lets an agent here put a question to an agent elsewhere without holding that peer’s credential.

Model access is itself pluggable. The backend is Claude’s API, any OpenAI-compatible endpoint (which covers Azure OpenAI, OpenRouter, LiteLLM, vLLM, and a locally hosted model), Gemini on Vertex AI as the machine’s own identity, or a deterministic stub for offline demonstrations and tests. Each provider is an adapter in its own package (llm/claude, llm/openai) that registers itself under the names configuration selects it by, and shares one retry policy (llm/llmhttp). A new provider is one more package, and it’s ready when it passes llmtest, the contract every adapter passes against a fake of its provider’s API.

9. Identity and trust boundaries

Milieu has three trust boundaries. Each has its own identity mechanism and its own log, and they are separate trust domains.

The sandbox boundary

An agent runs with least privilege: a granted list of commands, a fixed search path so the binary that was permission-checked is the binary that runs, a per-host egress allowlist, and no secrets unless an operator explicitly passed one. Isolation is Bubblewrap on Linux and the native sandbox on macOS. Only the agent’s own working directory is writable. Extra host paths appear only through operator-authored read-only mounts.

The API edge

External programs drive Milieu as an agent over TLS, authenticated by a bearer key that is stored only as a hash and can be revoked. The key resolves to exactly one agent and is interaction-only: it cannot manage agents or projects and cannot execute commands. A caller sees only its own conversations, and a request naming another agent’s identifier returns “not found”, so probing reveals nothing. A per-agent quota bounds how many conversations a key may hold open and how fast it may start new ones, and a refusal is distinguishable from an outage on both edges that honour it.

The gateway

The broker holds the secrets in a separate trusted zone. The agent talks only to a loopback proxy confined to its own port. The proxy re-originates the call over mutual TLS. The gateway authorises it by the client certificate, injects the credential, calls upstream, and logs the call but never the secret. A compromised agent can ask the gateway to act for it, within the routes that agent is authorised for, but cannot read the credential or reach anything no route describes.

You can also pass a secret directly into a sandbox through an agent’s [env] block. That path is for low-value credentials, gated by a host-level allowlist of variable names so an agent’s own configuration cannot pull arbitrary secrets. High-value credentials belong on the gateway.

10. Records and audit

Four separate records answer four separate questions.

The audit log holds one row per command invocation and per built-in tool call, carrying the agent, the command and arguments, the exit code, the timings, the isolation backend used, and a hash chained to the previous row. Editing, reordering, inserting, or deleting a record breaks every hash after it, and milieu audit verify reports the first break. Appends take an exclusive file lock and read the current tail under it, so a sandboxed command and the daemon extend one chain.

Rows for calls that reached a model also carry the tokens they spent, so spend is attributable per agent. Cost is computed at report time from a price table and is never frozen into the log, so a price change does not require rewriting history.

Without an audit key, the chain is unkeyed. It detects tampering by anyone who can’t rewrite the log file, but someone with write access to that file can recompute the whole chain. With an audit key held outside MILIEU_HOME, each record also carries an HMAC, so a rewritten chain fails verification, and a switch back to unkeyed records is reported as a downgrade. Either way, removing records from the end leaves a shorter chain that still verifies, and so does a log rewritten entirely without MACs. If that matters to your threat model, anchor the latest hash somewhere the host operator can’t rewrite. Messages between agents are signed by their sender and checked before delivery; audit records aren’t signed per agent. See SECURITY.

The agent log, the access log the REST API, A2A and MCP edges share, the gateway’s brokered-call log, the portal’s operator log, and the built-in issuer’s sign-in log are five files in different trust domains, with different actors, shapes, and readers. They share one chaining implementation, so a verifier cannot drift from a writer. Each is verified where it lives, including the gateway’s, which is checked in the trusted zone beside the log.

11. State and the filesystem

Everything resolves from one prefix, MILIEU_HOME, read at startup by every binary.

$MILIEU_HOME/var/
├── global/                       # global layer, shared and ungated
│   ├── knowledge/{axioms,articles,news,events}/
│   ├── memory/                   # + its index
│   └── index/                    # lexical + semantic
├── agents/<code>/
│   ├── agent.toml                # identity, role, permissions, network, env, mounts, gateway
│   ├── personality.md            # how it communicates
│   ├── responsibility.md         # what it owns
│   ├── skills/
│   ├── knowledge/  memory/  index/
│   ├── in.db  out.db             # message queues
│   └── state/<topic>/            # run transcripts, filed by topic
├── projects/<code>/
│   ├── project.toml              # code, name, instructions, members
│   ├── instructions.md           # prepended for runs in this project
│   └── knowledge/  memory/  index/
├── sessions/<id>/                # external conversations
├── skills/<agent>/<name>/<digest>/   # every build of a skill that has run
├── evals/<agent>/<skill>/        # recorded evaluation cases
├── schedules/                    # declarations + when each last fired
├── blobs/                        # content-addressed attachments
├── audit/audit.jsonl             # append-only, hash-chained
├── gateway/                      # trusted zone: CA, server identity, its own log
├── proxy/<code>/                 # an agent's proxy identity, host-only
├── signing/<code>/               # an agent's message-signing key, host-only
├── api/                          # CA, server cert, hashed keys, access log
├── auth/                         # built-in issuer: people, grants, clients, signing key, its log
└── web/                          # CA, server cert, operator accounts, portal log

One volume root captures all state, so backup, migration, and relocation are single-path operations, and multiple installations coexist on one host by prefix. This matches how self-contained server applications are usually packaged. The container mount plan is in DEPLOYMENT.

Two constraints follow from the storage choices, and both belong in a runbook:

• Never put the state directory on NFS. The embedded databases need proper POSIX locking.
• Derived data can be deleted, but authored data cannot. Transcripts, indexes, skill build snapshots, and settled messages are rebuildable. Knowledge, memory, briefs, skills, and the audit log are not.

Storage interfaces

The services don’t talk to SQLite. Each one that keeps records reaches them through an interface in its own package, and the SQLite code is an adapter in a sqlite subpackage beside it:

An interface speaks the package’s own types and errors, never database/sql types or file paths. Its comment lists what any backend must guarantee: idempotent delivery, a refresh token spent at most once, a rate bucket updated atomically, and so on. A test package beside it (messagetest, apikeystest, and the rest) checks those guarantees. The SQLite adapters run it, and a new backend is ready when it passes the same suite. The logic stays in the service: hashing, pricing, refill arithmetic, and who may do what never move into an adapter.

Files (agent and project TOML, transcripts, schedules, blobs), the vector index, and the hash-chained logs aren’t behind interfaces yet.

12. Availability and recovery

• Steps are idempotent. Routing, waking, and the reconcile are all safe to re-run. A message is consumed only after its reply is sent, so a crashed run leaves the message to be processed again.
• Loops are bounded. Reply depth caps agent-to-agent chains. A per-agent wake deadline bounds a hung model call so it cannot wedge the poll loop. A run’s tool-use loop is capped by the skill and by the installation.
• Retention is a configured policy. The daemon prunes settled messages and old transcripts on an hourly pass, against a window you set. Unread and undelivered messages are never pruned, and authored data is never touched.
• Supervision is conventional. In development, process-compose supervises the stack with ordered startup and health gating. In production, systemd supervises it, or Kubernetes does. The daemon exits cleanly on a termination signal after finishing the tick in flight.
• Shutdown is bounded. The network servers stop accepting connections on a termination signal and allow in-flight requests ten seconds to finish. That bounds how long a restart may block. It doesn’t promise that the work completes. A run longer than the grace period is cut off and its caller sees a dropped connection. Nothing is corrupted, because audit and session records are written before the reply is returned, but the answer is lost.

13. Technology choices

The main technology choices:

The internal structure is a conventional layered Go codebase: one package per subcommand and front end, use cases that orchestrate a piece of work (a turn, a wake, a lint pass), domain services that each own one noun and its rules, and a platform layer for cross-cutting infrastructure such as configuration, model access, sandboxing, identity, network policy, and the audit chain. A test enforces the direction of every import between them.

All four network servers share one listen, serve, and shutdown implementation, so timeouts and graceful-shutdown behaviour are uniform. Routing, authentication and how a refusal is rendered stay with each server, because those differ between a REST API, a JSON-RPC endpoint, and a browser portal.