Architecture

Milieu is an AI agent ecosystem in which agents perceive and act through command-line interfaces. This document describes the system as built: the processes that run, how they coordinate, where state lives, and how the principles of Observability, Explainability, and Accountability are made concrete.

This describes shipped behaviour. For things that are planned but not yet implemented, see ROADMAP.md. For how to use what is described here, see the USER-GUIDE and OPERATIONS guides.

Contents

• 1. Overview
• 2. Principles in practice
• 3. Processes & commands
• 4. Coordination: messages & routing
• 5. Sandboxing
• 6. Knowledge & memory layers
• 7. Skills & tools
• 8. Identity, trust boundaries & audit
• 9. Roles
• 10. Filesystem layout
• 11. Failure & recovery

1. Overview

Milieu is not a single daemon with embedded agents. It is one small Go binary over a shared on-disk state tree ($MILIEU_HOME/var), invoked per command. Agents are directories, not processes — an agent “runs” only for the duration of a skill invocation, inside a sandbox, then exits. Messages between agents are durable rows in per-agent SQLite files; a poller (milieu daemon) moves them and triggers replies.

    flowchart TB
    subgraph edge["Edges"]
      OP["Operator / CI<br/>milieu · milieu apply"]
      EXT["External programs<br/>(REST clients)"]
    end
    subgraph host["Milieu host — $MILIEU_HOME/var"]
      API["API server<br/>(internal/api)"]
      DCMD["milieu daemon<br/>poll · route · wake"]
      subgraph agents["Agents (sandboxed, transient)"]
        A1["agent: in.db / out.db<br/>skills · knowledge · memory"]
      end
      STORE["Knowledge & memory<br/>(global · agent · project)<br/>+ per-layer FTS5/vector index"]
      AUD["Audit log<br/>(append-only, hash-chained)"]
      BLOB["Blob store<br/>(content-addressed)"]
    end
    GW["Gateway broker<br/>(trusted zone, mTLS)<br/>holds secrets"]

    OP --> DCMD
    OP --> agents
    EXT -->|TLS + API key| API
    API --> agents
    DCMD --> agents
    agents <--> STORE
    agents -->|attachments| BLOB
    agents -->|brokered calls| GW
    agents --> AUD
    API --> AUD
    GW --> AUD
  

The three CLIs are the control surface:

• milieu — the single binary. Everything an operator or agent does directly: create agents, grant permissions, run sandboxed commands, send/read messages, run skills, manage knowledge/memory/projects, lint, inspect the audit log, and run the daemon, gateway and API servers.
• milieu daemon — the only long-running process in the core stack. It polls every agent’s inbox, runs a skill on unread messages, and routes replies. It owns no state of its own; it operates on the same var/ tree.
• milieu apply — declarative control plane (the kubectl analog). Applies a multi-document YAML manifest, reconciling agents, projects, knowledge, the gateway, and the API by calling the same services the CLI does.

2. Principles in practice

Per-agent cryptographic signing of each message and audit record (with the router verifying signatures) is on the roadmap, not yet shipped. Accountability today rests on the hash chain plus boundary-bound identity.

3. Processes & commands

One binary, cmd/milieu, which dispatches every subcommand against the service graph wired in internal/app. Most subcommands are per-invocation; four are long-running processes.

There is no separate per-agent runner process or resident agent loop. A skill runs in-process (milieu act / wake), spawning sandboxed subprocesses only for tool calls that need them. The agent “loop” is the poll-and-reply cycle that the daemon drives.

Evaluating a skill

Skill provenance records what ran — a declared version, a digest of the build, a snapshot under var/skills/, and skill rollback one command away. What it did not record is whether a change was an improvement. milieu eval grades a skill against recorded cases: a markdown file under var/evals/<agent>/<skill>/<case>.md whose frontmatter is the expectations and whose body is the input. Exit status is non-zero when a case fails, so a suite is a gate rather than a report.

Cases live outside every agent’s writable home, for the plainest reason in the system: an agent that can edit the tests it is measured by is not being measured. They are declared like everything else — evals: on an agent names one source directory per skill — because var/ is derived state an operator should be able to delete and rebuild, and a suite that lived only there would not survive it.

Two properties make the result mean something.

A graded run is isolated. Nothing is recalled into its prompt, no transcript is written, and no topic is classified or minted. This is correctness, not tidiness — a run’s transcript is recalled into the next run on the same topic, so a suite that recorded one would be graded, the second time, on a prompt containing its own previous answer, and topic classification would slowly reshape the catalogue that files the agent’s real work. The audit row is still written: an eval spends real tokens, and what a run cost is the account of it rather than a trace. It does not make a run read-only — a skill granted the memory tool still writes memory.

Assertions favour behaviour over prose. contains/excludes/matches are cheap and brittle: a model that rewords a correct answer breaks them. What tools a run consulted is what milieu observes exactly, and tools, forbids and max-tools are far more stable — whether the agent looked something up before answering is a property of its behaviour, not of its phrasing, and max-tools catches the regression no text assertion sees, a skill that still answers correctly but now takes eleven tool calls to do it. Every case also implicitly asserts stop: "", so a run cut off at its iteration cap fails rather than being graded on a fragment.

A judge criterion has a model grade the reply’s substance. It costs a call per case and is not deterministic, so it runs only under --judge and its tally is reported apart from the pass rate, never folded in. A verdict that cannot be parsed is a failure, never a pass. What --backend selects decides what the run measures at all: against a live model it measures the provider as much as the skill, and against echo it measures the harness and nothing about the reasoning — different runs, at different cadences.

Scheduled tasks

Everything above is arrival-driven: an agent acts when a message lands, when act is invoked, or when a request reaches an edge. A schedule is the one way work begins with nobody asking — a declaration under var/schedules/ naming an agent, a skill, an input and a cron expression, fired from the daemon’s tick.

It is off by default. MILIEU_SCHEDULER on the daemon decides whether any schedule fires, so an installation gets agents that only answer until it asks for agents that act. Declaring, linting and applying schedules works either way, which means an operator can prepare an installation’s recurring work on a host that will never run it.

Four properties are load-bearing:

• Declared, not scripted. kind: Schedule is reconciled by apply like every other resource, so an installation’s recurring behaviour is in its manifest rather than in a host crontab that lint cannot check and a backup of MILIEU_HOME does not capture.
• A cause in the log. The firing writes its own audit row (cmd: schedule) ahead of the act row it produces. Every other door into milieu binds a principal at a trust boundary; this one names the declaration instead, so “who asked for this?” still has an answer.
• At-most-once catch-up. A daemon that was down for a week fires each daily schedule once when it returns, not once per missed midnight — the clock advances to the moment of the claim, not to the slot it stood in for.
• Not a session. A scheduled run has no client, so it opens no conversation and is checked against no session quota. Its transcript is filed under the schedule’s own name.

The declaration and the clock are separate files (<name>.toml and state/<name>.json), so apply cannot reset a running schedule’s history and a firing cannot rewrite what an operator declared. A claim is taken under a flock, so two daemons over one MILIEU_HOME cannot both fire one slot — but schedules only fire where a daemon runs, and that is the coupling to know about in a multi-host deployment.

Internally the binaries share a layered design:

• internal/app — composition root; wires services and stores.
• internal/cli/* — one package per subcommand.
• internal/service/* — domain logic (see below).
• internal/platform/* — cross-cutting infrastructure (config, llm, embed, sandbox, identity, netpolicy, rule, httpx, auditchain, …).
• internal/api, internal/gateway — the two network servers.
• internal/setup — the milieu apply reconcile engine.

Services (internal/service): agents, message, router, wake, skills, permissions, knowledge, memory, projects, runner, audit, lint, blob, gwproxy, session, transcript, cleanup, daemon, act, skillrun, apikeys, admission, accesslog, gwclient, schedule, eval.

All four network servers run on platform/httpx: one listen/serve/shut-down sequence, one graceful-shutdown grace, one read-timeout profile, and one listener seam for tests. Routing, authentication, and how a refusal is rendered stay with each server, because those differ between a REST API, a JSON-RPC endpoint, and a browser portal for real reasons.

gwclient is an agent’s access to its gateway — where the broker is and what the agent proves itself with — which both ways of reaching it need first: the loopback forwarder (gwproxy) the runner fronts a sandboxed run with, and the direct host-side call the gateway and mcp built-in tools make. It was the runner’s, because the identity lookup happened to live there; running a process under a sandbox and making an mTLS HTTP request are two jobs.

admission is what the two API-key-authenticated edges — the REST control plane and the A2A endpoint — ask before doing any work: who is calling, and may they start another conversation. It is one service because two doors sharing one credential had drifted; only one of them checked that the agent a key names still exists, so a key outlived its agent on the other. It knows nothing about HTTP: it decides, and each edge renders the decision in its own protocol.

skills is the catalogue — what skills an agent has and what each contains — and skillrun executes one. Only the second needs an LLM backend, the sandbox runner, knowledge, memory, transcripts and the blob store, so lint and apply, which only read skills, no longer carry any of it.

act is the use case for one agent turn — resolve the skill, gate the project, resume the conversation, run, record. The three boundaries that let something outside ask an agent to work (milieu act, the REST API’s run routes, and the A2A endpoint) call it rather than each assembling the sequence themselves, so a turn is scoped, run and recorded identically whichever door the caller came in by.

4. Coordination: messages & routing

Each agent has two SQLite databases under its home (internal/service/message/store.go):

• in.db — the agent’s inbox (delivered messages).
• out.db — the agent’s outbox (messages it has sent, awaiting delivery).

Two files rather than one because SQLite serialises writers per file; the split keeps inbox and outbox writers from contending.

The flow for agent-to-agent collaboration:

1. send — milieu message send (or the API / a skill reply) writes a row to the sender’s out.db.
2. route — milieu route (one-shot) or milieu daemon (continuous) copies undelivered outbox rows into the recipient’s in.db.
3. wake — milieu wake or the daemon drains an inbox, runs a skill per unread message, and writes the reply to the agent’s out.db — which routes back.

Each message carries a reply_depth; wake increments it on reply and stops at MaxReplyDepth (5, in internal/service/wake/service.go), which breaks the otherwise-infinite reply loop between two agents. Messages are marked consumed only after a successful reply send, so wake is retry-safe.

ask is the synchronous front door: it sends a message stamped with a random correlation id and polls the caller’s inbox for the matching Re: reply. The ask: subject family is reserved so the daemon never auto-replies to a reply.

Attachments ride with messages by reference: the bytes are stored once in the content-addressed blob store (var/blobs/, keyed by SHA-256), and the message body envelope carries only a digest + media-type. wake hands them to the recipient’s skill as image/document content blocks.

5. Sandboxing

Command execution is isolated by a pluggable backend (internal/platform/sandbox), selected per platform and overridable with MILIEU_SANDBOX:

• Linux — bwrap (Bubblewrap): explicit --ro-bind / --bind / --unshare-*.
• macOS — sbexec (sandbox-exec).
• none — no isolation; never auto-selected, opt-in only.

The agent’s home is writable; protected directories are read-only; commands resolve against a fixed sandbox PATH (default /opt/homebrew/bin:/usr/local/bin:/usr/bin:/bin), never the caller’s $PATH, so the binary that is permission-checked, audited, and executed is always the same one. Operators set the PATH at build time (make build SANDBOX_PATH=…) or runtime (MILIEU_SANDBOX_PATH); agents can never change it. Read-only host paths can be exposed to an agent with operator-authored [[mount]] blocks in agent.toml.

Network egress is governed by a per-host allowlist ([network] allow); see §8 and SECURITY → Network egress for how each backend enforces it.

6. Knowledge & memory layers

Knowledge and memory are each scoped in three layers under var/:

Each knowledge layer is a tree of markdown organised as axioms/, articles/, news/, events/. Reads union the applicable layers most-specific-first (agent > project > global) and shadow on a path/name clash. --as <code> adds the agent layer; --in <project> adds the gated project layer (members only); global is always included. search merges layers; get returns the most-specific match.

Each layer carries its own index/ — an FTS5 lexical index plus an optional semantic vector index when an embedder is configured. milieu knowledge reindex [--scope agent|project|global] rebuilds one layer. Search modes: --lex (FTS5), --sem (vectors), --hybrid (both, fused via reciprocal-rank fusion, the default); with no embedder all modes fall back to lexical.

Embedding runs out-of-process: milieu is a client of an OpenAI-compatible or Voyage embeddings endpoint (internal/platform/embed). Switching backend or model changes the vector dimension, so semantic search refuses a mismatched index until you reindex.

Knowledge is hand-authored markdown (edit files, then reindex). Memory is tool-written: milieu memory save and the built-in memory tool target one layer by context (--in project, --global global, else the agent’s own). An auto-maintained per-layer MEMORY.md index is prepended to skill runs.

7. Skills & tools

A skill follows Anthropic’s Agent Skills format: a flat skills/<name>.md or a directory skills/<name>/SKILL.md with optional bundled resources. YAML frontmatter declares name, description, and optional model, allowed-tools, version, license, metadata, max-iterations. The body is the system prompt.

At run time (internal/service/skills):

• When a skill declares allowed-tools, Milieu exposes those tools — intersected with the agent’s granted permissions — to the model. The runner sandboxes each tool_use and feeds the result back.
• Six built-in tools need no subprocess and run host-side (audited as sandbox: "builtin"): knowledge and skill are read-only; transcript is read-only recall of the agent’s past runs and is permission-gated; memory can write and is also permission-gated; gateway makes a brokered HTTP call to the trusted-zone gateway using the agent’s proxy identity (mTLS, no sandbox round-trip), so agents reach brokered routes without shelling out to curl.
• code runs a short Starlark program (internal/platform/script) so a model can collapse a chain of dependent tool calls into one step — cheaper in tokens and in latency. Starlark is chosen for having no standard library at all: a program reaches only the functions milieu injects, and which ones those are is derived from the agent’s own grants, so a script is never a route around a permission. run() inside a script goes through the same runner and writes its own audit record. Runs are bounded by execution steps and wall clock, and the json, re, time and math modules are always present so a script can parse what a capability returned — with time.now() pinned per run so a script cannot read a live clock. See REFERENCE → Built-in tools.
• act --auto (and wake --auto, milieu daemon --auto) let the model pick the skill from the catalog’s names + descriptions (level-1 progressive disclosure); the choice is audited.
• Attachments (--attach) are sniffed, stored in the blob store, and passed as image/document blocks (requires a vision-capable model).
• The tool-use loop is bounded. A skill’s max-iterations says how many passes its work needs; a skill that declares none gets 10, and whatever is declared is clamped to the installation’s ceiling (MILIEU_MAX_ITERATIONS, default 50) — the author knows the task, the operator pays for it. A run that ends at the cap, at max_tokens, or for tool use that named no tool returns the partial reply and says so: the reason is on the result, the audit row (stop), the transcript frontmatter and the session turn, so a REST client reads stop in the body and an A2A peer reads milieu/stop in the task metadata. Truncation that only reached stderr was indistinguishable from a finished answer at every network boundary.

The LLM backend is pluggable (internal/platform/llm): claude (Anthropic Messages API, via ANTHROPIC_API_KEY), openrouter (a generic OpenAI-compatible chat backend defaulting to the OpenRouter gateway, also usable with LiteLLM, vLLM, or a local llama-server via MILIEU_LLM_URL), or echo (a deterministic stub for tests and offline demos).

8. Identity, trust boundaries & audit

Milieu has three distinct trust boundaries, each with its own identity mechanism and its own audit log. They are deliberately separate trust domains.

    flowchart LR
    subgraph untrusted["Untrusted / least privilege"]
      AG["Agent sandbox<br/>(no ambient secrets)"]
    end
    subgraph hostzone["Milieu host"]
      APIS["API server<br/>bearer key → agent<br/>1-way TLS"]
      AUDIT["hash-chained audit logs"]
    end
    subgraph trusted["Trusted zone (separate deployment)"]
      GWB["Gateway broker<br/>holds secrets<br/>mTLS, CN → agent"]
    end
    EXT["External program"] -->|API key over TLS| APIS
    APIS --> AG
    AG -->|local proxy, mTLS| GWB
    GWB -->|injects credential| UP["Upstream (REST/CLI/SQL)"]
    AG --> AUDIT
    APIS --> AUDIT
    GWB --> AUDIT
  

• Sandbox boundary — agents run with least privilege: a granted CLI allowlist, a fixed PATH, a per-host network allowlist, and no ambient secrets unless explicitly passed ([env] passthrough, gated by MILIEU_ENV_PASSTHROUGH).
• API edge (internal/api) — external programs drive milieu as an agent over one-way TLS. Auth is a bearer API key (sk_…) stored only as its SHA-256 hash in var/api/keys.db, revocable. The key resolves to one agent; interaction-only (no agent/project management, no command exec). CA + server cert via platform/identity (ECDSA P-256). Access log: var/api/audit.jsonl — shared with the A2A endpoint, which authenticates with the same keys, so an operator asking what a key has done reads one log rather than guessing which door it came in by. Every outcome is recorded, refusals included: those never become agent work and so appear in no other log. The keys themselves — and the session quota bounding what the principal behind one may consume — are service/apikeys, not the edge: the A2A endpoint authenticates with the same keys, the portal mints and revokes them, and apply provisions them, so the credential is a domain of its own rather than one server’s private table.
• Gateway (internal/gateway, service/gwclient, service/gwproxy) — runs in a separate trusted zone, holds secrets, and brokers calls so the credential never enters the sandbox. The agent talks only to a local proxy confined to its loopback port; the proxy re-originates over mutual TLS (Ed25519, TLS 1.3). The gateway is its own CA, authorizes each call by the client-cert CN (the agent identity), and logs it to its own hash-chained audit log — never the secret. Routes are REST (inject bearer/header, optional jq transform), CLI (run a whitelisted command with a curated env), or SQL (named, parameterized queries).

Audit log (internal/service/audit): every CLI invocation and built-in tool call appends one JSON row to var/audit/audit.jsonl with seq, agent, cmd, args, exit_code, started_at, ended_at, sandbox, and the chain fields prev_hash and hash. Appends take an exclusive file lock and read the current tail under it, so a sandboxed CLI run and the daemon extend one chain. Hash chaining detects edits/reorders/deletes but not tail truncation — anchor the latest hash externally if that matters. Record shape: REFERENCE → Audit records.

Four chains, one implementation. The agent log, the API access log, the gateway’s brokered-call log and the portal’s operator log are separate files in separate trust domains — different actors, different shapes, different readers — but the chaining, the locking, and the verification all live in internal/platform/auditchain; each package supplies only its own payload type. A verifier that drifts from the writer is invisible until it matters, which is why neither is left to the consumer. Each is checked where it lives: milieu audit verify, milieu api audit verify, milieu gateway audit verify (run in the trusted zone, beside the log), and the portal’s Audit screen.

9. Roles

Each agent has a role recorded in agent.toml:

• administrator — may dispatch work to any agent.
• associate (default) — does the work.
• assistant — like associate, narrower scope (convention).
• auditor — read-only; may not send messages (reply attempts are rejected at message send, so an auditor cannot wake meaningfully). Use auditors for read-only review through the audit log.

The load-bearing, enforced rule today is the auditor send-restriction. Broader role-based routing policy (e.g. who may address whom) is convention plus this restriction, not a general routing-layer ACL.

10. Filesystem layout

Everything resolves from a single prefix, MILIEU_HOME (default $PWD/.milieu for dev, /opt/milieu in the container image). All binaries read it at startup and resolve every path relative to it.

$MILIEU_HOME/var/
├── global/                       # global layer (shared, ungated)
│   ├── knowledge/{axioms,articles,news,events}/
│   ├── memory/                   # + MEMORY.md index
│   └── index/                    # FTS5 + semantic
├── agents/<code>/
│   ├── agent.toml                # identity, role, permissions, network, [env], [[mount]], [gateway]
│   ├── personality.md            # voice/tone brief (prepended to the system prompt)
│   ├── responsibility.md         # mandate/scope brief (prepended to the system prompt)
│   ├── skills/<name>.md | <name>/SKILL.md
│   ├── knowledge/  memory/  index/
│   ├── in.db  out.db             # message queues
│   └── state/<topic>/            # run transcripts, filed by topic (+ TOPICS.md)
├── projects/<code>/
│   ├── project.toml              # code, name, instructions, members
│   ├── instructions.md           # prepended to skill bodies for --in runs
│   └── knowledge/  memory/  index/
├── schedules/                    # recurring tasks: <name>.toml + state/<name>.json
├── blobs/                        # content-addressed attachments (SHA-256), shared, not GC'd
├── audit/audit.jsonl             # append-only, hash-chained
├── gateway/                      # trusted zone: ca/, server/, audit.jsonl
├── proxy/<code>/                 # an agent's proxy identity (leaf + pinned CA), host-only
└── api/                          # ca/, server/, keys.db (hashed), audit.jsonl

Why one prefix instead of FHS paths: one volume root captures all state; backup/migration/relocation are one-path operations; multiple instances coexist by prefix (/opt/milieu-staging); it matches self-contained server apps (Elasticsearch, Artifactory). The full container mount plan is in DEPLOYMENT.md.

Never put var/ on NFS. SQLite WAL needs proper POSIX locks.

11. Failure & recovery

• Idempotent steps. route, wake, and the apply reconcile are idempotent; re-running is safe. Messages are consumed only after a successful reply, so a crashed wake re-processes rather than drops.
• Bounded loops. reply_depth caps reply chains; milieu daemon --wake-timeout bounds a hung LLM call so it cannot wedge the poll loop.
• Retention. milieu daemon --retention hourly prunes settled messages (consumed inbound, delivered outbound) older than the window; unread/undelivered are never pruned. (Blob bytes are not pruned — see ROADMAP.)
• Supervision. In dev/demo, process-compose supervises the stack with ordered startup and health gating; in production, systemd supervises process-compose (see DEPLOYMENT.md). The daemon exits cleanly on SIGINT/SIGTERM after the in-flight tick.
• Shutdown is bounded, not drained. The network servers stop accepting connections on SIGINT/SIGTERM and allow in-flight requests 10s to finish. That bounds how long a restart may block; it does not promise the work completes. A skill run longer than the grace is cut off and its caller sees a dropped connection — nothing is corrupted, since the audit and session records are written before the reply is returned, but the answer is lost. See ROADMAP.