Design

(This page renders the repo-root DESIGN.md — the living design document.)

ml-party — Design

The living design document — when any other doc disagrees with this file, this file wins. User-facing how-to docs live in the docs/ user guide; planning lives in the repo’s GitHub Project.


1. Thesis

Every existing experiment tracker — W&B, MLflow, Neptune, Comet, Aim — is human-dashboard-first: excellent at scalars-over-time and config dicts, nearly silent about why a run exists and how it relates to the others. The information that rots first is not the metrics — it’s the reasoning and the lineage. ml-party bets that AI agents are the interface that will matter, so it is designed API-first for agents (usable by humans), and it treats a run’s abstract, typed relationships, and reproducibility contract as first-class data.

One-liner: a durable, agent-legible knowledge substrate for ML work — a lab notebook that agents write and query — that also happens to be a full local tracker.

The motivating case study is lived: ml-party was born inside a real research codebase that had accumulated dozens of cryptically named run directories of uncertain reproducibility, and fought it with a hand-rolled discipline of memory files, handoff docs, and [[links]]. ml-party is that discipline made structural, enforced, and queryable — so it no longer depends on an agent remembering to write a good handoff.

2. Principles

  1. System of record. ml-party owns knowledge (intent, lineage, verdicts) and the reproducibility contract. Telemetry mirrors are optional adapters; knowledge never lives primarily in a mirror target (§10).

  2. Agents-first, humans always. Every capability lands API/MCP-first and is mirrored in the CLI and web UI. The MCP server teaches its own workflow (§8.1).

  3. The write contract is the product — not the graph visualization. Enforcement is mechanical: a run cannot be finalized without its required knowledge fields, and refusals are machine-readable.

  4. Meet users where they are; win where we’re unique. Instrumentation choice belongs to the user (mlparty, W&B, TensorBoard, several at once). We don’t compete on charting primitives; the bet is agent-authored boards (§11.1).

  5. Orchestration is delegated, never owned. Jobs run under slurm / k8s / user scripts; ml-party tracks them wherever they execute and (later) exposes an audited control surface over user-registered actions (§11.2).

  6. Multi-user-shaped, single-user first. Every write carries an identity; the API is scoped so tenancy can be added without breaking shape; user management ships late.

  7. Local-first forever. The remote mode (§9) is a sync/transport layer over the same journal-first store; a laptop with no server keeps full functionality.

  8. Knowledge is append-only; correction is an edge. Nodes are never destructively edited. supersedes / refutes edges are the correction mechanism, and retrieval downranks corrected beliefs — enforcement produces completeness, not truth, so the correction story is structural.

3. Ontology

Fixed spine, extensible ribs: a small fixed core (so tools and cross-project reasoning work) plus custom node types and x--prefixed custom edges.

Core node types: project → experiment → run, plus note for distilled knowledge (kinds: feedback / reference / insight). An experiment answers one question; new question → new experiment. There is deliberately no sweep core type — a sweep is N runs sharing a commit, grouped by params_hash or a part-of edge.

Every node carries: ULID id, type, required human-legible title (the anti-cryptic-run-name rule), derived slug, tags, created_by (agent / human / librarian / ingest), timestamps, schema_version.

Core edge vocabulary: derives-from, supersedes, compares-to, produces, uses-data, part-of, confirms, refutes, duplicate-of. Every edge carries provenance (created_by, created_at, optional note) so a librarian-asserted edge is distinguishable from an author-asserted one.

The run node records the full execution:

  • status: open | finalized | failed | abandoned (abandoned = silent death, stamped by a TTL janitor — different knowledge from a declared failure).

  • parameters — the full config, mandatory and separate from the code commit (§5 explains why), canonicalized; params_hash indexed.

  • The repro tuple, auto-captured: code_ref (internal-git commit), snapshot_report, project_git (outer repo remote/HEAD/dirty), invocation (argv, cwd, entrypoint, allowlisted env — redacted), env_lock_ref (lockfile inside the snapshot commit), data_refs ({uri, fingerprint, role}), seed, hardware.

  • compute — where it runs, when that is not the machine that started it (§9): {system, job_id, url, host, note}, a pointer and nothing more. ml-party integrates with no job system and polls none; the field exists so a run submitted to a queue stays findable, and so the launcher’s hardware is not captured as the run’s when the run is declared remote up front (§9 — hardware.captured_by distinguishes the two views either way).

  • result at finalize: summary, verdict ∈ {confirmed, refuted, inconclusive}, headline metrics (knowledge — indexed and searchable; the step series is telemetry in the per-run journal, not the graph), surprises (often the highest-value field).

  • reproduce — exact re-run invocation, prefilled from the captured argv.

  • failure on failed runs: what_failed, failure_class ∈ {crash, oom, diverged, wrong-result, env, cancelled}, optional why/traceback.

Concrete pydantic schemas: src/mlparty/models.py (the code is the schema reference).

4. The write contract

Pre-registration split — intent is only honest before results exist:

  • run_start requires: title, purpose, hypothesis, parameters. "exploratory: <question>" is a valid hypothesis — forcing fake falsifiable claims on exploratory runs would generate slop. A run that dies before finalize therefore still carries intent + config + repro tuple instead of becoming a mystery directory.

  • run_finalize adds: method, result (summary / verdict / metrics — metrics_note may justify an empty dict / surprises), reproduce. Refusals are machine-readable {missing: [...], invalid: [...]} so an agent repairs in one round-trip.

  • run_fail records the lighter floor: enough that a future agent can decide whether to re-attempt without re-running it.

Validation is deterministic only (presence, minimum content, resolvable edge targets, enums). Semantic quality policing is the librarian’s job, never the write path’s — a slow or flaky gate teaches agents to route around the contract.

Because the contract’s cost lands on the author and its value on a reader weeks later, the write path gives value back: run_start returns hints (tree-identical prior runs, stale open runs) and auto-captures everything mechanically capturable, so the contract only demands what only the author knows. Retro ingest (provenance: retro) relaxes only code_ref — retro data must not weaken the live contract.

5. Code & reproducibility — the internal-git model

Each experiment owns a bare internal git repository (dulwich plumbing, no working checkout). At run_start the actually-running source — including uncommitted and project-gitignored files, where real ML scratch lives — is snapshotted as a commit. Load-bearing invariants:

  1. A run is not a commit — a run references a commit. Many runs → one commit is normal: a launch-arg sweep leaves the tree byte-identical, so runs share the commit and differ only in parameters. This is exactly why parameters is mandatory and separate — commit-only recording would collapse a sweep into one irreproducible blur.

  2. Commit identity = (tree, parents). Same tree + same declared derivation → same commit (dedup); same tree + different declared parent → distinct commit sharing the tree objects (~free).

  3. Parentage follows declared derivation, not launch order. Default is an orphan root; derives_from becomes the commit parent. The DAG never asserts false lineage — assist-don’t-assert hints suggest edges, the author confirms them.

  4. The knowledge graph’s derives-from edges are authoritative; the git DAG is a materialized view (for diff/ancestry). Derived views can be regenerated; the authoritative store cannot.

  5. One ref per run (refs/runs/<run_id>) — no shared branch tip, no ref races; this plus per-run metric journals and SQLite-WAL short transactions makes concurrency safe by construction.

  6. Nothing is captured that the user did not point at. source_root is explicit: without it a run records no source. With it, and when the root is a git repo, the file set is what git reports (tracked plus untracked-but-unignored) — .gitignore is a boundary the user already drew, so the snapshot inherits it instead of second-guessing it with an extension allowlist. A non-git root has no such boundary, so it falls back to the allowlist walk and requires confirm_snapshot; so does any capture past confirm_above_files / confirm_above_bytes. A root at or above $HOME is refused outright. Secret-shaped filenames are denied and size caps apply in both modes, with a hard refusal listing offenders. The snapshot report records what went in as well as what stayed out — capture fails in two directions (silently dropping the file that mattered; silently absorbing files that were never meant to leave the machine), and only an auditable report catches both. snapshot_preview() answers the same question before anything is written. Known trap: run outputs inside the source tree bloat snapshots; exclude them in store.toml.

  7. Env lock lives in the snapshot (.mlparty/env.lock — part of the program, diffed and dedup’d); instance facts (hardware, seed, data fingerprints, metrics) live on the run node.

  8. Secrets hygiene. Parameter/env values with secret-shaped keys (*_KEY, *TOKEN*, *SECRET*, …) are redacted before persist and the redactions noted in the report.

6. Storage — journal-first

The append-only journal.jsonl is the store’s source of truth (flock’d, fsync’d). SQLite (WAL + FTS5 + embedding cache) is a rebuildable index over it. This buys crash-safety, model-swappable embeddings, “ship missing events” sync (§9), and the librarian’s event bus.

.mlparty/
  store.toml                       # allowlist, caps, embedder, redaction rules
  journal.jsonl                    # append-only event log — source of truth
  index.sqlite                     # nodes/edges/FTS5/vectors — rebuildable
  repos/<experiment_id>.git/       # bare internal repos; refs/runs/<run_id>
  runs/<run_id>/metrics.jsonl      # per-run telemetry journals (byte-offset tailed)
  artifacts/sha256/ab/cd/<hash>    # content-addressed artifact store

Artifacts are a separate sha256 CAS — not git, not LFS (git zlib-packs binaries slowly for zero dedup gain on checkpoints; LFS drags in server plumbing). Whole-file dedup; ArtifactRef = {sha256, bytes, media_type, original_path}. Data itself lives anywhere — a DataRef stores uri + fingerprint, never the bytes.

7. Retrieval

Tiered and swappable, offline-capable at every tier. Default = SQLite FTS5/BM25 (zero extra deps); optional local embedder (fastembed / ONNX) and API embedders behind one interface, with cached embeddings. Hybrid = reciprocal-rank fusion of lexical + vector, then 1-hop typed graph expansion of seed hits, downranking superseded/refuted nodes (×0.35, annotated corrected_by). Results are compact cards; agents drill down via node_get, keeping query responses cheap in tokens.

8. Surfaces — thin frontends over one core

MlParty (core.py) is THE single API; the MCP server, the in-process client, the mlp CLI, and the read-only HTTP+SSE API are thin frontends over it.

8.1 MCP — the agent surface, self-teaching

13 tools (project_ensure, experiment_ensure/list, run_start, run_log_metric, run_log_artifact, run_finalize, run_fail, note_create, node_get, node_annotate, graph_query, run_diff). Contract refusals return as data ({ok: false, refusal: {...}}), never as protocol errors.

Zero-terminal setup is a product requirement: mlp init writes/merges ./.mcp.json so agents in the project pick the server up after a one-time approval; the full workflow (orient → prior-art query → pre-register → env-launch with instrumentation snippet → finalize → distill) is served by the workflow_guide tool. “Use ml-party for this run” is all a cold agent needs to hear.

The channel choice is load-bearing, and we got it wrong first. The workflow originally shipped as the server’s MCP instructions, which clients cap — Claude Code at exactly 2048 characters, mid-word, with no signal to either side — so a 6.5k-character manual was delivered 31% complete to every agent that connected. A teaching channel that can be silently truncated cannot hold the contract.

The cap is the client behaving correctly, not a bug to route around. The MCP schema defines instructions as “a hint to the model… MAY be added to the system prompt” — no length limit, but no delivery guarantee either, so a client that drops it entirely is conformant and truncating is more generous than the spec requires. And capping is right on the merits: server instructions are untrusted third-party text injected into the system prompt, so an uncapped field lets any connected server flood the context. That is the same reasoning we apply to retrieved node content (§ trust model); the client is doing to us what we tell agents to do with notes. The error was ours — putting a contract in a best-effort field.

Two consequences for anyone editing these strings. The cap is per client and absent from the spec, so 2048 is not a portable number: the router must survive any cap, which is why the pointer to workflow_guide is in the first line rather than merely inside the budget. And the same cap applies per tool description, truncated just as silently — run_start’s docstring is the one that creeps toward it, so both limits are pinned by tests.

That still leaves a hole, because “MAY be added to the system prompt” permits a client to drop instructions altogether: then no router arrives and nothing points at the guide. So the backstop is the one channel that cannot be truncated or dropped — the tool response itself. Every response carries a nudge to call workflow_guide() until it has been called, which makes the teaching independent of whether the connection brief survived. It is self-extinguishing, and it reaches a confused agent at exactly the moment it is confused, since refusals carry it too. Of the three channels MCP offers, only a tool is both complete and reachable on the agent’s own initiative: instructions is capped, and prompts are user-invoked slash commands, so falling back to one requires the user to already suspect the agent is under-briefed. So instructions is now a router — a summary whose first line is call workflow_guide(), held under budget by a test — and track_training remains as the user-facing entry point.

8.2 The two-writer architecture

An agent will not proxy 10k log_metric calls. The agent brackets the run over MCP (start / finalize); the training process is the second writer, streaming telemetry through the client lib: mlparty.attach() picks up the ML_PARTY_STORE / ML_PARTY_RUN env handshake, logs metrics/artifacts to the per-run journal, and auto-fails the run with the traceback on an unhandled crash. No contention: the writers touch disjoint files. Instrumentation is guarded (inert without ML_PARTY_RUN), so scripts stay runnable standalone.

Telemetry is optional; knowledge is mandatory. If the step series goes to another tool (or nowhere), the run still finalizes — result.metrics headline numbers are required regardless of telemetry route. This seam makes “metrics are up to the user” and “metrics are required” compatible.

8.3 Web UI

mlp ui serves a Vite+React+TS SPA from FastAPI, localhost-bound (tunnel over SSH, TensorBoard-style). Entity-first IA: experiments → runs → tabbed run page (Overview | Metrics | Artifacts | Code), live SSE metric charts, lineage DAG, search, compare, diff, annotate. Served SPA over a desktop shell because the store lives beside the runs on a headless training box; Tauri remains a packaging option once the remote API exists.

9. Remote mode — the three-machine architecture

Target scenario: the user’s laptop L (agent lives here), a compute box C (training runs here), a tracking server S (store + UI live here). Local mode is the degenerate case L = C = S.

  • Split repro capture: the machine that runs the code is the only honest witness to its environment. The agent’s machine (L) captures the code snapshot + intent at run_start; attach() on the compute host (C) patches env lock, hardware, and invocation onto the run. More accurate locally too (the launcher shell ≠ the training interpreter).

  • The compute reference closes the gap the split leaves: between run_start on L and attach() on C, nothing says where the job went — and a job that never attaches (queued, still pending, crashed before import) would say nothing at all. So a run carries compute: the job system, its handle, and the url a human opens to find the job again. Declared at run_start when already known, registered with run_set_compute once the submit command answers with an id (which is after start — hence a second call, not a start argument only), or handed to the job over the same env handshake as the run id (ML_PARTY_COMPUTE_SYSTEM / _JOB_ID / _URL) for attach() to record from C itself. Deliberately not a scheduler integration: ml-party neither submits nor polls, so it stays correct for every job system, including in-house ones it has never heard of. Declaring compute at run_start skips hardware capture outright: the launcher’s silicon is a wrong answer, not an approximate one, once the code runs elsewhere. Declaring it later does not retract the capture — captured_by already marks it as the launcher’s view, and deleting an honestly-labelled reading to satisfy an invariant would trade a labelled fact for no fact at all. So the rule is “never capture what is known to be wrong”, not “a remote run has no hardware”.

  • Heartbeats: the client lib heartbeats in a background thread; UI shows running/stale in near-real-time; the janitor uses heartbeat age, not mtime, to stamp abandoned.

  • Spool-and-flush: there is deliberately no RemoteStore class — every writer writes to a local spool store (a normal store), and a flusher ships the delta to an mlp serve instance as three idempotent streams: ULID-keyed journal events (preserved verbatim — authorship and timestamps survive the hop), content-addressed git objects (missing-negotiation push, refs/runs/* only) and artifacts, and offset-addressed metric appends (cursor advances only on ack; a 409 reports the server’s size and the client resumes from there). Cursors live in sync_state.json; losing them merely causes a safe replay. Reads use the ordinary HTTP read API. Configure via store.toml [sync] (url + token/token_file/ML_PARTY_TOKEN); mlparty.attach() flushes in a background thread with a synchronous final flush at finalize/fail; mlp sync ships leftovers after crashes or offline runs.

  • Legacy token auth: single shared bearer token per server (mlp serve --token/--generate-token, constant-time compare) but identity-shaped (token → created_by later), so multi-user auth changes keys, not shape. A token-less server (mlp ui) is read-only.

  • Interim bridge: without a served store, MCP-over-SSH-stdio reaches one with zero new protocol.

10. Mirrors — the W&B ownership rule

One instrumentation, fan-out output (future adapter): mlparty.log_metric() optionally mirrors to W&B/MLflow; finalize pushes the abstract/result into the mirror run’s summary. The rule that keeps the system of record coherent: telemetry may mirror anywhere; knowledge never lives primarily in the mirror target. Zero lock-in either direction; import/backfill from W&B is a stretch goal.

11. Forward design (agreed direction, later phases)

11.1 Boards — agent-authored views (the differentiator — shipped)

HTML artifacts become first-class boards: rendered by the web UI in a sandboxed iframe (CSP; boards are data, never trusted UI), listed per run/experiment, and allowed to call the read-only API from inside the sandbox. Incumbents render charts humans configure; ml-party’s agents author whole live views — comparison dashboards, audio/spectrogram galleries — per experiment. This, not charting parity, is the UX bet.

Settled: boards never enter the graph spine. A board is not a node and gets no produces edge — its provenance is already fully expressed by containment (it sits in the artifacts list of the run or experiment that produced it; experiments carry an artifacts field mirroring runs for exactly this). Minting a board node type or edge would grow the spine for a rendering/discovery concern. Discovery is solved directly instead: every carried artifact’s filename + note (plus a “board” token for text/html) is indexed into the carrying node’s FTS text, so graph_query surfaces the carrier; browsing goes through the board gallery (/api/boards, UI Boards page + per-experiment section). produces stays reserved for true node→node provenance (e.g. a future dataset node).

11.2 Run-control plane (shipped — local mode)

Users register action templates (shell with typed placeholders — start/stop/resume/requeue); agents invoke them with validated, shell-quoted parameter values, never free-form strings. Orchestration stays in slurm/k8s/scripts; ml-party is the uniform, audited interface agents use. Restart is never a mutation: “restart with lr halved” = a new run, derives-from the old, params delta explicit — taught by the MCP workflow.

Settled in the build: templates are the allowlist and live in <root>/actions.json (server-side ops config like auth, backed up, not knowledge); registration is a local-surface operation while the HTTP server only lists (read) and invokes (write-class: writer role / ingest token; never a sandboxed board). Invocations are action nodes — the CustomNode extension mechanism, no spine growth — carrying the rendered command, params, exit code and output tail, edged x-controls to the run they control; they replay from the journal and sync like any node. Execution is host-local and detached (outlives the caller); the remote runner (execute on a different compute host than the server; templates only ever execute where they were installed) is the deferred second half.

11.3 The librarian (post-1.0)

The first resident agent: consumes the event journal (its trigger bus), proposes dedup / supersedes / summary notes — additive-only, never edits or deletes, writes with created_by: librarian + confidence. Its prerequisites (journal, edge provenance) are already in the core design.

11.4 Vision

ml-party grows from a lab notebook into an ML-workflow-oriented harness for AI agents: MCP gives agents hands, the knowledge graph gives them memory, boards give them a voice, the control plane gives them agency over jobs, and the journal gives resident agents their trigger bus. The discipline: the core stays a clean substrate; agents plug in as clients and hooks, never as a bundled runtime.

12. Identity & auth (shipped)

Auth is off until the first user exists (mlp user add); a store without users behaves as if auth did not exist — mlp ui locally, the legacy single --token gating ingest on a served store. Once users exist, the served store is login-gated end to end: reads need a viewer, writes a writer, user management an admin (store-wide roles; per-project scoping is a later, additive layer). The legacy token is then ignored.

The layer splits in two, deliberately. Credential verification is one pluggable step; everything else — user records, roles, sessions, per-user API tokens, created_by stamping — is the shared identity substrate every authenticator feeds into. The three SSO-proofing guardrails:

  1. Users are keyed by an internal ULID, never username/email, and carry an identities list — (provider: password) today; an OIDC identity (Entra ID is the first planned provider) is appended later, so one human can hold both.

  2. authenticate_password() is a sibling-shaped function: an OIDC callback lands next to it and issues the same sessions. Never a rework.

  3. The password hash is a nullable field — SSO-provisioned users simply never have one. Roles live on the user record (IdP-group mapping is a later additive feature).

Mechanics: users + hashed tokens in <root>/auth/users.json (small, text, diffable, backed up with the store); passwords scrypt-hashed (stdlib — zero new dependencies); sessions are stateless HMAC-signed httpOnly cookies (a restart logs nobody out; a password reset invalidates that user’s sessions); machine access via mlp_… bearer tokens (mlp token create, shown once, stored hashed) — machines can’t SSO, so tokens are permanent infrastructure, not a stopgap. created_by on server writes is the authenticated principal, not what the request claimed (git’s author-vs- committer split: journal events keep their claimed author; the token user is the verified transport identity). Sandboxed boards run in opaque origins that carry no cookies, so the UI mints a short-lived signed read-only token (?bt=) for the iframe; boards-lib attaches it transparently, and it can neither write nor mint successors. MCP/stdio and the in-process client are local, same-trust-domain surfaces — authentication there is the OS user, not a login.

13. Stack

Python 3.11+ · pydantic v2 (every schema is a model) · SQLite WAL + FTS5 · dulwich (pure-Python git plumbing; snapshot trees are small, speed is irrelevant) · official mcp SDK · FastAPI + uvicorn (HTTP/SSE) · typer (mlp) · Vite + React + TS (UI) · pytest + ruff. Nothing in ml-party is performance-bound; the ecosystem (MCP SDK, git plumbing, ML tooling) is Python-native.


Validated 2026-08 on a private research corpus: the acceptance queries answer correctly; a real GPU training was tracked live end-to-end; a cold, unbriefed agent drove the MCP tools correctly.