Design¶
(This page renders the repo-root DESIGN.md — the living design document.)
ml-party — Design¶
The living design document — when any other doc disagrees with this file, this file wins. User-facing how-to docs live in the
docs/user guide; planning lives in the repo’s GitHub Project.
1. Thesis¶
Every existing experiment tracker — W&B, MLflow, Neptune, Comet, Aim — is human-dashboard-first: excellent at scalars-over-time and config dicts, nearly silent about why a run exists and how it relates to the others. The information that rots first is not the metrics — it’s the reasoning and the lineage. ml-party bets that AI agents are the interface that will matter, so it is designed API-first for agents (usable by humans), and it treats a run’s abstract, typed relationships, and reproducibility contract as first-class data.
One-liner: a durable, agent-legible knowledge substrate for ML work — a lab notebook that agents write and query — that also happens to be a full local tracker.
The motivating case study is lived: ml-party was born inside a real
research codebase that had accumulated dozens of cryptically named run
directories of uncertain reproducibility, and fought it with a
hand-rolled discipline of memory files, handoff docs, and [[links]]. ml-party is that discipline made
structural, enforced, and queryable — so it no longer depends on an agent
remembering to write a good handoff.
2. Principles¶
System of record. ml-party owns knowledge (intent, lineage, verdicts) and the reproducibility contract. Telemetry mirrors are optional adapters; knowledge never lives primarily in a mirror target (§10).
Agents-first, humans always. Every capability lands API/MCP-first and is mirrored in the CLI and web UI. The MCP server teaches its own workflow (§8.1).
The write contract is the product — not the graph visualization. Enforcement is mechanical: a run cannot be finalized without its required knowledge fields, and refusals are machine-readable.
Meet users where they are; win where we’re unique. Instrumentation choice belongs to the user (mlparty, W&B, TensorBoard, several at once). We don’t compete on charting primitives; the bet is agent-authored boards (§11.1).
Orchestration is delegated, never owned. Jobs run under slurm / k8s / user scripts; ml-party tracks them wherever they execute and (later) exposes an audited control surface over user-registered actions (§11.2).
Multi-user-shaped, single-user first. Every write carries an identity; the API is scoped so tenancy can be added without breaking shape; user management ships late.
Local-first forever. The remote mode (§9) is a sync/transport layer over the same journal-first store; a laptop with no server keeps full functionality.
Knowledge is append-only; correction is an edge. Nodes are never destructively edited.
supersedes/refutesedges are the correction mechanism, and retrieval downranks corrected beliefs — enforcement produces completeness, not truth, so the correction story is structural.
3. Ontology¶
Fixed spine, extensible ribs: a small fixed core (so tools and cross-project
reasoning work) plus custom node types and x--prefixed custom edges.
Core node types: project → experiment → run, plus note for distilled
knowledge (kinds: feedback / reference / insight). An experiment answers
one question; new question → new experiment. There is deliberately no
sweep core type — a sweep is N runs sharing a commit, grouped by
params_hash or a part-of edge.
Every node carries: ULID id, type, required human-legible title
(the anti-cryptic-run-name rule), derived slug, tags, created_by
(agent / human / librarian / ingest), timestamps, schema_version.
Core edge vocabulary: derives-from, supersedes, compares-to,
produces, uses-data, part-of, confirms, refutes,
duplicate-of. Every edge carries provenance (created_by, created_at,
optional note) so a librarian-asserted edge is distinguishable from an
author-asserted one.
The run node records the full execution:
status:open | finalized | failed | abandoned(abandoned = silent death, stamped by a TTL janitor — different knowledge from a declared failure).parameters— the full config, mandatory and separate from the code commit (§5 explains why), canonicalized;params_hashindexed.The repro tuple, auto-captured:
code_ref(internal-git commit),snapshot_report,project_git(outer repo remote/HEAD/dirty),invocation(argv, cwd, entrypoint, allowlisted env — redacted),env_lock_ref(lockfile inside the snapshot commit),data_refs({uri, fingerprint, role}),seed,hardware.compute— where it runs, when that is not the machine that started it (§9):{system, job_id, url, host, note}, a pointer and nothing more. ml-party integrates with no job system and polls none; the field exists so a run submitted to a queue stays findable, and so the launcher’s hardware is not captured as the run’s when the run is declared remote up front (§9 —hardware.captured_bydistinguishes the two views either way).resultat finalize:summary,verdict ∈ {confirmed, refuted, inconclusive}, headlinemetrics(knowledge — indexed and searchable; the step series is telemetry in the per-run journal, not the graph),surprises(often the highest-value field).reproduce— exact re-run invocation, prefilled from the captured argv.failureon failed runs:what_failed,failure_class ∈ {crash, oom, diverged, wrong-result, env, cancelled}, optional why/traceback.
Concrete pydantic schemas: src/mlparty/models.py (the code is the
schema reference).
4. The write contract¶
Pre-registration split — intent is only honest before results exist:
run_startrequires:title,purpose,hypothesis,parameters."exploratory: <question>"is a valid hypothesis — forcing fake falsifiable claims on exploratory runs would generate slop. A run that dies before finalize therefore still carries intent + config + repro tuple instead of becoming a mystery directory.run_finalizeadds:method,result(summary / verdict / metrics —metrics_notemay justify an empty dict / surprises),reproduce. Refusals are machine-readable{missing: [...], invalid: [...]}so an agent repairs in one round-trip.run_failrecords the lighter floor: enough that a future agent can decide whether to re-attempt without re-running it.
Validation is deterministic only (presence, minimum content, resolvable edge targets, enums). Semantic quality policing is the librarian’s job, never the write path’s — a slow or flaky gate teaches agents to route around the contract.
Because the contract’s cost lands on the author and its value on a reader
weeks later, the write path gives value back: run_start returns hints
(tree-identical prior runs, stale open runs) and auto-captures everything
mechanically capturable, so the contract only demands what only the author
knows. Retro ingest (provenance: retro) relaxes only code_ref — retro
data must not weaken the live contract.
5. Code & reproducibility — the internal-git model¶
Each experiment owns a bare internal git repository (dulwich plumbing,
no working checkout). At run_start the actually-running source — including
uncommitted and project-gitignored files, where real ML scratch lives — is
snapshotted as a commit. Load-bearing invariants:
A run is not a commit — a run references a commit. Many runs → one commit is normal: a launch-arg sweep leaves the tree byte-identical, so runs share the commit and differ only in
parameters. This is exactly whyparametersis mandatory and separate — commit-only recording would collapse a sweep into one irreproducible blur.Commit identity = (tree, parents). Same tree + same declared derivation → same commit (dedup); same tree + different declared parent → distinct commit sharing the tree objects (~free).
Parentage follows declared derivation, not launch order. Default is an orphan root;
derives_frombecomes the commit parent. The DAG never asserts false lineage — assist-don’t-assert hints suggest edges, the author confirms them.The knowledge graph’s
derives-fromedges are authoritative; the git DAG is a materialized view (for diff/ancestry). Derived views can be regenerated; the authoritative store cannot.One ref per run (
refs/runs/<run_id>) — no shared branch tip, no ref races; this plus per-run metric journals and SQLite-WAL short transactions makes concurrency safe by construction.Nothing is captured that the user did not point at.
source_rootis explicit: without it a run records no source. With it, and when the root is a git repo, the file set is what git reports (tracked plus untracked-but-unignored) —.gitignoreis a boundary the user already drew, so the snapshot inherits it instead of second-guessing it with an extension allowlist. A non-git root has no such boundary, so it falls back to the allowlist walk and requiresconfirm_snapshot; so does any capture pastconfirm_above_files/confirm_above_bytes. A root at or above$HOMEis refused outright. Secret-shaped filenames are denied and size caps apply in both modes, with a hard refusal listing offenders. The snapshot report records what went in as well as what stayed out — capture fails in two directions (silently dropping the file that mattered; silently absorbing files that were never meant to leave the machine), and only an auditable report catches both.snapshot_preview()answers the same question before anything is written. Known trap: run outputs inside the source tree bloat snapshots; exclude them instore.toml.Env lock lives in the snapshot (
.mlparty/env.lock— part of the program, diffed and dedup’d); instance facts (hardware, seed, data fingerprints, metrics) live on the run node.Secrets hygiene. Parameter/env values with secret-shaped keys (
*_KEY,*TOKEN*,*SECRET*, …) are redacted before persist and the redactions noted in the report.
6. Storage — journal-first¶
The append-only journal.jsonl is the store’s source of truth
(flock’d, fsync’d). SQLite (WAL + FTS5 + embedding cache) is a rebuildable
index over it. This buys crash-safety, model-swappable embeddings, “ship
missing events” sync (§9), and the librarian’s event bus.
.mlparty/
store.toml # allowlist, caps, embedder, redaction rules
journal.jsonl # append-only event log — source of truth
index.sqlite # nodes/edges/FTS5/vectors — rebuildable
repos/<experiment_id>.git/ # bare internal repos; refs/runs/<run_id>
runs/<run_id>/metrics.jsonl # per-run telemetry journals (byte-offset tailed)
artifacts/sha256/ab/cd/<hash> # content-addressed artifact store
Artifacts are a separate sha256 CAS — not git, not LFS (git zlib-packs
binaries slowly for zero dedup gain on checkpoints; LFS drags in server
plumbing). Whole-file dedup; ArtifactRef = {sha256, bytes, media_type, original_path}. Data itself lives anywhere — a DataRef stores
uri + fingerprint, never the bytes.
7. Retrieval¶
Tiered and swappable, offline-capable at every tier. Default = SQLite
FTS5/BM25 (zero extra deps); optional local embedder (fastembed / ONNX)
and API embedders behind one interface, with cached embeddings. Hybrid =
reciprocal-rank fusion of lexical + vector, then 1-hop typed graph
expansion of seed hits, downranking superseded/refuted nodes (×0.35,
annotated corrected_by). Results are compact cards; agents drill down via
node_get, keeping query responses cheap in tokens.
8. Surfaces — thin frontends over one core¶
MlParty (core.py) is THE single API; the MCP server, the in-process
client, the mlp CLI, and the read-only HTTP+SSE API are thin frontends
over it.
8.1 MCP — the agent surface, self-teaching¶
13 tools (project_ensure, experiment_ensure/list, run_start,
run_log_metric, run_log_artifact, run_finalize, run_fail,
note_create, node_get, node_annotate, graph_query, run_diff).
Contract refusals return as data ({ok: false, refusal: {...}}), never as
protocol errors.
Zero-terminal setup is a product requirement: mlp init writes/merges
./.mcp.json so agents in the project pick the server up after a one-time
approval; the full workflow (orient → prior-art query → pre-register →
env-launch with instrumentation snippet → finalize → distill) is served by
the workflow_guide tool. “Use ml-party for this run” is all a cold agent
needs to hear.
The channel choice is load-bearing, and we got it wrong first. The workflow
originally shipped as the server’s MCP instructions, which clients cap —
Claude Code at exactly 2048 characters, mid-word, with no signal to either
side — so a 6.5k-character manual was delivered 31% complete to every agent
that connected. A teaching channel that can be silently truncated cannot
hold the contract.
The cap is the client behaving correctly, not a bug to route around. The MCP
schema defines instructions as “a hint to the model… MAY be added to the
system prompt” — no length limit, but no delivery guarantee either, so a
client that drops it entirely is conformant and truncating is more generous
than the spec requires. And capping is right on the merits: server
instructions are untrusted third-party text injected into the system prompt,
so an uncapped field lets any connected server flood the context. That is the
same reasoning we apply to retrieved node content (§ trust model); the client
is doing to us what we tell agents to do with notes. The error was ours —
putting a contract in a best-effort field.
Two consequences for anyone editing these strings. The cap is per client and
absent from the spec, so 2048 is not a portable number: the router must
survive any cap, which is why the pointer to workflow_guide is in the first
line rather than merely inside the budget. And the same cap applies per tool
description, truncated just as silently — run_start’s docstring is the one
that creeps toward it, so both limits are pinned by tests.
That still leaves a hole, because “MAY be added to the system prompt” permits a
client to drop instructions altogether: then no router arrives and nothing
points at the guide. So the backstop is the one channel that cannot be
truncated or dropped — the tool response itself. Every response carries a
nudge to call workflow_guide() until it has been called, which makes the
teaching independent of whether the connection brief survived. It is
self-extinguishing, and it reaches a confused agent at exactly the moment it is
confused, since refusals carry it too. Of the three channels MCP offers, only a tool is both
complete and reachable on the agent’s own initiative: instructions is
capped, and prompts are user-invoked slash commands, so falling back to one
requires the user to already suspect the agent is under-briefed. So
instructions is now a router — a summary whose first line is call
workflow_guide(), held under budget by a test — and track_training
remains as the user-facing entry point.
8.2 The two-writer architecture¶
An agent will not proxy 10k log_metric calls. The agent brackets the
run over MCP (start / finalize); the training process is the second
writer, streaming telemetry through the client lib: mlparty.attach()
picks up the ML_PARTY_STORE / ML_PARTY_RUN env handshake, logs
metrics/artifacts to the per-run journal, and auto-fails the run with the
traceback on an unhandled crash. No contention: the writers touch disjoint
files. Instrumentation is guarded (inert without ML_PARTY_RUN), so
scripts stay runnable standalone.
Telemetry is optional; knowledge is mandatory. If the step series goes
to another tool (or nowhere), the run still finalizes — result.metrics
headline numbers are required regardless of telemetry route. This seam
makes “metrics are up to the user” and “metrics are required” compatible.
8.3 Web UI¶
mlp ui serves a Vite+React+TS SPA from FastAPI, localhost-bound (tunnel
over SSH, TensorBoard-style). Entity-first IA: experiments → runs → tabbed
run page (Overview | Metrics | Artifacts | Code), live SSE metric charts,
lineage DAG, search, compare, diff, annotate. Served SPA over a desktop
shell because the store lives beside the runs on a headless training box;
Tauri remains a packaging option once the remote API exists.
9. Remote mode — the three-machine architecture¶
Target scenario: the user’s laptop L (agent lives here), a compute box C (training runs here), a tracking server S (store + UI live here). Local mode is the degenerate case L = C = S.
Split repro capture: the machine that runs the code is the only honest witness to its environment. The agent’s machine (L) captures the code snapshot + intent at
run_start;attach()on the compute host (C) patches env lock, hardware, and invocation onto the run. More accurate locally too (the launcher shell ≠ the training interpreter).The compute reference closes the gap the split leaves: between
run_starton L andattach()on C, nothing says where the job went — and a job that never attaches (queued, still pending, crashed before import) would say nothing at all. So a run carriescompute: the job system, its handle, and the url a human opens to find the job again. Declared atrun_startwhen already known, registered withrun_set_computeonce the submit command answers with an id (which is after start — hence a second call, not a start argument only), or handed to the job over the same env handshake as the run id (ML_PARTY_COMPUTE_SYSTEM/_JOB_ID/_URL) forattach()to record from C itself. Deliberately not a scheduler integration: ml-party neither submits nor polls, so it stays correct for every job system, including in-house ones it has never heard of. Declaringcomputeatrun_startskips hardware capture outright: the launcher’s silicon is a wrong answer, not an approximate one, once the code runs elsewhere. Declaring it later does not retract the capture —captured_byalready marks it as the launcher’s view, and deleting an honestly-labelled reading to satisfy an invariant would trade a labelled fact for no fact at all. So the rule is “never capture what is known to be wrong”, not “a remote run has no hardware”.Heartbeats: the client lib heartbeats in a background thread; UI shows running/stale in near-real-time; the janitor uses heartbeat age, not mtime, to stamp
abandoned.Spool-and-flush: there is deliberately no RemoteStore class — every writer writes to a local spool store (a normal store), and a flusher ships the delta to an
mlp serveinstance as three idempotent streams: ULID-keyed journal events (preserved verbatim — authorship and timestamps survive the hop), content-addressed git objects (missing-negotiation push,refs/runs/*only) and artifacts, and offset-addressed metric appends (cursor advances only on ack; a 409 reports the server’s size and the client resumes from there). Cursors live insync_state.json; losing them merely causes a safe replay. Reads use the ordinary HTTP read API. Configure viastore.toml [sync](url + token/token_file/ML_PARTY_TOKEN);mlparty.attach()flushes in a background thread with a synchronous final flush at finalize/fail;mlp syncships leftovers after crashes or offline runs.Legacy token auth: single shared bearer token per server (
mlp serve --token/--generate-token, constant-time compare) but identity-shaped (token →created_bylater), so multi-user auth changes keys, not shape. A token-less server (mlp ui) is read-only.Interim bridge: without a served store, MCP-over-SSH-stdio reaches one with zero new protocol.
10. Mirrors — the W&B ownership rule¶
One instrumentation, fan-out output (future adapter): mlparty.log_metric() optionally
mirrors to W&B/MLflow; finalize pushes the abstract/result into the mirror
run’s summary. The rule that keeps the system of record coherent:
telemetry may mirror anywhere; knowledge never lives primarily in the
mirror target. Zero lock-in either direction; import/backfill from W&B is
a stretch goal.
11. Forward design (agreed direction, later phases)¶
11.2 Run-control plane (shipped — local mode)¶
Users register action templates (shell with typed placeholders —
start/stop/resume/requeue); agents invoke them with validated,
shell-quoted parameter values, never free-form strings. Orchestration
stays in slurm/k8s/scripts; ml-party is the uniform, audited interface
agents use. Restart is never a mutation: “restart with lr halved” = a
new run, derives-from the old, params delta explicit — taught by the MCP
workflow.
Settled in the build: templates are the allowlist and live in
<root>/actions.json (server-side ops config like auth, backed up, not
knowledge); registration is a local-surface operation while the HTTP
server only lists (read) and invokes (write-class: writer role / ingest
token; never a sandboxed board). Invocations are action nodes — the
CustomNode extension mechanism, no spine growth — carrying the rendered
command, params, exit code and output tail, edged x-controls to the run
they control; they replay from the journal and sync like any node.
Execution is host-local and detached (outlives the caller); the remote
runner (execute on a different compute host than the server; templates
only ever execute where they were installed) is the deferred second half.
11.3 The librarian (post-1.0)¶
The first resident agent: consumes the event journal (its trigger bus),
proposes dedup / supersedes / summary notes — additive-only, never
edits or deletes, writes with created_by: librarian + confidence. Its
prerequisites (journal, edge provenance) are already in the core design.
11.4 Vision¶
ml-party grows from a lab notebook into an ML-workflow-oriented harness for AI agents: MCP gives agents hands, the knowledge graph gives them memory, boards give them a voice, the control plane gives them agency over jobs, and the journal gives resident agents their trigger bus. The discipline: the core stays a clean substrate; agents plug in as clients and hooks, never as a bundled runtime.
12. Identity & auth (shipped)¶
Auth is off until the first user exists (mlp user add); a store
without users behaves as if auth did not exist — mlp ui locally, the legacy
single --token gating ingest on a served store. Once users exist, the
served store is login-gated end to end: reads need a viewer, writes a
writer, user management an admin (store-wide roles; per-project scoping is
a later, additive layer). The legacy token is then ignored.
The layer splits in two, deliberately. Credential verification is one
pluggable step; everything else — user records, roles, sessions, per-user
API tokens, created_by stamping — is the shared identity substrate
every authenticator feeds into. The three SSO-proofing guardrails:
Users are keyed by an internal ULID, never username/email, and carry an
identitieslist —(provider: password)today; an OIDC identity (Entra ID is the first planned provider) is appended later, so one human can hold both.authenticate_password()is a sibling-shaped function: an OIDC callback lands next to it and issues the same sessions. Never a rework.The password hash is a nullable field — SSO-provisioned users simply never have one. Roles live on the user record (IdP-group mapping is a later additive feature).
Mechanics: users + hashed tokens in <root>/auth/users.json (small, text,
diffable, backed up with the store); passwords scrypt-hashed (stdlib — zero
new dependencies); sessions are stateless HMAC-signed httpOnly cookies (a
restart logs nobody out; a password reset invalidates that user’s
sessions); machine access via mlp_… bearer tokens (mlp token create,
shown once, stored hashed) — machines can’t SSO, so tokens are permanent
infrastructure, not a stopgap. created_by on server writes is the
authenticated principal, not what the request claimed (git’s author-vs-
committer split: journal events keep their claimed author; the token user
is the verified transport identity). Sandboxed boards run in opaque origins
that carry no cookies, so the UI mints a short-lived signed read-only
token (?bt=) for the iframe; boards-lib attaches it transparently, and
it can neither write nor mint successors. MCP/stdio and the in-process
client are local, same-trust-domain surfaces — authentication there is the
OS user, not a login.
13. Stack¶
Python 3.11+ · pydantic v2 (every schema is a model) · SQLite WAL + FTS5 ·
dulwich (pure-Python git plumbing; snapshot trees are small, speed is
irrelevant) · official mcp SDK · FastAPI + uvicorn (HTTP/SSE) · typer
(mlp) · Vite + React + TS (UI) · pytest + ruff. Nothing in ml-party is
performance-bound; the ecosystem (MCP SDK, git plumbing, ML tooling) is
Python-native.
Validated 2026-08 on a private research corpus: the acceptance queries answer correctly; a real GPU training was tracked live end-to-end; a cold, unbriefed agent drove the MCP tools correctly.