Remote tracking (spool-and-flush)

Track runs on one machine, store and watch them on another. Design rationale: the design document §9. The model is spool-and-flush: nothing ever writes to the network directly — every writer uses a normal local store (the spool), and a flusher ships the delta to the server. Offline runs, crashes, and lost connections lose nothing; the spool keeps everything until it’s shipped, and every shipment is idempotent, so retrying is always safe.

Server (machine S)

# authenticated mode — users, roles, per-user tokens:
mlp user add paul --role admin --root /srv/mlparty/.mlparty
mlp token create --user paul --name train-box --root /srv/mlparty/.mlparty
mlp serve --root /srv/mlparty/.mlparty

# or legacy single-token mode (trusted networks only — reads stay open):
mlp serve --root /srv/mlparty/.mlparty --generate-token

mlp serve is mlp ui plus the authenticated sync-ingest API — same web viewer, same read API. With users configured, everything is login-gated: humans sign into the web UI, sync clients authenticate with a per-user token, and ingest needs the writer role (see Deployment for roles/administration). Keep it localhost-bound and tunnel over SSH, or put TLS in front before exposing it; credentials ride on every request. A server started with plain mlp ui refuses all sync writes.

Client (machine C or L)

Point the spool store at the server in store.toml:

[sync]
url = "http://127.0.0.1:7327"     # or the tunneled/TLS address
token_file = "server_token"        # or token = "…", or ML_PARTY_TOKEN env
interval_seconds = 10

The token is either a per-user mlp_… token (auth mode — must belong to a writer) or the legacy shared token; the client doesn’t care which.

That’s all. From then on:

  • mlparty.attach() flushes in a background thread while the training runs (metrics appear in the server UI within ~one interval) and does a synchronous final flush at finalize/fail.

  • mlp sync ships anything left over — after a crash, an offline run, or for stores nothing is attached to. Run it any time; it converges. Override ad hoc with mlp sync --to URL --token ….

What ships, and why it’s safe

Stream

Addressing

Retry behavior

journal events

ULID id per event

server dedupes; replays are no-ops

git snapshots

content (sha) + refs/runs/*

missing-negotiation; re-push skips

artifacts

content (sha256)

deduped on existence, hash-verified

metrics

byte offset per run

cursor advances only on ack; conflicts resume from the server’s size

Events are preserved verbatim — authorship (created_by) and timestamps survive the hop; the git model applies (the event’s created_by is the author, the authenticated token user is the transport identity that shipped it). Sync cursors live in the spool’s sync_state.json; deleting it merely causes a harmless full replay.

Caveats

  • One spool per run: two spools syncing the same run’s metrics to one server is detected and refused (offset conflict) — knowledge-graph events from many spools merge fine.

  • In legacy single-token mode the server trusts the token entirely and the read API is open — trusted networks only. Configure users for per-user identity and login-gated reads.