geth/docs/production-readiness-roadmap.md

# Production Readiness Roadmap

This roadmap tracks the work required before `geth` becomes a dependable base
layer for personal scripts, infrastructure automation, and other projects.
Deployment has not started, so large refactors are in scope when they reduce
long-term risk.

Status markers:

- `[ ]` Not started
- `[~]` In progress
- `[x]` Done

## Release Principle

Do not widen the feature surface until the existing daemon, storage, protocol,
authorization, and operational contracts are boring to test, upgrade, and
debug. The project already has enough prototype capability to validate the
architecture; the next milestone is making those capabilities stable.

## Phase 0: Current Quality Gate

Goal: make the documented local quality gate pass before deeper refactors.

- `[x]` Restore the full workspace quality gate.
  Acceptance criteria:
  - `[x]` `cargo fmt --all -- --check` passes.
  - `[x]` `cargo check --workspace --all-targets` passes.
  - `[x]` `cargo test --workspace` passes.
  - `[x]` `cargo clippy --workspace --all-targets -- -D warnings` passes.
  - `[x]` CI enforces the same required checks.

## Phase 1: Daemon Subsystem Refactor

Goal: split `geth-node` into reviewable daemon subsystems without changing
behavior.

- `[x]` Extract daemon startup and runtime ownership.
  Acceptance criteria:
  - `[x]` Daemon startup, shutdown, signal handling, socket setup, and Iroh
    endpoint ownership live outside the main feature handler module.
  - `[x]` Iroh accept-loop and background live-sync task spawning live outside
    the main feature handler module.
  - `[x]` Runtime state is represented by narrow structs with documented
    ownership and locking rules.
  - `[x]` Existing daemon startup and status tests pass unchanged.

- `[~]` Extract local control routing.
  Acceptance criteria:
  - `[x]` Local `ControlRequest` dispatch is a routing layer, not the home of
    every feature implementation.
  - `[ ]` Each command family has a small handler module or function group.
  - `[x]` Local-only behavior remains covered by existing integration tests.

- `[~]` Extract protected peer-control routing.
  Acceptance criteria:
  - `[x]` Shared bounded Iroh line-read and send-finish helpers live outside
    the main feature handler module.
  - `[x]` Outbound peer-control, pipe-wire, and overlay-wire request helpers
    live outside the main feature handler module.
  - `[~]` Iroh control ALPN handling, nonce checks, peer-card validation, and
    endpoint-binding validation are centralized.
  - `[x]` Feature handlers receive authenticated caller context rather than
    repeating peer-card boilerplate.
  - `[x]` Remote request tests still prove discovery alone grants no access.

- `[ ]` Extract resource module handlers.
  Acceptance criteria:
  - `[ ]` CAS/file-root, KV, DB, document, pubsub, pipe, SSH, and overlay
    handlers are separated enough that each can be reviewed independently.
  - `[ ]` Each module documents its resource IDs, capabilities, and mutation
    points.
  - `[ ]` No generic `geth-common` crate is introduced.

- `[x]` Extract live-sync engine.
  Acceptance criteria:
  - `[x]` Sync cursor keys, cursor persistence, stream health recording, and
    local sync-status reduction live in a sync-focused module.
  - `[x]` Sync stream selection, watermarks, and run result helpers live in a
    sync-focused module.
  - `[x]` Per-module sync handlers have consistent interfaces.
  - `[x]` `geth sync status --json` output remains stable.

## Phase 2: Stable Automation Contracts

Goal: make command, JSON, and protocol contracts explicit enough for scripts
and downstream projects.

- `[x]` Define compatibility policy.
  Acceptance criteria:
  - `[x]` CLI command compatibility is documented.
  - `[x]` `--json` output compatibility is documented.
  - `[x]` Local control JSONL compatibility is documented.
  - `[x]` Peer wire protocol compatibility is documented.
  - `[x]` SQLite and signed-operation compatibility are documented.

- `[x]` Add golden JSON tests.
  Acceptance criteria:
  - `[x]` Important script-facing commands have stable JSON fixture tests.
  - `[x]` Error JSON includes stable codes for common operator and automation
    failures.
  - `[x]` Fixture updates require intentional review.

- `[x]` Expand protocol roundtrip tests.
  Acceptance criteria:
  - `[x]` Every `ControlRequest` and `ControlResponse` variant roundtrips.
  - `[x]` Every `PeerControlRequest` and `PeerControlResponse` variant
    roundtrips.
  - `[x]` Pipe and overlay wire protocol variants roundtrip.
  - `[x]` Unknown or malformed protocol inputs fail safely.

- `[x]` Classify command stability.
  Acceptance criteria:
  - `[x]` Commands are marked stable, experimental, or prototype in docs.
  - `[x]` Help text avoids production claims for experimental paths.
  - `[x]` Prototype paths have explicit migration or removal expectations.

## Phase 3: Storage And Migration Hardening

Goal: treat local SQLite state as durable product data before users depend on
it.

- `[x]` Replace opportunistic schema setup with ordered migrations.
  Acceptance criteria:
  - `[x]` The store tracks numeric schema versions.
  - `[x]` Each migration is transactional where SQLite supports it.
  - `[x]` Fresh database creation and repeated opens are idempotent.
  - `[x]` Old schema fixtures migrate to the current schema in tests.

- `[x]` Make multi-table writes transactional.
  Acceptance criteria:
  - `[x]` Signed operation and signature imports commit atomically.
  - `[x]` Keychain/auth sync imports stage accepted records and commit accepted
    op/signature groups through store batch transactions.
  - `[x]` SSH distribution sync imports stage accepted records and commit
    requests, certificates, and revocations through one store transaction.
  - `[x]` Multi-record sync imports cannot leave partial state after a local
    error.
  - `[x]` Tests cover failure injection for at least one multi-table path.

- `[x]` Add backup and restore workflow.
  Acceptance criteria:
  - `[x]` `geth backup create` or an equivalent documented command captures
    metadata, identity public material, config, and CAS metadata expectations.
  - `[x]` Restore can write to a separate target home for validation.
  - `[x]` Backup output avoids copying private SSH admin keys.
  - `[x]` Docs explain what is and is not included.

- `[x]` Document database durability settings.
  Acceptance criteria:
  - `[x]` WAL and synchronous settings are chosen deliberately.
  - `[x]` Crash-recovery expectations are documented.
  - `[x]` `geth doctor` or status output reports obvious store issues.

## Phase 4: Security Boundary Closure

Goal: finish the authorization and remote-input audit before deployment.

- `[x]` Complete remote authorization matrix coverage.
  Acceptance criteria:
  - `[x]` Denied and allowed paths are tested for CAS, KV, DB, document,
    pubsub, pipe, SSH proxy/admin shell, SSH cert metadata, and revocations.
  - `[x]` Every remote mutating or service-opening operation has an explicit
    resource capability check before mutation or host access.
  - `[x]` The matrix fails tests when a new remote operation lacks a guard.

- `[x]` Add adversarial identity and bearer tests.
  Acceptance criteria:
  - `[x]` Tests cover discovered-only peers.
  - `[x]` Tests cover wrong endpoint bindings.
  - `[x]` Tests cover revoked nodes.
  - `[x]` Tests cover stale or mismatched peer cards.
  - `[x]` Tests cover bearer secrets with wrong capabilities.
  - `[x]` Tests cover bearer attempts to mutate trust graph state.

- `[x]` Bound all remote input paths.
  Acceptance criteria:
  - `[x]` Remote request payloads have documented size limits.
  - `[x]` Remote stream reads have timeouts or bounded behavior.
  - `[x]` Oversized messages are rejected without state mutation.
  - `[x]` Tests cover oversized payload denial for representative protocols.

- `[x]` Audit host-opening paths.
  Acceptance criteria:
  - `[x]` TCP forwarding remains loopback-only unless a later ADR expands it.
  - `[x]` Unix forwarding rejects unsafe paths.
  - `[x]` SSH proxy connects only to the intended local SSH endpoint.
  - `[x]` Overlay TUN setup remains explicit opt-in.
  - `[x]` Docs state host access risks and recovery commands.

## Phase 5: Production Cryptography Boundary

Goal: remove prototype cryptography from paths users may treat as real
confidential storage.

- `[x]` Replace prototype private CAS envelope.
  Acceptance criteria:
  - `[x]` The BLAKE3-XOR prototype envelope is not used for new private CAS
    writes.
  - `[x]` New private CAS writes use a reviewed AEAD construction or an
    established envelope format such as age.
  - `[x]` Key derivation, nonce generation, and envelope versioning are
    documented.
  - `[x]` Tests cover tamper detection, wrong resource, wrong key, and nonce
    uniqueness behavior.

- `[x]` Define pre-release encrypted blob migration behavior.
  Acceptance criteria:
  - `[x]` Existing prototype envelopes are either rejected with a clear error
    or migrated through an explicit command.
  - `[x]` Docs state that prototype envelopes made before deployment are not a
    durable security format.

## Phase 6: Sync Correctness And Fault Testing

Goal: make convergence and failure behavior predictable enough for automation.

- `[ ]` Build deterministic multi-daemon fault tests.
  Acceptance criteria:
  - `[ ]` Tests cover peer restart.
  - `[ ]` Tests cover endpoint rotation.
  - `[ ]` Tests cover temporary peer unavailability.
  - `[ ]` Tests cover partial stream failure.
  - `[ ]` Tests cover duplicate records and stale cursors.

- `[x]` Define per-resource conflict semantics.
  Acceptance criteria:
  - `[x]` CAS/file-root conflict semantics are documented and tested.
  - `[x]` KV conflict semantics are documented and tested.
  - `[x]` Document merge semantics are documented and tested.
  - `[x]` DB change application limits are documented and tested.

- `[x]` Add live-sync retry and backoff policy.
  Acceptance criteria:
  - `[x]` Failed peer streams back off without starving healthy streams.
  - `[x]` Retry state appears in `geth sync status --json`.
  - `[x]` Operators can trigger immediate retry with `geth sync now`.

## Phase 7: Script And Automation UX

Goal: make `geth` ergonomic and stable as a base layer for custom automation.

- `[x]` Stabilize JSON errors.
  Acceptance criteria:
  - `[x]` Common failures include stable machine-readable error codes.
  - `[x]` Human errors still include next-step recovery hints.
  - `[x]` Tests assert both code and operator-facing hint for representative
    failures.

- `[x]` Add wait commands for automation.
  Acceptance criteria:
  - `[x]` `geth wait daemon` can block until local control is ready.
  - `[x]` `geth wait peer <node>` can block until peer control succeeds or
    times out.
  - `[x]` `geth wait sync <node>` can block until required streams are healthy
    or stale.
  - `[x]` Wait commands support JSON output and timeout flags.

- `[x]` Make common commands idempotent.
  Acceptance criteria:
  - `[x]` Resource creation commands can be safely repeated where practical.
  - `[x]` Grant creation supports deterministic caller-provided IDs.
  - `[x]` Repeated sync and import commands report no-op state clearly.

- `[x]` Publish automation examples.
  Acceptance criteria:
  - `[x]` Docs include shell examples.
  - `[x]` Docs include Python examples using CLI JSON.
  - `[x]` Docs include user-service automation examples.
  - `[x]` Examples avoid private-key copying and privileged service mutation.

## Phase 8: Observability And Operations

Goal: make production failures diagnosable from the CLI and logs.

- `[x]` Improve structured tracing.
  Acceptance criteria:
  - `[x]` Logs include fields for command, peer node, resource, capability,
    stream, cursor, and stable error code where applicable.
  - `[x]` Sensitive bearer tokens and private material are not logged.

- `[x]` Expand health status.
  Acceptance criteria:
  - `[x]` `geth status --json` reports daemon uptime.
  - `[x]` Store schema version is visible.
  - `[x]` Native backend health is visible.
  - `[x]` Iroh relay/local-discovery state is visible without exposing
    unrelated config secrets.

- `[x]` Add `geth doctor`.
  Acceptance criteria:
  - `[x]` Doctor detects daemon not running.
  - `[x]` Doctor detects stale local socket.
  - `[x]` Doctor detects bad config.
  - `[x]` Doctor detects missing `ssh-keygen`.
  - `[x]` Doctor explains missing grants and endpoint mismatches when enough
    local metadata exists.
  - `[x]` Doctor has JSON output for scripts.

## Phase 9: Packaging, Upgrade, And Release Discipline

Goal: make first deployment the start of a controlled compatibility story.

- `[x]` Harden release artifacts.
  Acceptance criteria:
  - `[x]` Linux, macOS, and Windows archives are built from tags.
  - `[x]` Release archives are smoke-tested directly, not only through
    `cargo run`.
  - `[x]` Archives include relevant docs and license files.

- `[x]` Add upgrade tests.
  Acceptance criteria:
  - `[x]` Fixture homes from previous tagged pre-releases migrate forward.
    There are no tagged pre-releases yet, so the current test uses a synthetic
    pre-release home and the release policy requires retaining real tagged
    fixture homes once tags exist.
  - `[x]` Upgrade tests include store schema, config, keychain/auth logs, CAS
    metadata, and peer cards.
  - `[x]` Rollback expectations are documented.

- `[x]` Publish release and support policy.
  Acceptance criteria:
  - `[x]` Supported platforms are listed.
  - `[x]` Breaking-change policy is documented.
  - `[x]` Security update expectations are documented.
  - `[x]` User-service installation remains user-level only.

## Phase 10: Pre-Deployment Dogfood Gate

Goal: prove the system works as an actual base layer before broader use.

- `[ ]` Complete two-machine dogfood checklist.
  Acceptance criteria:
  - `[ ]` Fresh install works on at least two real machines.
  - `[ ]` Owner init, enrollment, peer exchange, grant, sync status, CAS, KV,
    document, pipe, SSH proxy, and service restart work end to end.
  - `[ ]` Denied remote operations do not mutate serving node state.
  - `[ ]` Backup and restore work.
  - `[ ]` Upgrade from one tagged pre-release to the next works.
  - `[ ]` `geth doctor` gives actionable output for intentionally broken
    setups.
  - `[ ]` Docs match actual commands.

## Working Order

1. `[x]` Finish Phase 0.
2. `[~]` Refactor `geth-node` into daemon subsystems.
3. `[x]` Add stable contract and golden JSON tests.
4. `[x]` Harden store migrations and backup.
5. `[x]` Complete security-boundary test coverage.
6. `[x]` Replace prototype private CAS cryptography.
7. `[ ]` Add fault-injection sync tests.
8. `[ ]` Improve automation commands and JSON errors.
9. `[ ]` Add operational health, doctor, and release gates.