operations
Failure, security, and observability
Failures are typed state transitions with visible watermarks, bounded consequences, and an owned recovery command.
- Question
- How does an operator prove the system is degraded rather than silently wrong?
- Binding decision
- Secrets never reach the browser; tenant, entitlement, query budget, generation, and provenance cross every boundary.
- Owner input
- Alert thresholds and error budgets remain proposals until production profiles are measured.
- 01Providerquota / outagepause bounded work
- 02QuestDBcommit laghold durable watermark
- 03Redisloss / trimquery authority or resync
- 04Ownerlease expiryfence and reacquire
- 05Viewerqueue pressureshed, resync, or close
Failure is observable state
The system emits traces, structured logs, and metrics for provider cost, ingest and commit watermarks, coverage gaps, query plans, cache staleness, owner epochs, broker lag, pending and reclaimed messages, viewer queue depth, reconnects, and resync outcomes.
- Cardinality budgets prevent instrument IDs from exploding every metric.
- Watermark lag locates the failing stage in one trace.
- Owner death, Redis loss, QuestDB lag, and subscriber stall have separate recovery semantics.
EvidencePrompt: observability and backpressure analysis requirements.
The gateway owns trust
The browser receives only an authorized query and stream contract. The gateway checks tenant, entitlement, query range, rate, and concurrency before work begins. Credentials for providers, stores, brokers, and workflows remain server-side and logs redact them.
- Per-tenant subjects and authorization prevent cross-tenant fanout.
- Malformed and oversized requests consume bounded parser and query budgets.
- Retention and data-rights decisions gate persistence and deletion.
Evidencedocs/SECURITY.md and target architecture contract.
mise x -- task fanout:chaosAcceptance target: Partitions and slow consumers produce bounded, typed recovery.