Chaos Room Simulator
Designing and shipping a distributed systems incident cockpit with simulated telemetry, live and cached Gemini AI analysis, and strict application-level spending guardrails.
The operational problem
When complex cloud systems fail, engineering and platform teams face high-stress, time-compressed situations. Responders must sort through noisy telemetry, form hypotheses about cascading dependencies, test mitigations, and communicate clearly under pressure.
Traditional game days and chaos engineering tests on real infrastructure are valuable, but they carry operational overhead and blast-radius risks. Organizations benefit from realistic, zero-risk simulation environments where engineers and platform leaders can practice diagnostic workflows, evaluate AI-assisted triage, and hone incident communication without risking customer-facing downtime.
The experience
Chaos Room Simulator provides an interactive cockpit that models distributed systems incidents against simulated, synthetic telemetry. The app includes three canonical incident scenarios:
- Incident 1: “Reservations are timing out” — A deployment bug in the reservation service strands database connections, saturating connection pools and creating API gateway timeouts.
- Incident 2: “Reservations succeed, confirmations stall” — Latency in an external notification vendor triggers unbounded worker retry loops, producing severe retry amplification and confirmation queue backlog.
- Incident 3: “A release leaves confirmations behind” — An unannounced event payload schema change produces consumer parse failures, requiring incompatible messages to be quarantined and replayed from the confirmation queue.
Engineers progress through four canonical checkpoints: 00:00 Orient (system topology and healthy baseline), 03:00 Alert (first customer impact and initial triage), 06:00 Evidence & Intervene (decision gate requiring intervention selection), and 15:00 Outcome & Debrief (post-mitigation stabilization and retrospective).
At each checkpoint, responders inspect simulated metrics (p95/p99 latency, error rates, queue age, and connection pool saturation), evaluate potential operational interventions, and choose between deterministic authored guidance, shared cached AI analysis, or newly generated live Gemini AI analysis.
Mike’s contribution
Mike established the product vision, scenario domain, UX architecture, and governance boundaries from concept through live launch.
Drawing on his background in platform delivery and product operations, he framed key trade-offs early:
- Bounded simulation over open-ended chaos: Structured scenarios around discrete, reproducible timeline checkpoints so engineering teams could compare hypotheses against clear ground truth.
- Internal model-generation allowances: Implemented an application-level $4.50 monthly generation ceiling with per-visitor and daily rate limits, keeping experimental AI costs predictable while decoupling model spend from cloud infrastructure billing.
- Dual-guard operational pause controls: Separated live model generation from cached analysis so public visitors can explore existing assessments without incurring marginal model costs when generation is paused.
- Rigorous verification protocols: Designed multi-phase test automation covering local deterministic emulators, headless browser verification, and production ledger accounting audits.
Mike specified the state reconstruction and verification models, partnered with Antigravity as an AI-assisted engineering co-developer to implement the full-stack architecture, and directed end-to-end testing across all release phases.
Architecture and operational guardrails
- Internal model-generation allowance ($4.50/month ceiling): To prevent unbounded model spending, the application tracks Gemini usage against an internal $4.50 monthly generation allowance. Using Cloud Firestore transactions, every generation attempt reserves worst-case token costs ($0.0074) before model dispatch. Quotas enforce strict limits: 2 requests/minute per visitor, 30 generations/day UTC, and 600 generations/month UTC. Hosting and cloud compute are billed separately under Google Cloud standard tiers.
-
Dual-guard runtime controls:
The backend incorporates two independent Firestore flags:
liveGenerationDisabledpauses new Gemini API calls while preserving access to previously cached analyses, whileaiDisabledserves as an operational kill switch that directs all analysis requests to authored baseline guidance without requiring code redeployment. - Deterministic baseline fallback: If cloud APIs, network connectivity, or generation quotas are unavailable, the client application serves deterministic, expert-authored baseline analyses without blocking the user, providing reliable continuity during disruptions.
- Shared 30-day operational cache: Canonical SHA-256 state hashing maps identical incident checkpoints to shared Firestore cache entries, allowing repeat inquiries across visitors to be served from cache at $0.00 marginal model spend without duplicate LLM calls.
- Automated schema & citation grounding checks: Incoming model responses are validated against strict TypeScript schemas and cross-checked against canonical scenario evidence. This verifies that citations reference actual simulated facts rather than hallucinated evidence IDs, without treating the model's diagnostic conclusions as infallible truth.
- Firebase App Check & reCAPTCHA Enterprise defense: Callable Cloud Functions verify incoming web App Check tokens before admitting requests, providing defense-in-depth attestation to mitigate unauthorized automated scripts and abusive scrapers.
What is live
The simulator is live in production at chaos.mikerestrepo.com, configured with Porkbun DNS routing, Google Cloud Functions 2nd Gen backend services, and automated Google-managed SSL certification.
Visitors can navigate active scenarios, test operational interventions, and request Gemini AI analysis validated for structural schema and citation grounding within daily quotas. Production smoke suites automatically audit ledger invariants and verify end-to-end telemetry.
Tools & Tech Stack
- React 19 & TypeScript & Vite for the responsive cockpit interface
- Google Cloud Functions 2nd Gen / Cloud Run (Node.js 22) for serverless backend execution
- Gemini API (
gemini-3.5-flash-lite) for structured incident root-cause analysis - Cloud Firestore for atomic accounting transactions and 30-day shared caching
- Firebase App Check & reCAPTCHA Enterprise for bot defense and endpoint attestation
- Firebase Hosting & Porkbun DNS for global edge delivery and custom domain routing
- Antigravity for AI-assisted development, headless Chrome automation, and end-to-end verification