Case study · Resilience lab
Payment Platform Disaster Lab
Production / resilience engineering around a distributed payment platform (XOF, dual PSP, Kafka, ledger) — prove financial correctness when the system is not 100% available.
Not a production PSP. A lab to show a payment system can stay financially correct under partial failures — no lost payment, none processed twice.
Signals
0
Double capture on timeout
Uncertainty is first-class — timeout never triggers blind failover.
D1–D10
Disaster scenarios
Documented failure hypotheses from PSP down to deploy canaries.
P0–P2
Shipped today
Compose, payment SM, dual PSP + chaos, gateway JWT, SRE control center.
Golden routing rule
PSP A down (503 / CB OPEN) → failover PSP B. PSP A timeout / UNKNOWN → IN_RECONCILIATION — never PSP B. Confusing D1 and D2 is a design flaw, not an ops detail.
Problem
Happy-path payment APIs are easy. Hard cases start when the PSP times out (captured or not?), the client retries after an HTTP timeout, a webhook arrives three times, a Kafka consumer dies while events pile up, or a deploy breaks 20% of traffic. Perceived high availability and financial correctness diverge — a blind failover to a second PSP can create a double debit; a non-idempotent retry can too.
Solution
One directing rule: no payment lost, none processed twice — even when the system is not fully available. Invariant chain: Identity → Authorization → Idempotency → State Machine → PSP (uncertainty first-class) → Ledger → Event → Reconciliation → Observability. Timeout PSP ≠ FAILED; uncertainty is IN_RECONCILIATION, not an excuse to call PSP B.
Constraints
- Timeout PSP ≠ FAILED — uncertainty is a business state (IN_RECONCILIATION).
- Failover A→B only on certain failure (503 / CB OPEN / business refuse).
- Never emit Kafka without committed payment state (transactional outbox).
- Database-per-service — isolate append-only ledger schema; no cross-DB joins.
- PSP simulators with hot-injectable chaos — not real Stripe network/SCA behavior.
Architecture flow
- 01Gateway
- 02Payment SM
- 03Dual PSP
- 04Outbox / Kafka
- 05Reconciliation
Architecture
Multi-service Spring Boot (failure domains you can kill in isolation). API Gateway: JWT + /v1/payments + X-Correlation-Id. payment-service: create/GET, state machine, idempotency, transactional outbox. Dual psp-simulator A/B with Resilience4j — safe failover vs IN_RECONCILIATION. Compose: Postgres, Kafka, Schema Registry, Keycloak, Kafka UI. Frontend is an SRE control center (Vite + React), not a merchant checkout — observe, inject, recover.
Decisions
- 01
Multi-services, not a monolith
Kill a consumer or a PSP in isolation — real failure domains for the lab.
- 02
Database-per-service
Isolate the append-only ledger schema; no cross-DB joins in the demo path.
- 03
Transactional outbox
Never publish a Kafka event without committed payment state.
- 04
Timeout PSP = UNKNOWN
No blind failover; reconcile before truth — prevents double capture.
- 05
Failover only on certain failure
503 / CB OPEN / business refuse — not timeout or ambiguous 5xx.
- 06
JSON Schema + Schema Registry
Readable contracts in demos; CI-compatible evolution without Avro ceremony yet.
- 07
Keycloak + API Gateway
Identity / JWT off the business path so chaos stays about money movement.
- 08
PSP simulators + FE control center
Hot chaos via PUT /admin/chaos; frontend observes and recovers — not a checkout product.
Trade-offs
Correctness before “API always 200” means more IN_RECONCILIATION and mandatory recon. Split services + outbox buy isolation and Kafka replay at the cost of heavier local ops. One Postgres with logical DBs keeps the lab simple — not multi-instance HA. JSON Schema favors demo readability over Avro’s battle-tested evolution. Simulators give deterministic chaos, not real network/SCA/settlement. Canary/K8s land in later phases (P6); FE can run in simulation and risk diverging from real BE behavior until wired.
Stack
What I’d do next
- 01
P3 — Ledger + Kafka consumers: double-entry append-only, idempotent consumer, DLT, Schema Registry end-to-end.
- 02
P4 — Notification + reconciliation: merchant webhooks; MATCHED / MISSING / DUPLICATE / AMOUNT_MISMATCH; poll PSP status out of IN_RECONCILIATION.
- 03
P5 — Observability: OTel traces gateway → payment → PSP → Kafka → ledger; KPI / SLO / CB / outbox lag dashboards.
- 04
P6 — Kubernetes + canary: Helm, probes, 5–20% canary, auto-rollback on error rate.
- 05
P7 — Documented chaos: fill D1–D10 (hypothesis → observed → recovery); wire FE to real backend.
Scoping a payment or resilience problem?
Share the failure modes you care about — I will map invariants, risks, and a realistic first slice.