Capital Markets · Data Engineering
Real-time trading analytics dashboard
500k+ events/sec market ingestion with sub-50ms end-to-end visibility for portfolio managers.
- Peak ingest
- 620k evt/s
- End-to-end latency
- <50ms
- Prior refresh
- 5 minutes
- Slippage impact
- ~28% ↓
Problem
A quant fund’s Tableau dashboard refreshed every five minutes. Traders were acting on stale P&L and exposure across 12 asset classes — expensive slippage.
They needed exchange-feed ingest at hundreds of thousands of events/sec, real-time derived metrics, and browser updates within 50ms, with an audit trail regulators could trust.
Approach
Kafka cluster partitioned by asset class; Python consumers with Numba-accelerated metric computation writing to ClickHouse for fast OLAP.
Node.js WebSocket fleet with delta diffing and Redis pub/sub for failover — only changed values hit the wire.
React + D3 OffscreenCanvas kept the UI responsive under high-frequency updates; Kafka’s log underpinned the regulatory trail.
Outcome
Latency moved from 5-minute polling to consistent sub-50ms push. The system sustained 620k events/sec peaks. First-quarter slippage costs fell an estimated 28%. The audit design passed a tier-1 FCA-oriented review with no findings.
Stack
- Apache Kafka
- ClickHouse
- Python / Numba
- Node.js WebSockets
- Redis
- React
- D3.js
- Kubernetes