Software & AI
99.99% availability in the migration from Event Hubs to Confluent Kafka
Migration to a streaming architecture on Confluent Kafka, with no impact on critical operations and more than 5 TB replicated per day in real time.
Case metrics
99.99%
Operational availability
-70%
Operational risks in streaming
5.18 TB/day
Replicated to the data lake in real time
The project story
Challenge
Evolve the data architecture to an event-driven model able to support large volumes with low latency and high availability, reducing operational risks in streaming.
The scenario required a safe transition from Event Hubs to Kafka, with no impact on business-critical operations.
Approach
We designed a highly available architecture on Confluent Kafka, defining topics, partitioning, retention policies, and replay strategies to ensure resilience and performance.
We also built automation and observability layers to increase operational control and anticipate incidents.
Result
We reached 99.99% availability, reduced operational risks in streaming by 70%, and started processing data with millisecond latency.
Today, more than 5 TB of data is replicated per day in real time, supporting fast decisions and a data-driven operation.
Project highlights
- Streaming architecture design
- Migration from Event Hubs to Kafka with no impact on critical operations
- Topics, partitions, retention, and replay policies
- Automation and observability
- ≈63 MB/s replicated, with latency under 100 ms
- Horizontal scalability
Technologies
- Azure
- Confluent (Kafka + Flink)
- Databricks
- Power BI
Other cases
Can your streaming grow without putting operations at risk?
Talk to Datatchê to migrate to Confluent Kafka with high availability and no impact on critical operations.
