Executive Summary

When a rapidly growing logistics scale-up expanded its active commercial fleet to over 75,000 vehicles, its core infrastructure hit a severe architectural ceiling. High-frequency telematics pings (GPS coordinates, engine diagnostics, temperature metrics) flooded the legacy PostgreSQL relational database, leading to disk IOPS saturation, Write-Ahead Log (WAL) contention, and connection pool starvation.

During peak dispatch hours, real-time vehicle tracking dashboards lagged by up to 15 minutes, causing missed carrier Service Level Agreements (SLAs) and daily operational outages.

Betta was brought in to audit, redesign, and execute a zero-downtime migration of the ingestion architecture. By replacing the monolithic REST/PostgreSQL setup with a event-driven streaming pipeline—powered by a high-concurrency Go gateway, Redpanda/Kafka message streams, stateful Apache Flink processing, and ClickHouse columnar storage—Betta eliminated the database bottleneck while cutting cloud infrastructure costs by 58.1%.

Key Impact Metrics

• P99 Ingestion Latency: Reduced from 12,400 ms to 82 ms (99.3% reduction).

• Ingestion Capacity: Scaled from 4,200 events/sec to 125,000+ events/sec (29.7x capacity increase).

• Tracking Data Freshness: Improved from 15+ minutes stale to < 450 milliseconds real-time updates.

• Cloud Infrastructure Spend: Reduced from $42,500/month to $17,800/month (58.1% cost savings).

• Platform Availability: Elevated from 98.2% to 99.999% during peak dispatch windows.

The Client Challenge: The Relational Bottleneck

As the client scaled from hundreds to tens of thousands of active connected vehicles, telemetry emission volume grew exponentially. Each vehicle transmitted state payloads every two seconds, generating over 37,500 continuous write requests per second during peak morning dispatch.

Underlying Technical Root Causes

1. B-Tree Index Fragmentation: High-cardinality telemetry data arriving out-of-order forced random I/O writes across PostgreSQL B-Tree indices. Index page splitting exhausted the shared_buffers cache, driving disk I/O utilization to 100%.

2. Write-Ahead Log (WAL) Lock Contention: Continuous unbuffered INSERT operations caused extreme WAL flush contention on disk, creating exponential write latencies across application workers.

3. Connection Pool Starvation: Slow database writes held HTTP connections open in the application tier, exhausting thread pools and causing cascading HTTP 504 Gateway Timeouts across public tracking APIs.

4. Out-of-Order Cellular Dead-Zone Bursts: When delivery trucks re-emerged from rural dead zones, they uploaded buffered pings in burst batches. These sudden spikes choked the ingestion tier, triggering severe cascading outages.

Betta’s Engineering Solution: Distributed Event-Driven Architecture

Betta’s team of senior SREs and cloud architects designed and deployed a decoupled, stream-first telemetry processing engine built for high-concurrency, horizontally scalable time-series ingestion.

Component Breakdown & Technical Strategy

1. Zero-Allocation Go Ingestion Gateway

Betta engineered a lightweight, non-blocking ingestion service written in Go using valyala/fasthttp and high-speed sonic JSON parsing. The gateway validates incoming payloads and pushes them directly into asynchronous producer memory buffers, decoupling network transport from disk storage durability.

2. Partitioned Stream Buffer (Redpanda/Kafka)

The incoming telemetry stream was routed into a 64-partition Redpanda topic explicitly hashed by vehicle_vin.

• Keyed Partitioning: Hashing on vehicle_vin ensured that every metric emitted by a given vehicle landed in the exact same stream partition in chronological order.

• Parallel Consumption: Enabled up to 64 independent, non-blocking consumer workers in downstream processing engines.

3. Stateful Stream Processing with Watermarking (Apache Flink)

To handle out-of-order cellular reconnection dumps without degrading live state calculation, Betta implemented an Apache Flink stream processing topology utilizing Bounded-Out-Of-Orderness Watermarks:

• Telemetry events within a 15-second delay window are processed immediately for live state tracking and written to a hot Redis state cache and ClickHouse analytics engine.

• Telemetry events older than 15 seconds (late backfills) are automatically separated via Flink Side Outputs and routed straight to an Amazon S3 Parquet Data Lake, completely bypassing and protecting the real-time operational database tier.

4. High-Performance Columnar Storage Engine (ClickHouse)

Relational storage was replaced with a ClickHouse cluster running the ReplacingMergeTree storage engine:

• Storage Efficiency: LZ4 data compression reduced physical disk utilization by 85% compared to raw PostgreSQL rows.

• Batch Ingestion: Engine outputs are aggregated into 1-second micro-batches before committing, allowing ClickHouse to ingest over 100,000 rows per second without incurring write lock contention.

Execution Strategy: Zero-Downtime Migration

Modifying a core pipeline operating 24/7 required a risk-averse, zero-downtime migration strategy executed across four phases:

Infrastructure as Code Automation: Betta codified the complete Kubernetes, Redpanda, Flink, and ClickHouse cluster setups into version-controlled Terraform modules and Helm values, allowing reproducible environment provisioning.

• eBPF Shadow Traffic Mirroring: Using Cilium eBPF network packet cloning at the Kubernetes Ingress layer, 100% of live telemetry traffic was duplicated and sent to the new streaming pipeline in parallel. SREs verified system performance, schema validation, and data integrity under true production loads without impacting live customer operations.

• Observability & Alerting Guardrails: OpenTelemetry, Prometheus, and Grafana dashboards were integrated to monitor core Service Level Indicators (SLIs). PagerDuty alerts were set for consumer lag exceeding 5,000 messages or P99 pipeline latencies exceeding 500ms.

• Seamless Read Cutover: Tracking portal APIs were updated to read real-time state from Redis and historical time-series analytics from ClickHouse. Once read latency parity was established, legacy PostgreSQL ingestion endpoints were safely decommissioned.

Results & Business Impact

Following the deployment, Betta ran comprehensive load testing to evaluate platform performance under simulated conditions representing 3x peak client scale (225,000 active connected assets).

Technical Performance Comparison

Strategic Business Outcomes

• Carrier SLA Compliance Restored: Sub-second positioning visibility enabled the client to exceed carrier delivery accuracy SLAs, preventing costly contractual penalty fees.

• Significant OPEX Reduction: By shifting away from over-provisioned, write-heavy relational database nodes, monthly cloud compute costs were reduced by $296,400 annually.

• Engineering Productivity Gain: Client SRE teams transitioned out of daily incident response loops, redirecting engineering cycles toward new product features like predictive maintenance alerts and AI route optimization.

Summary & Next Steps

Scaling high-throughput time-series workloads requires moving away from traditional database patterns and adopting stream-first primitives designed for high write volumes. By partnering with Betta, the client eliminated critical system bottlenecks, stabilized core platform services, and established an infrastructure foundation capable of supporting fleet expansion for years to come.

Partner with Betta

Betta provides specialized B2B engineering services in DevOps, SRE, Cloud Infrastructure, and Security. Whether you are addressing scaling bottlenecks, migrating legacy systems, or modernizing platform reliability, our embedded engineering teams deliver concrete results.

Ready to optimize your high-throughput cloud infrastructure?

Contact Betta's Architecture Team to schedule an infrastructure assessment.