A Kafka consumer that keeps up with the surge
A service consumes a Kafka topic and feeds Elasticsearch. Under a surge, its lag only grew.
Claudio Mascaro
PortuguêsMessages reach the topic faster than the service consumes them.
The gap is the lag.
Before: batches of up to 2,000.
But each message with its own query, 100 at a time.
The lag grows, and the data reaches search late.
Each message becomes its own sendMessage to the output topic.
With one commit per batch.
The HPA adds instances, one per partition, and the lag keeps growing.
After: I replaced the queries with one for the whole batch.
The same batches of up to 2,000.
Output goes out in chunks of 500, one at a time.
Each waits for the broker’s acknowledgement, so the Logstash topic gets a steady cadence, not a burst.
With the same CPU-based HPA, the lag rises more slowly.
In the surge, each instance has slack.
And the ones the HPA adds drain the queue.
Consumption and the data’s arrival in search rose 4.5× to 5×.
Got a consumer that can’t keep up with its surge?
A software supply chain.
Teams on npm, Go, PyPI, Maven and NuGet, each pulling from a source of its own.
An advisory flags a vulnerable npm version.
Blocking it means going source by source, while builds pull it through the ones still open.
I put a hub in front of the registries: Nexus.
One block there covers the teams that moved, and the old sources still bring the version to the rest.
After: the old sources are decommissioned.
One place is left to block, and one block at the hub covers every team.
The hub becomes one service of an internal development platform (IDP).
Only the pipeline publishes: an attested, signed build, a scan in OWASP Dependency-Track.
Consumers check the signature and the attestation.
A tampered copy of the internal package arrives.
Its digest does not match the attestation, and the hub holds it.
Had it got through, admission would refuse it.
Want one place to block what should not get in?
Claudio Mascaro
I build distributed systems, AI-native or not.And I restore the ones that are slow, expensive or failing, through diagnosis and surgery.Everything ships with the Reliability Pipeline: a person approves, the machine publishes.
Software engineer and SRE with broad experience in payment infrastructure. I apply the fundamentals of AI engineering at both scales: inside a large company, where resources are not the bottleneck, and on my products, where they are.
Senior SRE today, open to Staff SRE roles. Career
Where the pipeline already runs
Two problems I solved, and two systems of mine.
A software supply chain with one place to block
Teams on npm, Go, PyPI, Maven, and NuGet, each pulling packages from a source of its own.
Aurora Concierge
AI-powered service for a homecare operation, on a CRM of my own: inbox, takeover, human release. The final word belongs to a person, in a place built for it.
Acolhera
The same platform, white-labeled for health operations. The design takes a second brand without becoming a second system: same pipeline, different owner. It is where the per-project-plus-maintenance model already runs.