Faster database, lower cost
Archiving and optimising a legacy PostgreSQL database cut storage by 80% and average query latency by 74%. I set the approach and directed a senior engineer, who investigated and implemented it.
Next case studyMessaging at scale
A production Pub/Sub service that processed 9.2 million HTTP requests, including message responses and status updates, in a 30-day period.
- Role
- Designed and built
- Client
- A business-messaging platform
- Period
- Built in 2023
- Scope
- Production messaging service
01
The problem
A business-messaging platform needed a dependable service to send outbound messages and to handle a high volume of HTTP requests, including message responses and status updates.
Some messages fail on the first attempt. The service had to retry them at a sensible pace, keep hold of any message it still could not process, and make sure the team found out.
02
What I did
I designed and built the service on Google Cloud Pub/Sub and integrated it into the client’s campaign platform.
It read messages through a pull subscription, so it could control its own throughput. A message that failed was retried up to 10 times with exponential backoff, and one that still failed went to a dead-letter queue, which raised a developer alert. Fig. 1 follows one message along that path.
Diagram of the failure path. HTTP requests, including message responses and status updates, enter a Pub/Sub topic, and the subscriber pulls them through a pull subscription at a pace it controls. Messages that succeed are processed. One message keeps failing and is retried up to 10 times, with a longer wait before each retry (drawn not to scale). When the tenth retry also fails, the message, drawn hollow, follows a dashed dead-letter path into the dead-letter queue, and the queue raises a developer alert.
- message
- dead-lettered message
- dead-letter path
- backoff so far, not to scale
Trace: one message
Pull
The subscriber pulls the message from its Pub/Sub subscription, at a pace it controls, and tries to process it.
Retry with backoff
When processing fails, the message is retried, up to 10 times. Each wait is longer than the last: exponential backoff.
Dead-letter
If the tenth retry also fails, the message follows the dead-letter path into the dead-letter queue.
Alert
The dead-letter queue raises a developer alert, so the team knows a message needs attention.
03
Results
Traffic measured in a 30-day period
HTTP requests30 days
9.2 million
HTTP requests processed in a 30-day period, including message responses and status updates.
RetriesPer message
up to 10 retries
with exponential backoff. A message that still fails goes to the dead-letter queue, which raises a developer alert.
04
Stack
- Python
- asyncio
- Google Cloud Pub/Sub
- Redis
- GKE
- Pytest
Contact
Get in touch
Tell me about your project, your team or the problem you're working on.
Or find me on LinkedIn