What it does
Clients send messages through a portal or an API. Each message is routed to a carrier, tracked until its delivery report comes back, and billed to the right tenant.
How it’s built
An API service puts each job on a queue. An SMS gateway handles the carrier side and passes results back over RabbitMQ. A separate engine reads those events and keeps one record per message, matching delivery reports against it.
Sending and record-keeping are separate on purpose, so a problem in the engine doesn’t stop messages from going out.
The hard part
One message with no owning tenant made the routing cache try to load every phone number into memory, which is tens of millions of rows. The engine ran out of memory and crashed. The message then went back on the queue, so every restart hit it again.
Messages kept sending throughout, and the unprocessed records waited safely in the queue.
What I did
- Added a hard limit so the engine can never load the whole number table into memory.
- Built a monitor that reports to Slack and treats zero traffic as critical even when every service looks healthy, because a quiet platform can be a broken one.
- Added live job progress over Server-Sent Events.
Result
The platform runs live at 5 to 10 million messages a day across multiple tenants.