Between 07:17 and 14:57 UTC on 24 September 2026, outbound SMS were not sent to carriers. The API kept accepting messages and returning successful responses, but the messages stayed in our sending queue. They were delayed, not lost. Time-sensitive messages, such as one-time passwords, may have arrived after they expired.
Cause: A change to how our SMS service manages outgoing connections was deployed on Sep 23. It contained a concurrency bug: under high load, the same connection could be given to two send tasks at once. This corrupted the state of the connection pool, and the sending workers stopped processing. Because the workers did not crash, our health checks still reported them as healthy.
Resolution: We rolled back the change at 14:57 UTC. Sending resumed at once, and the full backlog was cleared by 14:59 UTC. About 3,000 messages that were in progress when the workers stopped were resent the same day, with no duplicate charges.
What we're changing: It took about 7 hours to detect this incident, and fixing that is our first priority. We are adding more alerting for this specific scenario. We are making concurrency and load testing a release requirement for changes to the sending path. The original fix will be reworked and tested at production-level concurrency before it ships again.