Vitelity Inbound SMS Outage March 27, 2022 0150 to 1720 UTC
Vitelity hasn't posted anything about this, so I will for those that care. Short Version: Inbound SMS to all Vitelity DIDs failed between 0150 UTC and 1720 UTC on March 27, 2022 Long Version: At 0150 UTC our Inbound SMS Monitoring detected that one of our DIDs did not receive a sent SMS. This continued until 0230 UTC where our system automatically cut a ticket (via email) to Vitelity that we were seeing 30+ minutes of zero inbound SMS messages to a random sample of our DIDs. At 0305 UTC I opened an emergency ticket about the issue. At 0341 UTC The ticket was responded to. At 0503 UTC I was able to talk to someone at Vitelity who confirmed that they also had sent SMS messages to their inbound DIDs and the messages were not received, and they were escalating. During that, I also provided DLRs for all the sent messages, showing that they were received by Onvoy, Iristel, Neutral Tandem, etc. Vitelity confirmed that they were esclating and "sounding the alarm." At 0810 UTC Vitelity updated the ticket and said they were still escalating. No change, 100% of SMS messages sent to Vitelity DIDs failed to reach Vitelity or us. At 1634 UTC I re-re-re-escalated the issue to the Inteliquent NOC and my Vitelity contact. I was informed that a tech had found that "a certificate had expired" and was causing all inbound SMS delivery to fail. Incidentally, this ALSO prevented an emails to Vitelity Support to ALSO not be delivered to the portal, and thus not open tickets. At 1720 UTC a test finally succeeded and we received our first SMS in over 15h 30m. Several tests after that across another sample of Vitelity DIDs were successful. At no point was status.vitelity.com actually updated with any sort of incident, and despite it being acknowledged to me, has still not posted any sign of the outage. Why I'm Posting This is the 48th SMS outage since August 2015 that Vitelity has experienced, and in most cases, I have been the first person to alert them to a problem. They do not monitor inbound SMS, they just wait for a customer complaint, and even then, I've been told "nobody else has opened a ticket," as if multiple customers need to complain before they take an outage seriously. In this case, it took 2 hours from opening an emergency ticket for Vitelity to actually test inbound SMS and confirm an issue, followed by another 12 hours to actually find the root cause and fix it. Beckman --------------------------------------------------------------------------- Peter Beckman Internet Guy beckman at angryox.com https://www.angryox.com/ ---------------------------------------------------------------------------
Sounds like you need to find another provider...? On Sun, Mar 27, 2022, 7:10 PM Peter Beckman <beckman at angryox.com> wrote:
Vitelity hasn't posted anything about this, so I will for those that care.
Short Version: Inbound SMS to all Vitelity DIDs failed between 0150 UTC and 1720 UTC on March 27, 2022
Long Version:
At 0150 UTC our Inbound SMS Monitoring detected that one of our DIDs did not receive a sent SMS.
This continued until 0230 UTC where our system automatically cut a ticket (via email) to Vitelity that we were seeing 30+ minutes of zero inbound SMS messages to a random sample of our DIDs.
At 0305 UTC I opened an emergency ticket about the issue.
At 0341 UTC The ticket was responded to.
At 0503 UTC I was able to talk to someone at Vitelity who confirmed that they also had sent SMS messages to their inbound DIDs and the messages were not received, and they were escalating. During that, I also provided DLRs for all the sent messages, showing that they were received by Onvoy, Iristel, Neutral Tandem, etc.
Vitelity confirmed that they were esclating and "sounding the alarm."
At 0810 UTC Vitelity updated the ticket and said they were still escalating. No change, 100% of SMS messages sent to Vitelity DIDs failed to reach Vitelity or us.
At 1634 UTC I re-re-re-escalated the issue to the Inteliquent NOC and my Vitelity contact. I was informed that a tech had found that "a certificate had expired" and was causing all inbound SMS delivery to fail.
Incidentally, this ALSO prevented an emails to Vitelity Support to ALSO not be delivered to the portal, and thus not open tickets.
At 1720 UTC a test finally succeeded and we received our first SMS in over 15h 30m. Several tests after that across another sample of Vitelity DIDs were successful.
At no point was status.vitelity.com actually updated with any sort of incident, and despite it being acknowledged to me, has still not posted any sign of the outage.
Why I'm Posting
This is the 48th SMS outage since August 2015 that Vitelity has experienced, and in most cases, I have been the first person to alert them to a problem. They do not monitor inbound SMS, they just wait for a customer complaint, and even then, I've been told "nobody else has opened a ticket," as if multiple customers need to complain before they take an outage seriously.
In this case, it took 2 hours from opening an emergency ticket for Vitelity to actually test inbound SMS and confirm an issue, followed by another 12 hours to actually find the root cause and fix it.
Beckman --------------------------------------------------------------------------- Peter Beckman Internet Guy beckman at angryox.com https://www.angryox.com/ --------------------------------------------------------------------------- _______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops
On Sun, Mar 27, 2022 at 07:08:01PM -0400, Peter Beckman wrote:
Why I'm Posting
This is the 48th SMS outage since August 2015 that Vitelity has experienced, and in most cases, I have been the first person to alert them to a problem. They do not monitor inbound SMS, they just wait for a customer complaint, and even then, I've been told "nobody else has opened a ticket," as if multiple customers need to complain before they take an outage seriously.
I wasn't aware that people used Vitelity SMS for anything serious. My few months of experience with them suggested that they didn't even support all ASCII characters (for example, '<' and '>' were simply replaced with spaces, at least when using XMPP), and their service silently failed to deliver SMS on a regular basis. I assumed they didn't see SMS as a production feature for those reasons (and a few others) so stopped using it fairly soon after. Has their service improved recently on other metrics? For example, do they support UTF-8 now? Denver https://jmp.chat/
I integrated with Vitelity for SMS in August 2013. My first evidence that UTF-8 was supported was January 13, 2017. There are weird issues with Vitelity, such as odd length limits to SMS messages, but we worked around them. SMS is reliable enough, though the 48 outages since August 2015 should give you an idea on how high a priority the service is to them compared to voice. When did you last test SMS with Vitelity? I have several years of SMS metric data on Vitelity SMS: +------+----------+---------+----------+ | yr | success% | failed% | delayed% | +------+----------+---------+----------+ | 2015 | 98.42% | 1.58% | 3.17% | | 2016 | 96.31% | 3.69% | 5.30% | | 2017 | 96.68% | 3.32% | 2.79% | | 2018 | 92.22% | 7.78% | 6.24% | | 2019 | 94.46% | 5.54% | 1.50% | | 2020 | 98.41% | 1.59% | 0.71% | | 2021 | 97.75% | 2.25% | 2.51% | | 2022 | 98.61% | 1.39% | 0.24% | +------+----------+---------+----------+ 2018-2019 sucked! This is based on a baseline of sending an SMS once every 10 minutes to one of our active DIDs that has least recently received an Inbound SMS message. Delayed is any SMS that took more than 10 seconds to receive. The average receipt time for 2022 is 2.28 seconds. Bandwidth beats that at 1.18 seconds average for 2022. Both are acceptable. ** NOTE these numbers may be attributable to outages by the sending carrier. They are not easily excluded, and so the numbers may be higher. This also is only Inbound, not Outbound. Beckman On Mon, 28 Mar 2022, Denver Gingerich wrote:
On Sun, Mar 27, 2022 at 07:08:01PM -0400, Peter Beckman wrote:
Why I'm Posting
This is the 48th SMS outage since August 2015 that Vitelity has experienced, and in most cases, I have been the first person to alert them to a problem. They do not monitor inbound SMS, they just wait for a customer complaint, and even then, I've been told "nobody else has opened a ticket," as if multiple customers need to complain before they take an outage seriously.
I wasn't aware that people used Vitelity SMS for anything serious. My few months of experience with them suggested that they didn't even support all ASCII characters (for example, '<' and '>' were simply replaced with spaces, at least when using XMPP), and their service silently failed to deliver SMS on a regular basis. I assumed they didn't see SMS as a production feature for those reasons (and a few others) so stopped using it fairly soon after.
Has their service improved recently on other metrics? For example, do they support UTF-8 now?
Denver https://jmp.chat/
--------------------------------------------------------------------------- Peter Beckman Internet Guy beckman at angryox.com https://www.angryox.com/ ---------------------------------------------------------------------------
On Sun, Mar 27, 2022 at 10:32:05PM -0400, Peter Beckman wrote:
My first evidence that UTF-8 was supported was January 13, 2017.
Using which of the methods that Vitelity provides (XMPP, email, other?)?
There are weird issues with Vitelity, such as odd length limits to SMS messages, but we worked around them. SMS is reliable enough,
I'm not sure what your metrics are for "reliable enough", but given the below chart, I don't think it would meet my bar. :)
When did you last test SMS with Vitelity? I have several years of SMS metric data on Vitelity SMS:
+------+----------+---------+----------+ | yr | success% | failed% | delayed% | +------+----------+---------+----------+ | 2015 | 98.42% | 1.58% | 3.17% | | 2016 | 96.31% | 3.69% | 5.30% | | 2017 | 96.68% | 3.32% | 2.79% | | 2018 | 92.22% | 7.78% | 6.24% | | 2019 | 94.46% | 5.54% | 1.50% | | 2020 | 98.41% | 1.59% | 0.71% | | 2021 | 97.75% | 2.25% | 2.51% | | 2022 | 98.61% | 1.39% | 0.24% | +------+----------+---------+----------+
2018-2019 sucked!
This is based on a baseline of sending an SMS once every 10 minutes to one of our active DIDs that has least recently received an Inbound SMS message. [...] Both are acceptable.
** NOTE these numbers may be attributable to outages by the sending carrier. They are not easily excluded, and so the numbers may be higher.
This also is only Inbound, not Outbound.
Given the lack of proper delivery receipts with most North American carriers, my assumption then would be that these "failed%" messages are all silent failures (i.e. the sender (on some random network) never knew that your Vitelity number didn't get their SMS). I can see how Vitelity's cost for SMS might encourage some use cases where reliability requirements are low, but especially since many (most?) carriers of this nature provide $0 incoming SMS, it's hard for me to see a good reason to use Vitelity for inbound. Perhaps your use case involves a lot of outbound too, though. Denver https://jmp.chat/
participants (3)
-
beckman＠angryox.com -
denver＠ossguy.com -
ross＠tajvar.io