Hey Peter, Thanks for reporting here, actually. I didn't even need to open a ticket, and stayed in bed. Haha It would surely help some of us out here if you do, and thanks again! *Glenn Geller* *VDOTel* On Sun, Feb 9, 2020 at 7:09 PM Peter Beckman <beckman at angryox.com> wrote:

START: 02-09-2020 20:31:39 UTC (~3:31pm EST) END: 02-09-2020 23:40:47 UTC (~6:41pm EST)

Services recovered approximately 3 hours and 10 minutes after we initially detected the network issues. Round Trip Times dropped from 50% packet loss and 375ms RTT back down to 68ms RTT with 0% packet loss.

We're waiting on an RFO from Vitelity/Voyant. Would that be an appropriate post here?

On Sun, 9 Feb 2020, Peter Beckman wrote:

...

START: 02-09-2020 20:31:39 UTC END: ongoing

Vitelity is experiencing some sort of network issue. While the Voyant NOC says they are aware, their portal is unstable due to the 40-50% packet loss and jump from 40-50ms to 375ms round trip times.

This is affecting all services AFAICT: API endpoints, Portal, incoming and outgoing calls.

We have moved most of our traffic away from Vitelity, except for phone numbers they host, as there is no ETA and 375ms plus 40-50% packet loss means pretty terrible call quality.

I'll update as I learn more.

--------------------------------------------------------------------------- Peter Beckman Internet Guy beckman at angryox.com http://www.angryox.com/ --------------------------------------------------------------------------- _______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops