AT&T / Onvoy / Vonage call routing screwup after LNP
So here's a weird one: we took over a small business account from Vonage. Vonage was using Onvoy for origination, and we elected to keep the TNs with Onvoy (through a wholesaler). So the "port" only consisted of Onvoy repointing traffic for those TNs internally away from Vonage and to our reseller, with no LRN change. The weird bit is that we definitely are seeing some traffic for those numbers hitting us, but it's been nearly 72 hours now and some calls are still ringing their Vonage ATAs. I couldn't tell you definitively where the delineation is, but I can tell you, for example, that if I call any of the TNs from my AT&T cell, those calls still hit Vonage, so I can at least reproduce the problem at-will. This is for a local real-estate office, and AT&T is big in our relatively rural market, so even if it turns out that AT&T is the only provider that is affected, that is still a huge percentage of our end-user's client base. And the frustrating bit is that traffic is now effectively being "forked", which is a huge inconvenience for our end-user since they have an old key system with analog trunks and so we have to choose between having our IAD hooked up to their KSU or having their stack of Vonage ATAs hooked up. (For now, we have left the Vonage ATAs in place, and we are forwarding calls that come to us to a single line from the ILEC that this office ended up keeping. I don't know what we would have done if they didn't have that line.) Onvoy swears up and down that everything is configured correctly on their side, and given that we are at least getting *some* calls, I am inclined to believe them. When I give them call examples from my cell phone, they say that they don't even see those calls hitting their systems at all. At this point, the running theory is that AT&T must have some kind of direct peering with Vonage, and Onvoy isn't in the loop at all on those calls. If that's the case, then perhaps everything magically works itself out once I have the end-user call up Vonage and have them close out the account completely. But I'm not sure it is worth the risk of having them take that step with things as they are, on the off-chance that I guessed wrong (instead of the problem getting fixed, calls from AT&T start going to /dev/null). Has anybody encountered anything like this before, or heard of national wireless carriers doing direct peering with national VoIP providers while completely bypassing PSTN switching infrastructure? Are there any AT&T, Onvoy, and/or Vonage reps reading this who can help un-**** this cluster? Thanks, -- Nathan Anderson First Step Internet, LLC nathana at fsr.com
Had similar experiences, but with different vendor. I would try to open a ticket with ATT to fix their routing. I know, it won't be easy. I would also try to speak with Vonage. I wouldn't have the customer disconnect before calls are flowing correctly. If this doesn't work, and you wait another day or two with no results, I may try to port the numbers away from convoy. Interested to know how you solved it... Good luck. On Dec 5, 2016 8:06 AM, "Nathan Anderson" <nathana at fsr.com> wrote:
So here's a weird one: we took over a small business account from Vonage. Vonage was using Onvoy for origination, and we elected to keep the TNs with Onvoy (through a wholesaler). So the "port" only consisted of Onvoy repointing traffic for those TNs internally away from Vonage and to our reseller, with no LRN change.
The weird bit is that we definitely are seeing some traffic for those numbers hitting us, but it's been nearly 72 hours now and some calls are still ringing their Vonage ATAs. I couldn't tell you definitively where the delineation is, but I can tell you, for example, that if I call any of the TNs from my AT&T cell, those calls still hit Vonage, so I can at least reproduce the problem at-will. This is for a local real-estate office, and AT&T is big in our relatively rural market, so even if it turns out that AT&T is the only provider that is affected, that is still a huge percentage of our end-user's client base. And the frustrating bit is that traffic is now effectively being "forked", which is a huge inconvenience for our end-user since they have an old key system with analog trunks and so we have to choose between having our IAD hooked up to their KSU or having their stack of Vonage ATAs hooked up. (For now, we have left the Vonage ATAs in place, and we are forwarding calls tha t come to us to a single line from the ILEC that this office ended up keeping. I don't know what we would have done if they didn't have that line.)
Onvoy swears up and down that everything is configured correctly on their side, and given that we are at least getting *some* calls, I am inclined to believe them. When I give them call examples from my cell phone, they say that they don't even see those calls hitting their systems at all. At this point, the running theory is that AT&T must have some kind of direct peering with Vonage, and Onvoy isn't in the loop at all on those calls. If that's the case, then perhaps everything magically works itself out once I have the end-user call up Vonage and have them close out the account completely. But I'm not sure it is worth the risk of having them take that step with things as they are, on the off-chance that I guessed wrong (instead of the problem getting fixed, calls from AT&T start going to /dev/null).
Has anybody encountered anything like this before, or heard of national wireless carriers doing direct peering with national VoIP providers while completely bypassing PSTN switching infrastructure? Are there any AT&T, Onvoy, and/or Vonage reps reading this who can help un-**** this cluster?
Thanks,
-- Nathan Anderson First Step Internet, LLC nathana at fsr.com
_______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops
Sounds like a possible ENUM routing issue if it's bypassing the PSTN. Was Vonage notified of the port away by Onvoy? Regards, Andrew Paolucci Sent with [ProtonMail](https://protonmail.com) Secure Email. -------- Original Message -------- Subject: Re: [VoiceOps] AT&T / Onvoy / Vonage call routing screwup after LNP Local Time: 5 December 2016 9:16 AM UTC Time: 5 December 2016 14:16 From: orenyny at gmail.com To: voiceops at voiceops.org <VoiceOps at voiceops.org> Had similar experiences, but with different vendor. I would try to open a ticket with ATT to fix their routing. I know, it won't be easy. I would also try to speak with Vonage. I wouldn't have the customer disconnect before calls are flowing correctly. If this doesn't work, and you wait another day or two with no results, I may try to port the numbers away from convoy. Interested to know how you solved it... Good luck. On Dec 5, 2016 8:06 AM, "Nathan Anderson" <nathana at fsr.com> wrote: So here's a weird one: we took over a small business account from Vonage. Vonage was using Onvoy for origination, and we elected to keep the TNs with Onvoy (through a wholesaler). So the "port" only consisted of Onvoy repointing traffic for those TNs internally away from Vonage and to our reseller, with no LRN change. The weird bit is that we definitely are seeing some traffic for those numbers hitting us, but it's been nearly 72 hours now and some calls are still ringing their Vonage ATAs. I couldn't tell you definitively where the delineation is, but I can tell you, for example, that if I call any of the TNs from my AT&T cell, those calls still hit Vonage, so I can at least reproduce the problem at-will. This is for a local real-estate office, and AT&T is big in our relatively rural market, so even if it turns out that AT&T is the only provider that is affected, that is still a huge percentage of our end-user's client base. And the frustrating bit is that traffic is now effectively being "forked", which is a huge inconvenience for our end-user since they have an old key system with analog trunks and so we have to choose between having our IAD hooked up to their KSU or having their stack of Vonage ATAs hooked up. (For now, we have left the Vonage ATAs in place, and we are forwarding calls tha t come to us to a single line from the ILEC that this office ended up keeping. I don't know what we would have done if they didn't have that line.) Onvoy swears up and down that everything is configured correctly on their side, and given that we are at least getting *some* calls, I am inclined to believe them. When I give them call examples from my cell phone, they say that they don't even see those calls hitting their systems at all. At this point, the running theory is that AT&T must have some kind of direct peering with Vonage, and Onvoy isn't in the loop at all on those calls. If that's the case, then perhaps everything magically works itself out once I have the end-user call up Vonage and have them close out the account completely. But I'm not sure it is worth the risk of having them take that step with things as they are, on the off-chance that I guessed wrong (instead of the problem getting fixed, calls from AT&T start going to /dev/null). Has anybody encountered anything like this before, or heard of national wireless carriers doing direct peering with national VoIP providers while completely bypassing PSTN switching infrastructure? Are there any AT&T, Onvoy, and/or Vonage reps reading this who can help un-**** this cluster? Thanks, -- Nathan Anderson First Step Internet, LLC nathana at fsr.com _______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops
Just to update everybody, I have been advised by a Vonage engineer that there is in fact a common peering partner between them and AT&T that does not route on LRN (whether using ENUM/DNS or not he didn't say). He was kind enough to open a ticket with them on our behalf to have the TNs in question removed from their switch, which they have since done. We are now waiting for propagation. I will retest later this afternoon, at which point I fully expect that we will be in business. I do not have inside knowledge as to whether Onvoy appropriately notified Vonage of the port-out, but I was also informed that in the case of this particular carrier, even if proper notification was done, this carrier receives a bulk updated TN list from Vonage only on a weekly basis. Outside of that, tickets need to be raised. So it is entirely possible that unless one raises a flag, even if everybody else in the chain does everything by the book when executing the port, calls can still be misrouted for up to a week. Grr. -- Nathan From: VoiceOps [mailto:voiceops-bounces at voiceops.org] On Behalf Of Andrew Paolucci Sent: Monday, December 05, 2016 8:30 AM To: Oren Yehezkely Cc: voiceops at voiceops.org Subject: Re: [VoiceOps] AT&T / Onvoy / Vonage call routing screwup after LNP Sounds like a possible ENUM routing issue if it's bypassing the PSTN. Was Vonage notified of the port away by Onvoy? Regards, Andrew Paolucci Sent with ProtonMail<https://protonmail.com> Secure Email. -------- Original Message -------- Subject: Re: [VoiceOps] AT&T / Onvoy / Vonage call routing screwup after LNP Local Time: 5 December 2016 9:16 AM UTC Time: 5 December 2016 14:16 From: orenyny at gmail.com<mailto:orenyny at gmail.com> To: voiceops at voiceops.org<mailto:voiceops at voiceops.org> <VoiceOps at voiceops.org<mailto:VoiceOps at voiceops.org>> Had similar experiences, but with different vendor. I would try to open a ticket with ATT to fix their routing. I know, it won't be easy. I would also try to speak with Vonage. I wouldn't have the customer disconnect before calls are flowing correctly. If this doesn't work, and you wait another day or two with no results, I may try to port the numbers away from convoy. Interested to know how you solved it... Good luck. On Dec 5, 2016 8:06 AM, "Nathan Anderson" <nathana at fsr.com<mailto:nathana at fsr.com>> wrote: So here's a weird one: we took over a small business account from Vonage. Vonage was using Onvoy for origination, and we elected to keep the TNs with Onvoy (through a wholesaler). So the "port" only consisted of Onvoy repointing traffic for those TNs internally away from Vonage and to our reseller, with no LRN change. The weird bit is that we definitely are seeing some traffic for those numbers hitting us, but it's been nearly 72 hours now and some calls are still ringing their Vonage ATAs. I couldn't tell you definitively where the delineation is, but I can tell you, for example, that if I call any of the TNs from my AT&T cell, those calls still hit Vonage, so I can at least reproduce the problem at-will. This is for a local real-estate office, and AT&T is big in our relatively rural market, so even if it turns out that AT&T is the only provider that is affected, that is still a huge percentage of our end-user's client base. And the frustrating bit is that traffic is now effectively being "forked", which is a huge inconvenience for our end-user since they have an old key system with analog trunks and so we have to choose between having our IAD hooked up to their KSU or having their stack of Vonage ATAs hooked up. (For now, we have left the Vonage ATAs in place, and we are forwarding calls tha t come to us to a single line from the ILEC that this office ended up keeping. I don't know what we would have done if they didn't have that line.) Onvoy swears up and down that everything is configured correctly on their side, and given that we are at least getting *some* calls, I am inclined to believe them. When I give them call examples from my cell phone, they say that they don't even see those calls hitting their systems at all. At this point, the running theory is that AT&T must have some kind of direct peering with Vonage, and Onvoy isn't in the loop at all on those calls. If that's the case, then perhaps everything magically works itself out once I have the end-user call up Vonage and have them close out the account completely. But I'm not sure it is worth the risk of having them take that step with things as they are, on the off-chance that I guessed wrong (instead of the problem getting fixed, calls from AT&T start going to /dev/null). Has anybody encountered anything like this before, or heard of national wireless carriers doing direct peering with national VoIP providers while completely bypassing PSTN switching infrastructure? Are there any AT&T, Onvoy, and/or Vonage reps reading this who can help un-**** this cluster? Thanks, -- Nathan Anderson First Step Internet, LLC nathana at fsr.com<mailto:nathana at fsr.com> _______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org<mailto:VoiceOps at voiceops.org> https://puck.nether.net/mailman/listinfo/voiceops
So frustrating to hear Nathan, but thank you for sharing!! I've had similar experiences, though usually the winning carrier has "republished" or "reannounced" their ownership and that has cleared things up for us. To save others the time and headache, can you let us know the process you used to reach out to Vonage (call, email) and the wording you used that eventually got you an engineer who understood what you were talking about? And if you find out the "common peering partner" name, would love to know. Beckman On Mon, 5 Dec 2016, Nathan Anderson wrote:
Just to update everybody, I have been advised by a Vonage engineer that there is in fact a common peering partner between them and AT&T that does not route on LRN (whether using ENUM/DNS or not he didn't say). He was kind enough to open a ticket with them on our behalf to have the TNs in question removed from their switch, which they have since done. We are now waiting for propagation. I will retest later this afternoon, at which point I fully expect that we will be in business.
I do not have inside knowledge as to whether Onvoy appropriately notified Vonage of the port-out, but I was also informed that in the case of this particular carrier, even if proper notification was done, this carrier receives a bulk updated TN list from Vonage only on a weekly basis. Outside of that, tickets need to be raised. So it is entirely possible that unless one raises a flag, even if everybody else in the chain does everything by the book when executing the port, calls can still be misrouted for up to a week. Grr.
-- Nathan
From: VoiceOps [mailto:voiceops-bounces at voiceops.org] On Behalf Of Andrew Paolucci Sent: Monday, December 05, 2016 8:30 AM To: Oren Yehezkely Cc: voiceops at voiceops.org Subject: Re: [VoiceOps] AT&T / Onvoy / Vonage call routing screwup after LNP
Sounds like a possible ENUM routing issue if it's bypassing the PSTN. Was Vonage notified of the port away by Onvoy?
Regards, Andrew Paolucci
Sent with ProtonMail<https://protonmail.com> Secure Email.
-------- Original Message -------- Subject: Re: [VoiceOps] AT&T / Onvoy / Vonage call routing screwup after LNP Local Time: 5 December 2016 9:16 AM UTC Time: 5 December 2016 14:16 From: orenyny at gmail.com<mailto:orenyny at gmail.com> To: voiceops at voiceops.org<mailto:voiceops at voiceops.org> <VoiceOps at voiceops.org<mailto:VoiceOps at voiceops.org>>
Had similar experiences, but with different vendor. I would try to open a ticket with ATT to fix their routing. I know, it won't be easy. I would also try to speak with Vonage. I wouldn't have the customer disconnect before calls are flowing correctly. If this doesn't work, and you wait another day or two with no results, I may try to port the numbers away from convoy.
Interested to know how you solved it... Good luck.
On Dec 5, 2016 8:06 AM, "Nathan Anderson" <nathana at fsr.com<mailto:nathana at fsr.com>> wrote: So here's a weird one: we took over a small business account from Vonage. Vonage was using Onvoy for origination, and we elected to keep the TNs with Onvoy (through a wholesaler). So the "port" only consisted of Onvoy repointing traffic for those TNs internally away from Vonage and to our reseller, with no LRN change.
The weird bit is that we definitely are seeing some traffic for those numbers hitting us, but it's been nearly 72 hours now and some calls are still ringing their Vonage ATAs. I couldn't tell you definitively where the delineation is, but I can tell you, for example, that if I call any of the TNs from my AT&T cell, those calls still hit Vonage, so I can at least reproduce the problem at-will. This is for a local real-estate office, and AT&T is big in our relatively rural market, so even if it turns out that AT&T is the only provider that is affected, that is still a huge percentage of our end-user's client base. And the frustrating bit is that traffic is now effectively being "forked", which is a huge inconvenience for our end-user since they have an old key system with analog trunks and so we have to choose between having our IAD hooked up to their KSU or having their stack of Vonage ATAs hooked up. (For now, we have left the Vonage ATAs in place, and we are forwarding calls t ha t come to us to a single line from the ILEC that this office ended up keeping. I don't know what we would have done if they didn't have that line.)
Onvoy swears up and down that everything is configured correctly on their side, and given that we are at least getting *some* calls, I am inclined to believe them. When I give them call examples from my cell phone, they say that they don't even see those calls hitting their systems at all. At this point, the running theory is that AT&T must have some kind of direct peering with Vonage, and Onvoy isn't in the loop at all on those calls. If that's the case, then perhaps everything magically works itself out once I have the end-user call up Vonage and have them close out the account completely. But I'm not sure it is worth the risk of having them take that step with things as they are, on the off-chance that I guessed wrong (instead of the problem getting fixed, calls from AT&T start going to /dev/null).
Has anybody encountered anything like this before, or heard of national wireless carriers doing direct peering with national VoIP providers while completely bypassing PSTN switching infrastructure? Are there any AT&T, Onvoy, and/or Vonage reps reading this who can help un-**** this cluster?
Thanks,
-- Nathan Anderson First Step Internet, LLC nathana at fsr.com<mailto:nathana at fsr.com>
_______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org<mailto:VoiceOps at voiceops.org> https://puck.nether.net/mailman/listinfo/voiceops
--------------------------------------------------------------------------- Peter Beckman Internet Guy beckman at angryox.com http://www.angryox.com/ ---------------------------------------------------------------------------
I do not have inside knowledge as to whether Onvoy appropriately notified Vonage of the port-out, ...
Ah, Vonage. Back when they were young, they didn't plan adequately for growth so their service was chaotic and their customer service was worse. So I ported my number to another VoIP provider, despite Vonage claiming that one couldn't do that. (Presumably in the chaos they neglected to veto the port-out.) A little later, I started getting calls from a crabby guy who asked what I was doing to his daughter. As it turned out, Vonage lost all track of the port-out and they assigned my number to a new customer, a local student. When she called her father, my number showed up on the caller ID, when he called that number, he got me. This was long enough ago that few people knew about number portability, and in that era the explanation "her phone company is run by nitwits" did not yet have the resonance it does today. I told him to tell her to tell Vonage to give her another number, but good luck with that. By fortunate coincidence, I happened to know a member of Vonage's board of directors, dropped him an e-mail and soon got a lengthy brown-nosing apology (on my cell phone while in England for $2/minute.) Lacking that contact, I expect I'd still be arguing with her father. R's, John
Lol, what a funny story)) ?????????? ? iPhone 5 ???. 2016 ?., ? 22:30, John Levine <johnl at taugh.com> ???????(?):
I do not have inside knowledge as to whether Onvoy appropriately notified Vonage of the port-out, ...
Ah, Vonage.
Back when they were young, they didn't plan adequately for growth so their service was chaotic and their customer service was worse. So I ported my number to another VoIP provider, despite Vonage claiming that one couldn't do that. (Presumably in the chaos they neglected to veto the port-out.)
A little later, I started getting calls from a crabby guy who asked what I was doing to his daughter. As it turned out, Vonage lost all track of the port-out and they assigned my number to a new customer, a local student. When she called her father, my number showed up on the caller ID, when he called that number, he got me. This was long enough ago that few people knew about number portability, and in that era the explanation "her phone company is run by nitwits" did not yet have the resonance it does today. I told him to tell her to tell Vonage to give her another number, but good luck with that.
By fortunate coincidence, I happened to know a member of Vonage's board of directors, dropped him an e-mail and soon got a lengthy brown-nosing apology (on my cell phone while in England for $2/minute.) Lacking that contact, I expect I'd still be arguing with her father.
R's, John _______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops
LOL, hilarious! :) Am 05.12.2016 um 20:30 schrieb John Levine:
I do not have inside knowledge as to whether Onvoy appropriately notified Vonage of the port-out, ...
Ah, Vonage.
Back when they were young, they didn't plan adequately for growth so their service was chaotic and their customer service was worse. So I ported my number to another VoIP provider, despite Vonage claiming that one couldn't do that. (Presumably in the chaos they neglected to veto the port-out.)
A little later, I started getting calls from a crabby guy who asked what I was doing to his daughter. As it turned out, Vonage lost all track of the port-out and they assigned my number to a new customer, a local student. When she called her father, my number showed up on the caller ID, when he called that number, he got me. This was long enough ago that few people knew about number portability, and in that era the explanation "her phone company is run by nitwits" did not yet have the resonance it does today. I told him to tell her to tell Vonage to give her another number, but good luck with that.
By fortunate coincidence, I happened to know a member of Vonage's board of directors, dropped him an e-mail and soon got a lengthy brown-nosing apology (on my cell phone while in England for $2/minute.) Lacking that contact, I expect I'd still be arguing with her father.
R's, John _______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops
The fact that Vonage hasn't disconnected the ATAs makes me feel like their disconnect workflow hasn't quite worked, and if they have a direct peering arrangement with AT&T this could be involved in that. They may try to contact Vonage as well and ask them about it, they certainly have experienced it somewhere before I'm sure, and might have a workflow for fixing it. Of course, opening a ticket with the originating carrier (AT&T) definitely needs to happen, since they're making the routing decision they're the ones that can fix it. -Paul On 12/05/2016 09:16 AM, Oren Yehezkely wrote:
Had similar experiences, but with different vendor. I would try to open a ticket with ATT to fix their routing. I know, it won't be easy. I would also try to speak with Vonage. I wouldn't have the customer disconnect before calls are flowing correctly. If this doesn't work, and you wait another day or two with no results, I may try to port the numbers away from convoy.
Interested to know how you solved it... Good luck.
On Dec 5, 2016 8:06 AM, "Nathan Anderson" <nathana at fsr.com <mailto:nathana at fsr.com>> wrote:
So here's a weird one: we took over a small business account from Vonage. Vonage was using Onvoy for origination, and we elected to keep the TNs with Onvoy (through a wholesaler). So the "port" only consisted of Onvoy repointing traffic for those TNs internally away from Vonage and to our reseller, with no LRN change.
The weird bit is that we definitely are seeing some traffic for those numbers hitting us, but it's been nearly 72 hours now and some calls are still ringing their Vonage ATAs. I couldn't tell you definitively where the delineation is, but I can tell you, for example, that if I call any of the TNs from my AT&T cell, those calls still hit Vonage, so I can at least reproduce the problem at-will. This is for a local real-estate office, and AT&T is big in our relatively rural market, so even if it turns out that AT&T is the only provider that is affected, that is still a huge percentage of our end-user's client base. And the frustrating bit is that traffic is now effectively being "forked", which is a huge inconvenience for our end-user since they have an old key system with analog trunks and so we have to choose between having our IAD hooked up to their KSU or having their stack of Vonage ATAs hooked up. (For now, we have left the Vonage ATAs in place, and we are forwarding calls tha t come to us to a single line from the ILEC that this office ended up keeping. I don't know what we would have done if they didn't have that line.)
Onvoy swears up and down that everything is configured correctly on their side, and given that we are at least getting *some* calls, I am inclined to believe them. When I give them call examples from my cell phone, they say that they don't even see those calls hitting their systems at all. At this point, the running theory is that AT&T must have some kind of direct peering with Vonage, and Onvoy isn't in the loop at all on those calls. If that's the case, then perhaps everything magically works itself out once I have the end-user call up Vonage and have them close out the account completely. But I'm not sure it is worth the risk of having them take that step with things as they are, on the off-chance that I guessed wrong (instead of the problem getting fixed, calls from AT&T start going to /dev/null).
Has anybody encountered anything like this before, or heard of national wireless carriers doing direct peering with national VoIP providers while completely bypassing PSTN switching infrastructure? Are there any AT&T, Onvoy, and/or Vonage reps reading this who can help un-**** this cluster?
Thanks,
-- Nathan Anderson First Step Internet, LLC nathana at fsr.com <mailto:nathana at fsr.com>
_______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org <mailto:VoiceOps at voiceops.org> https://puck.nether.net/mailman/listinfo/voiceops <https://puck.nether.net/mailman/listinfo/voiceops>
_______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops
participants (8)
-
a.slastenov＠gmail.com -
andrew＠paolucci.ca -
beckman＠angryox.com -
johnl＠taugh.com -
nathana＠fsr.com -
orenyny＠gmail.com -
paul＠timmins.net -
universe＠truemetal.org