In the last 3 months I've been consistently frustrated by my carriers. "3-4 minutes is acceptable delay for delivery of SMS messages." "Our termination change now throws 504s instead of 503s 20-50% of the time and adds 2-3 seconds of delay to your call attempts. We didn't notice, and though you did, it's been 3 days and we haven't fixed it. Sorry!" "There was an outage? Works for me now!" "Someone upstream is intentionally dropping SMS messages. But we can't say who it is, and we can't get them to fix it because they aren't our carrier." Does the industry just suck at knowing when their stuff is broken, or only react when enough customers complain? Do carriers simply not instrument, monitor or graph metrics of their operations and proactively monitor and fix issues? My Thresholds: * SMS delivery end-to-end: Under 10 seconds * 503 Route Advance: Under 1 second * Response/Notice to termination/API/origination/server outage: 20 minutes * Fix a major issue (or provide a fix timeline): 3 days Too many times in the last few months _I_ have been the canary in the carrier's coal mine bringing attention to places where their operations are broken or delayed. And even then, unless I escalate to management (like CEO level), things move at the speed of a sloth. Most of the time it seems I monitor my carrier's infrastructure more closely than they do. I hope and dream of the unicorn carrier -- such great operational awareness and execution that it doesn't matter how great their customer service is, I'll never have to talk to them. Beckman --------------------------------------------------------------------------- Peter Beckman Internet Guy beckman at angryox.com http://www.angryox.com/ ---------------------------------------------------------------------------
This is a problem that has plagued the wholesale industry for some time now. Wholesale customers often demand the lowest pricing, then expect the best service. The two are not compatible in my experience. As an actual carrier (and not a reseller for most services), we provide all of these services wholesale. Our customers (wholesale or retail) can do most things in real time in our portal or using our API and certainly don't have to escalate to executive level to get traction. But we're often not the lowest price; you can't fund a responsive operation at loss-leader prices. We can meet your "unicorn carrier" goal below, and I would be happy to discuss specific needs off-list for you or anyone who might be interested to see if we're a fit. Regards, Mike Mike Ray, MBA, CNE, CTE Communications Authority, Inc. 11523 Palm Brush Trail #401 Lakewood Ranch, FL? 34202 DIRECT: call or text 941 600-0207 http://www.commauthority.com -----Original Message----- From: VoiceOps [mailto:voiceops-bounces at voiceops.org] On Behalf Of Peter Beckman Sent: Tuesday, October 6, 2015 11:28 AM To: VoiceOps <voiceops at voiceops.org> Subject: [VoiceOps] Lack of Quality Industry-wide In the last 3 months I've been consistently frustrated by my carriers. "3-4 minutes is acceptable delay for delivery of SMS messages." "Our termination change now throws 504s instead of 503s 20-50% of the time and adds 2-3 seconds of delay to your call attempts. We didn't notice, and though you did, it's been 3 days and we haven't fixed it. Sorry!" "There was an outage? Works for me now!" "Someone upstream is intentionally dropping SMS messages. But we can't say who it is, and we can't get them to fix it because they aren't our carrier." Does the industry just suck at knowing when their stuff is broken, or only react when enough customers complain? Do carriers simply not instrument, monitor or graph metrics of their operations and proactively monitor and fix issues? My Thresholds: * SMS delivery end-to-end: Under 10 seconds * 503 Route Advance: Under 1 second * Response/Notice to termination/API/origination/server outage: 20 minutes * Fix a major issue (or provide a fix timeline): 3 days Too many times in the last few months _I_ have been the canary in the carrier's coal mine bringing attention to places where their operations are broken or delayed. And even then, unless I escalate to management (like CEO level), things move at the speed of a sloth. Most of the time it seems I monitor my carrier's infrastructure more closely than they do. I hope and dream of the unicorn carrier -- such great operational awareness and execution that it doesn't matter how great their customer service is, I'll never have to talk to them. Beckman --------------------------------------------------------------------------- Peter Beckman Internet Guy beckman at angryox.com http://www.angryox.com/ --------------------------------------------------------------------------- _______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops
You are correct Peter. ?It's a ?constant struggle. ?For years I used a product from Empirix called Intelisight. ?It gave us the ability to monitor carriers (and other managed objects) with defined sets of KPI's. ?Once I saw a carrier have issues (PDD, quality, excessive 503's) I used to call them and tell them they had a problem. ?Hours or days later they would resolve. ?I got away from that practice and just routed around them. ? Funny how they notice that. ?Once the issue was resolved I put them in a penalty box and slowly added traffic. ?Once they realized I was measuring their performance and there would be financial repercussions it's amazing how the quality of their under lying carriers improved. ?My carrier ticket count dropped by 25-30%. From: Peter Beckman <beckman at angryox.com> To: VoiceOps <voiceops at voiceops.org> Sent: Tuesday, October 6, 2015 10:27 AM Subject: [VoiceOps] Lack of Quality Industry-wide In the last 3 months I've been consistently frustrated by my carriers. ? ? "3-4 minutes is acceptable delay for delivery of SMS messages." ? ? "Our termination change now throws 504s instead of 503s 20-50% of the ? ? ? time and adds 2-3 seconds of delay to your call attempts. We didn't ? ? ? notice, and though you did, it's been 3 days and we haven't fixed it. ? ? ? Sorry!" ? ? "There was an outage? Works for me now!" ? ? "Someone upstream is intentionally dropping SMS messages. But we can't ? ? ? say who it is, and we can't get them to fix it because they aren't our ? ? ? carrier." Does the industry just suck at knowing when their stuff is broken, or only react when enough customers complain? Do carriers simply not instrument, monitor or graph metrics of their operations and proactively monitor and fix issues? My Thresholds: ? ? * SMS delivery end-to-end: Under 10 seconds ? ? * 503 Route Advance: Under 1 second ? ? * Response/Notice to termination/API/origination/server outage: 20 minutes ? ? * Fix a major issue (or provide a fix timeline): 3 days Too many times in the last few months _I_ have been the canary in the carrier's coal mine bringing attention to places where their operations are broken or delayed. And even then, unless I escalate to management (like CEO level), things move at the speed of a sloth. Most of the time it seems I monitor my carrier's infrastructure more closely than they do. I hope and dream of the unicorn carrier -- such great operational awareness and execution that it doesn't matter how great their customer service is, I'll never have to talk to them. Beckman --------------------------------------------------------------------------- Peter Beckman? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? Internet Guy beckman at angryox.com? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? http://www.angryox.com/ --------------------------------------------------------------------------- _______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops
That's a great story Anthony! It's similar to my issues. I already do this for termination. DID and SMS issues are more difficult to manage. And the issues you have with one carrier that drives you to mass port your DIDs that you hope to escape start happening with your new carrier. I wish carriers would have the same level of visibility in their service and infrastructure as you seemed to have with Intellisight, and then fix the problems proactively when they find them. "What gets measured gets managed." So true. So elusive in this business. Beckman On Tue, 6 Oct 2015, Anthony Orlando wrote:
You are correct Peter. ?It's a ?constant struggle. ?For years I used a product from Empirix called Intelisight. ?It gave us the ability to monitor carriers (and other managed objects) with defined sets of KPI's. ?Once I saw a carrier have issues (PDD, quality, excessive 503's) I used to call them and tell them they had a problem. ?Hours or days later they would resolve. ?I got away from that practice and just routed around them. ? Funny how they notice that. ?Once the issue was resolved I put them in a penalty box and slowly added traffic. ?Once they realized I was measuring their performance and there would be financial repercussions it's amazing how the quality of their under lying carriers improved. ?My carrier ticket count dropped by 25-30%.
--------------------------------------------------------------------------- Peter Beckman Internet Guy beckman at angryox.com http://www.angryox.com/ ---------------------------------------------------------------------------
Just to be clear that was my OB traffic not my IB. ?That's a whole different ball game.? From: Peter Beckman <beckman at angryox.com> To: Anthony Orlando <avorlando at yahoo.com> Cc: VoiceOps <voiceops at voiceops.org> Sent: Tuesday, October 6, 2015 11:34 AM Subject: Re: [VoiceOps] Lack of Quality Industry-wide That's a great story Anthony! It's similar to my issues. I already do this for termination. DID and SMS issues are more difficult to manage. And the issues you have with one carrier that drives you to mass port your DIDs that you hope to escape start happening with your new carrier. I wish carriers would have the same level of visibility in their service and infrastructure as you seemed to have with Intellisight, and then fix the problems proactively when they find them. ? ? "What gets measured gets managed." So true. So elusive in this business. Beckman On Tue, 6 Oct 2015, Anthony Orlando wrote:
You are correct Peter. ?It's a ?constant struggle. ?For years I used a product from Empirix called Intelisight. ?It gave us the ability to monitor carriers (and other managed objects) with defined sets of KPI's. ?Once I saw a carrier have issues (PDD, quality, excessive 503's) I used to call them and tell them they had a problem. ?Hours or days later they would resolve. ?I got away from that practice and just routed around them. ? Funny how they notice that. ?Once the issue was resolved I put them in a penalty box and slowly added traffic. ?Once they realized I was measuring their performance and there would be financial repercussions it's amazing how the quality of their under lying carriers improved. ?My carrier ticket count dropped by 25-30%.
--------------------------------------------------------------------------- Peter Beckman? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? Internet Guy beckman at angryox.com? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? http://www.angryox.com/ -------------------------------------------------------------------------
Ditto. We do the same, but it doesn't get us very far. We've recently fired two carriers for performance. On Tue, Oct 6, 2015 at 12:34 PM, Peter Beckman <beckman at angryox.com> wrote:
That's a great story Anthony! It's similar to my issues. I already do this for termination. DID and SMS issues are more difficult to manage. And the issues you have with one carrier that drives you to mass port your DIDs that you hope to escape start happening with your new carrier.
I wish carriers would have the same level of visibility in their service and infrastructure as you seemed to have with Intellisight, and then fix the problems proactively when they find them.
"What gets measured gets managed."
So true. So elusive in this business.
Beckman
On Tue, 6 Oct 2015, Anthony Orlando wrote:
You are correct Peter. It's a constant struggle. For years I used a
product from Empirix called Intelisight. It gave us the ability to monitor carriers (and other managed objects) with defined sets of KPI's. Once I saw a carrier have issues (PDD, quality, excessive 503's) I used to call them and tell them they had a problem. Hours or days later they would resolve. I got away from that practice and just routed around them. Funny how they notice that. Once the issue was resolved I put them in a penalty box and slowly added traffic. Once they realized I was measuring their performance and there would be financial repercussions it's amazing how the quality of their under lying carriers improved. My carrier ticket count dropped by 25-30%.
--------------------------------------------------------------------------- Peter Beckman Internet Guy beckman at angryox.com http://www.angryox.com/ ---------------------------------------------------------------------------
_______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops
What would help, and what has been done in other industries (think JD Power, Gartner? ok maybe not Gartner) is to have this data aggregated and available to view as a part of an industry report. So that you can look at Cheap-o-voip.com and say?. ?The rates sure look good, but their PDD, completion, etc are dismal, I?ll pass?? That would quickly help separate the good from the bad and hopefully drive the industry overall to improve those key metrics. I would pay for that sort of report, just to avoid wasting time on duds. Or? I would be open to sharing data, either as summary data or sanitized CDRs to help drive that sort of database. Now? if we could only agree on what the key metrics are... it seems that everyone has a different measuring stick. Best Regards, Ivan Kovacevic Vice President, Client Services Star Telecom | www.startelecom.ca | SIP Based Services for Contact Centers *From:* VoiceOps [mailto:voiceops-bounces at voiceops.org] *On Behalf Of *Pete E *Sent:* Tuesday, October 06, 2015 5:14 PM *To:* Peter Beckman <beckman at angryox.com> *Cc:* VoiceOps <voiceops at voiceops.org> *Subject:* Re: [VoiceOps] Lack of Quality Industry-wide Ditto. We do the same, but it doesn't get us very far. We've recently fired two carriers for performance. On Tue, Oct 6, 2015 at 12:34 PM, Peter Beckman <beckman at angryox.com> wrote: That's a great story Anthony! It's similar to my issues. I already do this for termination. DID and SMS issues are more difficult to manage. And the issues you have with one carrier that drives you to mass port your DIDs that you hope to escape start happening with your new carrier. I wish carriers would have the same level of visibility in their service and infrastructure as you seemed to have with Intellisight, and then fix the problems proactively when they find them. "What gets measured gets managed." So true. So elusive in this business. Beckman On Tue, 6 Oct 2015, Anthony Orlando wrote: You are correct Peter. It's a constant struggle. For years I used a product from Empirix called Intelisight. It gave us the ability to monitor carriers (and other managed objects) with defined sets of KPI's. Once I saw a carrier have issues (PDD, quality, excessive 503's) I used to call them and tell them they had a problem. Hours or days later they would resolve. I got away from that practice and just routed around them. Funny how they notice that. Once the issue was resolved I put them in a penalty box and slowly added traffic. Once they realized I was measuring their performance and there would be financial repercussions it's amazing how the quality of their under lying carriers improved. My carrier ticket count dropped by 25-30%. --------------------------------------------------------------------------- Peter Beckman Internet Guy beckman at angryox.com http://www.angryox.com/ --------------------------------------------------------------------------- _______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops
With DIDs migrating to VoIP could ENUM registry be the solution to this issue? As predicted if PSTN shuts down there will only peer to peer routes. Kumudu Suriyaarachchi ksuriyaarachchi at alteva.com<mailto:ksuriyaarachchi at alteva.com> P 484.534.4427 [Alteva-Logo-horizontal] From: VoiceOps [mailto:voiceops-bounces at voiceops.org] On Behalf Of Ivan Kovacevic Sent: Tuesday, October 06, 2015 11:15 PM To: Pete E; Peter Beckman Cc: VoiceOps Subject: Re: [VoiceOps] Lack of Quality Industry-wide What would help, and what has been done in other industries (think JD Power, Gartner? ok maybe not Gartner) is to have this data aggregated and available to view as a part of an industry report. So that you can look at Cheap-o-voip.com and say?. ?The rates sure look good, but their PDD, completion, etc are dismal, I?ll pass?? That would quickly help separate the good from the bad and hopefully drive the industry overall to improve those key metrics. I would pay for that sort of report, just to avoid wasting time on duds. Or? I would be open to sharing data, either as summary data or sanitized CDRs to help drive that sort of database. Now? if we could only agree on what the key metrics are... it seems that everyone has a different measuring stick. Best Regards, Ivan Kovacevic Vice President, Client Services Star Telecom | www.startelecom.ca<http://www.startelecom.ca/> | SIP Based Services for Contact Centers From: VoiceOps [mailto:voiceops-bounces at voiceops.org<mailto:voiceops-bounces at voiceops.org>] On Behalf Of Pete E Sent: Tuesday, October 06, 2015 5:14 PM To: Peter Beckman <beckman at angryox.com<mailto:beckman at angryox.com>> Cc: VoiceOps <voiceops at voiceops.org<mailto:voiceops at voiceops.org>> Subject: Re: [VoiceOps] Lack of Quality Industry-wide Ditto. We do the same, but it doesn't get us very far. We've recently fired two carriers for performance. On Tue, Oct 6, 2015 at 12:34 PM, Peter Beckman <beckman at angryox.com<mailto:beckman at angryox.com>> wrote: That's a great story Anthony! It's similar to my issues. I already do this for termination. DID and SMS issues are more difficult to manage. And the issues you have with one carrier that drives you to mass port your DIDs that you hope to escape start happening with your new carrier. I wish carriers would have the same level of visibility in their service and infrastructure as you seemed to have with Intellisight, and then fix the problems proactively when they find them. "What gets measured gets managed." So true. So elusive in this business. Beckman On Tue, 6 Oct 2015, Anthony Orlando wrote: You are correct Peter. It's a constant struggle. For years I used a product from Empirix called Intelisight. It gave us the ability to monitor carriers (and other managed objects) with defined sets of KPI's. Once I saw a carrier have issues (PDD, quality, excessive 503's) I used to call them and tell them they had a problem. Hours or days later they would resolve. I got away from that practice and just routed around them. Funny how they notice that. Once the issue was resolved I put them in a penalty box and slowly added traffic. Once they realized I was measuring their performance and there would be financial repercussions it's amazing how the quality of their under lying carriers improved. My carrier ticket count dropped by 25-30%. --------------------------------------------------------------------------- Peter Beckman Internet Guy beckman at angryox.com<mailto:beckman at angryox.com> http://www.angryox.com/ --------------------------------------------------------------------------- _______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org<mailto:VoiceOps at voiceops.org> https://puck.nether.net/mailman/listinfo/voiceops This communication (including any attachments) may contain privileged or confidential information of Alteva and is intended solely for the use of the individual to whom it is addressed. If you are not the intended recipient, you should notify the author and delete this communication from your system, including any attachments. Any disclosure, copying, saving or distribution of this communication, or the taking of any action based on it, is strictly prohibited. Any views or opinions expressed are solely those of the author and do not necessarily represent those of Alteva.
Is there a centralized report of Tier 1 ISPs? It would be great to know who has the best performance... I think mass data like that would be difficult to analyze. Something similar to http://www.internettrafficreport.com/ - not sure what this really shows... Aryn H. K. Nakaoka anakaoka at trinet-hi.com Direct: 808.356.2901 Fax : 808.356.2919 Tri-net Solutions 733 Bishop St. #1170 Honolulu, HI 96813 http://www.trinet-hi.com https://twitter.com/AlohaTone Aloha Tone PBX <https://www.youtube.com/watch?v=96YWPY9wCeU> https://www.youtube.com/watch?v=96YWPY9wCeU <http://youtu.be/27v2wbnFIDs> Aloha Tone (HA) High Availability <http://youtu.be/rJsr4k0RBH8> http://youtu.be/rJsr4k0RBH8 CONFIDENTIALITY NOTICE: The information contained in this email and any attachments may be privileged, confidential and protected from disclosure. Any disclosure, distribution or copying of this email or any attachments by persons or entities other than the intended recipient is prohibited. If you have received this email in error, please notify the sender immediately by replying to the message and deleting this email and any attachments from your system. Thank you for your cooperation. On Tue, Oct 6, 2015 at 5:14 PM, Ivan Kovacevic < ivan.kovacevic at startelecom.ca> wrote:
What would help, and what has been done in other industries (think JD Power, Gartner? ok maybe not Gartner) is to have this data aggregated and available to view as a part of an industry report.
So that you can look at Cheap-o-voip.com and say?. ?The rates sure look good, but their PDD, completion, etc are dismal, I?ll pass?? That would quickly help separate the good from the bad and hopefully drive the industry overall to improve those key metrics. I would pay for that sort of report, just to avoid wasting time on duds. Or? I would be open to sharing data, either as summary data or sanitized CDRs to help drive that sort of database.
Now? if we could only agree on what the key metrics are... it seems that everyone has a different measuring stick.
Best Regards,
Ivan Kovacevic
Vice President, Client Services
Star Telecom | www.startelecom.ca | SIP Based Services for Contact Centers
*From:* VoiceOps [mailto:voiceops-bounces at voiceops.org] *On Behalf Of *Pete E *Sent:* Tuesday, October 06, 2015 5:14 PM *To:* Peter Beckman <beckman at angryox.com> *Cc:* VoiceOps <voiceops at voiceops.org> *Subject:* Re: [VoiceOps] Lack of Quality Industry-wide
Ditto. We do the same, but it doesn't get us very far. We've recently fired two carriers for performance.
On Tue, Oct 6, 2015 at 12:34 PM, Peter Beckman <beckman at angryox.com> wrote:
That's a great story Anthony! It's similar to my issues. I already do this for termination. DID and SMS issues are more difficult to manage. And the issues you have with one carrier that drives you to mass port your DIDs that you hope to escape start happening with your new carrier.
I wish carriers would have the same level of visibility in their service and infrastructure as you seemed to have with Intellisight, and then fix the problems proactively when they find them.
"What gets measured gets managed."
So true. So elusive in this business.
Beckman
On Tue, 6 Oct 2015, Anthony Orlando wrote:
You are correct Peter. It's a constant struggle. For years I used a product from Empirix called Intelisight. It gave us the ability to monitor carriers (and other managed objects) with defined sets of KPI's. Once I saw a carrier have issues (PDD, quality, excessive 503's) I used to call them and tell them they had a problem. Hours or days later they would resolve. I got away from that practice and just routed around them. Funny how they notice that. Once the issue was resolved I put them in a penalty box and slowly added traffic. Once they realized I was measuring their performance and there would be financial repercussions it's amazing how the quality of their under lying carriers improved. My carrier ticket count dropped by 25-30%.
--------------------------------------------------------------------------- Peter Beckman Internet Guy beckman at angryox.com http://www.angryox.com/ ---------------------------------------------------------------------------
_______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops
_______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops
We have had to cook up our own monitoring and right now it consists of: a) OPTIONS/ping/latency/jitter monitoring of IP addresses for each vendor b) Near real time monitoring with email alerts and CDR examples whenever each vendor falls out of bounds on: a. Percentage of connected calls (with a breakdown of TOP 5 SIP failure responses (503, 404, 403, etc). b. Percentage of PDD calls (8 seconds or more) This is a pretty blunt tool, but it has helped identify cases where a vendor goes completely down or has systemic problems that are noticeable to customers. We have been reaching out to carriers proactively, problem CDRs are sent along with the alert email, which makes it easy to forward. However I am not sure, doing so actually improves anything. The penalty box approach may work better. Best Regards, Ivan Kovacevic Vice President, Client Services Star Telecom | www.startelecom.ca | SIP Based Services for Contact Centers | LinkedIn <http://www.linkedin.com/company/star-telecom-www.startelecom.ca-?trk=top_nav...> *From:* VoiceOps [mailto:voiceops-bounces at voiceops.org] *On Behalf Of *Anthony Orlando via VoiceOps *Sent:* Tuesday, October 06, 2015 12:19 PM *To:* Peter Beckman <beckman at angryox.com>; VoiceOps <voiceops at voiceops.org> *Subject:* Re: [VoiceOps] Lack of Quality Industry-wide You are correct Peter. It's a constant struggle. For years I used a product from Empirix called Intelisight. It gave us the ability to monitor carriers (and other managed objects) with defined sets of KPI's. Once I saw a carrier have issues (PDD, quality, excessive 503's) I used to call them and tell them they had a problem. Hours or days later they would resolve. I got away from that practice and just routed around them. Funny how they notice that. Once the issue was resolved I put them in a penalty box and slowly added traffic. Once they realized I was measuring their performance and there would be financial repercussions it's amazing how the quality of their under lying carriers improved. My carrier ticket count dropped by 25-30%. ------------------------------ *From:* Peter Beckman <beckman at angryox.com> *To:* VoiceOps <voiceops at voiceops.org> *Sent:* Tuesday, October 6, 2015 10:27 AM *Subject:* [VoiceOps] Lack of Quality Industry-wide In the last 3 months I've been consistently frustrated by my carriers. "3-4 minutes is acceptable delay for delivery of SMS messages." "Our termination change now throws 504s instead of 503s 20-50% of the time and adds 2-3 seconds of delay to your call attempts. We didn't notice, and though you did, it's been 3 days and we haven't fixed it. Sorry!" "There was an outage? Works for me now!" "Someone upstream is intentionally dropping SMS messages. But we can't say who it is, and we can't get them to fix it because they aren't our carrier." Does the industry just suck at knowing when their stuff is broken, or only react when enough customers complain? Do carriers simply not instrument, monitor or graph metrics of their operations and proactively monitor and fix issues? My Thresholds: * SMS delivery end-to-end: Under 10 seconds * 503 Route Advance: Under 1 second * Response/Notice to termination/API/origination/server outage: 20 minutes * Fix a major issue (or provide a fix timeline): 3 days Too many times in the last few months _I_ have been the canary in the carrier's coal mine bringing attention to places where their operations are broken or delayed. And even then, unless I escalate to management (like CEO level), things move at the speed of a sloth. Most of the time it seems I monitor my carrier's infrastructure more closely than they do. I hope and dream of the unicorn carrier -- such great operational awareness and execution that it doesn't matter how great their customer service is, I'll never have to talk to them. Beckman --------------------------------------------------------------------------- Peter Beckman Internet Guy beckman at angryox.com http://www.angryox.com/ --------------------------------------------------------------------------- _______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops
You guys are discovering the dirty little secrets of voice service and why Quality of Service (QoS) has fallen of a cliff... I have been doing int?l voice termination and networks for nearly 30 years now and back then it was ?big-iron? switches and dedicated circuits costing lots of money... with monthly expense high, a carrier had to properly cost justify every connection and pay attention to the results..QoS was NOT a problem and pretty much everything connected and sounded well. Then VoIP came along and as the MBA?say, ?Barriers to entry? disappeared... lots and lots of small operators came along with little QoS rules and started hawking their wares...on the demand side, telco?s became indiscriminate about who and how many they inter-connected with since it did not cost much for a voip connection (compared to nailed up TDM circuits!) and on it went,, 10 years later and no real standardization of QoS has been created. Companies have services with names like ?standard quality service? translate this to ?cheap but nothing works? and ?premium quality? which is better.. Nowadays, with USA AND Int?l voice termination, carriers spend their efforts on managing the rates (not the QoS) lest they screw up and go negative margins, get cherry picked or increase rate disputes (the bane of our industry..) We need standardization like you reference and our company is building an online system to develop exactly those standard of quality with Int?l voice. as a short term solution, we do NOT deal with ?a-z standard rates? since there is no way we can test the thousands of rate destinations. Instead we accept only one or two int?l destinations from any one vendor and then we manually test prior to production routing.. this is pretty much the only way you can now guarantee any sort of QoS. Hopefully we will come up with better long term solutions in the years to come. Paul Stamoulis +1 212 444 3003 www.OneStopTel.net<http://www.onestoptel.net/> Thousands of Carriers...One Connection! Onestopcorp and Onestoptel monitors and reads all emails as only business emails are permitted to be sent and received. Onestopcorp also monitors emails for compliance reasons and to ensure that no unauthorized disclosure of confidential information is passed via the email system. This email and any attachments are confidential. It is intended for the recipient only. If you are not the intended recipient, any use, disclosure, distribution, printing or copying of this email is unauthorized. If you have received this email in error please immediately notify the sender by replying to this email and delete the email from your computer. The contents of any attachment to this email may contain software viruses, which could damage your own computer system. While Onestopcorp has taken every reasonable precaution to minimize this risk, we cannot accept liability for any damage, which you sustain as a result of software viruses. You should carry out your own virus checks before opening the attachment. From: VoiceOps [mailto:voiceops-bounces at voiceops.org] On Behalf Of Ivan Kovacevic Sent: Tuesday, October 06, 2015 2:29 PM To: Anthony Orlando <avorlando at yahoo.com>; Peter Beckman <beckman at angryox.com>; voiceops at voiceops.org Subject: Re: [VoiceOps] Lack of Quality Industry-wide We have had to cook up our own monitoring and right now it consists of: a) OPTIONS/ping/latency/jitter monitoring of IP addresses for each vendor b) Near real time monitoring with email alerts and CDR examples whenever each vendor falls out of bounds on: Percentage of connected calls (with a breakdown of TOP 5 SIP failure responses (503, 404, 403, etc). Percentage of PDD calls (8 seconds or more) This is a pretty blunt tool, but it has helped identify cases where a vendor goes completely down or has systemic problems that are noticeable to customers. We have been reaching out to carriers proactively, problem CDRs are sent along with the alert email, which makes it easy to forward. However I am not sure, doing so actually improves anything. The penalty box approach may work better. Best Regards, Ivan Kovacevic Vice President, Client Services Star Telecom | www.startelecom.ca<http://www.startelecom.ca/> | SIP Based Services for Contact Centers | LinkedIn<http://www.linkedin.com/company/star-telecom-www.startelecom.ca-?trk=top_nav...> From: VoiceOps [mailto:voiceops-bounces at voiceops.org<mailto:voiceops-bounces at voiceops.org>] On Behalf Of Anthony Orlando via VoiceOps Sent: Tuesday, October 06, 2015 12:19 PM To: Peter Beckman <beckman at angryox.com<mailto:beckman at angryox.com>>; VoiceOps <voiceops at voiceops.org<mailto:voiceops at voiceops.org>> Subject: Re: [VoiceOps] Lack of Quality Industry-wide You are correct Peter. It's a constant struggle. For years I used a product from Empirix called Intelisight. It gave us the ability to monitor carriers (and other managed objects) with defined sets of KPI's. Once I saw a carrier have issues (PDD, quality, excessive 503's) I used to call them and tell them they had a problem. Hours or days later they would resolve. I got away from that practice and just routed around them. Funny how they notice that. Once the issue was resolved I put them in a penalty box and slowly added traffic. Once they realized I was measuring their performance and there would be financial repercussions it's amazing how the quality of their under lying carriers improved. My carrier ticket count dropped by 25-30%. ________________________________ From: Peter Beckman <beckman at angryox.com<mailto:beckman at angryox.com>> To: VoiceOps <voiceops at voiceops.org<mailto:voiceops at voiceops.org>> Sent: Tuesday, October 6, 2015 10:27 AM Subject: [VoiceOps] Lack of Quality Industry-wide In the last 3 months I've been consistently frustrated by my carriers. "3-4 minutes is acceptable delay for delivery of SMS messages." "Our termination change now throws 504s instead of 503s 20-50% of the time and adds 2-3 seconds of delay to your call attempts. We didn't notice, and though you did, it's been 3 days and we haven't fixed it. Sorry!" "There was an outage? Works for me now!" "Someone upstream is intentionally dropping SMS messages. But we can't say who it is, and we can't get them to fix it because they aren't our carrier." Does the industry just suck at knowing when their stuff is broken, or only react when enough customers complain? Do carriers simply not instrument, monitor or graph metrics of their operations and proactively monitor and fix issues? My Thresholds: * SMS delivery end-to-end: Under 10 seconds * 503 Route Advance: Under 1 second * Response/Notice to termination/API/origination/server outage: 20 minutes * Fix a major issue (or provide a fix timeline): 3 days Too many times in the last few months _I_ have been the canary in the carrier's coal mine bringing attention to places where their operations are broken or delayed. And even then, unless I escalate to management (like CEO level), things move at the speed of a sloth. Most of the time it seems I monitor my carrier's infrastructure more closely than they do. I hope and dream of the unicorn carrier -- such great operational awareness and execution that it doesn't matter how great their customer service is, I'll never have to talk to them. Beckman --------------------------------------------------------------------------- Peter Beckman Internet Guy beckman at angryox.com<mailto:beckman at angryox.com> http://www.angryox.com/ --------------------------------------------------------------------------- _______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org<mailto:VoiceOps at voiceops.org> https://puck.nether.net/mailman/listinfo/voiceops
Maybe its show lack of quality/affordable products in this area (monitoring voip)? On 6/10/2015 18:27, Peter Beckman wrote:
In the last 3 months I've been consistently frustrated by my carriers.
"3-4 minutes is acceptable delay for delivery of SMS messages."
"Our termination change now throws 504s instead of 503s 20-50% of the time and adds 2-3 seconds of delay to your call attempts. We didn't notice, and though you did, it's been 3 days and we haven't fixed it. Sorry!"
"There was an outage? Works for me now!"
"Someone upstream is intentionally dropping SMS messages. But we can't say who it is, and we can't get them to fix it because they aren't our carrier."
Does the industry just suck at knowing when their stuff is broken, or only react when enough customers complain? Do carriers simply not instrument, monitor or graph metrics of their operations and proactively monitor and fix issues?
My Thresholds: * SMS delivery end-to-end: Under 10 seconds * 503 Route Advance: Under 1 second * Response/Notice to termination/API/origination/server outage: 20 minutes * Fix a major issue (or provide a fix timeline): 3 days
Too many times in the last few months _I_ have been the canary in the carrier's coal mine bringing attention to places where their operations are broken or delayed. And even then, unless I escalate to management (like CEO level), things move at the speed of a sloth. Most of the time it seems I monitor my carrier's infrastructure more closely than they do.
I hope and dream of the unicorn carrier -- such great operational awareness and execution that it doesn't matter how great their customer service is, I'll never have to talk to them.
Beckman --------------------------------------------------------------------------- Peter Beckman Internet Guy beckman at angryox.com http://www.angryox.com/ --------------------------------------------------------------------------- _______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops
participants (9)
-
anakaoka＠trinet-hi.com -
avorlando＠yahoo.com -
beckman＠angryox.com -
ivan.kovacevic＠startelecom.ca -
ksuriyaarachchi＠alteva.com -
mike＠astrocompanies.com -
peeip989＠gmail.com -
pstamoulis＠onestoptel.net -
shopik＠inblock.ru