troubleshooting RTP loss - and does anyone use webair/opti9?
HI team we recently noticed packet loss happening every morning from 6AM to 11AM, our MOS drops during that time our datacenter in garden City Long island (formerly Webair / now Opti9) claims it is outside of their network and can't do anything - but they have been unable to identify where the loss is occurring so i have 2 questions 1) Is there anyone in this group using the same datacenter, and if so, do you have any RTP issues ? maybe the issue is with one of our SIP providers, etc and not the Datacenter directly 2) What's a good tool / method to determine where the packet loss is happening ? i am using VoipMonitor - but willing to start using other tools to find the root cause -- Izzy Goldstein Chief Technology Officer - DevOps Master Main: (212) 477-1000 x 2085 <(212)%20477-1000> Direct: (929) 477-2085 Website: https://telego.com/ Confidentiality Notice: This e-mail may contain confidential and/or privileged information. If you are not the intended recipient or have received this e-mail in error please notify us immediately by email reply and destroy this e-mail. Any unauthorized copying, disclosure or distribution of the material in this e-mail is strictly forbidden. Any views or opinions presented in this email are solely those of the author and do not necessarily represent those of TeleGo (T). Employees of TeleGo are expressly required not to make defamatory statements and not to infringe or authorize any infringement of copyright or any other legal right by email communications. Any such communication is contrary to TeleGo policy and outside the scope of the employment of the individual concerned. TeleGo will not accept any liability in respect of such communication, and the employee responsible will be personally liable for any damages or other liability arising. TeleGo Hosted PBX <https://youtu.be/DaT8YAZ4V0w>
Izzy ? feel your pain ? this is exactly why we built out our active testing capabilities so we can point you exactly @ where this degradation is occurring. SQmediator / DVQattest creates synthetic traffic that emulates voice as well as video transactions and supports a wide range of traditional and modern day codecs. It emulates the adaptive codecs used by webRTC as well as SILK, SATIN and many others. The platform enables one to test fully end-to-end ? plus you can deploy ?hub? agents in the core that can respond to multiple incoming call requests from edge agents to create a variety of deployment configurations. More info available @ www.telchemy.com/sqmediator.php. Will follow up with you directly to dig deeper ? but responded here as well in case anyone else on the list can leverage this info and benefit from this diagnostic functionality. -anthony Anthony Caiozzo | 617-312-5189 Support: 678-387-3000 or <http://www.telchemy.com/custportal> www.telchemy.com/custportal to open a ticket From: VoiceOps <voiceops-bounces at voiceops.org> On Behalf Of Izzy Goldstein - TeleGo via VoiceOps Sent: Monday, October 30, 2023 2:06 PM To: Voiceops.org <voiceops at voiceops.org> Subject: [VoiceOps] troubleshooting RTP loss - and does anyone use webair/opti9? HI team we recently noticed packet loss happening every morning from 6AM to 11AM, our MOS drops during that time our datacenter in garden City Long island (formerly Webair / now Opti9) claims it is outside of their network and can't do anything - but they have been unable to identify where the loss is occurring so i have 2 questions 1) Is there anyone in this group using the same datacenter, and if so, do you have any RTP issues ? maybe the issue is with one of our SIP providers, etc and not the Datacenter directly 2) What's a good tool / method to determine where the packet loss is happening ? i am using VoipMonitor - but willing to start using other tools to find the root cause -- Izzy Goldstein Chief Technology Officer - DevOps Master Main: <tel:(212)%20477-1000> (212) 477-1000 x 2085 Direct: <tel:(929)%20477-2085> (929) 477-2085 Website: <https://telego.com/> https://telego.com/ <https://telego.com/wp-content/uploads/telego-signature-logo.png> Confidentiality Notice: This e-mail may contain confidential and/or privileged information. If you are not the intended recipient or have received this e-mail in error please notify us immediately by email reply and destroy this e-mail. Any unauthorized copying, disclosure or distribution of the material in this e-mail is strictly forbidden. Any views or opinions presented in this email are solely those of the author and do not necessarily represent those of TeleGo (T). Employees of TeleGo are expressly required not to make defamatory statements and not to infringe or authorize any infringement of copyright or any other legal right by email communications. Any such communication is contrary to TeleGo policy and outside the scope of the employment of the individual concerned. TeleGo will not accept any liability in respect of such communication, and the employee responsible will be personally liable for any damages or other liability arising. <https://youtu.be/DaT8YAZ4V0w> TeleGo Hosted PBX
Not too experienced, but my 2 cents If it's based on a faulty router. do a traceroute in the morning and in the afternoon a few days maybe you will find a correlation to a specific route. In morning hours try to ping every hop from the traceroute with BIG packets so you could find how far you can get without packet loss. the latter will also help If it is based on congestion On Mon, Oct 30, 2023 at 2:07?PM Izzy Goldstein - TeleGo via VoiceOps < voiceops at voiceops.org> wrote:
HI team
we recently noticed packet loss happening every morning from 6AM to 11AM, our MOS drops during that time our datacenter in garden City Long island (formerly Webair / now Opti9) claims it is outside of their network and can't do anything - but they have been unable to identify where the loss is occurring
so i have 2 questions
1) Is there anyone in this group using the same datacenter, and if so, do you have any RTP issues ? maybe the issue is with one of our SIP providers, etc and not the Datacenter directly
2) What's a good tool / method to determine where the packet loss is happening ?
i am using VoipMonitor - but willing to start using other tools to find the root cause
--
Izzy Goldstein
Chief Technology Officer - DevOps Master
Main: (212) 477-1000 x 2085 <(212)%20477-1000>
Direct: (929) 477-2085
Website: https://telego.com/
Confidentiality Notice: This e-mail may contain confidential and/or privileged information. If you are not the intended recipient or have received this e-mail in error please notify us immediately by email reply and destroy this e-mail. Any unauthorized copying, disclosure or distribution of the material in this e-mail is strictly forbidden. Any views or opinions presented in this email are solely those of the author and do not necessarily represent those of TeleGo (T). Employees of TeleGo are expressly required not to make defamatory statements and not to infringe or authorize any infringement of copyright or any other legal right by email communications. Any such communication is contrary to TeleGo policy and outside the scope of the employment of the individual concerned. TeleGo will not accept any liability in respect of such communication, and the employee responsible will be personally liable for any damages or other liability arising.
TeleGo Hosted PBX <https://youtu.be/DaT8YAZ4V0w> _______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops
-- *Pinchas S. Neiman* Software Engineer At ESEQ Technology Corp. 845.213.1229 #2
An HTML attachment was scrubbed... URL: <https://puck.nether.net/pipermail/voiceops/attachments/20231115/6eb79939/att...>
+1 for MTR also iperf in both directions. And if the only tool you have is ping. Change ping frequency and packet size to match G711. Ping each individual hop along the way, that you can, and look for discrepancies. Keep in mind it's ICMP so it will get deprioritized, but if it works fine sometimes and other times it does not, it's a load or fault-at-load that causes it to go karplunk. My favourite real life experience troubleshooting network weirdness was with a fortune 1000 client's network team early in the VoIP era. Same type of time-of-day weirdness. They were swearing up and down that their network gear was clean and was not running high on memory or cpu use... Until we painstakingly determined that it must be the network gear.... At which point they found their queue limits were too low for VoIP. The reason their network gear was clean was because it was ignoring packets when it got busy. That was a decade+ ago, I am sure the current network requirements have evolved (video and all)... But still... On Wed, Nov 15, 2023, 9:19 a.m. Richard Jobson via VoiceOps < voiceops at voiceops.org> wrote:
You either need to monitor the segment of the network surrounding the packet loss to isolate it
2. or you could use an active tool such as our VPP to make PESQ measurement to call from endPoint into your net work, and proven end to end exactly where & where the problem is happening.
Sent from my iPhone
On Oct 30, 2023, at 12:44 PM, Pinchas Neiman via VoiceOps < voiceops at voiceops.org> wrote:
? Not too experienced, but my 2 cents
If it's based on a faulty router. do a traceroute in the morning and in the afternoon a few days maybe you will find a correlation to a specific route.
In morning hours try to ping every hop from the traceroute with BIG packets so you could find how far you can get without packet loss.
the latter will also help If it is based on congestion
On Mon, Oct 30, 2023 at 2:07?PM Izzy Goldstein - TeleGo via VoiceOps < voiceops at voiceops.org> wrote:
HI team
we recently noticed packet loss happening every morning from 6AM to 11AM, our MOS drops during that time our datacenter in garden City Long island (formerly Webair / now Opti9) claims it is outside of their network and can't do anything - but they have been unable to identify where the loss is occurring
so i have 2 questions
1) Is there anyone in this group using the same datacenter, and if so, do you have any RTP issues ? maybe the issue is with one of our SIP providers, etc and not the Datacenter directly
2) What's a good tool / method to determine where the packet loss is happening ?
i am using VoipMonitor - but willing to start using other tools to find the root cause
--
Izzy Goldstein
Chief Technology Officer - DevOps Master
Main: (212) 477-1000 x 2085 <(212)%20477-1000>
Direct: (929) 477-2085
Website: https://telego.com/
Confidentiality Notice: This e-mail may contain confidential and/or privileged information. If you are not the intended recipient or have received this e-mail in error please notify us immediately by email reply and destroy this e-mail. Any unauthorized copying, disclosure or distribution of the material in this e-mail is strictly forbidden. Any views or opinions presented in this email are solely those of the author and do not necessarily represent those of TeleGo (T). Employees of TeleGo are expressly required not to make defamatory statements and not to infringe or authorize any infringement of copyright or any other legal right by email communications. Any such communication is contrary to TeleGo policy and outside the scope of the employment of the individual concerned. TeleGo will not accept any liability in respect of such communication, and the employee responsible will be personally liable for any damages or other liability arising.
TeleGo Hosted PBX <https://youtu.be/DaT8YAZ4V0w> _______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops
-- *Pinchas S. Neiman* Software Engineer At ESEQ Technology Corp. 845.213.1229 #2
_______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops
_______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops
Ivan Kovacevic Chief Revenue Officer and Co-CEO STAR TELECOM - Cloud Communications and Customer Experience Solutions Toronto - Canada | Fort Lauderdale - USA | Belgrade - Serbia O: +1 855-782-7835 x723 | M (CAN) +1-647-771-4773 | M (USA) +1-954-305-5665 | ivan.kovacevic at startelecom.ca www.startelecom.ca -- NOTE: This email message and any attachments are for the sole use of the intended recipient(s) and may contain confidential and/or privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by replying to this email, and destroy all copies of the original message.
AppNeta by Broadcom is an awesome tool and includes this type of troubleshooting https://techdocs.broadcom.com/us/en/ca-enterprise-software/it-operations-man... [cid:image001.jpg at 01DA0B40.4D766C30] From: VoiceOps <voiceops-bounces at voiceops.org> On Behalf Of Izzy Goldstein - TeleGo via VoiceOps Sent: Monday, October 30, 2023 2:06 PM To: Voiceops.org <voiceops at voiceops.org> Subject: [VoiceOps] troubleshooting RTP loss - and does anyone use webair/opti9? HI team we recently noticed packet loss happening every morning from 6AM to 11AM, our MOS drops during that time our datacenter in garden City Long island (formerly Webair / now Opti9) claims it is outside of their network and can't do anything - but they have been unable to identify where the loss is occurring so i have 2 questions 1) Is there anyone in this group using the same datacenter, and if so, do you have any RTP issues ? maybe the issue is with one of our SIP providers, etc and not the Datacenter directly 2) What's a good tool / method to determine where the packet loss is happening ? i am using VoipMonitor - but willing to start using other tools to find the root cause -- Izzy Goldstein Chief Technology Officer - DevOps Master Main: (212) 477-1000 x 2085<tel:(212)%20477-1000> Direct: (929) 477-2085<tel:(929)%20477-2085> Website: https://telego.com/ [https://telego.com/wp-content/uploads/telego-signature-logo.png] Confidentiality Notice: This e-mail may contain confidential and/or privileged information. If you are not the intended recipient or have received this e-mail in error please notify us immediately by email reply and destroy this e-mail. Any unauthorized copying, disclosure or distribution of the material in this e-mail is strictly forbidden. Any views or opinions presented in this email are solely those of the author and do not necessarily represent those of TeleGo (T). Employees of TeleGo are expressly required not to make defamatory statements and not to infringe or authorize any infringement of copyright or any other legal right by email communications. Any such communication is contrary to TeleGo policy and outside the scope of the employment of the individual concerned. TeleGo will not accept any liability in respect of such communication, and the employee responsible will be personally liable for any damages or other liability arising. TeleGo Hosted PBX<https://youtu.be/DaT8YAZ4V0w>
mtr https://en.wikipedia.org/wiki/MTR_(software) It isn't specific to VoIP. But it is still a very good first step in troubleshooting which hop the packet loss begins at. Unlike the basic traceroute tools that come with Linux or Windows, it can run continuously for however long you need, even days. This can be especially useful if the packet loss is intermittent.
Mike - appreciate the pointer - seems very similar to Windows pathping. It's also identical the methodology we leverage inside the DVQattest tool I described earlier today - but instead we send RTP packets with decreasing TTLs to effectively map out the exact path that the voice traffic took. -anthony -----Original Message----- From: VoiceOps <voiceops-bounces at voiceops.org> On Behalf Of Mike Johnston via VoiceOps Sent: Monday, October 30, 2023 4:38 PM To: Voiceops.org <voiceops at voiceops.org> Subject: Re: [VoiceOps] troubleshooting RTP loss - and does anyone use webair/opti9? mtr https://en.wikipedia.org/wiki/MTR_(software) It isn't specific to VoIP. But it is still a very good first step in troubleshooting which hop the packet loss begins at. Unlike the basic traceroute tools that come with Linux or Windows, it can run continuously for however long you need, even days. This can be especially useful if the packet loss is intermittent. _______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops
Anybody else experiencing outages in Chicago and Ashburn, VA today? Marc Rys Rystec Inc
participants (5)
-
anthony.caiozzo＠telchemy.com -
igoldstein＠telego.net -
ivan.kovacevic＠startelecom.ca -
m.rys＠rystec.com -
mjohnston＠wiktel.com -
neimanpinchas＠gmail.com -
richard＠teraquant.com -
twolf＠wolftechgroup.com