Trust but verify. We monitor and alert on: - Every API connection we rely on for production - Every SMSC connection - Call Metrics - Inbound and Outbound SMS Volume against trailing 3 weeks of data on the same day and time to be alerted of spikes or unexpected drops - Test inbound and outbound SMS regularly on every DID when there has not been recent activity - Test inbound calling to every DID when there has not been recent activity - Test outbound carrier calling endpoints to ensure they are responding and reachable - automatedly review logs for problem indicators, spikes in errors, etc - and other stuff! Successes: - Detected a situation where a tier 0 carrier was completely unaware that there would be 20-40 minute drops in a bind to their inbound short code carrier/partner that we highlighted and detected based on our trailing 3 weeks of comparative data based on a small volume. - Alerted staff to periods of API unavailability that their own monitoring missed, due to a Web Application Firewall bug - Caught countless situations where a number ported away, or an NNID assignment was "lost" and our carrier did not detect nor notify us of these actions I'm fascinated by the automation of call quality and packet loss in calls, and look forward to hearing more from other people about how and what they are monitoring, and stories about how that proactive testing has found issues and improved customer experience. :-) Beckman, operationally nerdy On Wed, 4 Nov 2026, Mike Hammett via VoiceOps wrote:
Hi all,
We just chased two weeks of intermittent inbound RTP loss (carrier toward our SBC, business hours only) that nothing we had could see. Our SBC's counters and VoIPmonitor agreed on what arrived; the carrier's egress probe showed zero loss, so the loss was somewhere on the public path between them, and the carrier called it clean. We are also mid-migration from Metaswitch to NetSapiens, so whatever we set up has to make sense on both.
Before this, we had VoIPmonitor on a mirror of the SBC traffic and Smokeping and LibreNMS watching our own routers, which told us nothing about the carrier path. What we are putting in now: MOS and loss alerts in VoIPmonitor, Smokeping with call-shaped pings (1 per second, 172 bytes, EF) to every hop toward each carrier that answers, and SIP OPTIONS every minute to the carriers' SBCs (they answer with a 500, which is fine; it is a timed reply over the real path). The gap is anything that measures inbound traffic to us continuously.
I am after persistent monitoring people have left running for months, not one-off tests. A few things I would like to hear about:
- Scheduled test calls through the carrier, scored by your passive monitor. Who places them, how often, and did it ever catch something before customers did?
- Carriers that give you ongoing quality data: RTCP from their side, a looking glass, per-trunk stats. Does anyone get this without a fight?
- Measuring one-way loss toward you. A VM near the carrier running irtt or similar? Something a NetSapiens shop already does out of the box?
- OPTIONS to carrier SBCs as a standing probe. Any carrier pushback?
The 2023 "troubleshooting RTP loss" thread covered the one-off side (mtr, sized pings, Telchemy). I will summarize the replies and add them to the list.
Thanks, Mike
----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest Internet Exchange http://www.midwest-ix.com
--------------------------------------------------------------------------- Peter Beckman Internet Guy beckman@angryox.com https://www.angryox.com/ ---------------------------------------------------------------------------