Hi Groupers, This has been a wild month over here. Our peer SBC has been lit up with alarms with bounces from this peer or that. We'll lose one gateway from a peer for a minute or so, then it'll come right back up. Sometimes the SIP bounce is accompanied with a BGP route flap, sometimes not. It almost always happens late at night, but a few sneak in during the day. We've got plenty of redundancy to cover the events, so ASR remains high. Part of our root-cause analysis is seeking an uderstanding of the experiences of similar networks, so I need to ask... Is anyone else seeing a general increase in internet-based peer bounces? -B
What SBC and what kinds of traffic volume? I saw a similarly bizarre issue on mine. On Fri, 2010-08-20 at 06:50 -0700, Beth Johnson wrote:
Hi Groupers, This has been a wild month over here. Our peer SBC has been lit up with alarms with bounces from this peer or that. We'll lose one gateway from a peer for a minute or so, then it'll come right back up. Sometimes the SIP bounce is accompanied with a BGP route flap, sometimes not. It almost always happens late at night, but a few sneak in during the day.
We've got plenty of redundancy to cover the events, so ASR remains high.
Part of our root-cause analysis is seeking an uderstanding of the experiences of similar networks, so I need to ask... Is anyone else seeing a general increase in internet-based peer bounces?
-B
_______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops
Acme SBCs: moderately loaded, very healthy, very stable. All traffic graphs show no interesting increase or decrease in total traffic; there is no traffic-volume based attack detectable by our equipment. A couple of RFOs have discovered planned, unannounced changes that occurred. Maybe everyone has decided to sneak upgrades and other changes in before school starts and they lose that summer intern on which everything can be conveniently blamed? Interns and dogs are most handy in that regard... Hopefully, however, its just the NSA dropping in more Einstein taps. -B On Fri, Aug 20, 2010 at 7:27 AM, anorexicpoodle <anorexicpoodle at gmail.com>wrote:
What SBC and what kinds of traffic volume? I saw a similarly bizarre issue on mine.
On Fri, 2010-08-20 at 06:50 -0700, Beth Johnson wrote:
Hi Groupers, This has been a wild month over here. Our peer SBC has been lit up with alarms with bounces from this peer or that. We'll lose one gateway from a peer for a minute or so, then it'll come right back up. Sometimes the SIP bounce is accompanied with a BGP route flap, sometimes not. It almost always happens late at night, but a few sneak in during the day.
We've got plenty of redundancy to cover the events, so ASR remains high.
Part of our root-cause analysis is seeking an uderstanding of the experiences of similar networks, so I need to ask... Is anyone else seeing a general increase in internet-based peer bounces?
-B
_______________________________________________ VoiceOps mailing listVoiceOps at voiceops.orghttps://puck.nether.net/mailman/listinfo/voiceops
Well i recently got bitten by a small issue where the Acme BCP setting in the media manager for "Max Signaling Bandwidth" wasn't high enough to support our burst traffic, so it would drop signaling traffic during spikes. Of course this is a global setting so the traffic that it dropped was not even isolated to a single realm, but across the entire SD, and would often include keep-alive traffic. All the traffic policing in the media manager is implemented in the network processor so short of being in full debug mode, it generates absolutely no messages at all that it is doing this. Traffic just goes into a black hole and peers seem to fall off the earth for a while. Of course those thinking creatively will quickly realize this will generate retransmissions, which will drive up the signaling traffic and make the problem worse. I'm not saying this is your issue, but i chased BGP updates, upstream routing issues, switching issues, solar flares, etc forever trying to figure out what would cause this. Once i figured out this was the problem i just cranked the setting to max, and let my trust mode settings handle DOS protection, since its realm specific and far better suited to the task. On Fri, 2010-08-20 at 08:04 -0700, Beth Johnson wrote:
Acme SBCs: moderately loaded, very healthy, very stable.
All traffic graphs show no interesting increase or decrease in total traffic; there is no traffic-volume based attack detectable by our equipment.
A couple of RFOs have discovered planned, unannounced changes that occurred. Maybe everyone has decided to sneak upgrades and other changes in before school starts and they lose that summer intern on which everything can be conveniently blamed? Interns and dogs are most handy in that regard...
Hopefully, however, its just the NSA dropping in more Einstein taps.
-B
On Fri, Aug 20, 2010 at 7:27 AM, anorexicpoodle <anorexicpoodle at gmail.com> wrote:
What SBC and what kinds of traffic volume? I saw a similarly bizarre issue on mine.
On Fri, 2010-08-20 at 06:50 -0700, Beth Johnson wrote:
> > Hi Groupers, > This has been a wild month over here. Our peer SBC has been > lit up with alarms with bounces from this peer or that. > We'll lose one gateway from a peer for a minute or so, then > it'll come right back up. Sometimes the SIP bounce is > accompanied with a BGP route flap, sometimes not. It almost > always happens late at night, but a few sneak in during the > day. > > We've got plenty of redundancy to cover the events, so ASR > remains high. > > Part of our root-cause analysis is seeking an uderstanding > of the experiences of similar networks, so I need to ask... > Is anyone else seeing a general increase in internet-based > peer bounces? > > -B > > > > _______________________________________________ > VoiceOps mailing list > VoiceOps at voiceops.org > https://puck.nether.net/mailman/listinfo/voiceops
Come to think of it, I've been more annoyed in the past month or so... Anyone know of a tool that can maintain/display a traceroute-like history? Not all network problems cause bgp route updates, and would be nice to monitor traffic paths to peers for possible correlation of path changes to SIP events. David On Fri, Aug 20, 2010 at 6:50 AM, Beth Johnson <bethjohnson5060 at gmail.com> wrote:
Hi Groupers, This has been a wild month over here.? Our peer SBC has been lit up with alarms with bounces from this peer or that.? We'll lose one gateway from a peer for a minute or so, then it'll come right back up.? Sometimes the SIP bounce is accompanied with a BGP route flap, sometimes not.? It almost always happens late at night, but a few sneak in during the day.
We've got plenty of redundancy to cover the events, so ASR remains high.
Part of our root-cause analysis is seeking an uderstanding of the experiences of similar networks, so I need to ask... Is anyone else seeing a general increase in internet-based peer bounces?
-B
_______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops
Smokeping? On 8/20/2010 11:03 AM, David Hiers wrote:
Come to think of it, I've been more annoyed in the past month or so...
Anyone know of a tool that can maintain/display a traceroute-like history? Not all network problems cause bgp route updates, and would be nice to monitor traffic paths to peers for possible correlation of path changes to SIP events.
David
On Fri, Aug 20, 2010 at 6:50 AM, Beth Johnson<bethjohnson5060 at gmail.com> wrote:
Hi Groupers, This has been a wild month over here. Our peer SBC has been lit up with alarms with bounces from this peer or that. We'll lose one gateway from a peer for a minute or so, then it'll come right back up. Sometimes the SIP bounce is accompanied with a BGP route flap, sometimes not. It almost always happens late at night, but a few sneak in during the day.
We've got plenty of redundancy to cover the events, so ASR remains high.
Part of our root-cause analysis is seeking an uderstanding of the experiences of similar networks, so I need to ask... Is anyone else seeing a general increase in internet-based peer bounces?
-B
_______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops
_______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops
-- Lee Riemer Director of Technical Operations Bestline Communications, L.P. Voice: 1+512.328.9095 Fax: 1+512.328.0038
This may be more than you're looking for: http://www.renesys.com/ Very detailed analysis of peer connections. A bit spendy, but I had a demo license for a while and found it incredibly useful. If you run an ISP with clueful customers who demand to know what's going on, it's worth the investment to back up your statements. JT On Aug 20, 2010, at 9:03 AM, David Hiers wrote:
Come to think of it, I've been more annoyed in the past month or so...
Anyone know of a tool that can maintain/display a traceroute-like history? Not all network problems cause bgp route updates, and would be nice to monitor traffic paths to peers for possible correlation of path changes to SIP events.
David
On Fri, Aug 20, 2010 at 6:50 AM, Beth Johnson <bethjohnson5060 at gmail.com
wrote: Hi Groupers, This has been a wild month over here. Our peer SBC has been lit up with alarms with bounces from this peer or that. We'll lose one gateway from a peer for a minute or so, then it'll come right back up. Sometimes the SIP bounce is accompanied with a BGP route flap, sometimes not. It almost always happens late at night, but a few sneak in during the day.
We've got plenty of redundancy to cover the events, so ASR remains high.
Part of our root-cause analysis is seeking an uderstanding of the experiences of similar networks, so I need to ask... Is anyone else seeing a general increase in internet-based peer bounces?
-B
Beth Johnson wrote:
Part of our root-cause analysis is seeking an uderstanding of the experiences of similar networks, so I need to ask... Is anyone else seeing a general increase in internet-based peer bounces?
We've seen a large number of issues at one of our locations which is primarily served by Highwinds. BGP flaps, a 10GE gone down altogether for days, etc etc. I think we've had two major issues and four minor ones in the last 30-45 days, which is unprecedented. -- Carlos Alvarez TelEvolve 602-889-3003
participants (6)
-
anorexicpoodle＠gmail.com -
bethjohnson5060＠gmail.com -
carlos＠televolve.com -
hiersd＠gmail.com -
jtodd＠loligo.com -
lriemer＠bestline.net