Does anyone have any information on the TNS SS7 outage going on now? Both of our A-links go to different facilities (Chicago and Ashburn) over different VPNs (which are up). TNS has acknowledged that it's a carrier issue. That they're investigating. ----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest Internet Exchange http://www.midwest-ix.com
The links have since come back. This is the second major failure of TNS within 6 months. No RFO yet, but I will certainly be on them for some detail. My Metaswitch support rep verified that (pardon my lack of using the correct terminology, I'm not that familiar with it) our SS7 links never actually went down, it just lost connectivity beyond that. Sorta like my BGP peer stayed up, but I lost the global routing table behind it. ----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest Internet Exchange http://www.midwest-ix.com From: "Mike Hammett" <voiceops at ics-il.net> To: "voiceops" <voiceops at voiceops.org> Sent: Wednesday, January 19, 2022 4:05:32 PM Subject: [VoiceOps] TNS SS7 Outage Does anyone have any information on the TNS SS7 outage going on now? Both of our A-links go to different facilities (Chicago and Ashburn) over different VPNs (which are up). TNS has acknowledged that it's a carrier issue. That they're investigating. ----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest Internet Exchange http://www.midwest-ix.com _______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops
That's never a good thing. We were isolated for ~ 24 hours a couple of years ago and that's something I never want to repeat. Funny story -- AT&T even called us and said "do you guys know that both your A-Kinks are down"? We replied with the AT&T ticket numbers on the circuits that were down and the SS7 folks said "yeah, we're not allowed to talk to those guys, hopefully they fix it soon". I don't think that escallate button on the ticket does what you think it does.... It was also a complete pain to try to get them to change routing for us. We had a neighboring company offer to give us a temporary link to their switch -- If memory serves that's a F-Link, but AT&T could never get the routing right. Unfortunately, changing SS7 providers wasn't an option. One of the other SS7 providers (to remain nameless) could never actually get their circuits sent to the correct place. This is making me think we should look at getting 2 more A-Links, so we have 4 again...... On Wed, Jan 19, 2022 at 5:20 PM Mike Hammett <voiceops at ics-il.net> wrote:
The links have since come back. This is the second major failure of TNS within 6 months. No RFO yet, but I will certainly be on them for some detail.
My Metaswitch support rep verified that (pardon my lack of using the correct terminology, I'm not that familiar with it) our SS7 links never actually went down, it just lost connectivity beyond that. Sorta like my BGP peer stayed up, but I lost the global routing table behind it.
----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com
Midwest Internet Exchange http://www.midwest-ix.com
------------------------------ *From: *"Mike Hammett" <voiceops at ics-il.net> *To: *"voiceops" <voiceops at voiceops.org> *Sent: *Wednesday, January 19, 2022 4:05:32 PM *Subject: *[VoiceOps] TNS SS7 Outage
Does anyone have any information on the TNS SS7 outage going on now? Both of our A-links go to different facilities (Chicago and Ashburn) over different VPNs (which are up).
TNS has acknowledged that it's a carrier issue. That they're investigating.
----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com
Midwest Internet Exchange http://www.midwest-ix.com
_______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops _______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops
That's really odd for all of your SS7 services to go down because carriers typically connect to 2 A-links which are backed up by 2 D-links. So there are a total of 4 STPs (Signaling Transfer Points) programmed to back each other up. It could be down for one of two reasons. In the SS7 network, each switch is assigned a unique point code (9 digit number that identifies the switch location). In order for the ISUP trunks to work, the SS7 provider has to establish a route between your switch and the other carrier's switch. To make it a little easier to understand let's pretend you have a switch in CA and you are connecting it to 3 of AT&Ts San Francisco tandems. If each switch had the following point code assignments: Your switch's point code: 005-123-456 San Francisco tandem: 222-000-111 Oakland tandem: 222-000-222 Santa Rosa tandem: 222-000-333 Then your SS7 provider would build 3 point code routes: Your switch 005-123-456 to AT&T San Francisco tandem 222-000-111 Your switch 005-123-456 to AT&T Oakland tandem 222-000-222 Your switch 005-123-456 to AT&T Santa Rosa tandem 222-000-333 When your customer makes a call that is served by the Oakland tandem, the call would be routed over the 005-123-456 to 222-000-222 route set. If your links work but the call doesn't complete either your route set failed for some reason, or either AT&T's SS7 links or trunks went down. I'd ask them where the failure occurred. MARY LOU CAREY BackUP Telecom Consulting Office: 615-791-9969 Cell: 615-796-1111 On 2022-01-19 04:13 PM, Mike Hammett wrote:
The links have since come back. This is the second major failure of TNS within 6 months. No RFO yet, but I will certainly be on them for some detail.
My Metaswitch support rep verified that (pardon my lack of using the correct terminology, I'm not that familiar with it) our SS7 links never actually went down, it just lost connectivity beyond that. Sorta like my BGP peer stayed up, but I lost the global routing table behind it.
----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com
Midwest Internet Exchange http://www.midwest-ix.com
-------------------------
From: "Mike Hammett" <voiceops at ics-il.net> To: "voiceops" <voiceops at voiceops.org> Sent: Wednesday, January 19, 2022 4:05:32 PM Subject: [VoiceOps] TNS SS7 Outage
Does anyone have any information on the TNS SS7 outage going on now? Both of our A-links go to different facilities (Chicago and Ashburn) over different VPNs (which are up).
TNS has acknowledged that it's a carrier issue. That they're investigating.
----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com
Midwest Internet Exchange http://www.midwest-ix.com
_______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops _______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops
Currently the story is that there was a power supply issue in Chicago. Okay, so Chicago went down. What about Ashburn? Just how big was that power supply? ;-) I know TNS has more facilities, so I'm going to try to get another link out of them gratis. ----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest Internet Exchange http://www.midwest-ix.com ----- Original Message ----- From: "Mike Hammett" <voiceops at ics-il.net> To: "voiceops" <voiceops at voiceops.org> Sent: Wednesday, January 19, 2022 4:13:08 PM Subject: Re: [VoiceOps] TNS SS7 Outage The links have since come back. This is the second major failure of TNS within 6 months. No RFO yet, but I will certainly be on them for some detail. My Metaswitch support rep verified that (pardon my lack of using the correct terminology, I'm not that familiar with it) our SS7 links never actually went down, it just lost connectivity beyond that. Sorta like my BGP peer stayed up, but I lost the global routing table behind it. ----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest Internet Exchange http://www.midwest-ix.com ----- Original Message ----- From: "Mike Hammett" <voiceops at ics-il.net> To: "voiceops" <voiceops at voiceops.org> Sent: Wednesday, January 19, 2022 4:05:32 PM Subject: [VoiceOps] TNS SS7 Outage Does anyone have any information on the TNS SS7 outage going on now? Both of our A-links go to different facilities (Chicago and Ashburn) over different VPNs (which are up). TNS has acknowledged that it's a carrier issue. That they're investigating. ----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest Internet Exchange http://www.midwest-ix.com _______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops _______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops
Mike- An extra-large wall-wart :-) -Jeff Quoting Mike Hammett <voiceops at ics-il.net>:
Currently the story is that there was a power supply issue in Chicago. ? Okay, so Chicago went down. What about Ashburn? Just how big was that power supply? ?;-) ? I know TNS has more facilities, so I'm going to try to get another link out of them gratis.
----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com
Midwest Internet Exchange http://www.midwest-ix.com
?
------------------------- FROM: "Mike Hammett" <voiceops at ics-il.net> TO: "voiceops" <voiceops at voiceops.org> SENT: Wednesday, January 19, 2022 4:13:08 PM SUBJECT: Re: [VoiceOps] TNS SS7 Outage
The links have since come back. This is the second major failure of TNS within 6 months. No RFO yet, but I will certainly be on them for some detail. ? My Metaswitch support rep verified that (pardon my lack of using the correct terminology, I'm not that familiar with it) our SS7 links never actually went down, it just lost connectivity beyond that. Sorta like my BGP peer stayed up, but I lost the global routing table behind it. ?
----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com
Midwest Internet Exchange http://www.midwest-ix.com ? ?
------------------------- FROM: "Mike Hammett" <voiceops at ics-il.net> TO: "voiceops" <voiceops at voiceops.org> SENT: Wednesday, January 19, 2022 4:05:32 PM SUBJECT: [VoiceOps] TNS SS7 Outage ? Does anyone have any information on the TNS SS7 outage going on now? Both of our A-links go to different facilities (Chicago and Ashburn) over different VPNs (which are up).
TNS has acknowledged that it's a carrier issue. That they're investigating.
----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com
Midwest Internet Exchange http://www.midwest-ix.com ?
_______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops
_______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops
That sounds as believable as the Christmas Day bombing in Nashville that took out local and 911 phone services out all the way up to St Louis. Anyone who's familiar with the network knows St Louis isn't in the same LATA and a building built to withstand natural disasters etc wouldn't be damaged that badly by a bomb going off down the street from the building. You'd better be asking for a link to a different set of STPs because if the power in Chicago took out all 4 of their STPs, they have some serious problems with their network redundancy plan! MARY LOU CAREY BackUP Telecom Consulting Office: 615-791-9969 Cell: 615-796-1111 On 2022-01-20 05:44 AM, Mike Hammett wrote:
Currently the story is that there was a power supply issue in Chicago.
Okay, so Chicago went down. What about Ashburn? Just how big was that power supply? ;-)
I know TNS has more facilities, so I'm going to try to get another link out of them gratis.
----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com
Midwest Internet Exchange http://www.midwest-ix.com
-------------------------
From: "Mike Hammett" <voiceops at ics-il.net> To: "voiceops" <voiceops at voiceops.org> Sent: Wednesday, January 19, 2022 4:13:08 PM Subject: Re: [VoiceOps] TNS SS7 Outage
The links have since come back. This is the second major failure of TNS within 6 months. No RFO yet, but I will certainly be on them for some detail.
My Metaswitch support rep verified that (pardon my lack of using the correct terminology, I'm not that familiar with it) our SS7 links never actually went down, it just lost connectivity beyond that. Sorta like my BGP peer stayed up, but I lost the global routing table behind it.
----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com
Midwest Internet Exchange http://www.midwest-ix.com
-------------------------
From: "Mike Hammett" <voiceops at ics-il.net> To: "voiceops" <voiceops at voiceops.org> Sent: Wednesday, January 19, 2022 4:05:32 PM Subject: [VoiceOps] TNS SS7 Outage
Does anyone have any information on the TNS SS7 outage going on now? Both of our A-links go to different facilities (Chicago and Ashburn) over different VPNs (which are up).
TNS has acknowledged that it's a carrier issue. That they're investigating.
----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com
Midwest Internet Exchange http://www.midwest-ix.com
_______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops
_______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops _______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops
Our investigation determined that a router in our Ashburn data center where the STP is located was incorrectly learning the route back to our Ashburn VPN site via the IPX core in Chicago ( though Chicago was not actually in the path). The power issue in Chicago that took down an IPX switch, impacted the path from our Dallas VPN site to the Chicago STP as expected, but the loss of the route to Ashburn was not expected. TNS has corrected the Ashburn routing as an emergency break-fix (non-impacting). TNS engineers have confirmed that the router in Ashburn is now learning the route from our IPX core in Ashburn as designed. Please let us know if you have any questions. ----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest Internet Exchange http://www.midwest-ix.com ----- Original Message ----- From: "Mary Lou Carey" <marylou at backuptelecom.com> To: "Mike Hammett" <voiceops at ics-il.net> Cc: "voiceops" <voiceops at voiceops.org> Sent: Thursday, January 20, 2022 6:23:28 PM Subject: Re: [VoiceOps] TNS SS7 Outage That sounds as believable as the Christmas Day bombing in Nashville that took out local and 911 phone services out all the way up to St Louis. Anyone who's familiar with the network knows St Louis isn't in the same LATA and a building built to withstand natural disasters etc wouldn't be damaged that badly by a bomb going off down the street from the building. You'd better be asking for a link to a different set of STPs because if the power in Chicago took out all 4 of their STPs, they have some serious problems with their network redundancy plan! MARY LOU CAREY BackUP Telecom Consulting Office: 615-791-9969 Cell: 615-796-1111 On 2022-01-20 05:44 AM, Mike Hammett wrote:
Currently the story is that there was a power supply issue in Chicago.
Okay, so Chicago went down. What about Ashburn? Just how big was that power supply? ;-)
I know TNS has more facilities, so I'm going to try to get another link out of them gratis.
----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com
Midwest Internet Exchange http://www.midwest-ix.com
-------------------------
From: "Mike Hammett" <voiceops at ics-il.net> To: "voiceops" <voiceops at voiceops.org> Sent: Wednesday, January 19, 2022 4:13:08 PM Subject: Re: [VoiceOps] TNS SS7 Outage
The links have since come back. This is the second major failure of TNS within 6 months. No RFO yet, but I will certainly be on them for some detail.
My Metaswitch support rep verified that (pardon my lack of using the correct terminology, I'm not that familiar with it) our SS7 links never actually went down, it just lost connectivity beyond that. Sorta like my BGP peer stayed up, but I lost the global routing table behind it.
----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com
Midwest Internet Exchange http://www.midwest-ix.com
-------------------------
From: "Mike Hammett" <voiceops at ics-il.net> To: "voiceops" <voiceops at voiceops.org> Sent: Wednesday, January 19, 2022 4:05:32 PM Subject: [VoiceOps] TNS SS7 Outage
Does anyone have any information on the TNS SS7 outage going on now? Both of our A-links go to different facilities (Chicago and Ashburn) over different VPNs (which are up).
TNS has acknowledged that it's a carrier issue. That they're investigating.
----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com
Midwest Internet Exchange http://www.midwest-ix.com
_______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops
_______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops _______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops
participants (4)
-
jbrower＠signalogic.com -
marylou＠backuptelecom.com -
shawn＠rmrf.us -
voiceops＠ics-il.net