VoiceOps Digest, Vol 63, Issue 3
Jim and Mark - I read the original question to be centered around peering vs Registration interfaces. I get the auto-blacklist argument for Registration interfaces. I struggle to understand the same rationale for peering interfaces. SBCs are supposed to be hardened security devices, but as Mark mentioned - they do take fixes for bugs in the security context. IF the company you're helping doesn't have the expertise to properly configure or maintain their SBC, why introduce risk by not having a router ACL in front of the peering interface(s)? Regards, Russ On Wed, Sep 3, 2014 at 2:08 PM, <voiceops-request at voiceops.org> wrote:
Send VoiceOps mailing list submissions to voiceops at voiceops.org
To subscribe or unsubscribe via the World Wide Web, visit https://puck.nether.net/mailman/listinfo/voiceops or, via email, send a message with subject or body 'help' to voiceops-request at voiceops.org
You can reach the person managing the list at voiceops-owner at voiceops.org
When replying, please edit your Subject line so it is more specific than "Re: Contents of VoiceOps digest..."
Today's Topics:
1. Re: Mitigating SIP threats with SBC policies, configuration settings (Gast, Jim) 2. Mitigating SIP threats with SBC policies, configuration settings (Patrick McNeil)
----------------------------------------------------------------------
Message: 1 Date: Wed, 3 Sep 2014 18:43:10 +0000 From: "Gast, Jim" <jim.gast at tdstelecom.com> To: Mark R Lindsey <lindsey at e-c-group.com>, Robert Nystrom <ronystrom at gmail.com> Cc: "voiceops at voiceops.org" <VoiceOps at voiceops.org> Subject: Re: [VoiceOps] Mitigating SIP threats with SBC policies, configuration settings Message-ID: <24E30310A043EA45BA0B7366E86A3C390EB85666 at cmailbox5> Content-Type: text/plain; charset="us-ascii"
Hi, Robert (and Mark) -
Re: Mitigating threats with SBC policies
I agree with everything Mark said. And the Metaswitch Perimeta also does auto-blacklisting and the end result is comparable to the Acme Packet result.
Re: different errors other than 403 Forbidden when headers have unexpected values . . . very risky.
I agree. The bad guys can catalog the different responses and use those to fine tune attacks to exploit weaknesses unique to your combination of devices. I would like to suggest a solution that I don't deploy (and then tell you why I don't). If you put a traditional firewall in front of your SBC, you can drop all traffic from places like the APNIC IP space. That way the intruder learns nothing. You can also put layer 2 ACLs in your edge routers to drop traffic from implausible sources destined for VoIP SBCs.
I don't put a firewall in front of my SBCs. When high-volume attacks come, the up-front firewall dulls the attack so much that the SBC does not auto-blacklist effectively. In the very bad attacks, the dulled attack might not trip the alarms that it should. Also, if you put a firewall in front of your SBC, you will have to teach the firewall team when to yawn and when not to.
I am sorry if my answer is a little short on quantifiable details. We (kudos to Mark) have be debating whether to front an SBC with an external firewall for years. As far as I remember the answer was always to read up on the auto-blacklisting in your SBC. And then use auto-blacklisting (even though the bad guys may be able to use error responses to figure out what equipment you have).
I wish I had a better answer.
Cheers,
/ Jim Gast, TDS Telecom
From: VoiceOps [mailto:voiceops-bounces at voiceops.org] On Behalf Of Mark R Lindsey Sent: Wednesday, September 03, 2014 11:35 AM To: Robert Nystrom Cc: voiceops at voiceops.org Subject: Re: [VoiceOps] Mitigating SIP threats with SBC policies, configuration settings
Robert -- These are good questions.
On Sep 3, 2014, at 11:49 , Robert Nystrom <ronystrom at gmail.com<mailto: ronystrom at gmail.com>> wrote: First: Customer's ACME is visible from public Internet on udp/5060, and SIP trunk is only being used to interconnect to SIP trunk carrier for inbound and outbound dialing. I've tested the SBC from the Internet and it actually responds to INVITE and REGISTER messages (with 403 Forbidden). They are alsonot supposed to be allowing any REGISTER for remote user MD5 Digest Authentication - but it does respond. Question: Is there any operational need or business usage case that you would see that would make this setup a good idea?
Yes; you often do want to be able to REGISTER with the SBC from random places on the Internet. Does your specific customer need to allow registration from anywhere on the Internet? Maybe not. One popular place to blacklist in advance is APNIC IP space.
The Acme Packet has auto-blacklisting features that can be setup so that if a specific source IP sends several SIP messages without successfully registering or completing a phone call, then the SBC can blacklist the source for a while. E.g., if you don't successfully register within your first three SIP messages, then blacklist you for an hour.
If you do know in advance all the right places from which you 'd be sending SIP to the SBC, then it *is* best to setup the SBC for default-deny so that traffic only from those sources is permitted. That's straightforward to do by matching access-control-trust-levels between the realm-config and the access-control objects.
Second: I have written some SIP software that sends malformed message headers, and have noticed that the SBC responds with different errors other than 403 Forbidden when headers have unexpected values. For example, when I send an INVITE with extra CRLFs, I get a 400 CSeq missing header. When I send a Contact header of "None", I get 400 Invalid Contact. This leads me to believe that the SBC sip parser is parsing all of the SIP message rather than always sending a 403 Forbidden to an IP address sourced from the untrusted public Internet...this also seems to be very risky. Is there a specific security configuration with the policies that you would recommend? It seems like this introduces the risk of DoS and fuzzing attacks if the SBC is parsing more of the SIP message rather than just dropping the message based on invalid source IP. Could lead to cpu and memory issues if the queues are filled from invalid and fuzzed traffic.
The auto-blacklisting functions can help mitigate this risk. In addition to blacklisting based on failure to successfully REGISTER (or do anything else allowed by policy), you can auto-blacklist sources of "invalid signaling". I haven't found a good written definition of "invalid signaling", but I have definitely seen devices that sent slightly-malformed SIP trigger it and get blacklisted. Non-RFC3261-compliant CR's and LF's are definitely a popular way to do it. Bria/EyeBeam/X-Lite's "UDP Keepalives" (0-byte UDP datagrams) have been another way.
Further, there are internal resources (primarily, CPU capacity) allocated differently for trusted sources versus untrusted sources. An untrusted source could be a device that hasn't yet successfully registered , for example, and we might only want to give 2% of total system capacity to all untrusted sources.
To your bigger point, though, Oracle/Acme Packet does try to be a security device, and I know they test it against fuzzers. Their release notes show when they fixed a security bug or a SIPd crash that was found through fuzzer testing. Yes, it means they have to be extremely careful with how they parse the SIP in order to do so safely.
mark at ecg.co<mailto:mark at ecg.co> +1-229-316-0013 http://ecg.co/lindsey
Yes, Robert Nystrom the OP was talking mostly about a peering. And I inadvertently changed the subject. Sorry about that. There are some good reasons to consider auto-blacklisting of peers: -- Sometimes peers have bugs and do nasty things -- Sometimes peers have routing loops (or as RFC3261 calls them, "spirals") -- Sometimes peers decide to do load testing at 10am on a Tuesday -- Sometimes peers get hacked But short of blacklisting, you might just want to limit the rate of traffic they can send, then block their traffic if they exceed the limit. The Acme Packet has an access-control object that lets you configure the default-deny policy. But you suggested putting a router ACL in front of the peering interface. + Upside: Some defense in depth. Maybe if the SBC (or its admin) got something wrong, the firewall (and its admin) won't make the same mistake. - Downside: Divergent change: to reconfigure the IP addresses or UDP ports of the peering you have to do it in multiple places - Downside: You have to prevent the firewall/router from changing anything useful about the packet - Downside: You have to buy and operate and patch two distinct devices with capacity to inspect every packet's IP and UDP headers. Here's an example where that might really matter: If you could depend on the SBC to be the security appliance, you might use a Cisco Catalyst 3560X as the router between the SBC and the Internet. But if you have to buy a router that can actually inspect every packet and keep up with nontrivial workload, you'll need something much more capable and thus expensive.
mark at ecg.co +1-229-316-0013 http://ecg.co/lindsey
On Sep 3, 2014, at 18:23 , par <slocoach at gmail.com> wrote:
Jim and Mark - I read the original question to be centered around peering vs Registration interfaces. I get the auto-blacklist argument for Registration interfaces. I struggle to understand the same rationale for peering interfaces. SBCs are supposed to be hardened security devices, but as Mark mentioned - they do take fixes for bugs in the security context. IF the company you're helping doesn't have the expertise to properly configure or maintain their SBC, why introduce risk by not having a router ACL in front of the peering interface(s)?
Regards, Russ
Message: 1 Date: Wed, 3 Sep 2014 18:43:10 +0000 From: "Gast, Jim" <jim.gast at tdstelecom.com> To: Mark R Lindsey <lindsey at e-c-group.com>, Robert Nystrom <ronystrom at gmail.com> Cc: "voiceops at voiceops.org" <VoiceOps at voiceops.org> Subject: Re: [VoiceOps] Mitigating SIP threats with SBC policies, configuration settings Message-ID: <24E30310A043EA45BA0B7366E86A3C390EB85666 at cmailbox5> Content-Type: text/plain; charset="us-ascii"
Hi, Robert (and Mark) -
Re: Mitigating threats with SBC policies
I agree with everything Mark said. And the Metaswitch Perimeta also does auto-blacklisting and the end result is comparable to the Acme Packet result.
Re: different errors other than 403 Forbidden when headers have unexpected values . . . very risky.
I agree. The bad guys can catalog the different responses and use those to fine tune attacks to exploit weaknesses unique to your combination of devices. I would like to suggest a solution that I don't deploy (and then tell you why I don't). If you put a traditional firewall in front of your SBC, you can drop all traffic from places like the APNIC IP space. That way the intruder learns nothing. You can also put layer 2 ACLs in your edge routers to drop traffic from implausible sources destined for VoIP SBCs.
I don't put a firewall in front of my SBCs. When high-volume attacks come, the up-front firewall dulls the attack so much that the SBC does not auto-blacklist effectively. In the very bad attacks, the dulled attack might not trip the alarms that it should. Also, if you put a firewall in front of your SBC, you will have to teach the firewall team when to yawn and when not to.
I am sorry if my answer is a little short on quantifiable details. We (kudos to Mark) have be debating whether to front an SBC with an external firewall for years. As far as I remember the answer was always to read up on the auto-blacklisting in your SBC. And then use auto-blacklisting (even though the bad guys may be able to use error responses to figure out what equipment you have).
I wish I had a better answer.
Cheers,
/ Jim Gast, TDS Telecom
From: VoiceOps [mailto:voiceops-bounces at voiceops.org] On Behalf Of Mark R Lindsey Sent: Wednesday, September 03, 2014 11:35 AM To: Robert Nystrom Cc: voiceops at voiceops.org Subject: Re: [VoiceOps] Mitigating SIP threats with SBC policies, configuration settings
Robert -- These are good questions.
On Sep 3, 2014, at 11:49 , Robert Nystrom <ronystrom at gmail.com<mailto:ronystrom at gmail.com>> wrote: First: Customer's ACME is visible from public Internet on udp/5060, and SIP trunk is only being used to interconnect to SIP trunk carrier for inbound and outbound dialing. I've tested the SBC from the Internet and it actually responds to INVITE and REGISTER messages (with 403 Forbidden). They are alsonot supposed to be allowing any REGISTER for remote user MD5 Digest Authentication - but it does respond. Question: Is there any operational need or business usage case that you would see that would make this setup a good idea?
Yes; you often do want to be able to REGISTER with the SBC from random places on the Internet. Does your specific customer need to allow registration from anywhere on the Internet? Maybe not. One popular place to blacklist in advance is APNIC IP space.
The Acme Packet has auto-blacklisting features that can be setup so that if a specific source IP sends several SIP messages without successfully registering or completing a phone call, then the SBC can blacklist the source for a while. E.g., if you don't successfully register within your first three SIP messages, then blacklist you for an hour.
If you do know in advance all the right places from which you 'd be sending SIP to the SBC, then it *is* best to setup the SBC for default-deny so that traffic only from those sources is permitted. That's straightforward to do by matching access-control-trust-levels between the realm-config and the access-control objects.
Second: I have written some SIP software that sends malformed message headers, and have noticed that the SBC responds with different errors other than 403 Forbidden when headers have unexpected values. For example, when I send an INVITE with extra CRLFs, I get a 400 CSeq missing header. When I send a Contact header of "None", I get 400 Invalid Contact. This leads me to believe that the SBC sip parser is parsing all of the SIP message rather than always sending a 403 Forbidden to an IP address sourced from the untrusted public Internet...this also seems to be very risky. Is there a specific security configuration with the policies that you would recommend? It seems like this introduces the risk of DoS and fuzzing attacks if the SBC is parsing more of the SIP message rather than just dropping the message based on invalid source IP. Could lead to cpu and memory issues if the queues are filled from invalid and fuzzed traffic.
The auto-blacklisting functions can help mitigate this risk. In addition to blacklisting based on failure to successfully REGISTER (or do anything else allowed by policy), you can auto-blacklist sources of "invalid signaling". I haven't found a good written definition of "invalid signaling", but I have definitely seen devices that sent slightly-malformed SIP trigger it and get blacklisted. Non-RFC3261-compliant CR's and LF's are definitely a popular way to do it. Bria/EyeBeam/X-Lite's "UDP Keepalives" (0-byte UDP datagrams) have been another way.
Further, there are internal resources (primarily, CPU capacity) allocated differently for trusted sources versus untrusted sources. An untrusted source could be a device that hasn't yet successfully registered , for example, and we might only want to give 2% of total system capacity to all untrusted sources.
To your bigger point, though, Oracle/Acme Packet does try to be a security device, and I know they test it against fuzzers. Their release notes show when they fixed a security bug or a SIPd crash that was found through fuzzer testing. Yes, it means they have to be extremely careful with how they parse the SIP in order to do so safely.
mark at ecg.co<mailto:mark at ecg.co> +1-229-316-0013 http://ecg.co/lindsey
All good points Mark but make sure to separate the layer 4 (transport - IP/port) vs. layer 7 (application - protocol validation and content inspection). All routers an ITSP would deploy in a data center will do layer 4 ACLs which is pretty lightweight and won't touch the packet content when allowing whereas a traditional ITSP firewall by nature will want to inspect known protocols contents and potentially change them depending on settings. My somewhat biased opinion having never had any luck with firewalls and VoIP, is that layer 4 ACLs are a no-brainer must have at the router level. The SBC should be hardened as if the ACLs didn't exist but frankly commercial SBCs are very expensive and CPU time on them should be considered an expensive resource. Anything you can do to easily offload known invalid/unwanted traffic at the layer 4 level is a huge win - the sheer volume of bots snooping for live systems alone should justify this. The router is there anyway so this is a win no matter what and there is zero reason to not maintain this especially if you only expect traffic from specific geographic regions as others have alluded to. As I said I am biased against firewalls in this scenario. By the time you look at the cost of a firewall that can handle the packet inspection at the packets per second rate required for any kind of volume, you could have bought yourself another SBC. If money is not an issue then a commercial IPS is more useful than a traditional firewall to make you feel warm and fuzzy. Just my 2 cents - firewalls in front of the SBC aren't worth the effort unless you have a team dedicated to just firewalls. Others experiences may/will vary. Now I will concede that my biggest issues with firewalls have been with packets per second handling and their learning curve. Inevitably you will break things when inserting a firewall in front of anything which is the learning curve. Firewalls also tend to be underpowered at the CPU and memory levels for anything cost effective to handle the packets per second traffic of a high volume system. That being said, if you have the resources to continually manage a firewall and money to support that initiative there is no logical reason NOT to put a firewall in front of your SBCs especially if you subscribe to the "more layers are better" philosophy. Jesse From: Mark R Lindsey [mailto:lindsey at e-c-group.com] Sent: Wednesday, September 03, 2014 10:16 PM To: Russ Penar Cc: voiceops at voiceops.org Subject: Re: [VoiceOps] VoiceOps Digest, Vol 63, Issue 3 Yes, Robert Nystrom the OP was talking mostly about a peering. And I inadvertently changed the subject. Sorry about that. There are some good reasons to consider auto-blacklisting of peers: -- Sometimes peers have bugs and do nasty things -- Sometimes peers have routing loops (or as RFC3261 calls them, "spirals") -- Sometimes peers decide to do load testing at 10am on a Tuesday -- Sometimes peers get hacked But short of blacklisting, you might just want to limit the rate of traffic they can send, then block their traffic if they exceed the limit. The Acme Packet has an access-control object that lets you configure the default-deny policy. But you suggested putting a router ACL in front of the peering interface. + Upside: Some defense in depth. Maybe if the SBC (or its admin) got something wrong, the firewall (and its admin) won't make the same mistake. - Downside: Divergent change: to reconfigure the IP addresses or UDP ports of the peering you have to do it in multiple places - Downside: You have to prevent the firewall/router from changing anything useful about the packet - Downside: You have to buy and operate and patch two distinct devices with capacity to inspect every packet's IP and UDP headers. Here's an example where that might really matter: If you could depend on the SBC to be the security appliance, you might use a Cisco Catalyst 3560X as the router between the SBC and the Internet. But if you have to buy a router that can actually inspect every packet and keep up with nontrivial workload, you'll need something much more capable and thus expensive.
mark at ecg.co<mailto:mark at ecg.co> +1-229-316-0013 http://ecg.co/lindsey
On Sep 3, 2014, at 18:23 , par <slocoach at gmail.com<mailto:slocoach at gmail.com>> wrote: Jim and Mark - I read the original question to be centered around peering vs Registration interfaces. I get the auto-blacklist argument for Registration interfaces. I struggle to understand the same rationale for peering interfaces. SBCs are supposed to be hardened security devices, but as Mark mentioned - they do take fixes for bugs in the security context. IF the company you're helping doesn't have the expertise to properly configure or maintain their SBC, why introduce risk by not having a router ACL in front of the peering interface(s)? Regards, Russ Message: 1 Date: Wed, 3 Sep 2014 18:43:10 +0000 From: "Gast, Jim" <jim.gast at tdstelecom.com<mailto:jim.gast at tdstelecom.com>> To: Mark R Lindsey <lindsey at e-c-group.com<mailto:lindsey at e-c-group.com>>, Robert Nystrom <ronystrom at gmail.com<mailto:ronystrom at gmail.com>> Cc: "voiceops at voiceops.org<mailto:voiceops at voiceops.org>" <VoiceOps at voiceops.org<mailto:VoiceOps at voiceops.org>> Subject: Re: [VoiceOps] Mitigating SIP threats with SBC policies, configuration settings Message-ID: <24E30310A043EA45BA0B7366E86A3C390EB85666 at cmailbox5> Content-Type: text/plain; charset="us-ascii" Hi, Robert (and Mark) - Re: Mitigating threats with SBC policies I agree with everything Mark said. And the Metaswitch Perimeta also does auto-blacklisting and the end result is comparable to the Acme Packet result. Re: different errors other than 403 Forbidden when headers have unexpected values . . . very risky. I agree. The bad guys can catalog the different responses and use those to fine tune attacks to exploit weaknesses unique to your combination of devices. I would like to suggest a solution that I don't deploy (and then tell you why I don't). If you put a traditional firewall in front of your SBC, you can drop all traffic from places like the APNIC IP space. That way the intruder learns nothing. You can also put layer 2 ACLs in your edge routers to drop traffic from implausible sources destined for VoIP SBCs. I don't put a firewall in front of my SBCs. When high-volume attacks come, the up-front firewall dulls the attack so much that the SBC does not auto-blacklist effectively. In the very bad attacks, the dulled attack might not trip the alarms that it should. Also, if you put a firewall in front of your SBC, you will have to teach the firewall team when to yawn and when not to. I am sorry if my answer is a little short on quantifiable details. We (kudos to Mark) have be debating whether to front an SBC with an external firewall for years. As far as I remember the answer was always to read up on the auto-blacklisting in your SBC. And then use auto-blacklisting (even though the bad guys may be able to use error responses to figure out what equipment you have). I wish I had a better answer. Cheers, / Jim Gast, TDS Telecom From: VoiceOps [mailto:voiceops-bounces at voiceops.org<mailto:voiceops-bounces at voiceops.org>] On Behalf Of Mark R Lindsey Sent: Wednesday, September 03, 2014 11:35 AM To: Robert Nystrom Cc: voiceops at voiceops.org<mailto:voiceops at voiceops.org> Subject: Re: [VoiceOps] Mitigating SIP threats with SBC policies, configuration settings Robert -- These are good questions. On Sep 3, 2014, at 11:49 , Robert Nystrom <ronystrom at gmail.com<mailto:ronystrom at gmail.com><mailto:ronystrom at gmail.com<mailto:ronystrom at gmail.com>>> wrote: First: Customer's ACME is visible from public Internet on udp/5060, and SIP trunk is only being used to interconnect to SIP trunk carrier for inbound and outbound dialing. I've tested the SBC from the Internet and it actually responds to INVITE and REGISTER messages (with 403 Forbidden). They are alsonot supposed to be allowing any REGISTER for remote user MD5 Digest Authentication - but it does respond. Question: Is there any operational need or business usage case that you would see that would make this setup a good idea? Yes; you often do want to be able to REGISTER with the SBC from random places on the Internet. Does your specific customer need to allow registration from anywhere on the Internet? Maybe not. One popular place to blacklist in advance is APNIC IP space. The Acme Packet has auto-blacklisting features that can be setup so that if a specific source IP sends several SIP messages without successfully registering or completing a phone call, then the SBC can blacklist the source for a while. E.g., if you don't successfully register within your first three SIP messages, then blacklist you for an hour. If you do know in advance all the right places from which you 'd be sending SIP to the SBC, then it *is* best to setup the SBC for default-deny so that traffic only from those sources is permitted. That's straightforward to do by matching access-control-trust-levels between the realm-config and the access-control objects. Second: I have written some SIP software that sends malformed message headers, and have noticed that the SBC responds with different errors other than 403 Forbidden when headers have unexpected values. For example, when I send an INVITE with extra CRLFs, I get a 400 CSeq missing header. When I send a Contact header of "None", I get 400 Invalid Contact. This leads me to believe that the SBC sip parser is parsing all of the SIP message rather than always sending a 403 Forbidden to an IP address sourced from the untrusted public Internet...this also seems to be very risky. Is there a specific security configuration with the policies that you would recommend? It seems like this introduces the risk of DoS and fuzzing attacks if the SBC is parsing more of the SIP message rather than just dropping the message based on invalid source IP. Could lead to cpu and memory issues if the queues are filled from invalid and fuzzed traffic. The auto-blacklisting functions can help mitigate this risk. In addition to blacklisting based on failure to successfully REGISTER (or do anything else allowed by policy), you can auto-blacklist sources of "invalid signaling". I haven't found a good written definition of "invalid signaling", but I have definitely seen devices that sent slightly-malformed SIP trigger it and get blacklisted. Non-RFC3261-compliant CR's and LF's are definitely a popular way to do it. Bria/EyeBeam/X-Lite's "UDP Keepalives" (0-byte UDP datagrams) have been another way. Further, there are internal resources (primarily, CPU capacity) allocated differently for trusted sources versus untrusted sources. An untrusted source could be a device that hasn't yet successfully registered , for example, and we might only want to give 2% of total system capacity to all untrusted sources. To your bigger point, though, Oracle/Acme Packet does try to be a security device, and I know they test it against fuzzers. Their release notes show when they fixed a security bug or a SIPd crash that was found through fuzzer testing. Yes, it means they have to be extremely careful with how they parse the SIP in order to do so safely.
mark at ecg.co<mailto:mark at ecg.co><mailto:mark at ecg.co<mailto:mark at ecg.co>> +1-229-316-0013<tel:%2B1-229-316-0013> http://ecg.co/lindsey
Hi, Russ - I agree. It adds to safety to put ACLs in the router(s) to discard hostile packets destined for the peering interface(s) of your SBC(s). Mark correctly mentioned the downsides. Router ACLs are simple and effective. Jim Gast, Network Architecture, TDS Telecom Good, Fast, Cheap. Pick 2. From: VoiceOps [mailto:voiceops-bounces at voiceops.org] On Behalf Of Mark R Lindsey Sent: Wednesday, September 03, 2014 10:16 PM To: Russ Penar Cc: voiceops at voiceops.org Subject: Re: [VoiceOps] VoiceOps Digest, Vol 63, Issue 3 Yes, Robert Nystrom the OP was talking mostly about a peering. And I inadvertently changed the subject. Sorry about that. There are some good reasons to consider auto-blacklisting of peers: -- Sometimes peers have bugs and do nasty things -- Sometimes peers have routing loops (or as RFC3261 calls them, "spirals") -- Sometimes peers decide to do load testing at 10am on a Tuesday -- Sometimes peers get hacked But short of blacklisting, you might just want to limit the rate of traffic they can send, then block their traffic if they exceed the limit. The Acme Packet has an access-control object that lets you configure the default-deny policy. But you suggested putting a router ACL in front of the peering interface. + Upside: Some defense in depth. Maybe if the SBC (or its admin) got something wrong, the firewall (and its admin) won't make the same mistake. - Downside: Divergent change: to reconfigure the IP addresses or UDP ports of the peering you have to do it in multiple places - Downside: You have to prevent the firewall/router from changing anything useful about the packet - Downside: You have to buy and operate and patch two distinct devices with capacity to inspect every packet's IP and UDP headers. Here's an example where that might really matter: If you could depend on the SBC to be the security appliance, you might use a Cisco Catalyst 3560X as the router between the SBC and the Internet. But if you have to buy a router that can actually inspect every packet and keep up with nontrivial workload, you'll need something much more capable and thus expensive.
mark at ecg.co<mailto:mark at ecg.co> +1-229-316-0013 http://ecg.co/lindsey
On Sep 3, 2014, at 18:23 , par <slocoach at gmail.com<mailto:slocoach at gmail.com>> wrote: Jim and Mark - I read the original question to be centered around peering vs Registration interfaces. I get the auto-blacklist argument for Registration interfaces. I struggle to understand the same rationale for peering interfaces. SBCs are supposed to be hardened security devices, but as Mark mentioned - they do take fixes for bugs in the security context. IF the company you're helping doesn't have the expertise to properly configure or maintain their SBC, why introduce risk by not having a router ACL in front of the peering interface(s)? Regards, Russ Message: 1 Date: Wed, 3 Sep 2014 18:43:10 +0000 From: "Gast, Jim" <jim.gast at tdstelecom.com<mailto:jim.gast at tdstelecom.com>> To: Mark R Lindsey <lindsey at e-c-group.com<mailto:lindsey at e-c-group.com>>, Robert Nystrom <ronystrom at gmail.com<mailto:ronystrom at gmail.com>> Cc: "voiceops at voiceops.org<mailto:voiceops at voiceops.org>" <VoiceOps at voiceops.org<mailto:VoiceOps at voiceops.org>> Subject: Re: [VoiceOps] Mitigating SIP threats with SBC policies, configuration settings Message-ID: <24E30310A043EA45BA0B7366E86A3C390EB85666 at cmailbox5> Content-Type: text/plain; charset="us-ascii" Hi, Robert (and Mark) - Re: Mitigating threats with SBC policies I agree with everything Mark said. And the Metaswitch Perimeta also does auto-blacklisting and the end result is comparable to the Acme Packet result. Re: different errors other than 403 Forbidden when headers have unexpected values . . . very risky. I agree. The bad guys can catalog the different responses and use those to fine tune attacks to exploit weaknesses unique to your combination of devices. I would like to suggest a solution that I don't deploy (and then tell you why I don't). If you put a traditional firewall in front of your SBC, you can drop all traffic from places like the APNIC IP space. That way the intruder learns nothing. You can also put layer 2 ACLs in your edge routers to drop traffic from implausible sources destined for VoIP SBCs. I don't put a firewall in front of my SBCs. When high-volume attacks come, the up-front firewall dulls the attack so much that the SBC does not auto-blacklist effectively. In the very bad attacks, the dulled attack might not trip the alarms that it should. Also, if you put a firewall in front of your SBC, you will have to teach the firewall team when to yawn and when not to. I am sorry if my answer is a little short on quantifiable details. We (kudos to Mark) have be debating whether to front an SBC with an external firewall for years. As far as I remember the answer was always to read up on the auto-blacklisting in your SBC. And then use auto-blacklisting (even though the bad guys may be able to use error responses to figure out what equipment you have). I wish I had a better answer. Cheers, / Jim Gast, TDS Telecom From: VoiceOps [mailto:voiceops-bounces at voiceops.org<mailto:voiceops-bounces at voiceops.org>] On Behalf Of Mark R Lindsey Sent: Wednesday, September 03, 2014 11:35 AM To: Robert Nystrom Cc: voiceops at voiceops.org<mailto:voiceops at voiceops.org> Subject: Re: [VoiceOps] Mitigating SIP threats with SBC policies, configuration settings Robert -- These are good questions. On Sep 3, 2014, at 11:49 , Robert Nystrom <ronystrom at gmail.com<mailto:ronystrom at gmail.com><mailto:ronystrom at gmail.com<mailto:ronystrom at gmail.com>>> wrote: First: Customer's ACME is visible from public Internet on udp/5060, and SIP trunk is only being used to interconnect to SIP trunk carrier for inbound and outbound dialing. I've tested the SBC from the Internet and it actually responds to INVITE and REGISTER messages (with 403 Forbidden). They are alsonot supposed to be allowing any REGISTER for remote user MD5 Digest Authentication - but it does respond. Question: Is there any operational need or business usage case that you would see that would make this setup a good idea? Yes; you often do want to be able to REGISTER with the SBC from random places on the Internet. Does your specific customer need to allow registration from anywhere on the Internet? Maybe not. One popular place to blacklist in advance is APNIC IP space. The Acme Packet has auto-blacklisting features that can be setup so that if a specific source IP sends several SIP messages without successfully registering or completing a phone call, then the SBC can blacklist the source for a while. E.g., if you don't successfully register within your first three SIP messages, then blacklist you for an hour. If you do know in advance all the right places from which you 'd be sending SIP to the SBC, then it *is* best to setup the SBC for default-deny so that traffic only from those sources is permitted. That's straightforward to do by matching access-control-trust-levels between the realm-config and the access-control objects. Second: I have written some SIP software that sends malformed message headers, and have noticed that the SBC responds with different errors other than 403 Forbidden when headers have unexpected values. For example, when I send an INVITE with extra CRLFs, I get a 400 CSeq missing header. When I send a Contact header of "None", I get 400 Invalid Contact. This leads me to believe that the SBC sip parser is parsing all of the SIP message rather than always sending a 403 Forbidden to an IP address sourced from the untrusted public Internet...this also seems to be very risky. Is there a specific security configuration with the policies that you would recommend? It seems like this introduces the risk of DoS and fuzzing attacks if the SBC is parsing more of the SIP message rather than just dropping the message based on invalid source IP. Could lead to cpu and memory issues if the queues are filled from invalid and fuzzed traffic. The auto-blacklisting functions can help mitigate this risk. In addition to blacklisting based on failure to successfully REGISTER (or do anything else allowed by policy), you can auto-blacklist sources of "invalid signaling". I haven't found a good written definition of "invalid signaling", but I have definitely seen devices that sent slightly-malformed SIP trigger it and get blacklisted. Non-RFC3261-compliant CR's and LF's are definitely a popular way to do it. Bria/EyeBeam/X-Lite's "UDP Keepalives" (0-byte UDP datagrams) have been another way. Further, there are internal resources (primarily, CPU capacity) allocated differently for trusted sources versus untrusted sources. An untrusted source could be a device that hasn't yet successfully registered , for example, and we might only want to give 2% of total system capacity to all untrusted sources. To your bigger point, though, Oracle/Acme Packet does try to be a security device, and I know they test it against fuzzers. Their release notes show when they fixed a security bug or a SIPd crash that was found through fuzzer testing. Yes, it means they have to be extremely careful with how they parse the SIP in order to do so safely.
mark at ecg.co<mailto:mark at ecg.co><mailto:mark at ecg.co<mailto:mark at ecg.co>> +1-229-316-0013<tel:%2B1-229-316-0013> http://ecg.co/lindsey
Thanks Mark, Jesse, and Jim, The blacklist feature on the peering makes sense given Mark's examples. I fully agree with the divergent change downside, the others are debatable as Jesse pointed out. I have not studied the resource drain of adding a simple ACL to a router - but expect the impact to be minimal vs requiring a hardware upgrade. I also have not studied the impact of attacks against a properly configured SBC, does it eat up extra resrouces or is sluffing off the attack traffic trivial? I don't know. Regards, Russ On Thu, Sep 4, 2014 at 9:49 AM, Gast, Jim <jim.gast at tdstelecom.com> wrote:
Hi, Russ ?
I agree. It adds to safety to put ACLs in the router(s) to discard hostile packets destined for the peering interface(s) of your SBC(s). Mark correctly mentioned the downsides. Router ACLs are simple and effective.
Jim Gast, Network Architecture, TDS Telecom
*Good, Fast, Cheap. Pick 2.*
*From:* VoiceOps [mailto:voiceops-bounces at voiceops.org] *On Behalf Of *Mark R Lindsey
*Sent:* Wednesday, September 03, 2014 10:16 PM *To:* Russ Penar *Cc:* voiceops at voiceops.org *Subject:* Re: [VoiceOps] VoiceOps Digest, Vol 63, Issue 3
Yes, Robert Nystrom the OP was talking mostly about a peering. And I inadvertently changed the subject. Sorry about that.
There are some good reasons to consider auto-blacklisting of peers:
-- Sometimes peers have bugs and do nasty things
-- Sometimes peers have routing loops (or as RFC3261 calls them, "spirals")
-- Sometimes peers decide to do load testing at 10am on a Tuesday
-- Sometimes peers get hacked
But short of blacklisting, you might just want to limit the rate of traffic they can send, then block their traffic if they exceed the limit.
The Acme Packet has an access-control object that lets you configure the default-deny policy. But you suggested putting a router ACL in front of the peering interface.
+ Upside: Some defense in depth. Maybe if the SBC (or its admin) got something wrong, the firewall (and its admin) won't make the same mistake.
- Downside: Divergent change: to reconfigure the IP addresses or UDP ports of the peering you have to do it in multiple places
- Downside: You have to prevent the firewall/router from changing anything useful about the packet
- Downside: You have to buy and operate and patch two distinct devices with capacity to inspect every packet's IP and UDP headers. Here's an example where that might really matter: If you could depend on the SBC to be the security appliance, you might use a Cisco Catalyst 3560X as the router between the SBC and the Internet. But if you have to buy a router that can actually inspect every packet and keep up with nontrivial workload, you'll need something much more capable and thus expensive.
*>>> mark at ecg.co <mark at ecg.co> +1-229-316-0013 <%2B1-229-316-0013> http://ecg.co/lindsey <http://ecg.co/lindsey>*
On Sep 3, 2014, at 18:23 , par <slocoach at gmail.com> wrote:
Jim and Mark - I read the original question to be centered around peering vs Registration interfaces. I get the auto-blacklist argument for Registration interfaces. I struggle to understand the same rationale for peering interfaces. SBCs are supposed to be hardened security devices, but as Mark mentioned - they do take fixes for bugs in the security context. IF the company you're helping doesn't have the expertise to properly configure or maintain their SBC, why introduce risk by not having a router ACL in front of the peering interface(s)?
Regards, Russ
Message: 1 Date: Wed, 3 Sep 2014 18:43:10 +0000 From: "Gast, Jim" <jim.gast at tdstelecom.com> To: Mark R Lindsey <lindsey at e-c-group.com>, Robert Nystrom <ronystrom at gmail.com> Cc: "voiceops at voiceops.org" <VoiceOps at voiceops.org> Subject: Re: [VoiceOps] Mitigating SIP threats with SBC policies, configuration settings Message-ID: <24E30310A043EA45BA0B7366E86A3C390EB85666 at cmailbox5> Content-Type: text/plain; charset="us-ascii"
Hi, Robert (and Mark) -
Re: Mitigating threats with SBC policies
I agree with everything Mark said. And the Metaswitch Perimeta also does auto-blacklisting and the end result is comparable to the Acme Packet result.
Re: different errors other than 403 Forbidden when headers have unexpected values . . . very risky.
I agree. The bad guys can catalog the different responses and use those to fine tune attacks to exploit weaknesses unique to your combination of devices. I would like to suggest a solution that I don't deploy (and then tell you why I don't). If you put a traditional firewall in front of your SBC, you can drop all traffic from places like the APNIC IP space. That way the intruder learns nothing. You can also put layer 2 ACLs in your edge routers to drop traffic from implausible sources destined for VoIP SBCs.
I don't put a firewall in front of my SBCs. When high-volume attacks come, the up-front firewall dulls the attack so much that the SBC does not auto-blacklist effectively. In the very bad attacks, the dulled attack might not trip the alarms that it should. Also, if you put a firewall in front of your SBC, you will have to teach the firewall team when to yawn and when not to.
I am sorry if my answer is a little short on quantifiable details. We (kudos to Mark) have be debating whether to front an SBC with an external firewall for years. As far as I remember the answer was always to read up on the auto-blacklisting in your SBC. And then use auto-blacklisting (even though the bad guys may be able to use error responses to figure out what equipment you have).
I wish I had a better answer.
Cheers,
/ Jim Gast, TDS Telecom
From: VoiceOps [mailto:voiceops-bounces at voiceops.org] On Behalf Of Mark R Lindsey Sent: Wednesday, September 03, 2014 11:35 AM To: Robert Nystrom Cc: voiceops at voiceops.org Subject: Re: [VoiceOps] Mitigating SIP threats with SBC policies, configuration settings
Robert -- These are good questions.
On Sep 3, 2014, at 11:49 , Robert Nystrom <ronystrom at gmail.com<mailto: ronystrom at gmail.com>> wrote: First: Customer's ACME is visible from public Internet on udp/5060, and SIP trunk is only being used to interconnect to SIP trunk carrier for inbound and outbound dialing. I've tested the SBC from the Internet and it actually responds to INVITE and REGISTER messages (with 403 Forbidden). They are alsonot supposed to be allowing any REGISTER for remote user MD5 Digest Authentication - but it does respond. Question: Is there any operational need or business usage case that you would see that would make this setup a good idea?
Yes; you often do want to be able to REGISTER with the SBC from random places on the Internet. Does your specific customer need to allow registration from anywhere on the Internet? Maybe not. One popular place to blacklist in advance is APNIC IP space.
The Acme Packet has auto-blacklisting features that can be setup so that if a specific source IP sends several SIP messages without successfully registering or completing a phone call, then the SBC can blacklist the source for a while. E.g., if you don't successfully register within your first three SIP messages, then blacklist you for an hour.
If you do know in advance all the right places from which you 'd be sending SIP to the SBC, then it *is* best to setup the SBC for default-deny so that traffic only from those sources is permitted. That's straightforward to do by matching access-control-trust-levels between the realm-config and the access-control objects.
Second: I have written some SIP software that sends malformed message headers, and have noticed that the SBC responds with different errors other than 403 Forbidden when headers have unexpected values. For example, when I send an INVITE with extra CRLFs, I get a 400 CSeq missing header. When I send a Contact header of "None", I get 400 Invalid Contact. This leads me to believe that the SBC sip parser is parsing all of the SIP message rather than always sending a 403 Forbidden to an IP address sourced from the untrusted public Internet...this also seems to be very risky. Is there a specific security configuration with the policies that you would recommend? It seems like this introduces the risk of DoS and fuzzing attacks if the SBC is parsing more of the SIP message rather than just dropping the message based on invalid source IP. Could lead to cpu and memory issues if the queues are filled from invalid and fuzzed traffic.
The auto-blacklisting functions can help mitigate this risk. In addition to blacklisting based on failure to successfully REGISTER (or do anything else allowed by policy), you can auto-blacklist sources of "invalid signaling". I haven't found a good written definition of "invalid signaling", but I have definitely seen devices that sent slightly-malformed SIP trigger it and get blacklisted. Non-RFC3261-compliant CR's and LF's are definitely a popular way to do it. Bria/EyeBeam/X-Lite's "UDP Keepalives" (0-byte UDP datagrams) have been another way.
Further, there are internal resources (primarily, CPU capacity) allocated differently for trusted sources versus untrusted sources. An untrusted source could be a device that hasn't yet successfully registered , for example, and we might only want to give 2% of total system capacity to all untrusted sources.
To your bigger point, though, Oracle/Acme Packet does try to be a security device, and I know they test it against fuzzers. Their release notes show when they fixed a security bug or a SIPd crash that was found through fuzzer testing. Yes, it means they have to be extremely careful with how they parse the SIP in order to do so safely.
mark at ecg.co<mailto:mark at ecg.co> +1-229-316-0013 http://ecg.co/lindsey
participants (4)
-
jhoward＠ShoreTel.com -
jim.gast＠tdstelecom.com -
lindsey＠e-c-group.com -
slocoach＠gmail.com