On Sat Aug 01, 2009 at 04:23:57PM -0500, Nathan Stratton wrote:

...

Yup! $dayjob runs a VoIP platform based around Broadworks and Asterisk.

Interesting combo.

It works well - we use Broadworks for all the call routing, and then bolted on Asterisk to do all the IVRs. Simon

On Aug 1, 2009, at 5:31 PM, Alex Balashov wrote:

Not that I would recommend using Asterisk as a general-purpose call router or transit element or anything resembling a switch.

Nor would I, but the setup I inherited is doing just that (registrations handled by a commercial SBC, codec based routing directly to an asterisk box or a cluster of Dialogic IMG 1010s, Asterisk using Realtime and PHP-AGI hitting Postgres for routing) and I'm easily passing 1.2 mil minutes a day. As mentioned in another thread, I do need 15 asterisk boxen to pull this off, but it works. And once you get things together, its surprisingly stable. You just need to jump off the upgrade bandwagon, pick one, make it work, and stick with it. Daryl

I have, however, seriously considered using SER/OpenSER/whatever-it's-called- this-week as call routing platform, backing off to Asterisk for any application stuff. We've got a commercial SBC to deal with the front-end registrations, etc.

That is the way that the service delivery platforms we help engineer work. The OpenSER technology stack is our bread and butter, and it's what we do 90% of the time. It is, however, a relatively abstruse and mildly programmatic art - not out-of-the-box / turn-key like Asterisk at all. -- Alex Balashov Evariste Systems Web : http://www.evaristesys.com/ Tel : (+1) (678) 954-0670 Direct : (+1) (678) 954-0671 Mobile : (+1) (678) 237-1775

Alex Balashov wrote:

Daryl G. Jurbala wrote:

...

Nor would I, but the setup I inherited is doing just that (registrations handled by a commercial SBC, codec based routing directly to an asterisk box or a cluster of Dialogic IMG 1010s, Asterisk using Realtime and PHP-AGI hitting Postgres for routing) and I'm easily passing 1.2 mil minutes a day. As mentioned in another thread, I do need 15 asterisk boxen to pull this off, but it works. And once you get things together, its surprisingly stable. You just need to jump off the upgrade bandwagon, pick one, make it work, and stick with it.

I don't mean to sound derogatory, but what is this enormous fixation that people have with building AGI scripts in PHP-AGI?

It's nothing personal; lots of my customers do it, it seems widely prevalent in the ITSP cottage industry.

PHP is a language designed for serving dynamic web pages; of all the runtime environments suitable for the development of standalone utilities, it is absolutely the worst and most perverse one you could use.

Secondly, calling AGI(somescript.php) causes the entire PHP interpreter and runtime library environment to be loaded *every* *single* *time* *the* *script* *is* *called*. Also, there's no application server or back end to provide pooling of persistent database connections on demand; every time you run it, it goes through the entire process of establishing a connection to PostgreSQL. Every time. Over and over.

I can only conclude that the reason PHP is so often used is because call shops hire developers chiefly with the aim of web portal/interface design in mind, and when it comes time to develop some of the backend logic used within the actual platform -- "Well, uh, we have a bunch of web developers who know PHP, so let's use that." It's one of those succinct examples of suit vs. technocracy decisions; it's a conclusion some "business guy" with "vision" would see as a very rationale instance of code reuse, economies of scale, and return on talent pool, but makes no sense from an engineering perspective.

It'll work for 1.2m minutes, but this won't cut it for high volumes of call setups.

The proper approach here would be to build a minor piece of middleware: a FastAGI service that provides AGI command responses over a network socket. Simultaneously, the service can maintain persistent database connections in order to speed up that process. It also gives you the ability to centralise all your call control and business layer logic into one "logic controller" and essentially utilise the Asterisk boxes as "dumb computing resources." In the most extreme and rewarding case, you would hardly have a dial plan; all aspects of the user experience would be dispatched through FastAGI, and any hostwise application-level constructs (such as Asterisk call Queues with a capital Q) would be parallelised and simulated through AGI-based call control (i.e. MoH, wait time announcements, etc.) Then you can just "farm" the work out across a cluster of Asterisk boxes of arbitrary size and get higher returns to scale. You would also not need to reprovision them every time you make a logic change.

You will also reduce your per-port cost requirement by making the process more efficient. It may not be a big deal because of the low price of commodity hardware, but every little bit helps, especially on a large scale. At that point, the limitation you run into becomes more the number of concurrent calls that Asterisk can reliably maintain inherently, rather than the enormous weight and latency of your setup process. That's a better place to be. -- Alex Balashov Evariste Systems Web : http://www.evaristesys.com/ Tel : (+1) (678) 954-0670 Direct : (+1) (678) 954-0671 Mobile : (+1) (678) 237-1775

Daryl G. Jurbala wrote:

On Aug 1, 2009, at 7:12 PM, Alex Balashov wrote:

...

I don't mean to sound derogatory, but what is this enormous fixation that people have with building AGI scripts in PHP-AGI?

Familiarity with web programming. Again, this is an inherited setup, not how I would design it from a clean slate.

Understood. I didn't mean it personally; I was sort of rhetorically inquiring as to the avenue of thinking taken by the folks preceding you who designed the setup that way, not indicting your willingness to maintain it in that state.

The rest of your post just makes you seem like you have a chip on your shoulder, and that there are only certain ways to do thing, and those are what you deem appropriate.

On a qualitative level, it certainly irritates me, but no, it's not really a chip on my shoulder as much as a very starkly significant trend I've identified in the course of my work. I'd say about 90% of my business comes from helping ITSPs who started off on naive, mostly (but not necessarily) Asterisk-based approaches scale upward once they hit the point where that approach - easy and low-barrier as it may be - has severely diminishing returns. Everyone who does that sort of thing - especially on a programmatic level - identifies certain "anti-patterns" that they then register as a common engineering pitfall and discourage others from doing. The anti-pattern I see here is that using PHP solves the wrong problem by answering the wrong question the wrong way. There are many other such patterns I constantly run into; I think the most common one is using the database as an IPC (interprocess communication) mechanism for high volumes of delay-sensitive, real-time data--the kind of stuff that should be passed around in shared memory, pipes, domain sockets, network sockets, semaphores, etc. ViciDIAL takes that to an absurd, ghastly extreme. Once again, I wasn't attacking you, and I apologise if that is the perception. I was just wondering aloud why people do this kind of stuff.

In the end, the technology isn't for its own sake. Its to make money. I'm in the back (for over 2 years), and growing over 10% per month, even through the downturn. How about you?

I'm not an ITSP, so I can't answer that on an apples-to-apples basis. What I can say is that until a certain point, there is a high correlation between good engineering and making more money, for reasons that are fairly self-evident; it eliminates various additional external costs down the road, it reduces brand destruction through declining performance (and the ensuing scramble to bandage it), and better design allows you to better and more efficiently develop scalable, reusable business processes that can be replicated at decreasing marginal cost better and faster. So, it's not a purely aesthetic issue; there is an objective weight to one methodology over another, and it has everything to do with making money. On a practical level, the way I see this playing out is that I'm the bomb technician that has to go in and defuse an ITSP that has grown to have lots of call setups per second (whether from dialer traffic or what have you) and finds that their large farm of Asterisk boxes - and all call flows intermediated by extensive PHP-AGI calls - is falling over once or more daily, and that this just isn't tenable from a business perspective because the whole architecture is flawed from the very beginning. That has a huge price tag. -- Alex -- Alex Balashov Evariste Systems Web : http://www.evaristesys.com/ Tel : (+1) (678) 954-0670 Direct : (+1) (678) 954-0671 Mobile : (+1) (678) 237-1775

Peter Beckman wrote:

Before I start, I strongly disagree that PHP is bad for handling Asterisk dialplan logic. You want to write your business logic in code once, then reuse it. We happen to expose our business logic via a PHP API that our Perl FastAGI uses, but not everyone has the talent/ability/forethought to build it that way. The easiest way to reuse that business logic code you wrote for the website is to build libraries you can reuse elsewhere. In this case, a PHP that talks to Asterisk in an AGI sort of way (not directly).

I suppose that if the rationale were so intricately bound up in the benign and understandable motive of code re-use, that is logical and forgivable. No argument there, given that premise. There are various observations one might make about PHP runtime performance that suggest that it really doesn't work well at all outside of mod_php and should not ever be used in standalone processes, but I'll stay away from that because that really does have the tendency to degenerate into a religious debate, and your point is fundamentally sound and cogent.

First off, calling AGI(somescript.php) is awful.

It is, but most of the broken implementations I am brought in to fix do exactly ... that! :-)

We implemented it as a FastAGI script, running as a TCP service that asterisk connects to on a listening local port. It handles all of our database connections, forks new processes off for each call, and does everything we need the way we want it done, not having to add on a bunch of hacked-together scripts and modules in Asterisk.

That sounds very workable. I suppose there might be something to be said in favour of non-blocking I/O / synchronous I/O multiplexing in a unitary process versus forking a new process or thread for every call, but that only becomes a concern with very, very large amounts of call setups. The mere fact that you have a FastAGI server versus invoking local AGI puts you like 1000% ahead of people who don't. I've built many FastAGI services - whether for call routing or more esoteric business logic - in Perl as well.

Second, if you know PHP and nothing else, it is definitely easy to get it done in PHP, especially if your business logic is in neat libraries you can include and make calls to. If you call it directly via AGI(), it can and likely will hurt or kill your resources, but using it as a long running FastAGI daemon it can be managed and connections to the DB can be shared.

There is a considerable body of evidence suggesting that although PHP does expose a socket API, using it to do sockets is a terrible idea because it just doesn't perform well at all. I don't know why anyone would write a server - FastAGI or otherwise - in PHP. Besides, PHP does not really have the appropriate low-level dimension for heavyweight or nuanced system calls; that's not what it was designed for. Then again, I suppose people make that argument for Java, too. On the other hand, I have no trouble believing it still works better than local AGI() script invocations.

Last, if all of your business logic is already done in PHP for your website, it's really easy to simply re-use this existing code in your AGI. And if you do it right and well, it may even scale fairly well.

True, very true.

Your point however is still very valid: most customers do NOT do it right and well, and they should be motivated to learn how to do it better. I do think your downplay of PHP as an AGI language is simply misdirected. PHP can do FastAGI, and you can pool connections, fork processes, do things elegantly, same as with a Perl or C++ compiled FastAGI script. Of course we all know C++ will run faster, but we also know good C++ folk are hard to find, and once they build it, it's hard to find them again to update it. PHP programmers are easier to find, and if we simply educate them about how to build a FastAGI written in PHP reusing business logic already build into their website, well, it seems like a pretty good path to me.

Maybe. I suppose it's just hard for me to see, as someone with a programming background that almost entirely consists of socket-layer programming and network protocol implementation (in C), how PHP - given its architecture - can do those things in a scalable fashion even if it does bother to publish APIs for the underlying system calls. Managing large amounts of I/O-bound processing just isn't what it was built for at all. Even my curl_*() calls out of PHP are dog-slow. It's a qualitative issue -- "just because you can, doesn't mean you should." The real answer lies in robust performance studies and in the details of PHP's runtime implementation. I have neither of these annals at my ready disposal in order to illustrate that point. There's just a general principle that high-level, non-typed web languages aren't designed to participate in the same problem domain as general-purpose programming languages and/or languages on which systems are built at that lower level of abstraction. It is within the realm of conception that you could write a virtual machine container/hypervisor in PHP, if only the interpreter exposed the system calls and other primitives you'd need to pull it off. But it's not done for a reason--surely you can agree. (Except perhaps for the novelty/Slashdot factor by someone with far too much time on their hands.)

For my Perl FastAGI, I use Net::Server::Fork and Asterisk::AGI (with a few fixes and modifications) and it works great.

Interesting. I've tended to use Asterisk::FastAGI in the past, but it never seems to obey my bind port parameter on the initial invocation. -- Alex -- Alex Balashov Evariste Systems Web : http://www.evaristesys.com/ Tel : (+1) (678) 954-0670 Direct : (+1) (678) 954-0671 Mobile : (+1) (678) 237-1775

Peter Beckman wrote:

...

There is a considerable body of evidence suggesting that although PHP does expose a socket API, using it to do sockets is a terrible idea because it just doesn't perform well at all. I don't know why anyone would write a server - FastAGI or otherwise - in PHP.

I haven't done any tests to see how well or badly PHP does sockets

I haven't either; I suppose there are so many points of view from which doing a serious server in PHP is considered preposterous and misguided that I couldn't find anyone interested in commissioning a study to measure the precise degree to which it is so. My evidence is anecdotal, and for that reason difficult to relate or cite. I see people trying to do this stuff - for the reasons you mention - and I see it 1) falling over and 2) becoming untenably complex / impossible to maintain / experiencing internally degenerative motion as a codebase.

...

Besides, PHP does not really have the appropriate low-level dimension for heavyweight or nuanced system calls; that's not what it was designed for. Then again, I suppose people make that argument for Java, too.

Untrue: http://us.php.net/manual/en/intro.sockets.php

"The socket extension implements a low-level interface to the socket communication functions based on the popular BSD sockets, providing the possibility to act as a socket server as well as a client."

I was talking about PHP's character as a rather abstract, application-level runtime environment, not the ability of the interpreter-makers to clunkily expose a low-level system call API somehow. That's done in a lot of languages.

PHP is more versatile than many people think. It's not as deep zen as Perl can be, and there are some frustrating inconsistencies in how PHP operates, but it can be powerful in a knowledgable programmer's hands.

The same can be said of almost any language and runtime environment; it is possible to implement almost anything in a language that has a sufficiently broad library, and it is possible to follow better or worse architectural and programming practices in ColdFusion and Bash, too. I do not know that the capabilities technically present in the runtime interpreter - which are dictated to a fairly large extent by a combination of user demands and the developers' own sense of creativity - are really a reflection of PHP's "versatility" paradigmatically. As I said, you can write a VM hypervisor in it if the right system calls and API hooks were just available. I think I once read an article about someone that stringed a primitive SSH server out of components all defined and invoked inline from a Bash script; are you moved to say that the fact that it can be done is a testament to Bash's immense power and versatility in the right hands? It's a fair statement in that context, to be sure, but is it versatile in the sort of way that leads to a viable recommendation as an implementational language for an SSH server? Perhaps the example is incommensurable because Bash certainly doesn't make it easier or more terse to implement one than in a language like Perl or PHP, but even if it did, by code volume, through various improvements to the interpreter's capabilities (much as in PHP): Is that trait the defining one, the only thing holding you back from saying, "Let's write our SSH server in PHP?"

I think that maybe PHP has improved since you last used it (or so it seems). Again, I'm sure there are better ways, but using PHP for FastAGI is not as bad as you make it seem. Then again, I'm not doing 1,200,000 minutes per day.

I don't know how bad I'm making it seem. I am not suggesting that it is outside the realm of feasibility, especially from a basically functional perspective.

...

It's a qualitative issue -- "just because you can, doesn't mean you should."

Maybe, but if you are a small mom-and-pop and just need it to work, reusing already written code for both your web and VoIP applications makes a lot of sense. And if they do it right, by using a PHP-based FastAGI vs straight AGI(), it can scale fairly well.

It's a fair economic point, although personally, to me, that just says that the business logic that is shared between web and VoIP needs to reside in a common place that still allows the right approaches to be taken on both sides and can be called. Stored procedures, triggers and the like are a common candidate for this in my work; a database call can both figure out which rate center destination a call should be routed to from the call router, and calculate that rate center and cost dynamically for various UI functionality, for example. Or, if the technical argument is there - stemming from the complexity and real-time character of the data needed to make the decisions - perhaps even a real middleware layer of some sort. Seems to me that the code reuse argument goes so far. Saying that using the same technology on both sides of a very large abstraction and performance domain divide is the most efficient way to lever existing code fails to take into account that maybe you just shouldn't have done the business layer in PHP either, or otherwise have put the code in the wrong place. What *are* business layers for something like an ITSP call processing platform doing in PHP, anyway PHP is largely a presentation language. To reiterate in the most emphatic terms, that is not a description of the capabilities of its (phenomenally large) API - as far as I can tell there is little that PHP doesn't technically "have" - but rather a judgment about the "sort of thing that it is" or was intended to be, a genus. Bash was intended to be a shell and to provide a scripting environment for simple batching of the sort of stuff you'd often do in a shell (complete with very rudimentary flow control and data primitives), even though you could certainly augment its, er, versatility by hacking in more plumbing to the outside (system) world. Was PHP designed to be a language for standalone scripts and processes? Does much of the work that goes on in PHP exist to further that particular aim? It's pretty convincing from a cursory survey of the way PHP is progressing that it is - and always has been - about the web. It seems to me that its use for standalone scripts really is an accident of someone deciding it would be nifty to let people run the interpreter from the CLI, which I guess is formally required anyway in order to allow PHP to run in a CGI-style way (not with mod_php) if that is desired.

...

It is within the realm of conception that you could write a virtual machine container/hypervisor in PHP, if only the interpreter exposed the system calls and other primitives you'd need to pull it off. But it's not done for a reason--surely you can agree. (Except perhaps for the novelty/Slashdot factor by someone with far too much time on their hands.)

Given a programmer talented in all languages, absolutely.

It still comes down to the fact that PHP is a very slow layer of interpreted fat and high-level abstraction, and furthermore that its virtues and strengths - of which there are many - have absolutely zero imaginable correlation to the goal of writing a reasonably performing hypervisor. It dimensions well to web applications, not low-level, back-end system processes. We aren't going to agree on this, although I think we largely agree in the breadth of the overall context. The bottom line really comes down to what you value in the craft. The values of the commercial world often dictate that you should do something in the simplest and cheapest way possible, with no attention paid to the quality, density or longevity of the final product nor the erudition that is brought to bear on the problem. To the extent that one shares those values, one can PHP-AGI one's way out of this problem, sure. Actually, with the amount of call setups and burst volume objectively required in most cottage-industry ITSP operations, you could probably do the whole thing in Bash: ACCT_ROUTE_VAR=$(echo \ "SELECT id, rt FROM rt_tbl WHERE account = $ACCT LIMIT 1" \ | mysql -u user --password=blah -D database \ | head -n 3 | tail -n 1 | awk -F '|' '{print $2}') It'll end up on TheDailyWTF.com, but it does work, and will probably continue to work for reasonably large "values of n" that the provider will experience in the near/foreseeable future. On the other side, there's considerable value to be reaped in the technical excellence of a little overengineering - not excessive, of course, there's always a balance to find. The value is both qualitative to those that value the aesthetic dimension of good programmatic craftsmanship and economic in that the system can grow to meet your needs, including both needs themselves and magnitudes of needs that cannot be reasonably anticipated. If you and I were charged with developing an integrated technology stack as a competitor to say, BroadWorks or Metaswitch, and the requirement was that it be able to process bursts of, say, 6500 call setups per minute, and you used PHP-AGI and I used something that I find to avoid the pitfalls of that approach, my implementation would win and yours would lose, per given unit of hardware capacity. That issue is completely academic and largely moot because other limitations will be encountered long before that becomes a concern, since Asterisk is in use. :-) Secondly, neither of us is actually building a system that has to handle very bulky logic for 100+ CPS--that's considerably more than most commercial softswitch gear will reliably do. But if we were, I don't think you'd use PHP. It is not so much more difficult to use the right tool for the right job to begin with; it's not a night or day issue in terms of capital requirements. Even if you'll never need it, it's nice to know that you can rest easy at a lower threshold because you went for the right architecture, as opposed to what many of my customers worry about, which is that doubling their traffic next year will render their architecture completely unusable because it was built with this kind of crap, and is already starting to crack. I don't make my living by selling customers a geostationary satellite guidance system when they need a horse--there's plenty of that out there, much of it in enterprise-size environments. I know, I know, ITSPs are small companies that have to take the most pragmatic route to achieve anything. But I think that appreciating the long-term benefits of good (perhaps even a little excessive) engineering, as opposed to just putting out today's fire today, is critical to any field of commercial endeavour in technology, regardless. I realise you don't disagree with me on most of this. I just think that these premises lead to the conclusion that there's no way something like PHP can be the ticket, whereas you clearly don't.

I like Net::Server::Fork because it allows me to HUP to restart (reload new code) the parent process without touching the child processes, which means I can put new FastAGI code into production without waiting for no calls or worrying about disconnecting existing active calls. It is also stable and have not had problems with it in years.

Good to know; I had not considered the soft reload angle. Thanks! -- Alex -- Alex Balashov Evariste Systems Web : http://www.evaristesys.com/ Tel : (+1) (678) 954-0670 Direct : (+1) (678) 954-0671 Mobile : (+1) (678) 237-1775

On Tue, 4 Aug 2009, Alex Balashov wrote a lot of things that Peter Beckman cut, not because they are not worthwhile, but because his response is significantly less verbose:

I realize you don't disagree with me on most of this. I just think that these premises lead to the conclusion that there's no way something like PHP can be the ticket, whereas you clearly don't.

Honestly I agree with you on almost all points -- PHP isn't the best, greatest or most scalable solution to scripting Asterisk via AGI or FastAGI. In my opinion PHP is a legitimate language that a FastAGI daemon can be written in and function, maybe even function well. I think Bash would be a worse choice to write a FastAGI daemon in, but you could. You might even be able to use a whole bunch of pipes (or socat) and some GNU tools to build one. PHP can and does work, and so does Perl, and so does C and C++. I can write a FastAGI daemon in all of them. I can likely build it with some extra work, a few extra tools, maybe a different method of compiling and actually make it scale well in all three languages. For customers who are not sophisticated, and who have limited budgets, they can hire limited programming resources who have less knowledge about the kinds of intricacies we have learned over the years. This leads to the "exten => _X.,n,AGI(asterisk.php)" lines in the config that we see and immediately cringe at upon seeing. We know that some languages are better at being long running processes than others, and when the programming resources are available, are able to choose the best one for the job. PHP is a choice, an option, one that I think is perfectly valid for some situations. I've written a lot of cron jobs and command line scripts that work in PHP just fine. Since there are resources on the servers to handle the bloat of using an interpreted language to run these scripts, there isn't any real problem with it. There is a PHP CLI for this reason. Given the same resources, we'd probably both rule out PHP as the ideal solution for a FastAGI implementation. It is, IMO, still a legit option.

Good to know; I had not considered the soft reload angle. Thanks!

No problem! Just be sure that if your FastAGI is using HTTP API calls to your backend that you are able to manage DB changes gracefully. Having an existing call make a call to an internal backend API that now requires an additional variable will fail unless that case is handled. This is one of the reasons the FastAGI does not talk to the DB directly, but through HTTP API calls that are handled quickly by PHP. Beckman --------------------------------------------------------------------------- Peter Beckman Internet Guy beckman at angryox.com http://www.angryox.com/ ---------------------------------------------------------------------------

On Wed, 5 Aug 2009, Alex Balashov wrote:

I abdicate the question with a parting thought: Why is it that more high-capacity, commercial-grade stuff (of the standalone, not web front-end variety) isn't written in PHP? Why doesn't someone like Metaswitch or Broadsoft implement internals in PHP? To some extent, the answer is accounted for by the stubbornness of large-corporate fashions and entrenched interests surrounding certain technology stacks, as we well know, but I don't think that's the whole explanation. It's not like these guys didn't get the memo on PHP or something.

I really think it has to do with the existing code-base and the existing developers. Each company is beholden to the geeks that build, maintain and manage the mess of code that exists. Those geeks either get to choose what technologies are used, or do not have the resources to rewrite working code and are forced to maintain the code already written, maybe adding things on in different more familiar languages along the way, until you're in a giant pile of spaghetti code in 9 languages, all tangled loosely but functionally together by a very thin thread. What do Metaswitch or Broadsoft use? C? Perl? Are they using Asterisk or did they build their own telecom server stack? How do you know they are NOT using PHP internally or even externally? Disabling the web management server's tag (not returning "Apache/1.2.47 PHP/5.2.5" on requests, even though that is exactly what it is running) is trivial and a good security choice, assuming these devices are configured via a web-based interface.

...

...

Good to know; I had not considered the soft reload angle. Thanks!

No problem! Just be sure that if your FastAGI is using HTTP API calls to your backend that you are able to manage DB changes gracefully. Having an existing call make a call to an internal backend API that now requires an additional variable will fail unless that case is handled. This is one of the reasons the FastAGI does not talk to the DB directly, but through HTTP API calls that are handled quickly by PHP.

That's a smart setup. And, PHP is a fine language to use for an HTTP and/or RESTful API dispatcher. :)

Why thank you! :-) Beckman --------------------------------------------------------------------------- Peter Beckman Internet Guy beckman at angryox.com http://www.angryox.com/ ---------------------------------------------------------------------------

Peter Beckman wrote:

I really think it has to do with the existing code-base and the existing developers. Each company is beholden to the geeks that build, maintain and manage the mess of code that exists. Those geeks either get to choose what technologies are used, or do not have the resources to rewrite working code and are forced to maintain the code already written, maybe adding things on in different more familiar languages along the way, until you're in a giant pile of spaghetti code in 9 languages, all tangled loosely but functionally together by a very thin thread.

As far as I see, large, well-capitalised companies selling into the enterprise market generally have the resources to recruit, hire and integrate developers in whatever platform and/or software development ecosystem they please. If they want to switch, they can switch. If they want to rewrite or refactor large chunks of it, they can do that, although certainly not with any kind of speed, and that can lead to very valuable lost time to market. Still, I think the real criteria there have to do with, in addition to leveraging existing code and/or licensed core technologies (such as commercial SIP stacks, for example) - as you say - also with issues that are fundamentally cultural and somewhat religious in nature. The belief systems and experiential biases of CTOs and senior directors of this and that certainly play a major role.

What do Metaswitch or Broadsoft use? C? Perl? Are they using Asterisk or did they build their own telecom server stack? How do you know they are NOT using PHP internally or even externally? Disabling the web management server's tag (not returning "Apache/1.2.47 PHP/5.2.5" on requests, even though that is exactly what it is running) is trivial and a good security choice, assuming these devices are configured via a web-based interface.

For all my talking smack about PHP, I don't actually have a detailed knowledge of what those two platforms use, although I know for a fact they don't use PHP. Given the strangehold that Java has on big telecom, I would imagine it plays a dominant role. As Peter Childs mentioned, BroadWorks seems to revolve primarily around Java, which also makes sense in many ways given the vertical integration possibilities it offers. For example, a core management layer in Java can contain an embedded application server serving JSP that directly accesses runtime data from the former process, saving the need to implement costly and complicated APIs and/or middleware for those two components to communicate. BroadWorks certainly appears to work on that principle. On the other hand, I cannot imagine that every component of the system is written in Java; for instance, I have a hard time believing that the high-capacity media relay and media generation portions of the Broadsoft assembly are done in Java. That wouldn't make any sense. Most likely, those are done in C. I would hypothesise something similar about Metaswitch, although their stuff doesn't scream Java nearly so obviously apart from the switch EMS client itself. Doing management front-end GUIs that provision the and monitor the switch via SNMP in Java seems to be fairly standard fare for almost everyone. Also, most SIP stacks and signaling agents for other IP signaling protocols (H.323, H.248/MEGACO, MGCP, etc.) are written in C or C++, if for no other reason than performance, along with access to low-level system calls used to trigger various kernel-side APIs and subsystems[1]. And, of course, any sort of TDM hardware interfaces such as would be necessary on media gateway hardware (this is relevant to Metaswitch rather than Broadsoft) would of almost certain necessity have to be done in C or C++ - I just don't see the device driver access being plumbed up to the level of abstraction in which the JVM operates, with the VM and the sandbox and the security model aspects in mind alongside the more obvious restrictions and performance implications. I do not know if there's anything ASIC-assisted in the Metaswitch aside from the TDM ports, but if so that would almost certainly call for C or C++ as well. So, if I had to take a stab at it, it sounds like some combination of C/C++ on the low level, Java on the general level, and perhaps miscellaneous Perl throughout. -- Alex [1] Although, it is an interesting case in point that one of the most popular open-source media relays, AG Projects' MediaProxy 2.0, is written in Python but uses kernel APIs plumbed through C to Python that reach into the Linux kernel's IP forwarding subsystem to provide faster RTP relay than would be possible in pure userspace, let alone in an interpreted language like Python. That's about as close to relay at wire speed as you're going to get. -- Alex Balashov Evariste Systems Web : http://www.evaristesys.com/ Tel : (+1) (678) 954-0670 Direct : (+1) (678) 954-0671 Mobile : (+1) (678) 237-1775