Whats your Biggest deployment?

[screwloose]

Hi Guys

I'm interested in seeing how you guys are using fusionpbx in real life and how big is the deployment footprints you look after?

We have been using fusionpbx for years for customers and our selves. we have a hosted PBX that we run for customers that have 500+ end points attached.

We also have a few smaller 50 - 60 seat call centres around the place.

I love fusionpbx because its open and if something is broken you can just open the code and see whats wrong or change the code to suit the deployment.

 

[yukon]

Man. I wish I could get ONE 500 seat deployment. My biggest is 100 right now.

 

[JamesBorne]

None at 500, but we have multiple over 200.

This is going to turn into a massive d**k measuring competition haha

 

[screwloose]

Its not a competition (we all know who has the biggest d**k) but i'm more interested in how big solutions people have deployed and if they have run in to any issues. As we have approached 600 endpoints we have noticed more random issues.

The over 500 deployment is a multi tenet box that has lots of different customers on it.

None at 500, but we have multiple over 200.

This is going to turn into a massive d**k measuring competition haha

 

[JamesBorne]

. As we have approached 600 endpoints we have noticed more random issues.

Same. That's our PBX server extension cap.

 

[screwloose]

200 is your cap? or 600?

 

[Adrian Fretwell]

This is a very interesting discussion. My gut feeling is that the limits revolve around the number of concurrent calls rather than the number of end points registered. We have one tenant with over 300 phones but most of them are rarely used so the concurrent call count is low.

Capacity will very much depend on hardware, but I seem to remember reading a post that stated 1000 concurrent calls is the limit.

Our busiest server is a virtual machine running on XCP-Ng, It has 12 virtual CPUs, 16Gb virtual memory and 100Gb of storage.
The fusion instance has 800 endpoints registered spread across 50 domains. Thus far we have not encountered any issues, not even with RTP timing given that it is a VM.

Of the storage allocated, 14% is currently used. VM CPU usage is around 2.7% during busy periods, as reported by the Fusion dashboard. The underlying hardware is a Dual Intel Xeon 2.4GHz (12 cores/24 threads) - 128GB ECC Memory.

Calculating theoretical limits has its place but I do like seeing real world examples.
I would be very interested to hear other peoples comments about what they have in service and how it is performing.

Adrian.

 

[bcmike]

This is a very interesting discussion. My gut feeling is that the limits revolve around the number of concurrent calls rather than the number of end points registered. We have one tenant with over 300 phones but most of them are rarely used so the concurrent call count is low.

Capacity will very much depend on hardware, but I seem to remember reading a post that stated 1000 concurrent calls is the limit.

Our busiest server is a virtual machine running on XCP-Ng, It has 12 virtual CPUs, 16Gb virtual memory and 100Gb of storage.
The fusion instance has 800 endpoints registered spread across 50 domains. Thus far we have not encountered any issues, not even with RTP timing given that it is a VM.

Of the storage allocated, 14% is currently used. VM CPU usage is around 2.7% during busy periods, as reported by the Fusion dashboard. The underlying hardware is a Dual Intel Xeon 2.4GHz (12 cores/24 threads) - 128GB ECC Memory.

Calculating theoretical limits has its place but I do like seeing real world examples.
I would be very interested to hear other peoples comments about what they have in service and how it is performing.

Adrian.

What hypervisor do you use?

 

[Adrian Fretwell]

What hypervisor do you use?

I use XCP-ng (https://xcp-ng.org/) which uses the Xen hypervisor (https://xenproject.org/developers/teams/xen-hypervisor/)

 

[markjcrane]

There are several companies that have 500 domains and some with a lot more than that. There is one company with one cluster of servers that handle 30,000 endpoints for a single company. There many people with 500 endpoints up to 15,000 and higher. Those are a few I know about. You get into big numbers you have to optimize things and configure them to scale.

 

[ZPM]

My biggest is 264 endpoints on a dedicated physical server in our data center. All the smaller ones we run as VMs on VMware but are starting to put them on Hyper-V for new installs. I do single domain deployments.

 

[screwloose]

I use XCP-ng (https://xcp-ng.org/) which uses the Xen hypervisor (https://xenproject.org/developers/teams/xen-hypervisor/)

Ours is running on a XCP-ng host too. I have been s long time Xcp/Xenserver/XCP-NG user. I have always found it do be very good with Disk IO and network stack. I know a few people that have has setups on better hardware running VMware and Hyper V and not had as good experience. I got one customer that was having resource issues on a vmware cluster. I exported the machines and wiped vmware and put on xenserver 7.0 and imported the vmware machines and its been running much better ever since.

 

[JamesBorne]

200 is your cap? or 600?

600 is max and cap.
I was quoting 200 as per individual customer solution; we have many around that extension range.

 

[sokalsondha]

Hello
Can someone tell me please? A 4gb ram and 2vcpu digital ocean VPS can handle how many concurrent calls without transcoding?
Can I get double calls if I double my resources?

Thanks

 

[Adrian Fretwell]

@sokalsondha The short answers to your questions are: Unlikely, no idea, and no.

RTP (Real Time Protocol) is required for your audio transmission, is as it's name suggests, something that must be processed in real time. You cannot guarantee ANY real time operation on a vcpu, unless you have full control over the hypervisor. Having said that, FusionPBX generally works fine on vcpus .

It's not all about how many concurrent calls, I know there was a demonstration at Cluecon of a relatively low spec machine running 1 sofia profile and handling 1000 cps with 30,000 concurrent calls, but the number would be much much lower if a number of those calls were going to voicemail or an IVR.

When I look at my SIP traffic, the Internal Profile is much busier with Notify messages and Registrations that it is with call related SIP traffic.

There is no easy answer, you will just have to monitor your systems performance and plan upgrades accordingly.

One simple thing you can do is take a packet capture when your system is busy and use the VoIP statistics in Wireshark to see if you are getting and SIP retries because your system is overworked.

You can also look at /proc/net/udp (cat /proc/net/udp) and look at the rx_queue. You can look at netstat -su and check the RcvbufErrors.

You may get an answer by stress testing your system with something like sipp, but you really do need to know what you are doing to get meaningful results.

 

[sokalsondha]

@Adrian Fretwell
thanks very much sir for your kind reply.
i am talking without the transcoding, voicemail, recordings... just the live call
as you saying that with one sofia profile with very low config machine freeswitch achieved 30000 concurrent. which is just like a dream
my plan is 4GB RAM/ dual core xeon server . so any rough idea how many concurrent calls roughly i can see they are working.
it will be dedicated server. not VPS . thats why i am trying to figure out how many calls can this server handle

thanks again sir

 

[KonradSC]

We have a FusionPBX environment with 20,000+ extensions. If you are facilitating registered endpoints (phones) your number one bottleneck is mod_sofia. All I can say is get to know sipp (the testing app) so you can test the limits of sofia's handling of SIP packets. You can grow your environment by adding sip profiles, but at some point I feel that you will need to put a sip proxy out front.

Also, the SIP load is going to be different based on your clients and their needs. BLF's, MWI, Registration Expires, Feature Sync, etc. These all play a role in how many handsets per sip profile.

Running FusionPBX & FreeSWITCH at scale is a large (pun) topic.

 

[bcmike]

Good discussion.

Let me ask you something Konrad, I've come from an Asterisk background (migrating to fusion slowly) and we had boxes with fairly vanilla specs that were able to support 1000 + registrations, with MWI, BLFs, etc, do you think mod_sofia could do the same? I understand that all customers have different usage profiles, but what do you think the limits are generally speaking for standard client registrations?

Don't worry I won't hold you to any numbers, I just want to get a sense of what the pitfalls are.

 

[KonradSC]

Guys. It's time to put on your VoIP Engineer pants. There is a reason that every time the "how many ______ can my system handle" question gets asks that you don't get a specific answer.

Here's where I would start...

Each sofia sip profile is single threaded. That's the weakest link.

To me I tend to look at sip packets per second. (pps). You could argue sip transactions per second too, but for ease i'm going to say pps.

You kind of have to work backwards. I would start with two tests. Calls per second and registrations per second. Use sipp to generate both calls and registrations and see how many pps you can generate before you start getting retries.

Let's pretend that you start seeing retries at 30 pps. I would want my system to have a max burst of 30 pps for each sip profile. That would include calls, blf's, registrations, etc. If you have a production system I would setup a sip packet capture and look at my pps for a profile. In Wireshark use the IO graph to look at the Packets/1 sec graph. Also, in Wireshark look at the SIP Statistics table. Does it have many in the Resend column?

So maybe you see spikes in traffic that's more than 30 pps and that correlates to Resends. Time to add another sip profile so you can spread the load out.

It all depends on your environment. Just changing the expire timer from 30 to 120 or 600 seconds will greatly change the sip load on a profile with 500 phones. You could have 30 phones with 29 BLF's each. That's greatly different 1000 phones with an 8 hour expire and no BLF's.

If you setup FusionPBX for your local Dr's office then that's one thing. But if you make your living by running FusionPBX/FreeSWITCH as a hosted provider, then you have to load test your system!

 

[bcmike]

Usually the way I treat these discussions is that I use all of the anecdotal posts as data points to work out a hypothetical base line for testing and then test against that. For example from reading accounts in our discussion I'm thinking a virtual machine with certain specifications should be able to handle a thousand registrations, I then test against my hypothesis and adjust from there .

The objective of these conversations in my opinion is not get out of doing the work, but to get my initial hypothesis closer to the goal to reduce the amount of adjustments I have to do in testing.