handling 4000 sessions

[Nitesh]

Hi all

Good Evening

When I reach 2000 sessions i.e a 1000 calls ( I have set max sessions to 4000 )
the CPU usage shown on Dashboard is 23 % - 24 % , but my GUI gets all black and white and Nginx starts giving Bad gateway , when trying to login and when clicking on show all, when on Active calls tab .

All Voice calls, call recording on for most of the calls

This is TOP command CPU stats at 1000 calls ( 2000 sessions approx )

top - 09:36:14 up 3:50, 1 user, load average: 105.84, 89.37, 75.23
Tasks: 345 total, 5 running, 340 sleeping, 0 stopped, 0 zombie
%Cpu(s): 11.9 us, 82.6 sy, 0.0 ni, 3.6 id, 0.2 wa, 0.0 hi, 1.6 si, 0.0 st

KiB Mem: 66087976 total, 65712972 used, 375004 free, 78460 buffers
KiB Swap: 18989052 total, 308 used, 18988744 free. 60520480 cached Mem

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
959 www-data -2 -10 18.999g 2.488g 22268 S 3087 3.9 1505:46 freeswitch
810 root 20 0 1447296 17532 6188 S 6.2 0.0 139:46.94 fail2ban-server
3767 www-data 20 0 247680 21744 14160 S 1.9 0.0 0:19.04 php5-fpm
3496 www-data 20 0 247588 20212 13036 R 0.9 0.0 0:19.04 php5-fpm
12097 www-data 20 0 246808 19812 13036 R 0.9 0.0 0:10.90 php5-fpm
6505 www-data 20 0 247684 20236 13036 R 0.6 0.0 0:15.68 php5-fpm
10428 www-data 20 0 247648 20020 13036 S 0.6 0.0 0:11.89 php5-fpm
20861 postgres 20 0 234644 14328 12064 S 0.6 0.0 0:00.27 postgres

My Server configuration is

RAM 64GB
Bandwidth 33T/1G
8 core CPU

Call recording is enabled on most of the calls , I need to reach a 2500 to 3000 calls that is 5000 to 6000 approx sessions , can I reach that in current configuration , so I need to make some changes ,
I have already unloaded all the modules which I do not need , made them inactive .
How can adjust to take more calls

Please advice ,

Thanks

 

[KonradSC]

I have been told that a sofia profile only runs on one thread. Calls will be multi-threaded, but the profile will run on one thread. You might need to add some more profiles and run signalling on other ports like 5062, 5064, ect. One way to check this is to look at a packet capture and see how long the server's SIP responses take.

 

[Nitesh]

Thanks for advice Konrad,
I can make the calls distribute on other ports , but running on different ports will that decrease load on CPU ,are we sure ?

 

[DigitalDaz]

Nitesh,

You surely must have already done some tweaking?? Both the stock memcache and php limits should need changing, look at you nginx logs for errors.

 

[DigitalDaz]

Also, look the the postgres usage and logs

 

[Nitesh]

Thanks Digital Daz ,
I am doing the tweaking for getting calls on different ports , do I need to increase the memcache and what do I need to tweak on php limits ?

Regards

 

[Nitesh]

I am checking logs now , fs_cli getting these

Got error [502] posting to web server [http://127.0.0.1/app/xml_cdr/v_xml_cdr_import.php]
[ERR] mod_xml_cdr.c:402 Retry will be with url [http://127.0.0.1/app/xml_cdr/v_xml_cdr_import.php]
[ERR] mod_xml_cdr.c:418 Unable to post to web server, writing to file

 

[DigitalDaz]

Nitesh, how many concurrent you managed so far??

 

[Nitesh]

1000 calls - around 2000 sessions approx , with call recording enabled on each
for 700 - 900 no issues , when comes to 1000 plus it starts overloading CPU and congestion start coming up and call go to failover ,

and that is when I get these errors

also I get
switch_core_session.c:2282 Throttle Error! 1690
switch_time.c:1235 Over Session Rate of 40!  
when crossing 1000 calls

 

[DigitalDaz]

The above is probably settings in switch.conf.xml

But, load will start increasing because other thing will be failing, like I said, look at the memcache config I think that defaults to 1024 connections, and look at php config too.

Start doing things like turning off nginx access log, also reduce loglevel on freeswitch

 

[Nitesh]

Hi ,

Yes that is in switch.conf.xml

Issue is its getting overloaded , so giving me Throttle Error , as you can see its gettin 1690 sessions but due to overload , gives throttle error .

Yes memcache is 1024 , how do I increase it ?
I have turned off reduced FS logs in the starting , but yes have to turn off nginx access logs


What main settings I have to check alter in php config

 

[DigitalDaz]

switch_time.c:1235 Over Session Rate of 40! 

That should be in the switch.conf.xml