memcached and Postgres flooding at Loopback interface

[kwangmien]

Hi,

I am running Fusion PBX version 4.4.6 on Centos 7.
When I did a tcpdump on the Loopback interface of the Fusion PBX server, I discovered floods of memcached and postgres messages.

An example of the postgres message is Query : deallocate pdo_stmt_00000002. There are some other Postgres SQL queries in the flood.

For memcached, the GET API is used to retrieve directory information of the extension.

I could not find out which process is initiating these Postgres and memcache messages.

Can any one advise me on how to stop the loopback flooding ?


Thank you.

Regards,
Kwang Mien

 

[Adrian Fretwell]

The loopback interface is used by a number of processes including Postresql, Freeswitch and Nginx.
What you are describing is probably normal behavior.

 

[kwangmien]

I compared with my other Fusion PBX server. There are not a single memcache and postgres messages at loopback interface.

further analysis of the tcpdump at loopback interface shows that some processes are trying to get some information from memcache and postgres. at the point in time, there are no calls.

 

[Adrian Fretwell]

On a completely idle machine I still see packets on the loopback interface.

You can see what is listening with this:

netstat -ant | grep 127\.0\.0\.1
tcp        0      0 127.0.0.1:8021          0.0.0.0:*               LISTEN     
tcp        0      0 127.0.0.1:5432          0.0.0.0:*               LISTEN     
tcp        0      0 127.0.0.1:25            0.0.0.0:*               LISTEN     
tcp        0      0 127.0.0.1:11211         0.0.0.0:*               LISTEN     
tcp        0      0 127.0.0.1:5432          127.0.0.1:33260         ESTABLISHED
tcp        0      0 127.0.0.1:33260         127.0.0.1:5432          ESTABLISHED

You can see what process owns a port with:

ss -lptn 'sport = :8021'
State                     Recv-Q                    Send-Q                                        Local Address:Port                                         Peer Address:Port                   
LISTEN                    0                         5                                                 127.0.0.1:8021                                              0.0.0.0:*                        users:(("freeswitch",pid=885,fd=70))

All my systems are Debian, so I don't know exactly what you should see for other distributions.

 

[kwangmien]

i tried the netstat command.
there is a total of 1125 connections to memcached which result in TIME_WAIT. However, there are no processes listening to these ports.

tcp 0 0 127.0.0.1:11211 0.0.0.0:* LISTEN
tcp 0 0 127.0.0.1:8021 0.0.0.0:* LISTEN
tcp 0 0 127.0.0.1:5432 0.0.0.0:* LISTEN
tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN
tcp 0 0 127.0.0.1:37590 127.0.0.1:11211 TIME_WAIT
tcp 0 0 127.0.0.1:37718 127.0.0.1:11211 TIME_WAIT
tcp 0 0 127.0.0.1:36952 127.0.0.1:11211 TIME_WAIT
tcp 0 0 127.0.0.1:36676 127.0.0.1:11211 TIME_WAIT
tcp 0 0 127.0.0.1:58872 127.0.0.1:80 TIME_WAIT
tcp 0 0 127.0.0.1:36102 127.0.0.1:11211 TIME_WAIT
tcp 0 0 127.0.0.1:37548 127.0.0.1:11211 TIME_WAIT
tcp 0 0 127.0.0.1:36632 127.0.0.1:11211 TIME_WAIT
tcp 0 0 127.0.0.1:36072 127.0.0.1:11211 TIME_WAIT
tcp 0 0 127.0.0.1:36440 127.0.0.1:11211 TIME_WAIT
tcp 0 0 127.0.0.1:37174 127.0.0.1:11211 TIME_WAIT
tcp 0 0 127.0.0.1:36412 127.0.0.1:11211 TIME_WAIT
tcp 0 0 127.0.0.1:37248 127.0.0.1:11211 TIME_WAIT
tcp 0 0 127.0.0.1:59008 127.0.0.1:80 TIME_WAIT
tcp 0 0 127.0.0.1:37468 127.0.0.1:11211 TIME_WAIT
tcp 0 0 127.0.0.1:36364 127.0.0.1:11211 TIME_WAIT
tcp 0 0 127.0.0.1:36106 127.0.0.1:11211 TIME_WAIT
tcp 0 0 127.0.0.1:36772 127.0.0.1:11211 TIME_WAIT
tcp 0 0 127.0.0.1:36400 127.0.0.1:11211 TIME_WAIT
tcp 0 0 127.0.0.1:35986 127.0.0.1:11211 TIME_WAIT
tcp 0 0 127.0.0.1:36526 127.0.0.1:11211 TIME_WAIT
tcp 0 0 127.0.0.1:37462 127.0.0.1:11211 TIME_WAIT
tcp 0 0 127.0.0.1:36844 127.0.0.1:11211 TIME_WAIT
tcp 0 0 127.0.0.1:37022 127.0.0.1:11211 TIME_WAIT
tcp 0 0 127.0.0.1:36758 127.0.0.1:11211 TIME_WAIT
tcp 0 0 127.0.0.1:37438 127.0.0.1:11211 TIME_WAIT
tcp 0 0 127.0.0.1:37528 127.0.0.1:11211 TIME_WAIT
tcp 0 0 127.0.0.1:36858 127.0.0.1:11211 TIME_WAIT
tcp 0 0 127.0.0.1:36784 127.0.0.1:11211 TIME_WAIT
tcp 0 0 127.0.0.1:37578 127.0.0.1:11211 TIME_WAIT
tcp 0 0 127.0.0.1:36940 127.0.0.1:11211 TIME_WAIT
tcp 0 0 127.0.0.1:35898 127.0.0.1:11211 TIME_WAIT

 

[DigitalDaz]

On a new version of FusionPBX memcached shouldn't even be being used, it was replaced with another method some time ago.

 

[Adrian Fretwell]

I see a lot of TCP connections in this TIME_WAIT state on my live servers, the list looks very similar to what you have posted. This is quite normal for the way the TCP protocol works, see https://en.wikipedia.org/wiki/Transmission_Control_Protocol.

A connection can stay in TIME_WAIT for up to 4 minutes after either side has ended the connection, this is to allow for out of sequence packets to be recieved and also allow enough time to pass to be sure the remote TCP received the ACK of connnection termination.

 

[Adrian Fretwell]

On a new version of FusionPBX memcached shouldn't even be being used

I thought that but a memcache process still exists:

root@testbox-green:~# ps ax | grep mem
   350 ?        I      0:00 [kworker/3:2-memcg_kmem_cache]
   623 ?        Ssl    0:00 /usr/bin/memcached -m 64 -p 11211 -u memcache -l 127.0.0.1 -P /var/run/memcached/memcached.pid
  3390 ?        I      0:00 [kworker/6:1-memcg_kmem_cache]
  3606 pts/0    S+     0:00 grep mem

 

[Adrian Fretwell]

memcached is installed as part of the switch packages in the Fusion install:

/usr/src/fusionpbx-install.sh/debian/resources/switch/package-master-all.sh

apt-get update && apt-get install -y ntp curl memcached haveged

So although the FusionPBX code no longer uses mamcached, I guess Freeswitch still uses it for some function.