Fail2ban just parses log files for specific keywords. If your log files are rapidly growing then fail2ban will run more or take longer to run. You may want to watch your logs to see which one is the offender.
If it's the FreeSWITCH log, consider setting the log level lower. So if it's set to DEBUG, then change it to NOTICE. Something like that.