Linux kernel tuning for load balancing
Last modified: 09/01/2016

NOTICE: Any setting you make to change default kernel settings and/or
user limits must be implemented with caution. All changes should be
thoroughly tested on development servers whenever possible. If you are
not careful when adjusting default kernel settings you can render your
server inoperable.

Load balancing will require kernel tuning based the volume of traffic
and type of TCP service you are load balancing. NOTE: The values in
listed below are based on extremely high volume. Depending on your
application you may not need to set many of the parameters listed. But
please keep in mind that you will need to tune the kernel for any
server acting as a load balancer.

All setting listed below must be application tested and set based on
the number of connections to your frontend LB and backend hosts. For
each of the settings below use the sysctl command to backup the
orginal settings before making any changes, for example:

# sysctl fs.file-max | tee -a /root/sysctl_org_settings.txt

To make the changes persistent, add the setting(s) to the sysctl.conf
file:

CENTOS 5/6 or RHEL 5/6:
# vi /etc/sysctl.conf

CENTOS 7 or RHEL 7:
# vi /usr/lib/sysctl.d/00-system.conf

To load changes CENTOS 5/6 or RHEL 5/6:
# sysctl -p

To load changes CENTOS 7 or RHEL 7:
# sysctl -p /usr/lib/sysctl.d/00-system.conf

File Handle Limits:
------------------
Increase the maximum number of open file descriptors:

fs.file-max = 25000000

IPv4 Socket Tuning:
------------------
For a large numbers of concurent sessions:

net.ipv4.ip_local_port_range = 1024 65535
net.ipv4.tcp_tw_recycle = 1
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_fin_timeout = 60
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
net.ipv4.tcp_max_syn_backlog = 4096
net.ipv4.tcp_syncookies = 1
net.core.somaxconn = 1024

IPv4 Settings:
-------------
net.ipv4.ip_forward = 1
net.ipv4.ipfrag_max_dist = 4096
net.ipv4.ip_nonlocal_bind=1

Process Scheduler:
-----------------
Increase time for process run time before the kernel will consider
migrating it again to another core: 

CENTOS 5/6 or RHEL 5/6:
kernel.sched_migration_cost = 5000000

CENTOS 7 or RHEL 7:
kernel.sched_migration_cost_ns = 5000000

kernel.sched_autogroup_enabled = 0

Threads:
-------
Increase maximum number of threads that can be created by a process:

kernel.threads-max = 5000000
vm.max_map_count = 128000
kernel.pid_max = 65536

