Redis installation with three node and sentinel on each¶

This guide is based on these guides :
- Sean McGary : How to build a fault tolerant redis cluster with sentinel
- Data-Essential : Setup a secured redis cluster on CentOS7
~~-Pivotal : How to setup HAProxy and Redis Sentinal for auto failover and thus How to setp Redis Master Slave replication~~

We have three servers with theses configurations :

redis-01 : 172.16.1.43
redis-02 : 172.16.1.95
redis-03 : 172.16.1.182

All three are on the same subnet

Installation¶

Configuration Redis¶

We are going to use the following masterpassword “azerty123456789” (please choose a real password for your configuration !!!)

On the master (redis-01)¶

Edit the redis configuration

vi /etc/redis.conf

Find and edit the following elements :

bind 172.16.1.43

ports 6379

daemonize yes

masterauth "azerty123456789"

requirepass "azerty123456789"

maxmemory 3584mb
maxmemory-policy noeviction

On the first slave (redis-02)¶

bind 172.16.1.95

ports 6379

daemonize yes

masterauth "azerty123456789"

slaveof 172.16.1.43

requirepass "azerty123456789"

maxmemory 3584mb
maxmemory-policy noeviction

On the second slave (redis-03)¶

bind 172.16.1.182

ports 6379

daemonize yes

masterauth "azerty123456789"

slaveof 172.16.1.43

requirepass "azerty123456789"

maxmemory 3584mb
maxmemory-policy noeviction

Start or restart the redis service
service redis restart

We normally have a functional redis master/slave/slave replication.

We now need to add redis-sentinel to have a selection of a new master if redis-01 become not available.

/!\ If you need high availability with HAProxy for example, you need to set the slave as not readonly!

Here the parameters to modify :

Configuration Redis Sentinel¶

We are going to use the following masterpassword “azerty123456789” (please choose a real password for you configuration !!!)

On the master (redis-01)¶

Edit the redis configuration

vi /etc/redis.conf

Find and edit the following elements :

bind 172.16.1.43

ports 26379

daemonize yes

sentinel monitor redis-cluster 172.16.1.43 6379 2
sentinel auth-pass redis-cluster azerty123456789
sentinel down-after-milliseconds redis-cluster 3000
sentinel parallel-syncs redis-cluster 1
sentinel failover-timeout redis-cluster 6000

On the first slave (redis-02)¶

bind 172.16.1.95

ports 26379

daemonize yes

sentinel monitor redis-cluster 172.16.1.43 6379 2
sentinel auth-pass redis-cluster azerty123456789
sentinel down-after-milliseconds redis-cluster 3000
sentinel parallel-syncs redis-cluster 1
sentinel failover-timeout redis-cluster 6000

On the second slave (redis-03)¶

bind 172.16.1.182

ports 26379

daemonize yes

sentinel monitor redis-cluster 172.16.1.43 6379 2
sentinel auth-pass redis-cluster azerty123456789
sentinel down-after-milliseconds redis-cluster 3000
sentinel parallel-syncs redis-cluster 1
sentinel failover-timeout redis-cluster 6000

Start or restart the redis-sentinel service
service redis-sentinel restart

Testing¶

Now connect to redis on redis-01 and kill it to test the sentinel reaction (display the log of redis-sentinel on 2&3 at the same time to see the reaction).

redis-cli -h 172.16.1.43 -p 26379 -a azerty123456789

Check redis-cluster status :
172.16.1.43:26379> sentinel ckquorum redis-cluster

List the slaves :
172.16.1.43:26379> sentinel slaves redis-cluster

redis-cli -h 172.16.1.43 -a azerty123456789
Now kill the master :
172.16.1.43:6379> debug segfault

And see the reaction in the log of 2&3.
redis-cli -h 172.16.1.43 -p 26379 -a azerty123456789

Return on sentinel and get the name of the master :
172.16.1.43:26379> sentinel get-master-addr-by-name redis-cluster

Optimizations¶

From the official Redis setup hints1 :

Set overcommit memory to 1¶

Make sure to set the Linux kernel overcommit memory setting to 1. Add vm.overcommit_memory = 1 to /etc/sysctl.conf and then reboot or run the command sysctl vm.overcommit_memory=1 for this to take effect immediately.

Disable Transparent huge pages¶

Make sure to disable Linux kernel feature transparent huge pages, it will affect greatly both memory usage and latency in a negative way. This is accomplished with the following command: echo never > /sys/kernel/mm/transparent_hugepage/enabled.

However, this can be tricky so here the method from the MongoDB guide.

If tuned is available :

Create a new profile.¶

Create a new tuned profile directory:
sudo mkdir /etc/tuned/no-thp

Edit tuned.conf.¶

Create and edit vi /etc/tuned/no-thp/tuned.conf so that it contains the following:

[main]
include=virtual-guest

[vm]
transparent_hugepages=never

Enable the new profile.¶

Finally, enable the new profile by issuing:
sudo tuned-adm profile no-thp

Test Your Changes

You can check the status of THP support by issuing the following commands:

cat /sys/kernel/mm/transparent_hugepage/enabled
cat /sys/kernel/mm/transparent_hugepage/defrag

On Red Hat Enterprise Linux, CentOS, and potentially other Red Hat-based derivatives, you may instead need to use the following:

cat /sys/kernel/mm/redhat_transparent_hugepage/enabled
cat /sys/kernel/mm/redhat_transparent_hugepage/defrag

For both files, the correct output resembles:

always madvise [never]

If the defrag doesn’t show up with [never] then add this (courtesy of stackoverflow) :

echo 'never' > /sys/kernel/mm/transparent_hugepage/defrag

Make sure to setup some swap in your system (we suggest as much as swap as memory). If Linux does not have swap and your Redis instance accidentally consumes too much memory, either Redis will crash for out of memory or the Linux kernel OOM killer will kill the Redis process.
Set an explicit maxmemory option limit in your instance in order to make sure that the instance will report errors instead of failing when the system memory limit is near to be reached.
maxmemory 3584mb
If you are using Redis in a very write-heavy application, while saving an RDB file on disk or rewriting the AOF log Redis may use up to 2 times the memory normally used. The additional memory used is proportional to the number of memory pages modified by writes during the saving process, so it is often proportional to the number of keys (or aggregate types items) touched during this time. Make sure to size your memory accordingly.
Use daemonize no when run under daemontools.
Even if you have persistence disabled, Redis will need to perform RDB saves if you use replication, unless you use the new diskless replication feature, which is currently experimental.
If you are using replication, make sure that either your master has persistence enabled, or that it does not automatically restarts on crashes: slaves will try to be an exact copy of the master, so if a master restarts with an empty data set, slaves will be wiped as well.
By default Redis does not require any authentication and listens to all the network interfaces. This is a big security issue if you leave Redis exposed on the internet or other places where attackers can reach it. See for example this attack to see how dangerous it can be. Please check our security page and the quick start for information about how to secure Redis.

[Redis FAQ]
https://redis.io/topics/faq/