Redis

Redis Sentinel

Assume a scenario where you have only one Redis instance in your production and it fails at some point due to some reason. Your application caches data in the Redis data store and now your only data source is dead. One way to control these sorts of scenarios is to maintain master-slave architecture where slaves can replicate the master node until it comes back. Redis clusters support high availability up to some degree with the master-replica approach. Redis Sentinel is another approach that provides a more reliable way to maintain the high availability of Redis instances. It monitors the Redis master node for failures and triggers the failover process immediately which will promote an existing slave node to a brand new master.

Furthermore, Redis sentinel acts as a middle-man where clients connect and ask for the latest Master node IP address. So, the connected sentinel provides the master node address immediately.

In addition, a master node failure is confirmed if multiple sentinels agreed that a given master is not reachable or available. This concludes the failure detection phase and the failover process starts right away. Hence, the Redis sentinel can be seen as a distributed system with specific properties.

The agreement of sentinels is based on a quorum value which will be discussed in the following section.

Quorum Value

The Quorum value is the maximum number of sentinels that needs to be agreed on when the master node is down. This value is only used to identify a failure in the master node. The failover process starts with the authorization of multiple available sentinel nodes to proceed with a selected sentinel as the leader.

Features of Redis Sentinel

The sentinel is known for providing a high availability mechanism for the Redis data store. Apart from that, several other capabilities can be listed.

  • Sentinel continuously monitors the status of master and slave nodes in your Redis system.
  • Whenever there is a failure or something wrong with your Redis instances, the sentinel is capable of notifying the administrator or connected applications using sentinel API.
  • The failover phase is directed by the sentinel by promoting a replica as the new master. Remaining replicas configured to use the new master. Finally, the corresponding clients will get notified of the new master node address.
  • Also, the Redis sentinel is a configuration provider for the connected clients where clients can ask for the address of the currently available master instance and if a sudden collapse occurred, the sentinel is committed to pushing the new master node address immediately.

 

In the next section, we will be configuring Redis sentinels with master-replica instances and using the sentinel API to monitor the nodes.

Sentinel Configuration

First, we create two Redis instances at ports 7000 and 7001. Port 7000 is going to be the master node and the other one replicates the master. Both instances use the following config files respectively:

Master Node Configuration

port 7000
cluster-enabled no
cluster-config-file nodes.conf
cluster-node-timeout 5000
appendonly yes

 

Slave Node Configuration

port 7001
cluster-enabled no
cluster-config-file nodes.conf
cluster-node-timeout 5000
appendonly yes

 

Both instances will start by providing the configuration file associated with each. We can use the following command to start Redis instances separately:

redis-server redis.conf

 

Let’s connect to the Redis instance started at port 7001 as follows:

redis-cli -p 7001

 

Now, we can make this instance a replica of the master which is running at port 7000. The REPLICAOF command can be used as follows:

replicaof 127.0.0.1 7000

 

As expected, the instance running at port 7001 became the replica node of the master running at port 7000.

Now, we are ready to configure three Redis sentinels to monitor the above master instance. We need to have three configuration files to create three sentinel instances at ports 5000, 5001, and 5002 as shown in the following.

Each sentinel.conf file looks like the following except that the port number will be changed:

port 5000
sentinel monitor masternode 127.0.0.1 7000 2
sentinel down-after-milliseconds masternode 5000
sentinel failover-timeout masternode 60000

 

Now, it’s time to run the three sentinels. You can use the redis-sentinel executable along with the path to sentinel.conf configuration file to create a sentinel instance. Otherwise, we can still call the redis-server executable by specifying the path to sentinel.conf and the flag –sentinel.

Let’s start each sentinel using the following command:

redis-server sentinel.conf --sentinel

 

The first sentinel has been started at port 5000. Similarly, you can start the other two instances as well.

Now, our Redis sentinel setup is up and running as shown in the following illustration:

In the following section, we will be exploring more about Sentinel API and how we can utilize it to retrieve information related to the Redis master node.

Sentinel API

Redis provides a separate sentinel API to monitor associated masters and replicas, subscribe for notifications, and modify the sentinel settings. Furthermore, several usages are listed in the following.

 

  • Check the status of the monitored Redis master and slave instances
  • Details about other sentinels
  • Receive push-style notifications from sentinels in an event of a failover

 

The SENTINEL command can be used with its associated subcommands to query, update, or set Redis sentinels and monitored nodes.

Check the Status of Master Node

It is very important to monitor or check master node health from time to time. The following sentinel API command can be used to retrieve master details:

SENTINEL MASTER <monitored_master_name>

 

monitored_master_name: The name of the master node that is specified in the sentinel configuration file that we created in the earlier step.

Let’s use this command to query master status in our setup. In our case, the master node name is ‘masternode’.

SENTINEL MASTER masternode

 

Several pieces of information have been retrieved and a few of them are important such as num-slaves, flags, and num-other-sentinels.

The flags property is set to master which means the master is in good health. Whenever the master node is down, the s_down or o_down flag will be displayed. The property num-other-sentinels is set to 2 which means the Redis sentinel already recognized the other two sentinels for the master node. In addition, the num-slaves property displays the available replicas for the master node. In this case, it is set to 1 since we have only one replica.

Get Information About Connected Replicas

We can check the replicas connected with the master node using the following SENTINEL sub command:

SENTINEL REPLICAS <monitored_master_name>

 

In this example, the master name is ‘masternode’.

SENTINEL replicas masternode

 

As expected, the Sentinel detected the slave node running at port 7001.

Get Information About Associated Sentinels

Similarly, we can query the details related to other sentinels associated with the current master node using the following SENTINEL subcommand:

SENTINEL SENTINELS <master_node_name>

 

In this case, we will be fetching the information related to the master node named ‘masternode’.

SENTINEL sentinels masternode

 

Obtain the Master Node Address

As mentioned in the earlier section, Redis sentinel is a configuration provider for connected clients. So, it is capable of providing the currently running master node IP address and port to the requested clients. The following Sentinel API subcommand can be used to retrieve the mentioned information.

SENTINEL GET-MASTER-ADDR-BY-NAME <master_node_name>

 

Let’s execute the above command for our scenario as follows:

sentinel get-master-addr-by-name masternode

 

We only discussed a few sentinel API commands. Several other subcommands are available such as sentinel-failover, sentinel info-cache, sentinel masters, and etc. Furthermore, many commands are available to use for administration purposes as well. In the following section, we will be focusing on the Redis sentinel failover process.

Sentinel Failover Process

Since our sentinel is configured, we can test the fail-over phase. Let’s send our master node to sleep for 300 seconds which simulates a failure in the master node.

debug sleep 300

 

The master node which is running at port 7000 should be unreachable now. So, associated sentinels will notice the master is unavailable with the +sdown event. Then, this will be set to +odown where 2 sentinels confirm the master node is down according to the quorum value. Finally, the failover phase will start and ideally the replica should be promoted to the new master.

Let’s check the master node IP address and the port again.

sentinel get-master-addr-by-name masternode

 

As expected, the previous replica has been promoted to the new master which means the sentinel failover process is successful. This concludes the deployment and testing of our three sentinel setups for single master-replica pair.

Conclusion

Redis sentinel is the most reliable approach to ensure the high availability of a given Redis master replica instance. A sentinel is capable of monitoring, notifying, and initiating automatic failover without human intervention. Also, multiple sentinels together agree about the fact that the master node is unreachable and the quorum value is used as the maximum number of sentinels that need to be agreed upon when checking for the availability of the master instance. Redis sentinel offers an easy-to-use API to retrieve information about the health of the master node and associated replicas and perform administrative tasks as well.

About the author

Nimesha Jinarajadasa

Being a Full-stack Senior Software Engineer for more than five years, I love technology, as technology has the power to solve our many problems within just a minute. I try to learn more and create more opportunities for this new world.