>_ IzzyLab

Hands-on task

This task is step 11 of the course “Redis in production”

Sentinel and a manual failover

hard~25 min

A pair is running: a master on 6379 with data and a replica on 6380 replicating it. The master needs to go down for maintenance — switch the roles through Sentinel, so that clients asking it for the master address move to the new one.

  1. Configure Sentinel in /etc/redis/sentinel.conf: port 26379, monitoring master mymaster = 127.0.0.1:6379 with quorum 1 (we have a single Sentinel), down-after-milliseconds for mymaster — no more than 10000.
  2. Start Sentinel in the background and make sure it has found the replica.
  3. Perform a manual failover using Sentinel.

Result: Sentinel reports master 127.0.0.1:6380, instance 6380 is the master, the former master 6379 has become its replica, no data is lost. Don't switch the roles by hand with REPLICAOF — Sentinel must do it.

What you'll practise:
  • redis
  • Sentinel
  • failover
  • high availability
  • replication
  • quorum

Topic breakdown

Redis Sentinel is a separate process that watches a master and its replicas, discovers replicas by itself through the master's INFO, and when the master fails it performs an automatic failover: it picks a replica, promotes it to master (REPLICAOF NO ONE), reconfigures the others and gives clients the new address via SENTINEL GET-MASTER-ADDR-BY-NAME. A master is considered down when it hasn't responded for longer than down-after-milliseconds and quorum sentinels agree; in production there are at least three of them on different machines to avoid split-brain. SENTINEL FAILOVER <name> starts a switchover manually without an agreement — that is how a master is taken down for maintenance with no downtime for clients that get the address from Sentinel.

What's checked

  1. Sentinel on 26379 monitors mymaster (quorum 1, fast down-after)
  2. Sentinel reports master 127.0.0.1:6380
  3. 6380 became the master
  4. former master 6379 is a replica of 6380
  5. data on the new master is intact

What it looks like