>_ IzzyLab

Hands-on task

This task is step 13 of the course “Redis in production”

A distributed lock for cron scripts

hard~25 min

Nightly jobs are run by cron on several servers, and sometimes the same job runs twice at the same time. Write a wrapper /usr/local/bin/with-lock (bash, executable) that takes a lock in Redis:

with-lock <name> <command...>

Requirements:

  • the lock is the key lock:<name>, its value is a unique token of this run; the key is set with a single atomic command, only if it doesn't exist yet, and with a TTL of 30 seconds (so a crashed process doesn't hold the lock forever);
  • if the lock is taken — exit immediately with a non-zero code, do not run the command and don't wait;
  • otherwise — run the command and exit with its code;
  • after the command, release the lock only if the token in the key is still yours (while the command ran, the TTL may have expired and another process may have taken the lock). The compare and delete must be atomic, a single EVAL (a Lua script on the server), not a GET + DEL pair.

Redis is already running on 127.0.0.1:6379.

What you'll practise:
  • redis
  • SET NX EX
  • EVAL
  • Lua
  • distributed lock
  • bash

Topic breakdown

The simplest distributed lock in Redis is SET key token NX EX ttl: the key is created atomically and only if it doesn't exist, and the TTL guarantees that a crashed owner won't lock everyone out forever. The price of the TTL is that the lock can expire while the command is still running, and another process will take it. If the first owner then simply does DEL, it deletes someone else's lock, and two processes end up in the critical section. That is why the value is a unique token, and release is a "compare and delete" on the server in a single Lua script via EVAL: the script runs atomically, and nobody can squeeze in between GET and DEL. A separate GET + DEL pair from the client gives no such guarantee.

What's checked

  1. /usr/local/bin/with-lock exists and is executable
  2. the command runs, then the lock is released
  3. the command's exit code is passed through
  4. while running the key exists, TTL up to 30 s, token is unique
  5. lock taken: fail fast without running the command
  6. someone else's lock is not deleted on release
  7. acquire and release are atomic — SET with NX/EX and EVAL

What it looks like