>_ IzzyLab

Hands-on task

This task is step 15 of the course “Redis in production”

Reclaim messages from a crashed worker

medium~10 min

The stream jobs is processed by the group workers. Worker worker-old took several jobs and crashed — its messages have been sitting unacknowledged for a long time (over an hour). Worker worker-live is alive and is processing its job right now. Worker worker-slow is alive too — it has been computing a heavy report for about 30 minutes, which is normal for it.

Reassign to consumer worker-new all pending messages of the group that have been idle longer than 1 hour (3600000 ms), using XAUTOCLAIM. Then "process" them and acknowledge them.

Requirements:

  • worker-old must have no pending messages left;
  • don't touch the messages of worker-live and worker-slow — they must stay with their owners, unacknowledged;
  • don't change the stream itself: don't delete anything.
What you'll practise:
  • redis
  • streams
  • XAUTOCLAIM
  • XPENDING
  • XACK
  • consumer group

Topic breakdown

In a consumer group every delivered message is assigned to a specific consumer in the Pending Entries List (PEL) until it is acknowledged with XACK. If a worker crashes, its messages won't go anywhere on their own — they will hang there forever. So production needs a "collector": periodically look for messages idle longer than a reasonable threshold and reassign them to a live worker. XAUTOCLAIM does that in a single call: it scans the PEL from a cursor, claims everything idle longer than min-idle-time, and returns the next cursor (0-0 means the scan is complete). The threshold matters: too small, and it takes jobs away from live but slow workers, and one job gets done twice. The delivery counter (times-delivered in XPENDING) helps find "poison" messages that crash workers again and again.

What's checked

  1. worker-old has no pending messages
  2. messages transferred to worker-new via XAUTOCLAIM
  3. worker-new acknowledged the received messages
  4. worker-live and worker-slow messages untouched
  5. jobs stream unchanged

What it looks like