We learned this the hard way: one multi-hour fetch job got reclaimed every 15 minutes until all six of our workers were running it at once. That story is in One long job took down our whole Redis Streams worker pool. This post is the how-to.

The rule: idle time counts from delivery, not from activity

A pending message's idle time is the time since it was last delivered to a consumer. Your handler can be busy, healthy and logging away; Redis doesn't know and doesn't care. All it sees is a message that nobody has touched.

So this common reclaim loop:

r.xautoclaim(STREAM, GROUP, me, min_idle_time=15 * 60 * 1000, start_id="0-0")

doesn't mean "take over jobs whose worker died". It means "take over any job that has been running for more than 15 minutes". For short jobs those are the same thing. For long ones they are not.

Reproduce it

Two consumers on one stream. worker-a takes a job that runs for 6 seconds. worker-b runs the standard reclaim loop with a 2-second threshold (scaled down from 15 minutes so the demo is quick):

import threading, time, redis

r = redis.Redis(db=15)
STREAM, GROUP = "demo:jobs", "workers"
RECLAIM_MS = 2000
r.xgroup_create(STREAM, GROUP, id="0", mkstream=True)

r.xadd(STREAM, {"job": "big"})
[[_, [(mid, _)]]] = r.xreadgroup(GROUP, "worker-a", {STREAM: ">"}, count=1)

stolen = []
def reclaimer():
    t0, end = time.time(), time.time() + 6
    while time.time() < end:
        _, batch, *_ = r.xautoclaim(STREAM, GROUP, "worker-b",
                                    min_idle_time=RECLAIM_MS, start_id="0-0", count=1)
        if batch:
            stolen.append(round(time.time() - t0, 1))
        time.sleep(0.2)

t = threading.Thread(target=reclaimer); t.start()
time.sleep(6)          # worker-a is "working"
t.join()

p = r.xpending_range(STREAM, GROUP, min=mid, max=mid, count=1)[0]
print("claimed by worker-b at", stolen, "| owner", p["consumer"],
      "| times_delivered", p["times_delivered"])

Output:

claimed by worker-b at [2.0, 4.0] | owner b'worker-b' | times_delivered 3

worker-a never stopped working, yet the job was handed out twice more. In a real pool, each of those claims is a different worker starting the same job from scratch.

Why raising the timeout doesn't fix it

The obvious patch is min_idle_time=2 hours. That just moves the cliff: the next job that runs for 2 hours and 1 minute does the same thing. Meanwhile, a worker that really did crash keeps its job locked for 2 hours before anyone picks it up. You've made recovery slower without making long jobs safe.

What you want is for "idle" to mean "nobody has heard from this worker lately". That means the worker has to say something.

The fix: renew the lease with XCLAIM ... JUSTID

XCLAIM with min_idle_time=0 on a message resets its idle time to zero. With JUSTID it doesn't return the payload and doesn't increment the delivery counter, so it's a pure "I'm still here".

Run that from a background thread every couple of minutes while the handler works, and the reclaim loop never sees the job as idle. Stop it when the job finishes.

The check most heartbeats forget

A plain XCLAIM <id> 0 succeeds no matter who currently owns the message. Picture this:

1. worker-a stalls for 20 minutes (a long GC pause, a frozen VM, a network partition).

2. worker-b reclaims the job, correctly, and starts it.

3. worker-a wakes up and its heartbeat runs XCLAIM <id> 0 again.

Now the message belongs to worker-a again, and both workers are running the job. So before renewing, check that you still own it:

def renew(r, stream, group, consumer, message_id):
    pending = r.xpending_range(stream, group, min=message_id, max=message_id, count=1)
    if not pending:
        raise LeaseLost("acked elsewhere")
    if pending[0]["consumer"].decode() != consumer:
        raise LeaseLost("now owned by another worker")
    r.xclaim(stream, group, consumer, min_idle_time=0,
             message_ids=[message_id], justid=True)

(There is a tiny window between the two calls. It's acceptable here because the alternative failure, losing ownership for good, only happens after a stall far longer than the renewal interval.)

Using the library

We packaged that pattern, the thread and the ownership check into redis-streams-lease, one MIT-licensed file with no dependencies besides redis:

pip install git+https://github.com/goodluck8866/redis-streams-lease

A complete worker:

import socket, redis
from redis_streams_lease import Lease

STREAM, GROUP = "jobs", "workers"
ME = socket.gethostname()
RECLAIM_IDLE_MS = 15 * 60 * 1000   # a worker silent for 15 min is presumed dead
LEASE_INTERVAL = 120               # renew every 2 min

r = redis.Redis()

while True:
    # First, jobs whose worker really died (its lease stopped renewing).
    _, batch, *_ = r.xautoclaim(STREAM, GROUP, ME, min_idle_time=RECLAIM_IDLE_MS,
                                start_id="0-0", count=1)
    if not batch:
        resp = r.xreadgroup(GROUP, ME, {STREAM: ">"}, count=1, block=4000)
        batch = resp[0][1] if resp else []

    for message_id, fields in batch:
        with Lease(r, STREAM, GROUP, ME, message_id, interval=LEASE_INTERVAL) as lease:
            handle(fields)                     # may take hours
        if not lease.lost:
            r.xack(STREAM, GROUP, message_id)

Rerun the reproduction with the handler wrapped in Lease(..., interval=0.5):

lease renewals 11 lost False
claimed by worker-b at [] | owner b'worker-a' | times_delivered 1

No reclaims, and the delivery counter never moved.

Crash recovery still works

The renewal thread is a daemon thread. If the worker process dies, the thread dies with it, idle time starts growing again, and the ordinary XAUTOCLAIM loop takes over. Simulating that by stopping a lease without acking:

after crash, reclaimed in 1.5 s

That's the 2-second threshold minus the half-second since the last renewal, which is exactly the behaviour you want: a dead worker's job is recovered on the normal schedule, and a live worker's job is left alone.

lease.lost covers the remaining case. If the lease was lost (another worker took over after a long stall), the handler finishes but does not ack, so it doesn't acknowledge a message it no longer owns. Pass on_lost= if you want to cancel the work early instead.

Picking the numbers

The renewal interval has to be well below the reclaim threshold, so a few missed renewals don't hand the job away.

The reclaim threshold now only has one job, deciding how long a silent worker is given before it's presumed dead. It no longer caps how long a job may run, so you can keep it short.

Other approaches, compared

Splitting jobs is still worth doing when it's natural, and a status check makes a good second line of defence (we run one). But the lease is the only option that fixes the root cause in the stream itself, without new state to keep in sync.

FAQ

Does XCLAIM ... JUSTID increment the delivery counter? No. JUSTID claims without re-delivering, so times_delivered stays where it was. The reproduction above shows it staying at 1 through 11 renewals.

Which Redis version do I need?XAUTOCLAIM needs Redis 6.2+. The lease itself only uses XPENDING and XCLAIM, which exist since 5.0. We tested on Redis 7.0 with redis-py 8.1.

What if Redis restarts while a job is running? The renewal fails, gets logged, and is retried on the next tick. If Redis stays down long enough for the job to look idle, it gets reclaimed, which is the right outcome when a worker genuinely can't be reached.

Should I use threads or asyncio? The library uses a thread because most queue workers are synchronous, and a thread keeps renewing even while the handler blocks the event loop. In an asyncio worker, call renew() from a task on a timer instead.

What about jobs that were already duplicated? The lease prevents new duplicates. If your handlers aren't idempotent yet, make the writes upserts too: even with a lease, a worker that stalls past the threshold will lose its job to another one.

The workers this came from fetch Google search ad data for PPCEye, which shows which advertisers run search ads for any domain. The pattern itself has nothing to do with ads: if you run long jobs on Redis Streams, you probably want it. Issues and PRs are welcome on GitHub.