Celery Prefetch Multiplier Setting No Effect

I have a series of servers running multiple workers. These are long-running tasks, requiring anywhere from 10 minutes to 36 hours. As such, I'd like to avoid prefetching if at al

Solution 1:

I deal with the same problem this way.

Running worker with: celery worker --prefetch-multiplier=1 --concurrency=1 -Ofair

Under Django settings: CELERY_TASK_ACKS_LATE = True

Django==3.0.2
celery==4.4.0

---

It's important to have concurrency=1, because prefetch-multiplier multiplies on concurrency

-Ofair - might be redundant, need to check

Acks late param on worker didn't find (or didn't work) for me

Setting without django need to check (too lazy for it). Might be CELERYD_TASK_ACKS_LATE = True