In Celery Task Queue, Is Running Tasks In A Group Any Different Than Multiple Asyncs In A Loop?

Let's say I have a very simple task like this: @celery.task(ignore_result=True) def print_page(page): with open('path/to/page','w') as f: f.write(page) (Please ignore

Solution 1:

Your first choice seems like the better one. You have no desire to join the results (given that ignore_result=True) of the fanned-out print_pages tasks so a group adds unnecessary overhead/complexity. Just invoke the tasks individually as in choice A and you're fine.

Further though, I'd like to note that Python generators will not pickle so you cannot pass them asynchronously to Celery tasks.

Solution 2:

both solution is correct in your case there is no depending in the pages tasks but lets assume you have a task divided in sub tasks and all these subtasks are depedant sequentially in this case you should group it by choosing B