Python Celery - Get() Is Delayed

I am running the following simple example. Submit 20 jobs that take 2 seconds each using a single worker: celery -A celery_test worker --concurrency 10 -l INFO This should take 2

Solution 1:

It's because you wait for each job result in the loop. So you loose somewhat the benefits of concurrency because jobs results don't arrive in the same order as you request the results. See the example below with some timings added to get all time :

from celery_test import add
import time
results = []
for i in range(20):
    results.append(add.delay(i))

allTimeStart = time.time()

for result in results:
    timeStart = time.time()
    resultValue = result.get(timeout=10)
    timePassed = time.time() - timeStart
    allTimePassed = time.time() - allTimeStart
    print(allTimePassed, timePassed, resultValue)

Gives

(1.9835469722747803, 1.9835450649261475, 0)
(1.9858801364898682, 0.0022699832916259766, 1)
(1.988955020904541, 0.003039121627807617, 2)
(1.9928300380706787, 0.003849029541015625, 3)
(1.9935901165008545, 0.0007331371307373047, 4)
(1.9967319965362549, 0.0031011104583740234, 5)
(1.9973289966583252, 0.0005509853363037109, 6)
(2.0004770755767822, 0.003117084503173828, 7)
(2.0007641315460205, 0.00026702880859375, 8)
(3.00203800201416, 1.001255989074707, 9)
(3.9891350269317627, 0.9870359897613525, 10)
(3.9914891719818115, 0.0023059844970703125, 11)
(3.99283504486084, 0.001302957534790039, 12)
(3.99426007270813, 0.0013878345489501953, 13)
(3.997709035873413, 0.003403902053833008, 14)
(3.9984171390533447, 0.0006809234619140625, 15)
(4.000844955444336, 0.0024080276489257812, 16)
(4.004598140716553, 0.003731966018676758, 17)
(4.0053839683532715, 0.0007598400115966797, 18)
(5.006708145141602, 1.0012950897216797, 19)

But if you look the order of celery tasks results in celery log, you see that results don't arrive ordered as you request them :

[2017-05-31 01:06:39,067:INFO/PoolWorker-2] Taskcelery_test.add[01fe4581-7982-40f3-92d3-9f352d0b8eca]succeeded in 2.00315466001s:0
[2017-05-31 01:06:39,069:INFO/PoolWorker-8] Taskcelery_test.add[f468849c-76d9-4479-b7e2-850aab640437]succeeded in 2.003014307s:1
[2017-05-31 01:06:39,072:INFO/PoolWorker-3] Taskcelery_test.add[db6a0064-0a83-49dc-a731-54264651a32f]succeeded in 2.002590772s:3
[2017-05-31 01:06:39,072:INFO/PoolWorker-4] Taskcelery_test.add[421b1213-e1b7-4c73-8477-1554c53c4b14]succeeded in 2.002614007s:2
[2017-05-31 01:06:39,076:INFO/PoolWorker-7] Taskcelery_test.add[90bdde7f-9740-4d18-820d-dc4c66090b2b]succeeded in 2.00297982999s:4
[2017-05-31 01:06:39,077:INFO/PoolWorker-5] Taskcelery_test.add[661cba10-326a-4351-9fec-56d029847939]succeeded in 2.003134354s:5
[2017-05-31 01:06:39,080:INFO/PoolWorker-10] Taskcelery_test.add[31903dfe-4b35-49b8-bc66-8c8807a1ee53]succeeded in 2.00229301301s:6
[2017-05-31 01:06:39,080:INFO/PoolWorker-9] Taskcelery_test.add[60049a1b-009b-4d7b-ad4e-284f0d2e7147]succeeded in 2.00245238301s:7
[2017-05-31 01:06:39,084:INFO/PoolWorker-1] Taskcelery_test.add[4e673409-af0e-4a59-8a42-38f0179b495a]succeeded in 2.00299428699s:8
[2017-05-31 01:06:39,084:INFO/PoolWorker-6] Taskcelery_test.add[818bcea5-5654-4ec6-8706-1b6ca58f8735]succeeded in 2.002899974s:9
[2017-05-31 01:06:41,072:INFO/PoolWorker-2] Taskcelery_test.add[4ab62e6d-ada3-4e0d-82e2-356eb054631f]succeeded in 2.00349172599s:10
[2017-05-31 01:06:41,074:INFO/PoolWorker-8] Taskcelery_test.add[649c83db-a065-4cdd-9f5e-32ae1e5047f4]succeeded in 2.003091722s:11
[2017-05-31 01:06:41,076:INFO/PoolWorker-4] Taskcelery_test.add[f6a6e067-7f60-4c1f-b8f4-dce40a6094c0]succeeded in 2.00157168499s:12
[2017-05-31 01:06:41,077:INFO/PoolWorker-3] Taskcelery_test.add[ee7b0e01-2fa7-4bd0-b2f2-f5636155209b]succeeded in 2.00259804301s:13
[2017-05-31 01:06:41,081:INFO/PoolWorker-7] Taskcelery_test.add[521f7903-3594-4aab-b4df-3a4e723347cd]succeeded in 2.002994123s:14
[2017-05-31 01:06:41,081:INFO/PoolWorker-5] Taskcelery_test.add[26a3627c-7934-4613-b3c1-618784bbce26]succeeded in 2.003302467s:15
[2017-05-31 01:06:41,084:INFO/PoolWorker-9] Taskcelery_test.add[8e796394-b05f-439b-b695-6d3ff3230844]succeeded in 2.00281064s:17
[2017-05-31 01:06:41,084:INFO/PoolWorker-10] Taskcelery_test.add[13b40cd8-b0e4-4788-a3bb-4d050c1b6ad0]succeeded in 2.00298337401s:16
[2017-05-31 01:06:41,088:INFO/PoolWorker-6] Taskcelery_test.add[cb8f1303-4d05-4eae-9b40-b2d221f20140]succeeded in 2.00274520101s:19
[2017-05-31 01:06:41,088:INFO/PoolWorker-1] Taskcelery_test.add[0900bb54-8e2a-472c-99a8-ee18a8f4857c]succeeded in 2.003100015s:18

One solution : use group to get all results :

from celery_test import addfrom celery import group
import time
results = []
jobs = []
for i inrange(20):
    jobs.append(add.s(i))
result = group(jobs).apply_async()
timeStart = time.time()
print(result.join())
timePassed = time.time() - timeStart
print(timePassed)

Returns

[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
4.00328302383