Python Multiprocessing Apply_async "assert Left > 0" Assertionerror

I am trying to load numpy files asynchronously in a Pool: self.pool = Pool(2, maxtasksperchild = 1) ... nextPackage = self.pool.apply_async(loadPackages, (...)) for fi in np.arange

Solution 1:

There is a bug in Python C core code that prevents data responses bigger than 2GB return correctly to the main thread. you need to either split the data into smaller chunks as suggested in the previous answer or not use multiprocessing for this function

I reported this bug to python bugs list (https://bugs.python.org/issue34563) and created a PR (https://github.com/python/cpython/pull/9027) to fix it, but it probably will take a while to get it released (UPDATE: the fix is present in python 3.8.0+)

if you are interested you can find more details on what causes the bug in the bug description in the link I posted