Numpy Array Memory Issue

I believe I am having a memory issue using numpy arrays. The following code is being run for hours on end: new_data = npy.array([new_x, new_y1, new_y2, new_y3]) private.dat

Solution 1:

Use Python lists. Seriously, they grow far more efficiently. This is what they are designed for. They are remarkably efficient in this setting.

If you need to create an array out of them at the end (or even occasionally in the midst of this computation), it will be far more efficient to accumulate in a list first.

Solution 2:

Update: I incorporated @EOL's excellent indexing suggestion into the answer.

The problem might be the way row_stack grows the destination. You might be better off handling the reallocation yourself. The following code allocates a big empty array, fills it, and grows it as it fills an hour at a time

numcols = 4
growsize = 60*60 #60 samples/min * 60 min/hour
numrows = 3*growsize #3 hours, to start withprivate.data = npy.zeros([numrows, numcols]) #alloc one big memory block
rowctr = 0
while (recording):
    private.data[rowctr] = npy.array([new_x, new_y1, new_y2, new_y3])
    rowctr += 1
    if (rowctr == numrows): #full, grow by another hour's worth of dataprivate.data = npy.row_stack([private.data, npy.zeros([growsize, numcols])])
        numrows += growsize

This should keep the memory manager from thrashing around too much. I tried this versus row_stack on each iteration and it ran a couple of orders of magnitude faster.