Updating Csv File (add/remove Rows) With Python 3

G'day everyone, I've got a Raspberry Pi system which keeps track of tools being checked out by various users. I've set it up such that a scan of the system is performed when a user

Solution 1:

I don't think csv is the right structure here. You want to be able to look up a given tool, find out whether its tools_taken is True, or change its tools_taken, or remove a tool from the file, or add a tool to the file, right?

That's what a database is for, such as shelve:

import contextlib
import shelve

tools = shelve.open('Log.db', 'c', writeback=True)
with contextlib.closing(tools):
    # Add a tool
    tools['hammer'] = {'name': 'Hammer', 'owner': 'Joe', 'tools_taken': False}
    # Check if a tool is takenif tools['screwdriver']['tools_taken']:
        print('The screwdriver is taken!')
    # Change a tool's taken status:
    tools['screwdriver']['tools_taken'] = True# Remove a tooldel tools['jackhammer']

In other words, you can just it just like a dict (in this case, full of dicts), but it's automatically persistent across runs.

Solution 2:

From what I understand, you can't remove a single row from a CSV file? I would have to re-write the file, with that particular row being omitted?

Exactly. In fact, that's true for files in general. In order to remove stuff from the middle of a file, you have to move the whole rest of the file upward, then truncate the cruft left behind. Which you generally don't want to do, so the csv module won't help you do it.

---

So, how do you create a new CSV file? Three ways:

1. Open in read mode, read the whole file in, close, open in write mode, write the whole thing out, close.
2. Rename to Log.csv.bak, open that in read mode, open Log.csv in write mode, and copy from one to the other.
3. Open Log.csv in read mode, open a temporary file in write mode, copy from one to the other, then atomically rename the temporary file to Log.csv.

---

The third one is usually the best—but unfortunately, it's very hard to get it right in a cross-platform way, or even just for Windows. (It's very easy if you only care about Unix, however.) So, I'll show the second:

os.rename('Log.csv', 'Log.csv.bak')
withopen('Log.csv.bak') as infile, open('Log.csv', 'w') as outfile:
    reader = csv.reader(infile)
    writer = csv.writer(outfile)
    forrowin reader:
        if not supposed_to_be_removed(row):
            writer.writerow(row)

That's it.

---

This is similar to the way you'd write a copying algorithm in place of a mutating algorithm for a simple list:

newlist = [rowforrowin oldlist if not supposed_to_be_removed(row)]

Of course you could write that in terms of iterators instead of lists:

newlist = (rowforrowin oldlist if not supposed_to_be_removed(row))

And in fact, you can use the exact same iterator here:

os.rename('Log.csv', 'Log.csv.bak')
withopen('Log.csv.bak') as infile, open('Log.csv', 'w') as outfile:
    reader = csv.reader(infile)
    writer = csv.writer(outfile)
    newrows = (rowforrowin reader if not_supposed_to_be_removed(row))
    writer.writerows(newrows)

You could even turn it into a one-liner, if you wanted:

    writer.writerows(rowforrowin reader if not supposed_to_be_removed(row))

---

Finally, it might be worth considering whether a csv file is really the right answer here. If you're doing a whole bunch of operations, continually re-reading and re-writing the file over and over is going to be a pain—and be slow. Maybe you could keep it in memory and just read and write and startup and shutdown, but then you have to make sure you don't lose data on errors. See my other answer for another alternative.