Sed Command Run Using Os.system() Or Subprocess.call() Leaves Csv File Without A Delimiter

I am running a Python script which takes the dump of CSVs from a Postgres database and then I want to escape double quotes in all these files. So I am using sed to do so. In my Pyt

Solution 1:

Do not use intermediate shell when you do not need to. And check for return code of the subprocess to make sure it completed successfully (check_call does this for you)

path_to_file = ... # e.g. '/home/ubuntu/PTOR/csvdata1/' + table + '.csv'
subprocess.check_call(['sed', '-i', 's/"//g', path_to_file])

By "intermediate" shell I mean the shell process run by subprocess that parses the command (± splits by whitespace but not only) and runs it (runs sed in this example). Since you precisely know what arguments sed should be invoked with, you do not need all this and it's best to avoid that.

Solution 2:

Put your sed into a shell script, e.g.

#!/bin/bash# Parameter $1 = Filename
sed -i 's/"//g'"$1"

Call your shell script using subprocess:

sed_for_quotes = 'my_sed_script /home/ubuntu/PTOR/csvdata1/'+table+'.csv'

---

Use docs.python.org/3.6: shlex.split shlex.split(s, comments=False, posix=True) Split the string s using shell-like syntax.