How To Install And Import Python Modules At Runtime?

I want to write a script to automatically setup a brand new ubuntu installation and install a django-based app. Since the script will be run on a new server, the Python script need

Solution 1:

You can import pip instead of using subprocess:

import pip

definstall(package):
    pip.main(['install', package])

# Exampleif __name__ == '__main__':
    try:
        import pexpect
    except ImportError:
        install('pexpect')
        import pexpect

Another take:

import pip

defimport_with_auto_install(package):
    try:
        return__import__(package)
    except ImportError:
        pip.main(['install', package])
    return__import__(package)

# Exampleif __name__ == '__main__':
    pexpect = import_with_auto_install('pexpect')
    print(pexpect)

[edit]

You should consider using a requirements.txt along with pip. Seems like you are trying to automate deployments (and this is good!), in my tool belt I have also virtualenvwrapper, vagrant and ansible.

This is the output for me:

(test)root@vagrant:~/test# pip uninstall pexpect
Uninstalling pexpect:
  /usr/lib/python-environments/test/lib/python2.6/site-packages/ANSI.py
  /usr/lib/python-environments/test/lib/python2.6/site-packages/ANSI.pyc
  /usr/lib/python-environments/test/lib/python2.6/site-packages/FSM.py
  /usr/lib/python-environments/test/lib/python2.6/site-packages/FSM.pyc
  /usr/lib/python-environments/test/lib/python2.6/site-packages/fdpexpect.py
  /usr/lib/python-environments/test/lib/python2.6/site-packages/fdpexpect.pyc
  /usr/lib/python-environments/test/lib/python2.6/site-packages/pexpect-2.4-py2.6.egg-info
  /usr/lib/python-environments/test/lib/python2.6/site-packages/pexpect.py
  /usr/lib/python-environments/test/lib/python2.6/site-packages/pexpect.pyc
  /usr/lib/python-environments/test/lib/python2.6/site-packages/pxssh.py
  /usr/lib/python-environments/test/lib/python2.6/site-packages/pxssh.pyc
  /usr/lib/python-environments/test/lib/python2.6/site-packages/screen.py
  /usr/lib/python-environments/test/lib/python2.6/site-packages/screen.pyc
Proceed (y/n)? y
  Successfully uninstalled pexpect
(test)root@vagrant:~/test# python test.py
Downloading/unpacking pexpect
  Downloading pexpect-2.4.tar.gz (113Kb): 113Kb downloaded
  Running setup.py egg_info for package pexpect
Installing collected packages: pexpect
  Running setup.py install for pexpect
Successfully installed pexpect
Cleaning up...
<module 'pexpect' from '/usr/lib/python-environments/test/lib/python2.6/site-packages/pexpect.pyc'>
(test)root@vagrant:~/test#

Solution 2:

For those who are using pip version greater than 10.x, there is no main function for pip so the alternative approach is using import pip._internal as pip instead of import pip like :

Updated answer of Paulo

import pip._internal as pip

def install(package):
    pip.main(['install', package])

if__name__== '__main__':
    try:
        import pexpect
    except ImportError:
        install('pexpect')
        import pexpect

Solution 3:

I actually made a module for this exact purpose (impstall)

It's really easy to use:

import impstall
impstall.now('pexpect')
impstall.now('wx', pipName='wxPython')

Github link for issues/contributions

Solution 4:

I solved my problem using the imp module.

#!/usr/bin/env pythonimport pip
import imp

def install_and_load(package):
    pip.main(['install', package])

    path = '/usr/local/lib/python2.7/dist-packages'if path not in sys.path:
        sys.path.append(path)

    f, fname, desc = imp.find_module(package)
    return imp.load(package, f, fname, desc)

if __name__ == "__main__":
    try:
        import pexpect
    except:
        pexpect = install_and_load('pexpect')

    # More code...

Actually the code is less than ideal, since I need to hardcode the Python module directory. But since the script is intended for a known target system, I think that is ok.

Solution 5:

I had the same issue but none of Google's searches helped. After hours debugging, I found that it may be because the sys.path is not reloaded with new installation directory.

In my case on my Ubuntu Docker, I want to import dns.resolver at runtime for Python3.8 (pre-installed). I also created ubuntu user and run all things with this user (including my Python script).

• Before installing, sys.path doesn't have /home/ubuntu/.local/lib/python3.8/site-packages since I didn't install anything.
• While installing with subprocess or pip.main like above, it creates /home/ubuntu/.local/lib/python3.8/site-packages (as user installation).
• After installing , the sys.path should be refreshed to include this new location.

Since the sys.path is managed by site module, we should reload it (ref HERE):

import site
    from importlib import reload
    reload(site)

The full block for anyone that needs:

import subprocess
import sys

try:
    import dns.resolver
except ImportError:
    subprocess.check_call([sys.executable, "-m", "pip", "install", "dnspython"])
    import site
    from importlib import reload
    reload(site)
    import dns.resolver

I'm not Python developer so these code can be simplify more. This may help in cases such as fresh CI/CD environment for DevOps engineers like me.