trjhtr

Sometimes I download a Python script (from a trusted source), and can't run it because of missing dependencies, which I have to install one by one (no requirements.txt is provided). I'm looking for a way to automate this process. It's similar to this question, but I'd like to infer the requirements automatically. I.e. given the script, install all the packages it's importing.

If myscript.py looks like:

import pandas
import tensorflow as tf
from keras.models import Sequential

Then after running the script (e.g. bash extract_requirements.sh myscript.py) pandas, tensorflow, and keras will be installed.

Of course, a script can call another script, which has its own dependencies, package names don't always match to import names etc. So it's not a perfect solution, but better than nothing.

What I'm using now is a command-line script:

#!/env bash

set -u

usage() 
 echo "$0" '[-d] <MY_SCRIPT>'
 cat <<-EOF
 options:
 -d Dry-run. Print package names without installing.
EOF
 exit 1;


(( $# == 0 )) && usage
dry=0
if [[ "$1" == '-d' ]]; then
 dry=1
 shift
fi
(( $# != 1 )) && usage

script="$1"
imports1=$(egrep '^s*froms+[a-zA-Z_.-]+simport' "$script" | cut -f2 -d' ' | cut -f1 -d'.')
imports2=$(egrep '^s*imports+[a-zA-Z_-]+' "$script" | cut -f2 -d' ')
all=$(printf "$imports1n$imports2" | sort -u)
if (( dry )); then
 echo "$all" | xargs -L 1 echo
else
 echo "$all" | xargs -L 1 sudo pip install
fi

set +u

I was wondering if I'm missing some import cases, and more importantly, whether there's a simpler, more python-ic, solution.

!!! Use this at your own risk. Fiddling with user's packages is scary business !!!

You can monkey-patch the builtin import statement and upon encountering ModuleNotFoundError install whatever dependency is missing. Here is a small example for a script.py. You can of course extend this to feature multiple files or dynamically use the file given as input.

import builtins
import pip
old_import = __import__

def my_import(name, globals=None, locals=None, fromlist=(), level=0):
 if globals["__name__"] == "script":
 try:
 return_args = old_import(name, globals=globals, locals=locals, 
 fromlist=fromlist, level=level)
 except ModuleNotFoundError:
 builtins.__import__, temp = old_import, builtins.__import__
 pipcode = pip.main(['install', name])
 if pipcode != 0:
 builtins.__import__ = temp
 raise
 else:
 return_args = old_import(name, globals=globals, locals=locals, 
 fromlist=fromlist, level=level)
 builtins.__import__ = temp
 else:
 return_args = old_import(name, globals=globals, locals=locals, 
 fromlist=fromlist, level=level)

 return return_args

builtins.__import__ = my_import

import script

and the script.py is your custom script:

import os
from time import time

import argh # this should be installed

print(time())

Note: People sometimes do things like:

try:
 import optional_module
except ImportError:
 pass # optional all cool

Above code patches import itself. If you remove the if that limits install powers to script you allow recursive installation for everything that is imported. Ever. Even if it is an optional dependency for a 4th level dependency test case of script pip will try to install it.