Is there a tool to check what names I have used from a "wildly" imported module?

Question

I've been using python to do computations for my research. In an effort to clean up my terrible code, I've been reading Code Like a Pythonista: Idiomatic Python by David Goodger.

In this article, Goodger advises against "wild-card" imports of the form

from module import *

My code uses a lot of these. I'd like to clean my code up, but I'm not sure how. I'm curious if there is a way to check what names from module I have used in my code. This way I could either explicitly import these names or replace the names with module.name. Is there a tool designed to accomplish such a task?

Answer 1

Here's a simpler solution. It uses the ast module to strip the code out of a file and then compares it to the list of available functions found by the inspect module. Just replace the yourfilename and the yourmodulename before running.

import ast, inspect, yourmodulename as mymodule
filename='yourfilename.py'
tab = ' '*4

funcs = {m[0] for m in inspect.getmembers(mymodule) 
        if str(m[1])[1:].split(' ')[0] in ('function', 'class') and 
        inspect.getmodule(m[1]) == mymodule}
with open(filename) as f:
    code = ast.parse(f.read())
    words = {node.id for node in ast.walk(code) if isinstance(node, ast.Name)}
    print('from', mymodule.__name__, 'import (')
    out = tab
    for word in (', '.join((sorted(words & funcs)))+')').split():
        if len(out + word) > 80:
            print(out.rstrip())
            out = tab + word + ' '
        else:
            out += word + ' '
    if out:
        print(out.rstrip())

Edit: I checked it against my 3k+ line script with dozens of functions and it works.

Edit: Added check to make sure that It's only listing functions that are actually part of the desired module, not its submodules. Also, modified to make it list both classes and functions to import.

Answer 2

Overview

One approach is to:

• Manually (or even automatically) identify each of the modules you've imported from using the * approach,
• Import them in a separate file,
• And then do a search-and-replace if they appear in sys.modules[<module>].__dict__, which keeps track of which objects from a Python module have been loaded.

See for yourself sys.modules in action:

from numpy import *
import sys

sys.modules['numpy'].__dict__.keys()  # will display everything you just imported from `numpy`
>>> ['disp', 'union1d', 'all', 'issubsctype', 'savez', 'atleast_2d', 'restoredot', 'ptp', 'PackageLoader', 'ix_', 'mirr', 'blackman', 'FLOATING_POINT_SUPPORT', 'division', 'busdaycalendar', 'pkgload', 'void', 'ubyte', 'moveaxis', 'ERR_RAISE', 'void0', 'tri', 'diag_indices', 'array_equal', 'fmod', 'True_', 'indices', 'loads', 'round', 'set_numeric_ops', 'pmt', 'nanstd', '_mat', 'cosh', 'object0', 'argpartition', 'FPE_OVERFLOW', 'index_exp', 'append', 'compat', 'nanargmax', 'hstack', 'typename', 'diag', 'rollaxis', 'ERR_WARN', 'polyfit', 'version', 'memmap', 'nan_to_num', 'complex64', 'fmax', 'spacing', 'sinh', '__git_revision__', 'unicode_', 'sinc', 'trunc', 'vstack', 'ERR_PRINT', 'asscalar', 'copysign', 'less_equal', 'BUFSIZE', 'object_', 'divide', 'csingle', 'dtype', 'unsignedinteger', 'fastCopyAndTranspose', 'bitwise_and', 'uintc', 'select', 'deg2rad', 'nditer', 'eye', 'kron', 'newbuffer', 'negative', 'busday_offset', 'mintypecode', 'MAXDIMS', 'sort', 'einsum', 'uint0', 'zeros_like', 'int_asbuffer', 'uint8', 'chararray', 'linspace', 'resize', 'uint64', 'ma', 'true_divide', 'Inf', 'finfo', 'triu_indices', 'complex256', 'add_newdoc', 'seterrcall', 'logical_or', 'minimum', 'WRAP', 'tan', 'absolute', 'MAY_SHARE_EXACT', 'numarray', 'array_repr', 'get_array_wrap', 'polymul', 'tile', 'array_str', 'setdiff1d', 'sin', 'longlong', 'product', 'int16', 'str_', 'mat', 'fv', 'max', 'asanyarray', 'uint', 'npv', 'logaddexp', 'flatnonzero', 'amin', 'correlate', 'fromstring', 'left_shift', 'searchsorted', 'int64', 'may_share_memory', 'dsplit', 'intersect1d', 'can_cast', 'ppmt', 'show_config', 'cumsum', 'roots', 'outer', 'CLIP', 'fix', 'busday_count', 'timedelta64', 'degrees', 'choose', 'FPE_INVALID', 'recfromcsv', 'fill_diagonal', 'empty_like', 'logaddexp2', 'greater', 'histogram2d', 'polyint', 'arctan2', 'datetime64', 'complexfloating', 'ndindex', 'ctypeslib', 'PZERO', 'isfortran', 'asfarray', 'nanmedian', 'radians', 'fliplr', 'alen', 'recarray', 'modf', 'mean', 'square', 'ogrid', 'MAY_SHARE_BOUNDS', 'nanargmin', 'r_', 'diag_indices_from', 'hanning', 's_', 'allclose', 'extract', 'float16', 'ulonglong', 'matrix', 'asarray', 'poly1d', 'promote_types', 'rec', 'datetime_as_string', 'uint32', 'math', 'log2', '__builtins__', 'cumproduct', 'diagonal', 'atleast_1d', 'meshgrid', 'column_stack', 'put', 'byte', 'remainder', 'row_stack', 'expm1', 'nper', 'ndfromtxt', 'matmul', 'place', 'DataSource', 'newaxis', 'arccos', 'signedinteger', 'ndim', 'rint', 'number', 'rank', 'little_endian', 'ldexp', 'lookfor', 'array', 'vsplit', 'common_type', 'size', 'logical_xor', 'geterrcall', 'sometrue', 'exp2', 'bool8', 'msort', 'alltrue', 'zeros', 'False_', '__NUMPY_SETUP__', 'nansum', 'bool_', 'inexact', 'nanpercentile', 'broadcast', 'copyto', 'short', 'arctanh', 'typecodes', 'rot90', 'savetxt', 'sign', 'int_', 'std', 'not_equal', 'fromfunction', 'tril_indices_from', '__config__', 'double', 'require', 'rate', 'typeNA', 'str', 'getbuffer', 'abs', 'clip', 'savez_compressed', 'frompyfunc', 'triu_indices_from', 'conjugate', 'alterdot', 'asfortranarray', 'binary_repr', 'angle', 'lib', 'min', 'unwrap', 'apply_over_axes', 'ERR_LOG', 'right_shift', 'take', 'broadcast_to', 'byte_bounds', 'trace', 'warnings', 'any', 'shares_memory', 'compress', 'histogramdd', 'issubclass_', 'multiply', 'mask_indices', 'amax', 'logical_not', 'average', 'partition', 'nbytes', 'exp', 'sum', 'dot', 'int0', 'nanprod', 'longfloat', 'random', 'setxor1d', 'copy', 'FPE_UNDERFLOW', 'frexp', 'errstate', 'nanmin', 'swapaxes', 'SHIFT_OVERFLOW', 'infty', 'fft', 'ModuleDeprecationWarning', 'digitize', '__file__', 'NZERO', 'ceil', 'ones', 'add_newdoc_ufunc', '_NoValue', 'deprecate', 'median', 'geterr', 'convolve', 'isreal', 'where', 'isfinite', 'SHIFT_UNDERFLOW', 'MachAr', 'argmax', 'testing', 'deprecate_with_doc', 'full', 'polyder', 'rad2deg', 'isnan', '__all__', 'irr', 'sctypeDict', 'NINF', 'min_scalar_type', 'count_nonzero', 'sort_complex', 'nested_iters', 'concatenate', 'vdot', 'bincount', 'transpose', 'array2string', 'corrcoef', 'fromregex', 'vectorize', 'set_printoptions', 'isrealobj', 'trim_zeros', 'unravel_index', 'cos', 'float64', 'log1p', 'ushort', 'equal', 'cumprod', 'float_', 'vander', 'geterrobj', 'load', 'fromiter', 'poly', 'bitwise_or', 'polynomial', 'diff', 'iterable', 'array_split', 'get_include', 'pv', 'tensordot', 'piecewise', 'invert', 'UFUNC_PYVALS_NAME', 'SHIFT_INVALID', 'c_', 'flexible', 'pi', '__doc__', 'empty', 'VisibleDeprecationWarning', 'find_common_type', 'isposinf', 'arcsin', 'sctypeNA', 'imag', 'sctype2char', 'singlecomplex', 'SHIFT_DIVIDEBYZERO', 'matrixlib', 'apply_along_axis', 'reciprocal', 'tanh', 'dstack', 'cov', 'cast', 'logspace', 'packbits', 'issctype', 'mgrid', 'longdouble', 'signbit', 'conj', 'asmatrix', 'inf', 'flatiter', 'bitwise_xor', 'fabs', 'generic', 'reshape', 'NaN', 'cross', 'sqrt', '__package__', 'longcomplex', 'complex', 'pad', 'split', 'floor_divide', '__version__', 'format_parser', 'nextafter', 'polyval', 'flipud', 'i0', 'iscomplexobj', 'sys', 'mafromtxt', 'bartlett', 'polydiv', 'stack', 'identity', 'safe_eval', 'greater_equal', 'Tester', 'trapz', 'PINF', 'object', 'recfromtxt', 'oldnumeric', 'add_newdocs', 'RankWarning', 'ascontiguousarray', 'less', 'putmask', 'UFUNC_BUFSIZE_DEFAULT', 'unicode', 'half', 'NAN', 'absolute_import', 'typeDict', '__path__', 'shape', 'setbufsize', 'cfloat', 'RAISE', 'isscalar', 'character', 'bench', 'source', 'add', 'uint16', 'cbrt', 'bool', 'ufunc', 'save', 'ravel', 'float32', 'real', 'int32', 'tril_indices', 'around', 'lexsort', 'complex_', 'ComplexWarning', 'unicode0', 'ipmt', '_import_tools', 'atleast_3d', 'isneginf', 'integer', 'unique', 'mod', 'insert', 'bitwise_not', 'getbufsize', 'array_equiv', 'arange', 'asarray_chkfinite', 'in1d', 'interp', 'hypot', 'logical_and', 'get_printoptions', 'diagflat', 'float128', 'nonzero', 'kaiser', 'ERR_IGNORE', 'polysub', 'fromfile', 'prod', 'nanmax', 'core', 'who', 'seterrobj', 'power', 'bytes_', 'percentile', 'FPE_DIVIDEBYZERO', '__name__', 'subtract', 'print_function', 'nanmean', 'frombuffer', 'iscomplex', 'add_docstring', 'argsort', 'fmin', 'ones_like', 'is_busday', 'arcsinh', 'intc', 'float', 'ndenumerate', 'intp', 'unpackbits', 'Infinity', 'log', 'cdouble', 'complex128', 'long', 'round_', 'broadcast_arrays', 'inner', 'var', 'sctypes', 'log10', 'uintp', 'linalg', 'histogram', 'issubdtype', 'maximum_sctype', 'squeeze', 'int8', 'info', 'seterr', 'argmin', 'genfromtxt', 'maximum', 'record', 'obj2sctype', 'clongdouble', 'euler_gamma', 'arccosh', 'delete', 'tril', 'int', 'ediff1d', 'char', 'single', 'loadtxt', 'hsplit', 'ScalarType', 'triu', 'floating', 'expand_dims', 'floor', 'polyadd', 'nan', 'TooHardError', 'emath', 'arctan', 'bmat', 'isclose', 'ERR_DEFAULT', 'test', 'roll', 'string0', 'compare_chararrays', 'iinfo', 'real_if_close', 'repeat', 'nanvar', 'hamming', 'ALLOW_THREADS', 'ravel_multi_index', 'string_', 'isinf', 'ndarray', 'e', 'ERR_CALL', 'datetime_data', 'clongfloat', 'full_like', 'result_type', 'gradient', 'base_repr', 'argwhere', 'set_string_function']

You can either manually check if a function name you're not sure of appears here using if <function_name> in sys.modules[<module>].__dict__, or you can write a neat automated script that goes through each entry in sys.modules[<module>].

I would favour the latter for anything too sophisticated, and the former for diagnostic purposes.

Rough Implementation of Automatic Tool

A very, very, very quick-and-dirty example of how to write such an automated script:

import re
import sys

with open('file_I_want_to_change.py', 'r+') as f:
    file_contents = f.read()  # get the entire file as a string
    search_string = r"from ([a-zA-Z]+) import *"  # regex to find all loaded module names
    module_names = re.findall(search_string, file_contents)
    map(__import__, module_names)  # import ALL of these modules names at once

    for module in module_names:
        for function_name in sys.modules[module].__dict__:
            # do a very quick-and-dirty replace-all
            file_contents = file_contents.replace(function_name, "{0}.{1}".format(module, function_name)) 
    f.seek(0)  # move to start of file
    f.write(file_contents)

This is not very robust, and you shouldn't use it as-is! You may find yourself overwriting names not from the module but that are defined anyway.

It's probably best to allow some form of user interaction to confirm you want to apply a change for each function name found. But it gets the gist across.

This has been tested with the following simple example file:

from numpy import *
array([1])

becomes

from numpy import *
numpy.array([1])

EDIT: I have since created a much more robust and useful command line utility here

Answer 3

Use a tool like pyflakes (which you should use anyway) to note which names in your code become undefined after you replace from module import * with import module. Once you've positively identified every instance of a name imported from module, you can assess whether to

1. Always use import module and module.x for x imported from module.
2. Use import module as m and m.x if the module name is long.
3. Selectively import some names from module into the module namespace with from module import x, y, z

The above three are not mutually exclusive; as an extreme example, you can use all three in the same module:

import really.long.module.name
import really.long.module.name as rlmn
from really.long.module.name import obvious_name

really.long.module.name.foo()  # full module name
rlmn.bar()                     # module alias
obvious_name()                 # imported name

all in the same code. (I don't recommend using all three in the same module. Stick with either the full module name or the alias throughout a single module, but there is no harm importing common, obvious names directly and using the fully qualified name for more obscure module attributes.)