CODEWITHSUNDEEP
pythonpython-3.xpython-modulepython-packaging
Vivek Joshy
Vivek Joshy

Reputation: 934

How do you recursively get all submodules in a python package?

Problem

I have a folder structure like this:

- modules
    - root
        - abc
            hello.py
            __init__.py
        - xyz
            hi.py
            __init__.py
          blah.py
          __init__.py
      foo.py
      bar.py
      __init_.py

Here is the same thing in string format:

"modules",
"modues/__init__.py",
"modules/foo.py",
"modules/bar.py",
"modules/root",
"modules/root/__init__.py",
"modules/root/blah,py",
"modules/root/abc",
"modules/root/abc/__init__.py",
"modules/root/abc/hello.py",
"modules/root/xyz",
"modules/root/xyz/__init__.py",
"modules/root/xyz/hi.py"

I am trying to print out all the modules in the python import style format. An example output would like this:

modules.foo
modules.bar
modules.root.blah
modules.root.abc.hello
modules.root.xyz.hi

How can I do this is in python(if possible without third party libraries) easily?

What I tried

Sample Code

import pkgutil

import modules

absolute_modules = []


def find_modules(module_path):
    for package in pkgutil.walk_packages(module_path):
        print(package)
        if package.ispkg:
            find_modules([package.name])
        else:
            absolute_modules.append(package.name)


if __name__ == "__main__":
    find_modules(modules.__path__)
    for module in absolute_modules:
        print(module)

However, this code will only print out 'foo' and 'bar'. But not 'root' and it's sub packages. I'm also having trouble figuring out how to convert this to preserve it's absolute import style. The current code only gets the package/module name and not the actual absolute import.

Upvotes: 11

Views: 11563

Answers (4)

Ghostranger
Ghostranger

Reputation: 85

(Outdated Comment) The below code will give you the relative package module from the codes current working directory.

import os
import re

for root,dirname,filename in os.walk(os.getcwd()):
    pth_build=""
    if os.path.isfile(root+"/__init__.py"):
        for i in filename:
            if i <> "__init__.py" and i <> "__init__.pyc":
                if i.split('.')[1] == "py":
                    slot = list(set(root.split('\\')) -set(os.getcwd().split('\\')))
                    pth_build = slot[0]
                    del slot[0]
                    for j in slot:
                        pth_build = pth_build+"."+j
                    print pth_build +"."+ i.split('.')[0]

This code will display:

modules.foo
modules.bar
modules.root.blah
modules.root.abc.hello
modules.root.xyz.hi

If you run it outside the modules folder.

Upvotes: -1

Ibolit
Ibolit

Reputation: 9720

import my_module

from inspect import getmembers

print(getmembers(my_module))

This will list all the members in your module, including submodules, classes, functions, etc. You can then filter the list accordingly.

Upvotes: 1

rwst
rwst

Reputation: 2685

This uses setuptools.find_packages (for the packages) and pkgutil.iter_modules for their submodules. Python2 is supported as well. No need for recursion, it's all handled by these two functions used together.

import sys
from setuptools import find_packages
from pkgutil import iter_modules

def find_modules(path):
    modules = set()
    for pkg in find_packages(path):
        modules.add(pkg)
        pkgpath = path + '/' + pkg.replace('.', '/')
        if sys.version_info.major == 2 or (sys.version_info.major == 3 and sys.version_info.minor < 6):
            for _, name, ispkg in iter_modules([pkgpath]):
                if not ispkg:
                    modules.add(pkg + '.' + name)
        else:
            for info in iter_modules([pkgpath]):
                if not info.ispkg:
                    modules.add(pkg + '.' + info.name)
    return modules

Upvotes: 11

Vivek Joshy
Vivek Joshy

Reputation: 934

So I finally figured out how to do this cleanly and get pkgutil to take care of all the edge case for you. This code was based off python's help() function which only displays top level modules and packages.

import importlib
import pkgutil

import sys

import modules


def find_abs_modules(module):
    path_list = []
    spec_list = []
    for importer, modname, ispkg in pkgutil.walk_packages(module.__path__):
        import_path = f"{module.__name__}.{modname}"
        if ispkg:
            spec = pkgutil._get_spec(importer, modname)
            importlib._bootstrap._load(spec)
            spec_list.append(spec)
        else:
            path_list.append(import_path)
    for spec in spec_list:
        del sys.modules[spec.name]
    return path_list


if __name__ == "__main__":
    print(sys.modules)
    print(find_abs_modules(modules))
    print(sys.modules)

This will work even for builtin packages.

Upvotes: 2

Related Questions