CODEWITHSUNDEEP
pythonpathconfigfb-hydra
blue2609
blue2609

Reputation: 941

Referring to Hydra's `conf` directory from a Python sub/sub-sub-directory module

Suppose that we have a Python project with this structure:

hydra_config
├── conf
│   ├── api_key
│   │   ├── non_prod.yaml
│   │   └── prod.yaml
│   └── db
│       ├── mysql.yaml
│       └── postgresql.yaml
├── modules
│   └── module.py
└── my_app.py

Hydra's config documentation states that we need to add a @hydra.main decorator on top of a function that we want to give access to the configuration files. However, the docs only showed how to do this to a function in my_app.py which is the main module of the project. But how would I apply this decorator to a function in modules/module.py?

Here is the content of the files:

# modules/module.py

import hydra
from omegaconf import DictConfig, OmegaConf

@hydra.main(config_path="conf")
def module_function(cfg: DictConfig):
    print(OmegaConf.to_yaml(cfg))

# my_app.py

from modules.module import module_function

def main():
    module_function()

if __name__ == "__main__":
    main()

But when I ran python my_app.py, I got an error:

Primary config module 'modules.conf' not found.
Check that it's correct and contains an __init__.py file

Set the environment variable HYDRA_FULL_ERROR=1 for a complete stack trace.

I understand that this means the decorator added to module_function inside module.py couldn't find the conf directory which contains api_key and db config groups.

Does anyone here have any experience with this and know how to fix this error?

Upvotes: 4

Views: 10724

Answers (2)

Mateen Ulhaq
Mateen Ulhaq

Reputation: 27201

Another possibility is to make config_path an absolute path:

# modules/module.py

from pathlib import Path
import hydra
from omegaconf import DictConfig, OmegaConf

config_path = str(Path(__file__).parent.joinpath("../conf").resolve())

@hydra.main(version_base=None, config_path=config_path)
def module_function(cfg: DictConfig):
    print(OmegaConf.to_yaml(cfg))

Now, run python my_app.py.

Upvotes: 0

Jasha
Jasha

Reputation: 7639

First I'll go over some possible solutions, then I'll give an explanation. Here are three things you can do to make your example work:

Fix 1: create conf/__init__.py and change config_path="conf" to config_path="../conf":
├── conf
│   ├── config.yaml
│   └── __init__.py
├── modules
│   └── module.py
└── my_app.py

# modules/module.py
import hydra
from omegaconf import DictConfig, OmegaConf

@hydra.main(config_path="../conf", config_name="config")  # relative path
def module_function(cfg: DictConfig):
    print(OmegaConf.to_yaml(cfg))
Fix 2: move conf to modules/conf and create modules/conf/__init__.py:
├── modules
│   ├── conf
│   │   ├── config.yaml
│   │   └── __init__.py
│   └── module.py
└── my_app.py
Fix 3: define your decorated function in the module that will be called as __main__:
├── conf
│   └── config.yaml
└── my_app.py

# my_app.py
import hydra
from omegaconf import DictConfig, OmegaConf

@hydra.main(config_path="conf", config_name="config")
def module_function(cfg: DictConfig):
    print(OmegaConf.to_yaml(cfg))

if __name__ == "__main__":
    module_function()

Fix 3 will not be very satisfying to the OP :)

Explanation

In your example, since you've imported the decorated function (rather than defining it in the __main__ module), Hydra is interpreting the argument config_path="conf" as being relative to the parent of the module that defines the decorated function. In your example, the decorated function is defined in modules.module, whose parent is modules, so Hydra is looking for a python package called modules.conf. If found, this package will be searched for yaml files.

Fix 2 above takes the approach of making sure that modules/conf/__init__.py exists, so that modules.conf is in-fact a python package. Meanwhile, Fix 1 takes the approach of using a relative path config_path="../conf". Hydra then looks for a top-level package named conf. Since conf/__init__.py exists, this top-level conf package is found and searched for yaml files. To discover this package, Hydra uses the python import machinery. This means Hydra has the ability to discover yaml files located in python packages that are elsewhere on your $PYTHONPATH or that are installed via a package manager such as pip.

In the case of Fix 3, we have module_function.__module__ == "__main__", which means that Hydra cannot rely on python's import machinery to discover where module_function was defined. As a fallback, Hydra tries to find a directory conf (rather than a package conf); this is why an __init__.py file is not required for Fix 3. Here are the key lines of code implementing this behavior. Note that conf is still treated as a relative path, relative to the directory containing the file that defined the decorated function (rather than to the parent of the module that defined it). Here is the function implementating this relative path computation (for both the module case and the directory case).

Upvotes: 9

Related Questions