Bazel combine multiple rules in macro

Question

My project structure has multiple sub-modules within it, these are essentially folders that are projects within their own right too.

For a given python project, each with sub projects. I would like to be able to write a macro that can create a set of actions to be taken on a given sub-project.

Essentially something along these lines inside the WORKSPACE:

load("@io_bazel_rules_python//:python/pip.bzl",
   "pip_import"
)

pip_import(
   name = "foo",
   requirements = "@subprojectname//:requirements.txt",
)

load("@foo//:requirements.bzl", "pip_install")
pip_install()

There are multiple sub-projects I would like to do the above actions for. However, if I put all the code into a function inside a bzl file.

pyrules.bzl

load("@io_bazel_rules_python//:python/pip.bzl",
   "pip_import"
)


def doit(name):
    pip_import(
       name = "foo",
       requirements = "@{repo}//:requirements.txt".format(repo=name)
    )

    load("@foo//:requirements.bzl", "pip_install")
    pip_install()

I cannot use the second, load command due to "syntax error at 'load': expected expression". Without then having multiple bzl files that are loaded into the parent WORKSPACE, is there any other way to create reusable chunks of logic?

Update 1. Comments requested more information on the workflow. Warning, I have a non standard layout to how Bazel seems to be used else where i.e. not a monorepo. But it should come to a similar practical layout.

Projects are laid out as:

projectname/
    WORKSPACE
    BUILD
    src/
       stuff
    submodule/ # < git modules that are checked out into folder
        subproject1/
            BUILD
            src/
                stuff
        subproject2/
            BUILD
            src/ 
                stuff

Firstly, I have a repository_rule that is loaded in the WORKSPACE that finds all projects in the submodule folder and adds them as repositories. Within the subproject, some of these are python. I would like to load its requirements.txt files. Hence the overall question of where these requirements.txt files are and doing a pip install on them. Ideally I would like the py_library definitions in the subprojects BUILD files to know that there is a dependency on the requirements file of the subproject, but this likely doesn't matter as the parent BUILD file is the only thing that creates par_binaries and so on, so as long as the pip_install and setting the dependencies happens, the project itself should be usable.

Bazel doesn't appear to allow a child project to define their own repository actions like the above pip_install. I assume this is because you cant have repository actions in a BUILD file, and child WORKSPACE files don't seem to have any effect. So I ended up having to add this to the parent WORKSPACE file.

If it is in the parent WORKSPACE, I have to copy paste the pip actions for each of the subprojects I want to use. However, I would prefer to just set up a general rule which would locate the requirements file and pip install them. Trying to make a macro from this however means that I cannot use the load call within it. All the pip actions seem to need to interact with repository actions which then need to be called only from a parent WORKSPACE file.

Answer 1

Doesn't look like any reasonable solution exists for this scenario. However, I think it's important to realize the rules that you depend on can be modified if you are willing to support them internally. I ended up modifying the piptool.py in the rules_python repository to add support for multiple requirements.txt files in a single call. Then added the following rules into my own rules repository that I use internally.

#
# Python
#

def _pip_import_requirements_impl(repository_ctx):
  """Core implementation of pip_import."""
  # Add an empty top-level BUILD file.
  # This is because Bazel requires BUILD files along all paths accessed
  # via //this/sort/of:path and we wouldn't be able to load our generated
  # requirements.bzl without it.
  repository_ctx.file("BUILD", "")

  # To see the output, pass: quiet=False
  result = repository_ctx.execute([
    "python3", repository_ctx.path(repository_ctx.attr._script),
    "--name", repository_ctx.attr.name,
    "--input"
    ] +
    [repository_ctx.path(r) for r in repository_ctx.attr.requirements] +
    [
      "--output", repository_ctx.path("requirements.bzl"),
      "--directory", repository_ctx.path("")
    ],
  quiet = repository_ctx.attr.quiet)

  if result.return_code:
    fail("pip_import failed: %s (%s)" % (result.stdout, result.stderr))


pip_import_requirements = repository_rule(
  attrs = {
    "quiet" : attr.bool(default = False),
    "requirements": attr.label_list(
       allow_files = True,
       mandatory = True,
     ),
     "_script": attr.label(
       executable = True,
       default = Label("@io_bazel_rules_python//tools:piptool.par"),
       cfg = "host",
     ),
  },
  implementation = _pip_import_requirements_impl,
)

Then I can do the following in my WORKSPACE.

pip_import_requirements(
  name = "py_requirements",
  requirements = [
    "@mycore//:requirements.txt",
    "@myother//:requirements.txt"
  ]
)

load(
    "@py_requirements//:requirements.bzl",
    "pip_install",
)

pip_install()

And the following in any BUILD file that I happen to need. Note that the py_requirements reference then will always be available for any BUILD file in the project.

load(
    "@py_requirements//:requirements.bzl",
    "all_requirements"
)

par_binary(
    name = "funkyserver",
    main = "src/main.py",
    srcs = glob(["src/**/*.py"]),
    deps = [
        "@mycore//:core",
        "@myother//:other"
    ] + all_requirements,
)

For piptool.py. You will need to rebuild the piptool.par using the update_tools.sh in the rules_python repo.

diff --git a/rules_python/piptool.py b/rules_python/piptool.py
index f5d504a..ab520d8 100644
--- a/rules_python/piptool.py
+++ b/rules_python/piptool.py
@@ -87,7 +87,7 @@ def pip_main(argv):
 parser.add_argument('--name', action='store',
                     help=('The namespace of the import.'))

-parser.add_argument('--input', action='store',
+parser.add_argument('--input', action='store', nargs='+',
                     help=('The requirements.txt file to import.'))

 parser.add_argument('--output', action='store',
@@ -154,7 +154,8 @@ def main():
   args = parser.parse_args()

   # https://github.com/pypa/pip/blob/9.0.1/pip/__init__.py#L209
-  if pip_main(["wheel", "-w", args.directory, "-r", args.input]):
+  if pip_main(["wheel", "-w", args.directory] + [p for x in [
+      ('-r',  i) for i in args.input] for p in x]):
     sys.exit(1)

   # Enumerate the .whl files we downloaded.
@@ -219,7 +220,7 @@ def requirement(name):
   if name_key not in _requirements:
     fail("Could not find pip-provided dependency: '%s'" % name)
   return _requirements[name_key]
-""".format(input=args.input,
+""".format(input=" ".join(args.input),
            whl_libraries='\n'.join(map(whl_library, whls)) if whls else "pass",
            mappings=whl_targets))