SCons: How to generate dependencies after some targets have been built?

Question

I have a SCons project that builds set of Python modules (mostly as shared objects compiled from C++). This part works flawlessly, all dependencies seem to be fine.

However, now I reached point where I started trying to add tests for those modules. Those tests are supposed to be run as part of build. Tests are written in Python and when run they need to be run under environment that already has all modules built (so that import statements can find them).

I got that part working up to the point of dependencies for those tests. Tests are run, however I can't seem to find a way to generate dependencies for them from import statements.

I have found modulefinder which does exactly what I want. What's more I am able to run it in built environment and get expected results after my project is built. I wanted to use modulefinder in emitter to scan for files the test/Python script depends on.

Problem is that dependency scanning/building + running of emitters happens before SCons has built all modules and before it can set up environment properly for tests, hence also modulefinder can't work.

I can't seem to work out how to make SCons look for dependencies for specific targets after some other targets are already built.

edit:

I found ParseDepends in SCons documentation which seems to talk about the same type of issue (well, almost exactly same with exception to langauge).

This limitation of ParseDepends leads to unnecessary recompilations. Therefore, ParseDepends should only be used if scanners are not available for the employed language or not powerful enough for the specific task.

I am still hopeful though that there is some clean solution to my problem.

elmo · Accepted Answer

After a lot of playing I found, not-so-clean-but-not-too-horrible-way-that-seems-to-work which I've wrapped in a helper Scanner-derived class:

from SCons.Node import Node
from SCons import Scanner
import logging

_LOGGER = logging.getLogger(__name__)

class DeferredScanner(Scanner.Current):
    """
    This is a helper class for implementing source scanners that need
    to wait for specific things to be built before source scanning can happen.

    One practical example of usage is when you are you generating Python
    modules (i.e. shared libraries) which you want to test.

    You have to wait for all your modules are ready before dependencies
    of your tests can be scanned.

    To do this approach with this scanner is to collect all generated modules
    and `wait_for` them before scanning dependncies of whatever this scanner
    is used for.

    Sample usage:

        py_scanner = DeferredScanner(
            wait_for = all_generated_modules,
            function = _python_source_scanner,
            recursive = True,
            skeys = ['.py'],
            path_function = FindENVPathDirs('PYTHONPATH'),
        )

    """
    def __init__(self, wait_for, **kw):
        Scanner.Current.__init__(
            self,
            node_factory = Node,
            **kw
        )
        self.wait_for = wait_for
        self.sources_to_rescan = []
        self.ready = False

        env = wait_for[0].get_build_env()
        env.AlwaysBuild(wait_for)

        self._hook_implicit_reset(
            wait_for[0],
            'built',
            self.sources_to_rescan,
            lambda: setattr(self, 'ready', True),
        )

    def _hook_implicit_reset(self, node, name, on, extra = None):
        # We can only modify dependencies in main thread
        # of Taskmaster processing.
        # However there seems to be no hook to "sign up" for
        # callback when post processing of built node is hapenning
        # (at least I couldn't find anything like this in SCons
        #  2.2.0).
        # AddPostAction executes actions in Executor thread, not
        # main one, se we can't used that.
        #
        # `built` is guaranteed to be executed in that thread,
        # so we hijack this one.
        #
        node.built = lambda old=node.built: (
            old(),
            self._reset_stored_dependencies(name, on),
            extra and extra(),
        )[0]

    def _reset_stored_dependencies(self, name, on):
        _LOGGER.debug('Resetting dependencies for {0}-{1} files: {2}'.format(
                        # oh, but it does have those
            self.skeys, # pylint: disable=no-member
            name,
            len(on),
        ))
        for s in on:
            # I can't find any official way to invalidate
            # FS.File (or Node in general) list of implicit dependencies
            # that were found.
            # Yet we want to force that Node to scan its implicit
            # dependencies again.
            # This seems to do the trick.
            s._memo.pop('get_found_includes', None) # pylint: disable=protected-access

    def __call__(self, node, env, path = ()):
        ready = self.ready
        _LOGGER.debug('Attempt to scan {0} {1}'.format(str(node), ready))

        deps = self.wait_for + [node]

        if ready:
            deps.extend(Scanner.Current.__call__(self, node, env, path))

        self.sources_to_rescan.append(node)

        # In case `wait_for` is not dependent on this node
        # we have to make sure we will rescan dependencies when
        # this node is built itself.
        # It boggles my mind that SCons scanns nodes before
        # they exist, and caches result even if there was no
        # list returned.
        self._hook_implicit_reset(
            node,
            'self',
            [node],
        )

        return deps

This seems to work exactly as I'd have hoped and do the job. It is probably as efficient as you can get.

Probably should also note that this works with SCons 2.2.0, but I suspect it shouldn't be much different for newer ones.

SCons: How to generate dependencies after some targets have been built?

Answers (2)

Related Questions