How to run tests for a specific suite from included module configurations?

Question

I have a codeception.yml on my projects root folder. This has an include config directive to other codeception yml for my different modules

include:
    - tests/Modules/*

This is described here in the codeception doc as a multiple application setup

https://codeception.com/docs/08-Customization#One-Runner-for-Multiple-Applications

These sub-configs have different suite configs described in their own codeception.yml like this one in

tests/Modules/Frontend/codeception.yml

suites:
    Presentation:
        path: Presentation
        class_name: PresentationTester
        modules:
...

The problem: In my project root folder i cant run a specific suite for all included modules via

vendor/bin/codecept run Presentation

It says

Suite 'Presentation' could not be found

Running it pointing to a certain config does work

vendor/bin/codecept run Presentation -c tests/Modules/Frontend

But it just runs this one module. I want to run certain suites of all modules with one command.

Running a group (-g) does also work, so this indicates that the configs are loaded correctly.

Why does this not work for suites? All configs are included in my main yml, so the suites must be found in my opinion.

Whats the sense of having a setup like this, when no suites can be run from a central point?

Any ideas?

Answer 1

Update: found some problems and a solution; please refer to the end of my answer

Yesterday I found myself in the same situation as yours. I was refactoring our test suite from a suite-centric structure like this

tests/
  unit/
    Module1/
    Module2/
    ... (more modules)
  integration/
    Module1/
    Module2/
    ... (more modules)
  ... (more suites)

to a new module-centric structure like this

tests/
  Module1/
    Unit/
    Integration/
    ... (more suites)
  Module2/
    Unit/
    Integration/
    ... (more suites)
  ... (more modules)

However it seems to me that Codeception doesn't support this. After searching and debugging Codeception code, I found these infos.

Workaround

Here https://github.com/Codeception/Codeception/issues/5486 user LeeShan8 seems to issue the same problem and user vertexvaar talks about a patch he made that provides a --recurse-include option to run a specific suite in every submodule (patch can be downloaded from there).

So I searched the PR from vertexvaar to find out what happened to his patch. This is the PR https://github.com/Codeception/Codeception/pull/5737 but was closed due to inactivity by SamMousa.

SamMousa points out to a workaround: the --skip option.

If you want to run tests from unit suite in every submodule, you can do this:

codecept run -s suite1 -s suite2 ...

where suite1, suite2 ..., are all your test suite excluding unit. If you skip all the other suites, Codeception will run the remaining ones (unit in this case) in every submodule.

My solution

I really don't want to run unit suite by listing all the other ones in --skip options. It's so boring, slow to write and error prone.

I would really want to specify codecept run unit and see all the unit tests run, module by module. So I created a small patch (https://gist.github.com/mpallante/101b1508ffc5a2c4bf30d3344437ca0b) that fixes this behaviour.

It is similar to the one from vertexvar but to keep things simple I didn't add a new option. It just runs specified suite from each submodule.

I also automated the patch application using composer-patches (https://github.com/cweagans/composer-patches).

I installed composer-patches with

composer require cweagans/composer-patches

then put the patch in patches/allow-codeception-run-suites-from-submodules.patch and changed my composer.json to include the patch (in "extra" section):

  ...
  "extra": {
    ...
    "patches": {
      "codeception/codeception": {
        "Allow Codeception to run suites from submodules": "patches/allow-codeception-run-suites-from-submodules.patch"
      }
    }
  },
  ...

Finally, I ran composer update codeception/codeception: it will remove codeception, reinstall it, and apply the patch:

$ php which composer update codeception/codeception
Gathering patches for root package.
Removing package codeception/codeception so that it can be re-installed and re-patched.
  - Removing codeception/codeception (4.1.7)
Loading composer repositories with package information
Updating dependencies (including require-dev)
Package operations: 1 install, 0 updates, 0 removals
Gathering patches for root package.
Gathering patches for dependencies. This might take a minute.
  - Installing codeception/codeception (4.1.7): Loading from cache
  - Applying patches for codeception/codeception
    patches/allow-codeception-run-suites-from-submodules.patch (Allow Codeception to run suites from submodules)

Caution

Please note that this is solution seems to work in my case, but it is not fully tested. I usually run Codeception in very few ways (single suite, all suites, single test file), so don't know if there are some conflicts.

However it could be a starting point.

Maybe sometime in the future, I could contribute this patch to the project itself.

Update

I discovered a small problem: when a suite is defined in a submodule, but not globally, codecept run skips it.

I solved it by defining suites in global codeception.yml config file this way:

include:
    - modules/*/Tests
suites:
    integration:
    unit:
paths:

   ...

Then, in each submodules I define a ./modules/Mod1/Tests/codeception.yml file like this:

suites:
    unit:
        path: ./Unit
        modules:
            enabled:
                - Asserts
                - \Helper\Unit
    integration:
        path: ./Integration
        modules:
            enabled:
                - \Helper\Integration
namespace: Modules\Mod1\Tests
paths:
    tests: .
    output: ../../../tests/_output
    support: ../../../tests/_support
    data: ./_data

Hope this helped.