"lein test :only foo.bar.test/testme" not picking up function "testme" if "test-ns-hook" defined?

Question

I have defined a Clojure namespace with test functions and want to run them using Leiningen 2.9.1 via lein test.

The test functions are organized hierarchically. If I just run lein test, all the deftest will be picked up, leading to test duplication. For example:

(ns foo.bar.test
   (:require
      [clojure.test         :as t]
      [clojure.spec.alpha   :as s]
      [foo.bar.main         :as sut])) ; system under test

(t/deftest test-strip-empty
   (t/is
      (s/valid? ::sut/a-spec some-value)))

(t/deftest test-strip-several-squares
   (t/is
      (s/valid? ::sut/a-spec some-value)))

; collect subtests

(t/deftest testcollect-strip
   (test-strip-empty)
   (test-strip-several-squares))

lein test would run all three deftest entries, thus running test-strip-empty and test-strip-several-squares twice.

The function test-ns-hook can be defined to explicitly call the "top of the test tree".

(defn test-ns-hook []
   (testcollect-strip))

If exists, lein test will only call test-ns-hook:

Which is nice!

But once it exists, I cannot ran individual tests anymore.

lein test :only foo.bar.test/test-strip-several-squares

lein test foo.bar.test

Ran 0 tests containing 0 assertions.
0 failures, 0 errors.

Not nice!!

Remove the definition of test-ns-hook and it works:

lein test :only foo.bar.test/test-strip-several-squares

... 

Ran 1 tests containing 1 assertions.
1 failures, 0 errors.
Tests failed.

Can happyness be maximized by combining both features: leaving test-ns-hook defined and being able to run individual tests?

Answer 1

Don't group your tests like with testcollect-strip. I would call this an antipattern.

You can make individual assertions hierarchical within a single deftest form using the testing macro: https://clojuredocs.org/clojure.test/testing

(deftest t-math
  (testing "Arithmetic"
    (testing "with positive integers"
      (is (= 4 (+ 2 2)))
      (is (= 7 (+ 3 4))))
    (testing "with negative integers"
      (is (= -4 (+ -2 -2)))
      (is (= -1 (+ 3 -4))))))

---

~/expr/demo > lein clean ; lein test

lein test _bootstrap

-------------------------------
   Clojure 1.10.0    Java 12
-------------------------------

lein test tst.demo.core

Ran 2 tests containing 4 assertions.
0 failures, 0 errors.

You can also use test selectors to run only a subset of tests:

~ > lein help test
Run the project's tests.

Marking deftest or ns forms with metadata allows you to pick selectors to
specify a subset of your test suite to run:

    (deftest ^:integration network-heavy-test
      (is (= [1 2 3] (:numbers (network-operation)))))

Write the selectors in project.clj:

    :test-selectors {:default (complement :integration)
                     :integration :integration}

Arguments to this task will be considered test selectors if they are keywords,
otherwise arguments must be test namespaces or files to run. With no
arguments the :default test selector is used if present, otherwise all
tests are run. Test selector arguments must come after the list of namespaces.

A default :only test-selector is available to run select tests. For example,
`lein test :only leiningen.test.test/test-default-selector` only runs the
specified test. A default :all test-selector is available to run all tests.

Arguments: ([& tests])

So, adding metadata to the test definition

(deftest ^:basic-math t-math
  (testing "Arithmetic"
    (testing "with positive integers"
      (is (= 4 (+ 2 2)))
      (is (= 7 (+ 3 4))))
    (testing "with negative integers"
      (is (= -4 (+ -2 -2)))
      (is (= -1 (+ 3 -4))))))

And declaring test selector :basics to grab everything tagged with :basic-math in project.clj:

(defproject foo.bar "0.1.0-SNAPSHOT"
  ...
  :test-selectors {:basics :basic-math})

One can now run only the tests tagged with :basic-math via:

~ > lein test :basics

There is another trick to keep in mind. The namespace structure of your test code (dirs/files) doesn't need to match that of your source code. You could have a single source code ns super.calc, but a whole hierarchy of testing namespaces. I prefix them all with a root tst. prefix, which I think leads to a nicer naming structure than sticking a _test suffix on everything:

tst.super.calc
tst.super.calc.add
tst.super.calc.add.int
tst.super.calc.add.int.pos
tst.super.calc.add.int.neg
tst.super.calc.add.float
tst.super.calc.add.float.pos
tst.super.calc.add.float.neg
tst.super.calc.mult
...

So you can get as fine-grained as you desire. Mixing this with lein test selectors allows nearly infinitely fine-grained control.

---

Also,

Please checkout lein-test-refresh, my favorite way of doing testing in lein

https://github.com/jakemcc/lein-test-refresh