Common Lisp source code organization

Question

I'm new to CL and am using AllegroCL. I'm trying to figure out how to organize my source code to meet the following requirements:

1. I want to prevent the src code from including my test suites.
2. I want to declare project dependencies (both src and test deps) in a portable way so that other members on my team don't have to modify their systems.
3. I want to ease continuous integration on check-ins, including both builds and tests.

I've been trying to creatively use ASDF to meet these requirements, and I can't get it right. How do other people approach this problem? Are these 2 requirements just not "Lispy"?

Answer 1

IMPORTANT

The advice above is good, but you will be frustrated trying to test un-exported things. A simple work-around is not to define two packages. Just put your tests in the same package with your other sources. What's the harm?

If you think there will be harm, then you'll have to do it like this:

(defpackage #:sources
  (:use #:cl))

(defpackage #:tests
  (:use #:cl #:lisp-unit)
  (:import-from #:sources))

The important part is to :import-from your source package instead of :useing it.

Then you will have to qualify the symbols in your source package when you use them in your test package. For example, say you have this function:

(defun return-true () t)

Your test might look like:

(define-test test-return-true
  "Make sure it works"
  (assert-equal t (sources::return-true)))

The important part is your're saying (sources::return-true) instead of merely (return-true). The same goes for symbols like 'sym; refer to it as 'sources::sym.

Answer 2

If Quicklisp is installed you can use the built-in feature Quickproject.

(ql:quickload "quickproject")
(quickproject:make-project "~/src/lisp/swatchblade/"
                         :depends-on '(vecto hunchentoot))

This creates 4 files:

• package.lisp
• swatchblade.lisp
• swatchblade.asd
• README.txt

package.lisp defines package namespaces:

(defpackage #:swatchblade  
(:use #:cl)
  (:shadowing-import-from #:vecto
                          #:with-canvas
                          #:rounded-rectangle
                          #:set-rgb-fill
                          #:save-png-stream))

swatchblade.asd defines the system/project, source code files, dependencies, etc.

(asdf:defsystem #:swatchblade 
 :serial t
  :depends-on (#:vecto
               #:hunchentoot
               #:cl-colors)
  :components ((:file "package")
               (:file "swatchblade")))

swatchblade.lisp is where the source code goes.

You can load the the project via Quicklisp's quickload:

* (ql:quickload "swatchblade")
loading output
* (swatchblade:start-web-server :port 8080)
Server started on port 8080.

If you then create another project that depends on the swatchblade system:

quickproject:make-project "~/src/lisp/whimsytron/" 
                       :depends-on '(swatchblade))

Regarding tests, you can add another namespace in package.lisp for your tests:

    (defpackage #:swatchblade-tests  
      (:use #:cl #:swatchblade))

Create a test file, write the code, and add the file to the system definition:

(asdf:defsystem #:swatchblade 
 :serial t
  :depends-on (#:vecto
               #:hunchentoot
               #:cl-colors)
  :components ((:file "package")
               (:file "swatchblade")
               (:file "swatchglade-tests")))

Load the swatchblade-tests namespace to run the tests.

Sample project with tests here

If you want to avoid Quicklisp installing all dependencies into the system, you'll have to install the dependencies and load the system manually as far as I know.

The author of Quicklisp, Zach Beane, has a more detailed post on using quickproject.

Answer 3

Per Rainer's suggestion, I propose that you use the ASDF system definition facility to define two systems, your main system, foo, and the ancillary system foo-tests.

In the definition of the foo system, add a specification that in-order-to do the test-op on the foo, you need to do the test-op on foo-tests. This ensures that if you do (asdf:test-system "foo"), the corresponding test system, with its dependencies, will be loaded, and then ASDF will execute the test-op.

I find that FiveAM is an adequate library for building tests.

The above will get everything loaded, but now you need to make sure that doing the test-op on foo-tests actually runs the tests! To do that, you need to add a method on PERFORM for TEST-OP and (eql (find-system "foo-tests")). That PERFORM method should invoke all the FiveAM tests you have defined, and either succeed, or raise an error if the tests fail.

I have made a FiveAM-tester add-on for ASDF. I will try to see about making it publicly available.

Answer 4

I am using quicklisp which makes a "quicklisp"-folder in your home-folder in which a "local-project" folder can be found. This one contains a txt file in which you can insert the URIs to the .asd files.

How to use that utility:

• make a "project.asd" and "project-test.asd" in the project folder

project.asd (manages the includes for the pure project code)

(asdf:defsystem :project-name
  :description "description here"
  :version "version here"
  :author "your name here"
  :depends-on (:a
               :list 
               :of
               :dependencie
               :libraries)
  :components ((:file "sourcefileone")
               (:file "sourcefiletwo")))

project-test.asd (manages the includes for the test code)

(asdf:defsystem :project-name-test
  :description "testing"
  ...
  :depends-on (:project-name)
  :components ((:file "sourcefileone-test")
               (:file "sourcefiletwo-test")))

• now insert the URIs for those files into the above named local-projects.txt

• program parallel the project source in < filename>.lisp files and the test-calls in < filename>-test.lisp files (the *-test.lisp files have to contain a test-execute call)

• start your sbcl or whatever you use and then use (ql:quickload "project-name") or (ql:quickload "project-name-test") depending if you just want to load a project or test it.

The only thing you have to do porting this anywhere else, is to write the local-projects.txt on the computer the project is copied on. After that your colleges may depend on it using asdf-files and quickload in any other project they want. For copying the project folder you can either use ctr+c/v or maybe something more sophisticated as git.

For testing I programmed my own small test-suite, but I bet there are good ones out there. More information about quicklisp can be found here and about asdf here. Maybe this question can help you if you get stuck configuring quicklisp.

Answer 5

Use ASDF or use the Allegro CL defsystem tool.

1. make them two different systems. The test suite system depends on the software system.
2. Use relative pathnames, compute absolute pathnames based on the location of the system definition file or, the 'pro' version, use logical pathnames (which are pathnames in CL which can be remapped according to rules).
3. Probably there is an continuous integration tool for Common Lisp, but I haven't used any, yet. Having a defsystem description is a good start.