Recognizing local project / defining system with Quicklisp and Portacle

Question

I've been trying to get to grips with the package system in common-lisp, and I (sort of) understand the idea. However, like many (I have searched many other similar questions), there seems to be some implementation barrier I can't quite get past.

So: I am trying to define a package made of two lisp files. I'm using portacle, and all the files are in the same directory, both the system definition and the code I've written (/portacle/projects/):

ASDF File is shown below, and it's just named census-api.asd; there is a packages files named packages.lisp in the same directory.

However, when I try to (ql:quickload "census-api"), I get "System not found".

Fair enough, if I check the quicklisp local directories variable, it seems like it should work:

ql:*local-project-directories*
(#P"c:/Users/Brian/portacle/projects/"
 #P"c:/Users/Brian/portacle/all/quicklisp/local-projects/")

Similarly, I change the current path to the projects directory --

(setf *default-pathname-defaults* #P"c:/Users/Brian/portacle/projects/")
#P"c:/Users/Brian/portacle/projects/"

But none of those change the 'system not found' when I try to load via quicklisp. I feel like I'm missing some sort of pre-loading definition step?

What next?

I can go back to just loading the individual files themselves (They currently use an '''(in-package #:census-api)''' which errors out, but I could comment that line and load directly), but I'd prefer to be able to define systems.

Edit: Okay, I went and directly compiled 'packages.lisp' (definition below); quickload now is able to do `(ql:quickload "census-api")', and the relevant functions are available when called via the common lisp user package, eg, (census-api:set-fips etc). But I still can't use (in-package "census-api"), and if I have to pre-load all these files, I feel like I must be doing something wrong.

Edit 2: Okay, (in-package :census-api) does now work, because :census-api and "census-api" mean different things, which I know even if I don't always understand when one works vs the other). This solves my current use problem, but not the wider one -- how do I use quickload (or any other system utility) to avoid having to load packages manually? Or is that just the price of doing business?

census-api.asd

(defsystem "census-api"
  :description "A collection of utilities to pull Census, BLS, and Google DistanceMatrix API data.

For default use, evaluate the following --

Census: (set-fips), (set-hh-input), (print-hh-output (output-list))
BLS: (set-fips), (print-bls-data (prepare-hashed-bls-data))
Distancematric: (set-location-params), (print-dm-output (get-data (*api-response*)))"
  :author "Brian LastName <xxxx@gmail.com>"
  :license "To be determined"
  :version "0.5"
  :depends-on ("dexador" "shasht")
  :serial t
  :components ((:file "census-api")
               (:file "distancematrix-api")))

packages.lisp:

;;; Packages for Census Work

(defpackage #:distancematrix-api
  (:use #:cl)
  (:export #:set-location-params
           #:call-dm-api
           #:parse-response
           #:get-data
           #:print-dm-output))

(defpackage #:census-api
  (:use #:cl)
  (:export #:set-fips
           #:set-hh-output
           #:print-hh-output
           #:output-list
           #:prepare-hashed-bls-data
           #:print-bls-data))

Answer 1

cl-project can help. I once wrote also about Roswell and how to set up a Common lisp IDE. https://medium.com/towards-data-science/how-to-set-up-common-lisp-ide-in-2021-5be70d88975b?sk=b178a230dc2ba72c2a5af7ea86857fc0 (this is a friends link - so you don't have to be a medium member to read it).

I wrote there:

Start Common Lisp Packages/Projects using cl-projects

A package backbone can be automatically generated using cl-project.

Install it by Roswell:

$ ros install fukamachi/cl-project

Enter Roswell's local-projects folder, because we want to keep the package first on the local machine.

$ cd ~/.roswell/local-projects

Now create your project backbone — let’s call the project my-project and let’s assume it is dependent on the packages alexandria and cl-xlsx

$ make-project my-project --depends-on alexandria cl-xlsx

tree it to list its components:

$ tree my-project
my-project
├── my-project.asd
├── README.markdown
├── README.org
├── src
│   └── main.lisp
└── tests
    └── main.lisp2 directories, 5 files

The content of the meta-info project file my-project/my-project.asd which is built by the Common Lisp standard package system ASDF (Another System Definition Facility) is:

(defsystem "my-project"
  :version "0.1.0"
  :author ""
  :license ""
  :depends-on ("alexandria"
               "cl-xlsx")
  :components ((:module "src"
                :components
                ((:file "main"))))
  :description ""
  :in-order-to ((test-op (test-op "my-project/tests"))))(defsystem "my-project/tests"
  :author ""
  :license ""
  :depends-on ("my-project"
               "rove")
  :components ((:module "tests"
                :components
                ((:file "main"))))
  :description "Test system for my-project"
  :perform (test-op (op c) (symbol-call :rove :run c)))

Tests in Common Lisp are organized as separate packages — therefore, the .asd file prepares the tests for the project as a separate package with :rove added as one of its dependencies.

Fill in the author, license, and description parts.

During your project, when introducing new packages, they should be first included in the :depends-on listing of both package systems.

The :components section lists all files and folders belonging to this packaging system (the filenames are listed without the .lisp ending!). So whenever you add a new file or folder, you should update both defsystems in this file!

The content of my-projcet/src/main.lisp is:

(defpackage my-project
  (:use :cl))
(in-package :my-project);; blah blah blah.

You write your package code into ;; blah blah blah. .

Whenever you need dependencies, add them to the (:use :cl) listing.

In this case, it would be reasonable to complete it with (:use :cl :alexandria :cl-xlsx).

Or if you import single functions from these packages, you explicitly import only those by:

(defpackage my-project
  (:use #:cl
        #:cl-xlsx) ;; this package's entire content is imported
  (:import-from #:alexandria ;; package name
                #:alist-hash-table) ;; a function from :alexandria
  (:export #:my-new-function))

In the :export clause, you are declaring, which functions should be exported and made public — visible from outside this file.

Answer 2

You are nearly there!

ASDF system style

I advise to not write defsystem alone, but rather:

(asdf:defsystem "my-project" …)

and even:

(require 'asdf)
(in-package :asdf-user)
(defsystem …)

that way you can LOAD the .asd system definition and your Lisp will understand simple and complex ASDF system definitions.

The require is close to mandatory, but sometimes you feel you don't need it, just because it is written somewhere else, like in your Lisp startup file. While ASDF is included in Lisp implementations, it is necessary to require it. Same with other modules, for example sb-cover.

Even though you wrote the require, that doesn't mean that the Lisp knows where the defsystem symbol comes from. "require" isn't an "import". You write a top-level expression. Is that a standard top-level Lisp expression? No, it's from ASDF. Thus we need to import the symbol somehow, or reference it with its package prefix (asdf:defsystem), or "use" a package, or take advantage of a package done for this, :asdf-user, just like :cl-user, hence our in-package.

Here we only write one defsystem, so either solution should work fine. However we can be bitten soon enough by writing another ASDF declaration, after the defsystem, and so we would need to prefix it with the asdf: package prefix too. Using the :asdf-user package avoids potential debugging session. So I advise it.

Now, maybe a naked defsystem will work if you load the file with asdf:load-asd because ASDF knows ASDF… but that wouldn't work in all situations (does C-c C-k work now? I don't think so (edit) it does, given you have the slime-asdf extension, which you probably have), so I'd better take a couple extra steps that are more flexible.

So yeah we are dealing with symbols and packages, systems, ASDF mysteries… that's a lot, but if a system definition works for you keep it, it will work for a long time.

Best practices

(edit) The best practice is to use defsystem alone but to load the .asd with either C-c C-k (the Slime contrib slime-asdf uses asdf:load-asd) or programmatically with asdf:load-asd, instead of the built-in load, which can lead to issues given a complex enough system definition.

We can say my previous recommendations are kinda "wrong". They are simpler for newcomers though…

Manually load a project

When a system is not found you can compile and load the .asd file. Two solutions:

• in the .asd file buffer, use C-c C-k
• programmatically, evaluate load myfile.asd but better yet, following ASDF best practices, (asdf:load-asd /path/to/myfile.asd).

Then (ql:quickload …).

bug

You don't refer packages.lisp in the .asd.