Evaluate arguments passed to a macro that generates functions in lisp

Question

I'm trying to make a macro that defines an accessor function for every config based on the config object created by py-configparser:

(defmacro make-config-accessor (config section option)
  ; create an upper case function name then intern
  (let* ((fun-name (intern (string-upcase
                             (str:replace-all "_" "-"
                                              (str:concat "get-" option)))))) 
    `(defun ,fun-name (config)
       (py-configparser:get-option config ,section ,option))))

It works fine if option is passed into as a string, but not when it's a pair like (car ("db" . "test.db")), the form is passed as is and causes error.. How do I evaluate the option argument within the macro, without using eval.

Full example: Suppose I have a test.ini file:

[Settings]
db = "test.db"

Using py-configparser (you can install it with (ql:quickload "py-configparser"), you can turn the config file into a Lisp object:

(setf *test-config* (py-configparser:make-config))
(py-configparser:read-files *test-config* '("~/test.ini"))

This should be the output:

#S(PY-CONFIGPARSER:CONFIG
   :DEFAULTS #S(PY-CONFIGPARSER::SECTION :NAME "DEFAULT" :OPTIONS NIL)
   :SECTIONS (#S(PY-CONFIGPARSER::SECTION
                 :NAME "Settings"
                 :OPTIONS (("db" . "\"test.db\""))))
   :OPTION-NAME-TRANSFORM-FN #<FUNCTION STRING-DOWNCASE>
   :SECTION-NAME-TRANSFORM-FN #<FUNCTION IDENTITY>)
("~/test.ini")

Then, you can retrieve the db option like this:

(py-configparser:get-option *test-config* "Settings" "db")

The output:

"\"test.db\""

Now I am writing a macro to create a function for every option such as db, like (get-db *test-config*) should give me the same output.

I made it work with make-config-accessor macro above, but then when I passed a form like (car ("db" . "test.db")), I must use eval otherwise the str:concat fails.

I made a gen-accessors that loops over every options in a config object and generate an accessor for it:

(defun gen-accessors (config)
  (let ((sections (py-configparser:sections config)))
    (loop for s in sections
       do (loop for i in (py-configparser:items config s)
             do (let* ((o (car i)))
                  (make-config-accessor config s o))))))

Answer 1

You need two levels of evaluation.

Try:

(defmacro make-config-accessor (config section option)
  ; create an upper case function name then intern
  `(let* ((fun-name (intern (string-upcase 
                            (str:replace-all "_" "-" (str:concat "get-" ,option)))))) 
     (eval `(defun ,fun-name (config)
              (py-configparser:get-option config ,,section ,,option)))))

Now option is evaluated in the let* form. And the returned defun form then needs to be evaluated (which is always in the global scope, or null lexical environment, or toplevel) using eval.

Thats all the change that was needed for me to run your code correctly. Just for reference I'm adding the whole code I ran here (note: there is a change in gen-accessors, I think you meant to use config and not *config*).

(ql:quickload "str")
(ql:quickload "py-configparser")

(defmacro make-config-accessor (config section option)
  ; create an upper case function name then intern
  `(let* ((fun-name (intern (string-upcase 
                              (str:replace-all "_" "-" 
                                               (str:concat "get-" ,option)))))) 
     (eval `(defun ,fun-name (config)
              (py-configparser:get-option config ,,section ,,option)))))

(defun gen-accessors (config)
  (let ((sections (py-configparser:sections config)))
    (loop for s in sections
          do (loop for i in (py-configparser:items config s)
                   do (let* ((o (car i)))
                        (make-config-accessor config s o))))))

(setf *test-config* (py-configparser:make-config))
(py-configparser:read-files *test-config* '("~/Desktop/test.ini"))
(gen-accessors *test-config*)

(get-db *test-config*)

Answer 2

The first rule of writing macros is: if you find yourself using eval then you have almost certainly made a mistake. In this case the mistake you've made is that you don't want a macro at all: you want a function.

In particular you probably want this function or something like it:

(defun make-config-accessor (section option)
  ;; Make an accessor for OPTION in SECTION with a suitable name
  (let ((fun-name (intern (nsubstitute #\- #\_
                                       (format nil "GET-~A"
                                               (string-upcase option))))))
    (setf (symbol-function fun-name)
          (lambda (config)
            (py-configparser:get-option config section option)))
    fun-name)))

Then given a suitable config reader

(defun read-config (&rest files)
  (py-configparser:read-files (py-configparser:make-config)
                              files))

together with a rather simplified (less single-use bindings) version of your gen-accessors:

(defun gen-accessors (config)
  (loop for s in (py-configparser:sections config)
        appending (loop for i in (py-configparser:items config s)
                        collect (make-config-accessor s (car i)))))

Then, for instance if /tmp/x.ini contains

[Settings]
db = "test.db"
scrunge = 12

Then

 > (gen-accessors (read-config "/tmp/x.ini"))
(get-scrunge get-db)

> (get-scrunge (read-config "/tmp/x.ini"))
"12"

---

You can make the definition of make-config-accessor perhaps even nicer with something like this:

(defun curryr (f &rest trailing-args)
  (lambda (&rest args)
    (declare (dynamic-extent args))
    (apply f (append args trailing-args))))

(defun make-config-accessor (section option)
  ;; Make an accessor for OPTION in SECTION with a suitable name
  (let ((fun-name (intern (nsubstitute #\- #\_
                                       (format nil "GET-~A"
                                               (string-upcase option))))))
    (setf (symbol-function fun-name)
          (curryr #'py-configparser:get-option section option))
    fun-name))

Not everyone will find this nicer, of course.

Answer 3

It is one of the rare cases, where you have to use eval in combination with a backquoted macro call with unquoting the arguments.

(I stumbled over this construct once and called it myself eval-over-macro-call. - Following the naming tradition let-over-lambda. - Actually it should be named eval-over-backquoted-macro-call-with-unquoting. It allows you to use macros dynamically. Vsevolod Dyomkin also stumbled over it, independently. I answered him, because I stumbled over it around the same time or before. Macros - as you realized - don't allow arbitrary control over evaluation.)

But first, I generated some helper functions. (You can use your :str package functions, but I had problems to install it. Less dependencies is better. And me, personally, I would prefer cl-ppcre for replacements etc. However, in your case, one can get rid of any dependencies.

intern pollutes your namespace. You want only the function namespace have the get- function name entries. But not the variable namespace. Therefore, to only return symbols without interning them automatically, use read-from-string.

The dotted-list-p function requires :alexandria package. However, one needs it anyway mostly and since it is one of the most frequently used packages in common lisp shpere (together with :cl-ppcre) I think that doesn't count as "additional dependency".

For the dotted-pair-p function, I had to do some searches.

The dotted-list-to-list converter function, I wrote myself.

You could get rid of all the dotted-list functions, if you would use simple string lists for options.

In that case in the macro, simply use listp instead of dotted-list-p. And use option instead of (dotted-list-to-list option).

;; one character replacement
(substitute #\+ #\Space "a simple example")
            replacer find obj

(defun string-to-upper-symbol (str)
  (read-from-string (substitute #\- #\_ (format nil "get-~A" str))))

(ql:quickload :alexandria)

(defun dotted-list-p (x)
  (and (not (alexandria:proper-list-p x))
       (consp x)))
;; correct - but gives nil if empty list (or (null x) ...) would include empty list

(defun dotted-or-empty-list-p (x)
  (or (null x) (dotted-list-p x)))
;; this gives t for empty list and dotted lists

(defun dotted-pair-p (x)
  (and (not (listp (cdr x))) (consp x)))

(defun dotted-list-to-list (dotted-list &optional (acc '()))
  (cond ((null dotted-list) (nreverse acc))
        ((dotted-pair-p dotted-list) (dotted-list-to-list '() (cons (cdr dotted-list) 
                                                                    (cons (car dotted-list) 
                                                                          acc))))
        (t (dotted-list-to-list (cdr dotted-list) (cons (car dotted-list) acc)))))

Your macro contains in arguments list config which however is never used.

In case you just forgot to unquote the config in the macro, the correct solution will be:

(defmacro %make-config-accessor (config section option)
  ; create an upper case function name then intern
  (let* ((fun-name (string-to-upper-symbol option)))
    `(defun ,fun-name (,config)
       (py-configparser:get-option ,config ,section ,option)))))

(defun make-config-accessor (config section option)
  (if (dotted-list-p option)
      (loop for x in (dotted-list-to-list option)
            do (eval `(%make-config-accessor ,config ,section ,x)))
      (%make-config-accessor config section option)))

;; call with
;; (make-config-accessor '<your-config> '<your-section> '("option1" "option2" . "option3"))
;; test for existence
;; #'get-option1
;; #'get-option2
;; #'get-option3

In the other case, that you don't need config, the correct solution will be:

(defmacro %make-config-accessor (section option)
  ; create an upper case function name then intern
  (let* ((fun-name (string-to-upper-symbol option)))
    `(defun ,fun-name (config)
       (py-configparser:get-option config ,section ,option)))))

(defun make-config-accessor (section option)
  (if (dotted-list-p option)
      (loop for x in (dotted-list-to-list option)
            do (eval `(%make-config-accessor ,section ,x)))
      (%make-config-accessor section option)))

;; call with
;; (make-config-accessor '<your-section> '("option1" "option2" . "option3"))
;; test for existence
;; #'get-option1
;; #'get-option2
;; #'get-option3

Note, since you need a function, you have to quote in the call the arguments config and section (they wait for evaluation while in the function-round the option gets evaluated.

Thanks to quote and backquote and unquote and eval you have full control over evaluation levels in lisp.

Sometimes, one has to use more quotes in argument list, if one wants to have control over several rounds of evaluations.

You could also fuse helper-macro and function into one macro. However, then, every time you call the macro, you have to use this eval-over-backquoted-macro-call unquoting the desired argument.

(defmacro make-config-accessor (section option)
  (if (dotted-list-p option)
      (loop for x in (dotted-list-to-list option)
            do (eval `(make-config-accessor ,section ,x)))
      `(defun ,(string-to-upper-symbol c) (config)
         (py-configparser:get-option config ,section ,option))))

;; call it with
;; (eval `(make-config-accessor <your-section> ,<your-option>))
;; e.g.
;; (eval `(make-config-accessor <your-section> ,'("opt1" "opt2" . "opt3")))
;; test existence with
;; #'get-opt1
;; #'get-opt2
;; #'get-opt3

Btw. I don't buy anymore this "eval is forbidden" talking. In cases like this - mostly evaluation control in macros, one has to eval as the only alternative to have to write an extra mini interpreter for this problem ... which would be much more tedious (and very likely also more error prone).

You didn't give workable code. So I had to figure all this out with some tody functions/macros, I wrote.

(defmacro q (b c)
  `(defun ,(string-to-upper-symbol c) (a) (list a ,b ,c)))

(defun q-fun (b c)
  (if (dotted-list-p c)
      (loop for x in (dotted-list-to-list c)
            do (eval `(q ,b ,x)))
      (q b c)))

;; (q "b" "c")
;; (q "b" '("d" . "e"))
;; (macroexpand-1 '(q "b" '("d" . "e")))

(defmacro p (b c)
  (if (dotted-list-p c)
      (loop for x in (dotted-list-to-list c)
            do (eval `(p ,b ,x)))
      `(defun ,(string-to-upper-symbol c) (a) (list a ,b ,c))))