Dynamic lambda parameter list in a macro

Question

Background

I'm typically composing several function calls, the first of which requires a lambda function to be passed as an argument. I want to wrap the most common use cases with a macro, am having trouble with dynamically creating the lambda list for the lambda function. There was one previous discussion, but nothing there helps in this situation.

The intention is to call the macro like this example: (filter data (and (string= origin "USA") (> acceleration 10))) where origin and acceleration are columns in the data.

Current macro

(defmacro filter (data &body body)
  (with-unique-names (variables predicate)
   `(let* ((,variables (key-list ,data ',@body))
1.         (,predicate (lambda ,variables ,@body)) ;not working
2.      ;; (,predicate (lambda (origin acceleration) ,@body)) ; works
3.         (,predicate (eval (read-from-string
                               (format nil "#.(lambda ~A ~A)" 
                                            ,variables ',@body)))
      (choose ,data
              (mask ,data ,variables ,predicate)
              t))))

In this example, key-list is a function takes the data and body and produces a list containing all the variables in the predicate. For example:

CL-USER> (key-list vgcars '(and (string= origin "USA") (> acceleration 10)))
(ORIGIN ACCELERATION)

I've confirmed that it is a list, and correct.

In the let form:

1. is what should work
2. does work, but requires me to hand-code the lambda's parameter list, and this will be different for every predicate query.

Executing the macro as it is produces a compile time error:

Compile-time error:
  The lambda expression has a missing or non-list lambda list:
  (LAMBDA #:VARIABLES1199 (AND (STRING= ORIGIN "USA") (> ACCELERATION 10)))

Note the lack of substitution for ,variables. I've printed and type checked ,variables and it is a list with the correct values. As well, mask requires the variables to be passed in a similar manner, and in the hard-coded version of the lambda function this substitution works. I've tried all kinds of variations for the parameter list, e.g. (,@variables), etc., but none work.

The problem seems to be specific to creating the parameter list of the lambda function via macro substitution.

Anyone have any ideas?

Edit Based on a comment, I've added a version 3, using eval. This does work but requires escaping quote automatically. There is a risk in evaluating code like this, but since it's intended to be invoked from the REPL, no more risky than what already exists.

Answer 1

It should be obvious that nothing like this can work. More generally I think you are probably confused about what macros are for: see Rainer Joswig's answer for some guidelines about that.

Consider this case

(let ((d (read-data-from-file "my-data-file")))
  (filter d ...))

d is not known until the program is run, and hence any named columns in d are not known until that time. But if filter is a macro then it is expanded before that time and hence the lambda expression it creates is created before that time and hence the names of its arguments must be known before that time. Thus the macro can't know the names of the columns in the data.

Yet again remember that macros in Common Lisp are functions whose arguments are source code and whose values are source code. Macros, therefore, are called before the program runs and can not depend on things which are known only when the program runs.

(Of course Common Lisp is also a language which provides you eval & related things, by which you can blow your legs off at the neck, but don't do that: to repurpose a famous quote due to jwz:

a Lisp programmer once had a problem. 'I know, I'll use eval' they thought. Now they have an infinite number of problems.

---

The answer is not to use eval but to rethink the problem. As an example: let's assume that whatever data you have has fields which live in numbered columns of some kind. And maybe there's a column-ref accessor for that.

Once you have the data (so: properly after macroexpansion time) you can compute from it the names of fields which correspond to columns. So, for instance, you could construct some kind of table which maps from field names to column numbers:

(defun compute-key-table (data)
  (let ((keytable (make-hash-table)))
    ;; keytable maps from named field to its position perhaps?
    ...
    keytable))

And now you can write key-ref accessors:

(declaim (inline key-ref (setf key-ref))
(defun key-ref (data key keytable)
  ;; This does not check the key exists
  (column-ref data (gethash key keytable))

(defun (setf key-ref) (new data key keytable)
  (setf (column-ref data (gethash key keytable)) new)

And now, using these you can write this macro:

(defmacro with-data-field-accessors (data (&rest fields) &body forms)
  ;; data is either a symbol naming the data variable, or (variable <form>)
  ;; fields are the fields we want accessors for
  (let ((<kt> (make-symbol "KT"))
        (<data> (etypecase data
                  (symbol data)
                  (cons (first data)))))
    `(let* (,@(etypecase data
                  (symbol '())
                  (cons `((,@data))))
            (,<kt> (compute-key-table ,<data>)))
       (dolist (field ',fields)
         (unless (gethash field ,<kt>)
           (error "no data field ~S" field)))
       (symbol-macrolet ,(mapcar (lambda (field)
                                   `(,field (key-ref ,<data> ',field)))
                                 fields)
         ,@forms))))

So now you can use this:

(with-data-field-accessors (x (read-data)) (foo bar)
  (+ foo bar))

Which expands into:

(let* ((x (read-data)) (#:kt (compute-key-table x)))
  (dolist (field '(foo bar))
    (unless (gethash field #:kt) (error "no data field ~S" field)))
  (symbol-macrolet ((foo (key-ref x 'foo)) (bar (key-ref x 'bar))) (+ foo bar)))

Note that this macro:

• is given the field names, and does not try to compute them from information it cannot have;
• expands into code which will compute the field-name-to-column-number mapping at run-time, and will check, once, whether those mappings are valid;
• makes the field names look like variables by virtue of symbol-macrolet, which it can do since it is told the field names;
• doesn't call eval.

Obviously this is oversimplified: presumably the data has rows or something or all this would not be worth it. Whatever the answer is, it is not to use eval.

Answer 2

There are a few rules you need to apply when writing macros. Some mentioned here::

1. understand that a macro is a code generator and does not see runtime data

2. don't name a macro like a function. A macro should be named differently. Function names are active: print, append, filter, ... Macro names should be WITH-FILTER, FILTERING, ...

3. inside a macro definition make clear what a variable of the macro is. Don't use VARIABLES, but VARIABLES-SYMBOL. The value of VARIABLES is not the VARIABLES, but a SYMBOL. Use different namings for variables and functions of the macro-expansion phase.

4. clearly separate and DOCUMENT what are variables and functions of the macro expansion phase are.

5. clearly separate where code is generated and/or returned

6. test your macro with MACROEXPAND and MACROEXPAND-1 -> check the macro expansion

Now looking at your first version:

(defmacro filter (data &body body)
  (with-unique-names (variables predicate)

   ; here VARIABLES is FOO100
   ; here PREDICATE is FOO200
  
   ; now you want to return a computed list via a template

   `(let* ((,variables          ; <- FOO100

           (key-list ,data ',@body))   ; <- this is not evaluated, but a list is constructed

           ;  (key-list <value of DATA> '<spliced in value of BODY>)

           (,predicate          ; <- FOO200

            (lambda ,variables ,@body))   ; <- this is not evaluated, but a list is constructed

           ; (lambda FOO100 <spliced in value of BODY>)

           ; as you can see above makes no sense, since LAMBDA
           ; needs a list of variables, not a symbol.
           ; this its an invalid lambda list

)

      (choose ,data
              (mask ,data ,variables ,predicate)
              t))))

Now the other variant:

(defmacro filter (data &body body)
  (with-unique-names (variables predicate)

   ; here VARIABLES is FOO100
   ; here PREDICATE is FOO200
  
   ; now you want to return a computed list via a template

   `(let* ((,variables          ; <- FOO100

           (key-list ,data ',@body))   ; <- this is not evaluated,
                                       ; but a list is constructed

           ;  (key-list <value of DATA> '<spliced in value of BODY>)

           (,predicate          ; <- FOO200

            (eval (read-from-string
                               (format nil "#.(lambda ~A ~A)" 
                                            ,variables ',@body)))
            ; <- above is not evaluated, but a list is constructed

           ; (EVAL (READ-FROM-STRING
           ;         (FORMAT NIL "#.(lambda ~A ~A)
           ;                 FOO100 '<spliced in value of BODY)))

           ; Above then will run at runtime and create a function via EVAL.
           ; this is especially bad because you print code,
           ; read it and then evaluate it
           ; to generate a function -> very ugly 

)

      (choose ,data
              (mask ,data ,variables ,predicate)
              t))))


(defmacro filter (data &body body)
  (with-unique-names (variables predicate)
   `(let* ((,variables (key-list ,data ',@body))
1.         (,predicate (lambda ,variables ,@body)) ;not working
2.      ;; (,predicate (lambda (origin acceleration) ,@body)) ; works
3.         (,predicate (eval (read-from-string
                               (format nil "#.(lambda ~A ~A)" 
                                            ,variables ',@body)))
      (choose ,data
              (mask ,data ,variables ,predicate)
              t))))

And you call:

(filter data (and (string= origin "USA") (> acceleration 10)))

When the macro runs:

DATA is DATA

BODY is (AND (STRING= ORIGIN "USA") (> ACCELERATION 10))

VARIABLES is FOO100

PREDICATES is FOO200

Then you return a backquoted list. The backquoted list contains a mix of static data and data which is evaluated at macroexpansion time. Every expression with a comma in front is evaluated at macro expansion time, the rest is not evaluated at macro expansion time.

Not run at macro expansion time:

(KEY-LIST DATA '(and (string= origin "USA") (> acceleration 10))

Not run at macro expansion time:

(eval ...)

That means you have a lot of confusion:

1. new names introduced at macro-expansion time via WITH-UINQUE-NAMES

2. some code running at macro expansion time and some not, without systematic behind it

3. runtime evaluation, even with reading from text

A macro typically written:

(defmacro filtering (data &body body)

   (let* ((variable-names (key-list data body))

          (predicate-expression

               `(lambda ,variable-names ,@body)))

       ; The following is the template for the expression to generate
      
       `(choose ,data
                (mask ,data
                      ,variable-names
                      ,predicate-expression)
                t)))

That would then look like this:

(pprint (macroexpand-1 '(filter data
                                (and (string= origin "USA")
                                     (> acceleration 10)))))

(CHOOSE
 DATA
 (MASK
  DATA
  (ORIGIN ACCELERATION)
  (LAMBDA (ORIGIN ACCELERATION)
     (AND (STRING= ORIGIN "USA")
          (> ACCELERATION 10))))
 T)

Since I don't now what MASK or CHOOSE does, this is just a guess how to generate forms for them.

BUT THE BIG PROBLEM: KEY-LIST is run at macro expansion time

You really need to think this through, why you want code to be generated by a macro and how. MACROS run at COMPILE TIME and then can't see RUNTIME values. If a list of variables is only available at runtime, then the macro can't generate the code for a function, which needs these variables. The function then needs to be computed and compiled at runtime.