Does any Lisp allow mutually recursive macros?

Question

In Common Lisp, a macro definition must have been seen before the first use. This allows a macro to refer to itself, but does not allow two macros to refer to each other. The restriction is slightly awkward, but understandable; it makes the macro system quite a bit easier to implement, and to understand how the implementation works.

Is there any Lisp family language in which two macros can refer to each other?

Answer 1

Here's why mutually recursive macros can't work in any useful way.

Consider what a system which wants to evaluate (or compile) Lisp code for a slightly simpler Lisp than CL (so I'm avoiding some of the subtleties that happen in CL), such as the definition of a function, needs to do. It has a very small number of things it knows how to do:

• it knows how to call functions;
• it knows how to evaluate a few sorts of literal objects;
• it has some special rules for a few sorts of forms – what CL calls 'special forms', which (again in CL-speak) are forms whose car is a special operator;
• finally it knows how to look to see whether forms correspond to functions which it can call to transform the code it is trying to evaluate or compile – some of these functions are predefined but additional ones can be defined.

So the way the evaluator works is by walking over the thing it needs to evaluate looking for these source-code-transforming things, aka macros (the last case), calling their functions and then recursing on the results until it ends up with code which has none left. What's left should consist only of instances of the first three cases, which it then knows how to deal with.

So now think about what the evaluator has to do if it is evaluating the definition of the function corresponding to a macro, called a. In Cl-speak it is evaluating or compiling a's macro function (which you can get at via (macro-function 'a) in CL). Let's assume that at some point there is a form (b ...) in this code, and that b is known also to correspond to a macro.

So at some point it comes to (b ...), and it knows that in order to do this it needs to call b's macro function. It binds suitable arguments and now it needs to evaluate the definition of the body of that function ...

... and when it does this it comes across an expression like (a ...). What should it do? It needs to call a's macro function, but it can't, because it doesn't yet know what it is, because it's in the middle of working that out: it could start trying to work it out again, but this is just a loop: it's not going to get anywhere where it hasn't already been.

Well, there's a horrible trick you could do to avoid this. The infinite regress above happens because the evaluator is trying to expand all of the macros ahead of time, and so there's no base to the recursion. But let's assume that the definition of a's macro function has code which looks like this:

(if <something>
    (b ...)
    <something not involving b>)

Rather than doing the expand-all-the-macros-first trick, what you could do is to expand only the macros you need, just before you need their results. And if <something> turned out always to be false, then you never need to expand (b ...), so you never get into this vicious loop: the recursion bottoms out.

But this means you must always expand macros on demand: you can never do it ahead of time, and because macros expand to source code you can never compile. In other words a strategy like this is not compatible with compilation. It also means that if <something> ever turns out to be true then you'll end up in the infinite regress again.

---

Note that this is completely different to macros which expand to code which involves the same macro, or another macro which expands into code which uses it. Here's a definition of a macro called et which does that (it doesn't need to do this of course, this is just to see it happen):

(defmacro et (&rest forms)
  (if (null forms)
      't
    `(et1 ,(first forms) ,(rest forms))))

(defmacro et1 (form more)
  (let ((rn (make-symbol "R")))
    `(let ((,rn ,form))
       (if ,rn
           ,rn
         (et ,@more)))))

Now (et a b c) expands to (et1 a (b c)) which expands to (let ((#:r a)) (if #:r #:r (et b c))) (where all the uninterned things are the same thing) and so on until you get

(let ((#:r a))
  (if #:r 
      #:r 
      (let ((#:r b))
        (if #:r
            #:r 
            (let ((#:r c))
              (if #:r
                  #:r
                  t))))))

Where now not all the uninterned symbols are the same

And with a plausible macro for let (let is in fact a special operator in CL) this can get turned even further into

  ((lambda (#:r)
   (if #:r
       #:r
       ((lambda (#:r)
          (if #:r
              #:r
              ((lambda (#:r)
                 (if #:r 
                     #:r
                     t))
               c)))
        b)))
 a)

And this is an example of 'things the system knows how to deal with': all that's left here is variables, lambda, a primitive conditional and function calls.

One of the nice things about CL is that, although there is a lot of useful sugar, you can still poke around in the guts of things if you like. And in particular, you still see that macros are just functions that transform source code. The following does exactly what the defmacro versions do (not quite: defmacro does the necessary cleverness to make sure the macros are available early enough: I'd need to use eval-when to do that with the below):

(setf (macro-function 'et)
      (lambda (expression environment)
        (declare (ignore environment))
        (let ((forms (rest expression)))
          (if (null forms)
              't
            `(et1 ,(first forms) ,(rest forms))))))

(setf (macro-function 'et1)
      (lambda (expression environment)
        (declare (ignore environment))
        (destructuring-bind (_ form more) expression
          (declare (ignore _))
          (let ((rn (make-symbol "R")))
            `(let ((,rn ,form))
               (if ,rn
                   ,rn
                 (et ,@more)))))))

Answer 2

There have been historic Lisp systems that allow this, at least in interpreted code.

We can allow a macro to use itself for its own definition, or two or more macros to mutually use each other, if we follow an extremely late expansion strategy.

That is to say, our macro system expands a macro call just before it is evaluated (and does that each time that same expression is evaluated).

(Such a macro expansion strategy is good for interactive development with macros. If you fix a buggy macro, then all code depending on it automatically benefits from the change, without having to be re-processed in any way.)

Under such a macro system, suppose we have a conditional like this:

(if (condition)
  (macro1 ...)
  (macro2 ...))

When (condition) is evaluated, then if it yields true, (macro1 ...) is evaluated, otherwise (macro2 ...). But evaluation also means expansion. Thus only one of these two macros is expanded.

This is the key to why mutual references among macros can work: we are able rely on the conditional logic to give us not only conditional evaluation, but conditional expansion also, which then allows the recursion to have ways of terminating.

For example, suppose macro A's body of code is defined with the help of macro B, and vice versa. And when a particular invocation of A is executed, it happens to hit the particular case that requires B, and so that B call is expanded by invocation of macro B. B also hits the code case that depends on A, and so it recurses into A to obtain the needed expansion. But, this time, A is called in a way that avoids requiring, again, an expansion of B; it avoids evaluating any sub-expression containing the B macro. Thus, it calculates the expansion, and returns it to B, which then calculates its expansion returns to the outermost A. A finally expands and the recursion terminates; all is well.

What blocks macros from using each other is the unconditional expansion strategy: the strategy of fully expanding entire top-level forms after they are read, so that the definitions of functions and macros contain only expanded code. In that situation there is no possibility of conditional expansion that would allow for the recursion to terminate.

---

Note, by the way, that a macro system which expands late doesn't recursively expand macros in a macro expansion. Suppose (mac1 x y) expands into (if x (mac2 y) (mac3 y)). Well, that's all the expansion that is done for now: the if that pops out is not a macro, so expansion stops, and evaluation proceeds. If x yields true, then mac2 is expanded, and mac3 is not.

Answer 3

What is a macro?

A macro is just a function which is called on code rather than data.

E.g., when you write

(defmacro report (x)
  (let ((var (gensym "REPORT-")))
    `(let ((,var ,x))
       (format t "~&~S=<~S>~%" ',x ,var)
       ,var)))

you are actually defining a function which looks something like

(defun macro-report (system::<macro-form> system::<env-arg>)
  (declare (cons system::<macro-form>))
  (declare (ignore system::<env-arg>))
  (if (not (system::list-length-in-bounds-p system::<macro-form> 2 2 nil))
      (system::macro-call-error system::<macro-form>)
      (let* ((x (cadr system::<macro-form>)))
        (block report
          (let ((var (gensym "REPORT-")))
            `(let ((,var ,x)) (format t "~&~s=<~s>~%" ',x ,var) ,var))))))

I.e., when you write, say,

(report (! 12))

lisp actually passes the form (! 12) as the 1st argument to macro-report which transforms it into:

(LET ((#:REPORT-2836 (! 12)))
  (FORMAT T "~&~S=<~S>~%" '(! 12) #:REPORT-2836)
  #:REPORT-2836)

and only then evaluates it to print (! 12)=<479001600> and return 479001600.

Recursion in macros

There is a difference whether a macro calls itself in implementation or in expansion.

E.g., a possible implementation of the macro and is:

(defmacro my-and (&rest args)
  (cond ((null args) T)
        ((null (cdr args)) (car args))
        (t
         `(if ,(car args)
              (my-and ,@(cdr args))
              nil))))

Note that it may expand into itself:

(macroexpand '(my-and x y z))
==> (IF X (MY-AND Y Z) NIL) ; T

As you can see, the macroexpansion contains the macro being defined. This is not a problem, e.g., (my-and 1 2 3) correctly evaluates to 3.

However, if we try to implement a macro using itself, e.g.,

(defmacro bad-macro (code)
  (1+ (bad-macro code)))

you will get an error (a stack overflow or undefined function or ...) when you try to use it, depending on the implementation.