Is homoiconicity really needed for having macros?

Question

2012-02-04 is sponsored by word "homoiconicity" http://en.wikipedia.org/wiki/Homoiconicity .

Background: I am about to choose which book about Clojure to buy -- either "Clojure in Action" or (incoming at the end of April) "Clojure Programming" (you can read it via O'Reilly Rough Cuts, half of the pages are visible). What struck me, that in both books this property -- homoiconicity -- got so much emphasis.

Since Clojure roots are Lisp-based, I referred "dead sexy" book is this really so big deal. Well, I can see macros explained in detail but I didn't catch the mentioned emphasis. Now compare to this (quote from "Clojure in Action")

This homoiconicity is also what makes Clojure’s macro system possible.

It almost looks like without it, macros wouldn't be possible. Even wikipedia statement (link above) is more balanced, but none of those sources (*) counts human factors in favour of English syntax.

If I am not mistaken, macro syntax (Lisp-like) could be possible in C# (for example), only with more effort from C# team. But it is cost of the designers team, not users (?). Second -- habbits matter. If in real life you think "a + b", and in computer world you constantly translate it to "+ a b", productivity suffer (I can see this for myself, when I went from C++ functors to C# lambdas).

This praise, that the Clojure programmer writes programs almost directly as AST, frightens me as reading that "thanks to writing the code directly in hex code, not only you learn hex system by heart, but you are closer to machine".

To sum up -- I love metaprogramming, I love Lisp-like macros (I am not a Lisper though), but I see two things here -- macros and homoiconicity. The first one is without a doubt great, the second -- not so much (as for my understanding), because it makes human fit computer needs, and it should be other way around.

The question

Is homoiconicity really so beneficial for humans (end users of language) or it is actually almost solely beneficial to language designers? Examples are very welcome!

Or just in case I rephrase -- assuming given language has Lisp-macros, will "adding" homoiconicity improve productivity of end-user? Expresiveness? Or quite contrary?

(*) I cannot be 100% sure, because I see only a fraction of Clojure books, and I am not reading them, just evaluating them for purchase.

Update

Thank you all for the answers, pity I had to pick only one as a solution :-), it does not mean I value others less, this one is most complete for me.

Answer 1

Homoiconicity is a poorly defined concept.

What makes Lisp different is that its has a data representation of source code and the evaluator takes it as input. Processing source code as data is easy since all the usual functionality of Lisp applies.

What it provides:

• reading textual source and creating data structures of it
• processing of source as data
• printing of the 'source as data' as text, including things like pretty printing of code

Macros are functions which transform source code. Thus all you need is some representation of the source code and a step in the execution/compilation/interpretation of the programming language where this transformation step takes place.

The way Lisp works makes it convenient, since source code is a data structure, which can be manipulated with a multitude of built-in functions and facilities. Note that more complex transformations, where one needs to understand the syntax of the language Lisp, are also not easy to do in Lisp - though tools have been implemented (for example so-called code walkers).

There are also Lisps which have a syntax, which is not based on s-expressions. An example is RLISP, the implementation language of the Computer Algebra System REDUCE. RLISP also supports macros.

One could define a similar scheme based on strings and string manipulation. You can also define a macro system based on some kind of AST.

Example of languages with such macros:

1. Dylan. See: http://opendylan.org/books/dpg/macros.html
2. Julia. See: https://en.wikipedia.org/wiki/Julia_(programming_language)
3. Nemere. See: https://github.com/rsdn/nemerle/wiki/Macros-tutorial

Answer 2

While you need a language for macroes, it does not need to be the same language that the rest of your code is written in. all you need is "iconicity"*, the "homo" part is optional.

In Ocaml the macroes are writen in a different language called camlp4

it looks something like this:

#load "pa_extend.cmo";

value gram = Grammar.gcreate (Plexer.gmake ());
value exp = Grammar.Entry.create gram "expression";

EXTEND
  exp:
  [ "SUM"
    [ x = exp; "+"; y = exp -> Op "+" x y
     | x = exp; "-"; y = exp -> Op "-" x y]
  | "PROD"
    [ x = exp; "*"; y = exp -> Op "*" x y
    | x = exp; "/"; y = exp -> Op "/" x y]
  | "ATOM"
    [ x = INT -> Int (int_of_string x)
    | x = LIDENT -> Var x
    | "("; x = exp; ")" -> x ] ]
  ;
END;

where the language it is transforming looks like this:

let rec search ?(x=0) ?(y=0) f accu = match x, y with
    9, y -> search ~x:0 ~y:(y+1) f accu (* Next row *)
  | 0, 9 -> f accu                      (* Found a solution *)
  | x, y ->
      if m.(y).[x] <> '0' then search ~x:(x+1) ~y f accu else
        fold (fun accu n ->
                let n = Char.chr (n + 48) in
                if invalid x y n then accu else
                  (m.(y).[x] <- n;
                   let accu = search ~x:(x+1) ~y f accu in
                   m.(y).[x] <- '0';
                   accu)) accu 1 10

as you can see there is no homioconicity in Ocaml and it has a highly evolved macro(like) system.

• this is not a real term, i made it up in the hopes that someone would smile :)

Answer 3

Homoiconicity is not strictly required for macros (example: the C/C++ preprocessor implements a compile time macro system).

However homoiconcity makes macros much more effective and easy to use:

• You avoid the need for a syntax parsing step: In a homoiconic language like Clojure, the source code can be directly used as the abstract syntax tree of the compiler. Less performance overhead for the compiler, less conceptual overhead for the user.
• Code generation is easier - you just need to assemble the right data structure rather than carefully producing text source code in a form that can be successfully compiled. In general, textual code generation can be quite a tricky problem in languages that were not designed for it (you need to consider lots of things like symbol name generation, library imports, quoting rules, whether the compiler needs a source file to be written to disk etc.)
• Your macro language is the same as your source language. No need to learn a separate language or set of constructs for macros.
• IMHO homoiconicity makes macros much more simple, elegant and readable. In many cases, your macro ends up looking just like a template for the correct code.

As a small example, here's a macro that adds a Java/C# style "for" loop to Clojure.

(defmacro for-loop [[sym init check change :as params] & steps]
 `(loop [~sym ~init value# nil]
    (if ~check
      (let [new-value# (do ~@steps)]
        (recur ~change new-value#))
      value#)))

;; Usage:
(for-loop [i 0 (< i 10) (inc i)] 
  (println i))

And that's it - a new language construct added in six lines of code. You can see the homoiconicity clearly in action - the whole "loop" form is just a quoted Clojure data structure. The init, check and change parameters are also homoiconic clojure data structures passed to the macro that represent the loop control expressions. There was no need to do any parsing to support my new syntax.

I think it would be very hard to do this as concisely or simply in any non-homoiconic language.

Answer 4

Imagine using the old-and-awkward web template langage methodology of using <% code %> to use embedded java within python.

Then the macro language would be java and the target language would be python. Invoking a macro would mean escaping out into the <% code %> mode. You could even design such a system to support parameters, eval of strings, and such. It would be truly ugly (kind of like old jsp or php code).

A scary thought: could some jsp users find such a system more comfortable than a lisp system? Would this monster be improved if the macro/target language were the same?

So certainly it could be done. People have done some amazing things with C++ templates. Jay Freeman wrote a C++ program with Ackermann compile time and constant run time to demonstrate that C++ templates could do such a thing (presumably because Fibonacci examples don't blow up enough compilers?).

As for homoiconicity: it doesn't solve everything. Even with it, there is often some awkwardness: Quotes, quasi-quotes, etc. You need some way of switching between "this is macro code" and "this is target language code". This is an inherent problem to be tackled by language designers. Instead of a lispy language with lists as data structures or program code, you could think about just the annoyances of using strings with an eval method. Program source is kind of like a string. A string is a really horrible data structure. And strings within strings are even worse. Look up any quine code (peferably non-lisp). They can be thought of as homoiconicity done horribly wrong.

Answer 5

No, you don't necessarily need homoiconicity to create a powerful, Lisp-like macro system. Homoiconicity just makes such a system much easier to use; without it, you'd need another syntax to express the ASTs that macros produce when evaluated. In a homoiconic language, the macro system feels natural since programs and metaprograms look the same.