How can I format common lisp code (including newlines) from the command line?

Question

I'm looking for a way to pretty print/beautify/autoformat Common Lisp source code from the command line. I basically want a clone of the functionality of black for python (see https://github.com/psf/black). It would be a command line tool with minimal dependencies (I don't want to have to run it from within emacs, for example) that is idempotent and automatically inserts and removes newlines where appropriate, as well as doing the indenting.

Basically I'd like to be able to feed it source code consisting of just a single line and have it produce a readable file. Does anything satisfying all or even some of these requirements exist? I've already looked at most of the low-hanging fruit on github, and they only seem to do auto-indentation, not autoformatting (can't break up long lines). If not, is there any sort of precedent for this sort of thing in lisp, or is it unreasonably difficult for some reason that's specific to this language?

Answer 1

Using emacs as a Command Line Tool would be the best solution ...

The thing is, if you would use emacs with the correct settings, indenting is done for you by emacs.

True, emacs is not easy to use for a beginner. Very overwhelming. It took me several attempts with long periods of giving up inbetween.

I listed in a blogpost a set of minimal commands I use when writing lisp code. A list, which I would have loved someone gave me when starting emacs.

And I also wrote a blogpost for setting up a Common Lisp environment using Roswell the best Common Lisp implementations manager and package manager I have seen so far.

Emacs, you can then run in the terminal/command line - by starting it with $ emacs -nw which is the short for --no-window-system. Then it runs - like vim - inside the terminal and you have to use your keys only to move your cursor around within it and edit the code. - However you can still open new "windows" and buffers inside the terminal (vertically dissect your window by: C-x 3 and horizontally by C-x 2. Jump cursor from one window to the other: C-x o.

Such commands I listed all in a cheat-sheet manner in the firstly mentioned blogpost.

Whenever you install emacs there is a .emacs.d file in your home folder (at least in Linux and MacOs - in Windows I am very inexperienced, since I don't use it since more than 10 years ...). And inside that is an init.el file which controls the start behavior of emacs and where you configure the settings.

As kind of a minimal working init.el for Common Lisp programming - when using Roswell (means if you followed all installation instructions in my second blogpost, you can copy paste there (the language for the settings in this init.el file is emacs-lisp):

;; initialize/activate package management
(require 'package)
(setq package-enable-at-startup nil)
(setq package-archives '())

;; connect with melpa emacs lisp package repository
(add-to-list 'package-archives '("melpa"     . "http://melpa.org/packages/") t)

;; initialization of package list
(package-initialize)
(package-refresh-contents)

;; Ensure `use-package` is installed - install if not                                                                                        
(unless (package-installed-p 'use-package)
  (package-refresh-contents)
  (package-install 'use-package))

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;                                                                                       
;; slime for common-lisp                                                                                               
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;                                                                                       



;; to connect emacs with roswell
(load (expand-file-name "~/.roswell/helper.el"))

;; for connecting slime with current roswell Common Lisp implementation
(setq inferior-lisp-program "ros -Q run");; for slime;; and for fancier look I personally add:
(setq slime-contribs '(slime-fancy));; ensure correct indentation e.g. of `loop` form
(add-to-list 'slime-contribs 'slime-cl-indent);; don't use tabs
(setq-default indent-tabs-mode nil);; set memory of sbcl to your machine's RAM size for sbcl and clisp
;; (but for others - I didn't used them yet)
(defun linux-system-ram-size ()
  (string-to-number (shell-command-to-string 
                     "free --mega | awk 'FNR == 2 {print $2}'")))(setq slime-lisp-implementations 
   `(("sbcl" ("sbcl" "--dynamic-space-size"
                     ,(number-to-string (linux-system-ram-size))))
     ("clisp" ("clisp" "-m"
                       ,(number-to-string (linux-system-ram-size))
                       "MB"))
     ("ecl" ("ecl"))
     ("cmucl" ("cmucl"))))

Save it. And restart emacs. Then the correct indentation at least will work.

There are more settings which are convenient. (E.g. automatically showing you the expected function arguments while you are writing common lisp code etc. - To have them, you could git clone my .emacs.d folder by

git clone https://github.com/gwangjinkim/.emacs.d.git

And replace your .emacs.d temporarily by it. And start emacs. But at your own risk.

Answer 2

[This should be a comment but it's way too long.]

This is somewhere between hard and impossible. Consider the following form, here given on one long line:

(with-collectors (odd even) (iterate next ((i 0)) (when (< i 100) (if (evenp i) (even i) (odd i)) (next (1+ i)))))

How should this be indented? Well, here's how a fully-lisp-aware editor might indent it:

(with-collectors (odd even)
                 (iterate next ((i 0))
                          (when (< i 100) 
                            (if (evenp i)
                                (even i)
                              (odd i))
                            (next (1+ i)))))

and that's ... magnificently wrong. Here's how that same editor will indent it a little later on:

(with-collectors (odd even)
  (iterate next ((i 0))
    (when (< i 100) 
      (if (evenp i)
          (even i)
        (odd i))
      (next (1+ i)))))

This time it's got it right.

What changed? Well, what changed was the language: in particular the language in the second example has been extended to include a with-collectors form which the editor now knows how to process and also an iterate form which it also understands.

So this might seem like an obscure point, but it's not. Because Lisp's whole point (arguably) is that in order to solve problems you progressively and seamlessly extend the language from the base language you start with to the language you want to use to solve the problem.

This means that many Lisp programs consist of a series of extensions to the language, followed by a program in this new, extended language, in which the problem is solved. Lisp is a language-oriented programming-language.

What that means is that the only really reliable way of knowing how to indent a Lisp program is to ask the program. In the example above, initially the system thought that with-collectors was a function and it indented it like that. Later on, when it knew the definition, it realised that it was a let-style construct and indented it properly. Similarly for iterate.

What all this means is that a standalone tool really has no hope of indenting a substantial Lisp program well, because to do that it needs to know more about the program than it can without being the program. This, of course, is why Lisp encourages 'resident' development environments, where the program being developed is loaded into the development environment, rather than 'separated' ones where the development environment is more-or-less completely separated from the program being developed. It's possible that a standalone tool could get most of the way there by parsing the definitions in the program and spotting the ones which are extending the language. But to do that right, again, requires you to be the program.

Being a language-oriented programming language comes with significant benefits, but also with costs, of which this is unfortunately one.

---

If your task is very restricted and if you really want to take some big expression which is all on one line (and so, probably, has no comments) then something which will attempt to do this is below. You'd need to wrap this up into a program.

CAVEAT EMPTOR. This code is certainly unsafe and can execute arbitrary code depending on its input. Do not use it unless you are certain that the input you are feeding it is safe. So, don't use it, in fact.

;;;; Note horrid code, This is *certainly* unsafe
;;;
;;; This uses EVAL which I think is necessary here, but is what makes
;;; it unsafe.
;;;

(in-package :cl-user)

(eval-when (:compile-toplevel :load-toplevel :execute)
  (warn "UNSAFE CODE, USE AT YOUR OWN RISK."))

(defvar *tlf-handlers* (make-hash-table))

(defmacro define-tlf-handler (name ds-arglist &body forms)
  (let ((formn (make-symbol "FORM")))
    `(progn
       (setf (gethash ',name *tlf-handlers*)
             (lambda (,formn)
               (destructuring-bind ,ds-arglist (rest ,formn)
                 ,@forms)))
       ',name)))

(define-tlf-handler in-package (package)
  (let ((p (find-package package)))
    (if p
        (progn
          (format *debug-io* "~&Setting package ~S~%" package)
          (setf *package* p))
      (warn "no package ~S" package))))

(define-tlf-handler defpackage (package &body clauses)
  (format *debug-io* "~&Defining package ~S~%" package)
  (eval `(defpackage ,package ,@clauses)))

(define-tlf-handler defmacro (name arglist &body forms)
  (format *debug-io* "~&Defining macro ~S~%" name)
  (eval `(defmacro ,name ,arglist ,@forms)))

(define-tlf-handler eval-when (times &body forms)
  (declare (ignore times forms))
  (warn "Failing to handle eval-when"))

(define-condition pps-reader-error (reader-error simple-error)
  ())

(defparameter *pps-readtable* (copy-readtable nil))

(set-dispatch-macro-character
 #\# #\+
 (lambda (s c n)
   (declare (ignore c n))
   (error 'pps-reader-error
          :stream s
          :format-control "Can't handle #+"))
  *pps-readtable*)

(set-dispatch-macro-character
 #\# #\-
 (lambda (s c n)
   (declare (ignore c n))
   (error 'pps-reader-error
          :stream s
          :format-control "Can't handle #-"))
  *pps-readtable*)

(defun pp-stream (s &optional (to *standard-output*))
  (with-standard-io-syntax              ;note binds *package*
    (let ((*readtable* *pps-readtable*)
          (*read-eval* nil)
          (*print-case* :downcase))
      (do ((form (read s nil s) (read s nil s)))
          ((eq form s) (values))
        (format to "~&")
        (pprint form to)
        (when (and (consp form) (symbolp (car form)))
          (let ((handler (gethash (car form) *tlf-handlers*)))
            (when handler (funcall handler form))))))))

(defun pp-file (f &optional (to *standard-output*))
  (with-open-file (in f)
    (pp-stream in to)))