How can I easily find the source of error without the benefit of an interactive debugger?

Question

When I complained that Common Lisp implementations do not show line and column numbers in error backtraces, I was told that line and column numbers of errors are "not necessary in Common Lisp" because it is easy to jump to the location of the error by pressing v in SLIME. I feel uneasy about this regressive thinking, considering that almost all other programming language implementations (including Scheme and Racket) are able to show the file name, line number, and column number of errors.

I think line numbers and column numbers of errors are essential in these cases where SLIME is not used:

• When a Lisp program is running in batch mode (e.g. sbcl --script myprogram.lisp), the interactive debugger is disabled. When an error occurs, a backtrace may be printed and the program exits.

• In long-running applications such as web applications, it is common to "handle" errors by logging the backtrace to a logging facility (such as syslog) for later inspection, while continuing to serve requests despite the occurrence of the error.

Suppose an error occurs in one of the cases above. If the backtrace does not contain the file name, line number, and column number of the error, how can the source of the error be easily identified? Consider a Lisp project that has a thousand files, a few hundred thousand lines and ten thousand functions. If an error occurs and all I have is a backtrace, how do I identify the location of the error without the benefit of an interactive debugger (e.g. the SLIME debugger)?

I feel that I must be missing something, because I don't find non-interactive backtraces helpful even in my small 1000 line programs. I wonder how other Common Lisp programmers manage to read non-interactive backtraces in their million-line codebases that have thousands of files. How is it usually done?

Answer 1

In brief

You find the source exactly the same way you do so when running the program interactively:

1. have an editor which, either by some variant of tags / etags or by querying the Lisp system (not the running daemon, but another environment into which you have loaded your system), can build a mapping from symbol names to locations;
2. edit the backtrace with this editor;
3. find an interesting line in the backtrace, and type the 'edit the definition of this symbol' command at the editor;
4. ...;
5. profit.

Or, longer

Once upon a time, when the world was younger, a young and rather foolish physics student used to debug his FORTRAN programs using printed backtraces. And I do mean printed backtraces: when the machine crashed the chain printer attached to it would vomit out many sheets of paper which had procedure names and line numbers on them. And, after restarting the machine so the next user could make it crash in their turn he would take this printout and take his printout of his program and compare the line numbers: looking at the code and trying to work out what had gone wrong. He spent many hours late at night in this way.

Later on, this same student (now a maths student though) discovered a wonderful thing: a programming language called Lisp in which you could write programs to solve complex algebra problems which were of interest in his field. And although, in theory, if you had the kind of computer which maths departments could not afford, Lisp was an interactive language, this was not true in practice if all you had was the kind of computer that a maths department could afford. So things went on much as before: he would make some changes to his program, set up the equations it was going to try to solve, and then, late at night when there were no other users to inconvenience, set it off running. In the morning there might be, wonder of wonders, a solution, but more likely there would be only the corpse of the program in the form of an elaborate backtrace after it had been mortally wounded by some fierce bug (error handling was not yet thought of). This time, though, the backtrace would be in a log file from the run.

And the student made another discovery: there was a certain text editing environment used by some far-away people who had access to much bigger and better computers which purported to support Lisp programming rather well: certainly better than the editor he used then. And he managed to get a copy of this environment (legend has it it was version 17.64) on a tape from someone, and he managed to make it run, just, on the maths department's machine. And he taught it enough about the Lisp dialect he was using that it was indeed helpful, if often annoying to other users as it took rather a lot of the capacity of the machine to support it. And things were a bit better.

And this text editing system came with a rather wonderful tool: a program called 'tags' which would, for the languages it understood, make a database which mapped between definition names and their locations in the filesystem. And he modified this tags program to understand the dialect of Lisp he was using as well. Very wonderfully, the system would also cope with the case where the definition had moved, which it almost always had, and which made things like line and column numbers so brittle and useless (source control might have been invented by then, but the student knew nothing of that). This, of course, was the one of the primitive ancestors of the automatic systems which will find definitions of symbols that any reputable editor, and even some that are less than reputable, now has.

And now, when he came in in the morning to find a new backtrace from the previous night's run, he would edit this backtrace in the editing system, and find interesting lines in it, at which he would type the very wonderful 'meta-dot' or, as he knew it (not being blessed with a keyboard with a meta key), 'escape dot' command. And the disk light would come on for a little, and then he would be looking at the definition he was interested in.

Thus was the backtrace conquered. And from that day to this it has never dared raise its head again in polite company, but instead lurks, unheeded except by the few who now remember it, in the darker corners of the system. As for the student, well, no-one now remembers him at all.

Answer 2

One way to look at it is to ask "what do you really want?" from the error diagnosis -- i.e. what would the "location in source" give you?

It gives you

• a way to look at the existing code (you can navigate to it)
• a way to modify the existing code (you can make a change, and retry whatever was happening earlier)

With Common Lisp (and Smalltalk is similar in this respect) you already have the "image" with you -- which suggests two ways out.

1. with a REPL "into the image" you can jump to the functions in the stack and make the required changes "directly"
2. the standard supports (ed ...), which allows you to make a change to the form (a function, a top-level variable) without worrying about where it is located -- so you get to inspect and modify the source without really needing the row and column numbers.

In practice, I'd imagine (2) is less commonly used than (1), but its existence should allay fears of "missing source co-ordinates", since this is something the runtime already takes care of for you.

Edit #1:

• See http://www.sbcl.org/manual/#Source-Location-Printing for an example of the subtle issues involving "where the source is" for SBCL (and http://www.lispworks.com/documentation/lw71/LW/html/lw-50.htm#pgfId-897026 for a more sophisticated version in Lispworks)

• Still, if the problem is "how do I get to this function?", that is usually easily answered in most modern textual editing environments (not necessarily Emacs, even VSCode or SublimeText)

Edit #2:

• trivial-backtrace (available through Quicklisp, source here) can be seen as a portable solution to descriptive stacktraces.