A more high-level explanation of Forth

Question

Preamble

Forth, through the few manuals I've read, is often defined in extremely low-level terms, typically in assembly. Defining Forth in this way is extremely counter-intuitive for understanding implementations, and truly only shines for writing assembly-based or other low-level-language based ports of Forth to varying systems/architectures.

This can be seen in the famous JonesForth, where he implements words that are not necessary to implement in assembly for speed, and which blur the line between where the assembly ends and the Forth begins.

Starting Forth is a much better work that explains Forth in a more understandable manner, however, since it is geared towards teaching Forth itself, it is not very concise in the matter of how a Forth interpreter/compiler actually functions, and again blurs the Forth implementation.

Question

What I am truly asking is concisely what is involved in a functioning Forth implementation? What is the high-level model to how Forth works? So far I understand:

• Forth uses a dictionary to look up defined words
• Forth has two stacks, a parameter stack and return stack
• The parameter stack is used to hold values being operated on, whereas the return stack is used to hold return positions from jumps to nested words
• The primitive word NEXT is used to return to the state of the top of the return stack and typically appears after every defined word.

This is all I can truly say for sure because in the ANS FORTH-83 required word set, INTERPRET is not defined, however in Starting Forth, INTERPRET is defined as a primitive word used to check the dictionary for a word, if not check if it's a number, and if not abort. All of this conflicting information, and conversely the lack of information about Forth has overall made the guts of Forth complex to understand.

Answer 1

Any Forth implementation can be logically divided into the following layers (or mechanisms):

1. Forth processor. It includes access to the data stack, return stack, memory, call and return from subroutine, logical and arithmetic operations, threaded code interpreter (or "address interpreter", if any). Sometimes it is also called Forth Virtual Machine (FVM).

2. Code generator. It is responsible for access to the data space and code space areas, creating subroutines, incremental compilation of literals, subroutine calls, returns, and control flow.

3. Interpreter. It incorporates the abilities to create vocabularies and words (vocabulary entries), manage search order, and resolve lexemes (names, numbers, etc.) depending on context.

4. Translator. It parses a text, breaks into lexemes, resolves the lexemes (using the interpreter) and transforms them into various side-effects, depending on STATE (or mode of translation). Many standard words are part of the translator. Also it can be extended by user-defined (perhaps immediate) words, or by some more advanced (implementation-specific) methods.

Every next layer is based on (and uses) the previous layers. Also some additional reusable modules can be used under the hood (for example, stacks, lists, etc.).

Understanding the next layer requires understanding the previous layer. After clarification of the initial question, this answer can be also expanded.

Answer 2

Either look at FORTH as a user/programmer (and there the Brodie books "Starting FORTH" and "Thinking FORTH" shine, or take a peek at Pelc's tutorial). No need to worry about next, or assembly, or anything "low-level". Pick some FORTH system and use that.

Or futz around with it's innards, then jonesforth is a must. Other interesting systems to look at are lbForth (self-hosting, metacompiled for several different machines), or ff (also metacompiled, boostraps from minimal "pseudo-assembler" sources).

There are also several FORTHs written in C, if you don't want to mess with low-low level details.