Creating a Programming Language

Question

I was wondering how professional programmers create their own programming languages.
Do they only create a compiler which reads a text file and makes an executable out of it (considering there are no syntax errors etc)?

I'm not planning on creating my own programming language (I'm obviously too unexperienced for that) , I just wanna know how they do it.

Answer 1

I found much of the resources out there too bottom-up or too theoretical, for those not looking to get your hands deep into LLVM, C or assembly, but still like to write a programming language from "scratch" (without using parser generation tools).

You could check out my tutorial series Implementing a Programming Language in Swift.

I also liked these resources, they are also very beginner friendly:

• The Programming Language Pipeline

• Let's Build a Simple Interpreter

Answer 2

Most of the time when someone needs a programming language they are creating a 'domain specific language'. Basically they're creating a programming language that's purpose built for the problem they need to solve.

Usually these programming languages do not produce executable code. They usually analyze the program and produce a data structure in memory that's a representation of the program arranged to make it easy to evaluate the statements of the program. Then they evaluate those statements directly from that data structure rather than translating that data structure into a series of assembly language instructions.

Programming languages that do this are called 'interpreted'. And there are some very popular programming languages that fall into this category. Another term that's used for languages like this is a 'scripting language'. All scripting languages are interpreted, but not all interpreted languages get called scripting languages.

The strategy of producing an 'executable' or a series of instructions that the CPU executes directly is called 'compilation'. And languages that use this strategy are called 'compiled' languages. C and C++ are both compiled languages.

One interesting thing is that compilation or interpretation are just different execution strategies. It would be possible to make a C interpreter, and it's possible to compile Python programs. But languages that achieve wide acceptance using one execution strategy rarely see implementations that use another.

So, the most important things that a programmer does to create a new programming language are these:

1. Create a lexical analyzer and/or parser. (This is the part that transforms a series of characters into the internal data structure representing the program.)
2. Create an execution engine that executes the statements represented by the data structures.

Typically the syntax for a domain specific language is chosen to make the first step extraordinarily easy. Either a syntax is chosen that's very close to an existing language so a programmer can reuse an already existing parser, or the syntax is chosen specifically to make the language very easy to parse.

The second step is generally fairly trivial, though some language features may complicate things enormously.

Answer 3

If you're interested, there's a great free course you can take on Udacity that will give you a good idea: https://www.udacity.com/course/cs262 (Programming Languages - Building a Web Browser). I'm not halfway through the course yet, but we've learned some interesting concepts as well as the foundations of lexical analysis. You might think a web browser has nothing to do with a programming language (I did), but actually, they do pretty much the same things except for compiling the code to executable form. They both have to read, parse, and lex code, and interpret it according to the specification of the language. JavaScript is also a pretty powerful language built into every modern browser (and many other 'languages' are interpreted by browsers now as well).

To give you another example, the inaugural implementation of Python was implemented in the C programming language. This allows python programs to make use of C source code. There is also a Java take on Python (jython) that interoperates with Java programs. What makes Python Python (if you ignore the batteries-included aspect of it) is the language specification which includes things like what are the reserved words, how objects are stored in memory, what kind of expressions and control structures are valid, etc. etc. I'm by no means knowledgeable enough to develop a 'serious' language like Python. But someone who is would have to develop it in another language. Even if you did have the ability to develop another language, you would have to have something special for it to become widely used, as there are thousands of programming language, many of which are considered hobby languages (for example, there are languages designed so that their source code will resemble a Shakespeare play, or a recipe).

Answer 4

You should read up on compiler construction. Some of the main areas include:

• Lexical analysis
• Syntax analysis
• Semantic analysis
• Code optimisation