How are structural tokens like (), {}, and ; tracked if they are not included in the symbol table?

Question

I’m learning about compiler construction and have a question regarding the handling of structural tokens such as parentheses (), braces {}, and semicolons ;. I understand that these tokens are crucial for parsing but are not typically included in the symbol table. How does the compiler track and manage these tokens if they are not stored in the symbol table?

Here’s what I understand so far:

1. Lexical Analysis: Converts source code into a stream of tokens including identifiers, keywords, literals, operators, and structural tokens like (), {}, and ;.
2. Symbol Table: Primarily used to track entities like variables, functions, and literals that need to be referenced later. Does not typically include structural tokens.

What I’m curious about:

• How does the parser use structural tokens during syntax analysis if they are not recorded in the symbol table?
• What mechanisms or data structures are used to track the positions and usages of these tokens during parsing?

Example

Consider the following C++ code:

int main() {
    int a = 10;
    float b = 5.5;
    a = a + b;
    return 0;
}

• How are the parentheses () in main() and the braces {} around the function body tracked during the parsing of this code?

• What role do these tokens play in the construction of the parse tree or abstract syntax tree (AST)?

Also, check that my Symbol table is correct (Lexical Analysis Phase)

Answer 1

In a simplified/idealized flow, lexical analysis is followed by parsing, which is followed by static analysis, which is followed by other stages. The symbol table is created + modified as part of static analysis. So asking:

How does the parser use structural tokens during syntax analysis if they are not recorded in the symbol table?

is odd, because the parser doesn't care what is or isn't recorded in the symbol table, because that doesn't even exist yet.

You give an example of a 'symbol table', for the lexical analysis phase, but the output of lexical analysis isn't a symbol table, it's a sequence of tokens (every token in the source text, in the order they appear there), which isn't what appears in that table.

What mechanisms or data structures are used to track the positions and usages of these tokens during parsing?

The position of a token would normally be a member/property of the token data structure. Anything more specific would depend on the kind of parser you're using.

What role do these tokens play in the construction of the parse tree or abstract syntax tree (AST)?

Structural tokens are important to the parser's construction of the syntax tree, but once that's done, they're not very useful to subsequent stages. If you've got an AST, it probably doesn't even reference them.