CODEWITHSUNDEEP
juliametaprogrammingabstract-syntax-tree
Aleksei Matiushkin
Aleksei Matiushkin

Reputation: 121020

Getting the whole AST of the file / complex code

Julia manual states:

Every Julia program starts life as a string:

julia> prog = "1 + 1"
"1 + 1"

I can easily get the AST of the simple expression, or even a function with the help of quote / code_*, or using Meta.parse / Meta.show_sexpr if I have the expression in a string.

The question: Is there any way to get the whole AST of the codepiece, possibly including several atomic expressions? Like, read the source file and convert it to AST?

Upvotes: 3

Views: 816

Answers (2)

Mason
Mason

Reputation: 3051

If you want to do this from Julia instead of FemtoLisp, you can do 

function parse_file(path::AbstractString)
    code = read(path, String)
    Meta.parse("begin $code end")
end

This takes in a file path, reads it and parses it to a big expression that can be evaluated.

This comes from @NHDaly's answer, here: https://stackoverflow.com/a/54317201/751061

If you already have your file as a string and don’t want to have to read it again, you can instead do

parse_all(code::AbstractString) = Meta.parse("begin $code end")

It was pointed out on Slack by Nathan Daly and Taine Zhao that this code won't work for modules:

julia> eval(parse_all("module M x = 1 end"))
ERROR: syntax: "module" expression not at top level
Stacktrace:
 [1] top-level scope at REPL[50]:1
 [2] eval at ./boot.jl:331 [inlined]
 [3] eval(::Expr) at ./client.jl:449
 [4] |>(::Expr, ::typeof(eval)) at ./operators.jl:823
 [5] top-level scope at REPL[50]:1

This can be fixed as follows: 

julia> eval_all(ex::Expr) = ex.head == :block ? for e in ex eval_all(e) end : eval(e);

julia> eval_all(ex::Expr) = ex.head == :block ? eval.(ex.args) : eval(e);

julia> eval_all(parse_all("module M x = 1 end"));

julia> M.x
1

Since the question asker is not convinced that the above code produces a tree, here is a graph representation of the output of parse_all, clearly showing a tree structure.

enter image description here

In case you're curious, those leaves labelled #= none:1 =# are line number nodes, indicating the line on which each following expression takes place.

As suggested in the comments, one can also apply Meta.show_sexpr to an Expr object to get a more "lispy" representation of the AST without all the pretty printing julia does by default:

julia> (Meta.show_sexpr ∘ Meta.parse)("begin x = 1\n y = 2\n z = √(x^2 + y^2)\n end")
(:block,
  :(#= none:1 =#),
  (:(=), :x, 1),
  :(#= none:2 =#),
  (:(=), :y, 2),
  :(#= none:3 =#),
  (:(=), :z, (:call, :√, (:call, :+, (:call, :^, :x, 2), (:call, :^, :y, 2))))
)

Upvotes: 5

phipsgabler
phipsgabler

Reputation: 20970

There's jl-parse-file in the FemtoLisp implementation of the Julia parser. You can call it from the Lisp REPL (julia --lisp), and it returns an S-expression for the whole file. Since Julia's Expr is not much different from Lisp S-expressions, that might be enough for you purposes.

I still wonder how one would access the result of this from within Julia. If I understand correctly, the Lisp functions are not exported from libjulia, so there's no direct way to just use a ccall. But maybe a variant of jl_parse_eval_all can be implemented.

Upvotes: 1

Related Questions