CODEWITHSUNDEEP
pythonc++parsinglibclang
user6167676
user6167676

Reputation:

Using libclang to parse in C++ in Python

After some research and a few questions, I ended up exploring libclang library in order to parse C++ source files in Python.

Given a C++ source 

int fac(int n) {
    return (n>1) ? n∗fac(n−1) : 1;
}

for (int i = 0; i < linecount; i++) {
   sum += array[i];
}

double mean = sum/linecount;

I am trying to identify the tokens fac as a function name, n as variable name, i as a variable name, mean as variable name, along with each ones position. I interested in eventually tokenizing them.

I have read some very useful articles (eli's, Gaetan's) as well as some stack overflow questions 35113197, 13236500.

However, given I am new in Python and struggling to understand the basics of libclang, I would very much appreciate some example chunk of code which implements the above for me to pick up and understand from.

Upvotes: 6

Views: 16549

Answers (1)

Andrew Walker
Andrew Walker

Reputation: 42520

It's not immediately obvious from the libclang API what an appropriate approach to extracting token is. However, it's rare that you would ever need (or want) to drop down to this level - the cursor layer is typically much more useful.

However, if this is what you need - a minimal example might look something like:

import clang.cindex

s = '''
int fac(int n) {
    return (n>1) ? n*fac(n-1) : 1;
}
'''

idx = clang.cindex.Index.create()
tu = idx.parse('tmp.cpp', args=['-std=c++11'],  
                unsaved_files=[('tmp.cpp', s)],  options=0)
for t in tu.get_tokens(extent=tu.cursor.extent):
    print t.kind

Which (for my version of clang) produces

TokenKind.KEYWORD
TokenKind.IDENTIFIER
TokenKind.PUNCTUATION
TokenKind.KEYWORD
TokenKind.IDENTIFIER
TokenKind.PUNCTUATION
TokenKind.PUNCTUATION
TokenKind.KEYWORD
TokenKind.PUNCTUATION
TokenKind.IDENTIFIER
TokenKind.PUNCTUATION
TokenKind.LITERAL
TokenKind.PUNCTUATION
TokenKind.PUNCTUATION
TokenKind.IDENTIFIER
TokenKind.PUNCTUATION
TokenKind.IDENTIFIER
TokenKind.PUNCTUATION
TokenKind.IDENTIFIER
TokenKind.PUNCTUATION
TokenKind.LITERAL
TokenKind.PUNCTUATION
TokenKind.PUNCTUATION
TokenKind.LITERAL
TokenKind.PUNCTUATION
TokenKind.PUNCTUATION

Upvotes: 14

Related Questions