libtooling can't find stddef.h nor other headers

Question

I am writing a tool to parse C family source code projects basically following these two tutorials 1 2 on clang 3.4 (trunk 192426) on ubuntu 12.04.

Based on offical tutorial, it says I could pass compile_commands.json by -p, however, if I only type $ ./main -p [path of compile_commands.json], it will complain about missing positional arguments. It seems like I still need to pass all file names as arguments, which is not practical if the project is really huge. I prefer it could simply parse all files specified in compile_commands.json without asking but can't find out how to turn that on.

Since I can't find a tutorial for CommonOptionsParser to do any customize thing, I use CompilationDatabase class instead. There is a dummy visitor returning true for VisitStmt, VisitDecl and VisitType so I'll skip that. The main function is pretty simple:

int main(int argc, const char **argv) {
    string errorMsg = "";
    CompilationDatabase *cd = CompilationDatabase::autoDetectFromDirectory (argv[1], errorMsg);
    ClangTool Tool(*cd, cd->getAllFiles());

    int result = Tool.run(newFrontendActionFactory<ExampleFrontendAction>());

    return result;
}

I choose opencv to parse since using cmake gaurantee the correctness of compile_commands.json(right?). However, a lot of errors show up (attached in the end). LibTooling complain about it can't find stdarg.h, stddef.h nor emmintrin.h. It's a FAQ for clang, but it says why that will happen, but didn't say how to solve that while using libtooling. Pass all arguments for clang -### for clang could solve that, but how to pass those arguments while using libtooling?

# include <stdarg.h>
          ^
1 error generated.
Error while processing /home/jcwu/repos/opencv/3rdparty/openexr/IlmImf/ImfCompressionAttribute.cpp.
In file included from /home/jcwu/repos/opencv/3rdparty/libjpeg/jmemansi.c:16:
/home/jcwu/repos/opencv/3rdparty/libjpeg/jinclude.h:35:10: fatal error: 'stddef.h' file not found
#include <stddef.h>
         ^
1 error generated.
Error while processing /home/jcwu/repos/opencv/3rdparty/libjpeg/jmemansi.c.
error: no suitable precompiled header file found in directory '/home/jcwu/repos/opencv/modules/legacy/precomp.hpp.gch'
1 error generated.
Error while processing /home/jcwu/repos/opencv/modules/legacy/src/hmmobs.cpp.
In file included from /home/jcwu/repos/opencv/3rdparty/libwebp/enc/quant.c:17:
In file included from /home/jcwu/repos/opencv/3rdparty/libwebp/enc/../dsp/../enc/vp8enci.h:17:
/usr/include/string.h:34:10: fatal error: 'stddef.h' file not found
#include <stddef.h>
         ^
1 error generated.
Error while processing /home/jcwu/repos/opencv/3rdparty/libwebp/enc/quant.c.
In file included from /home/jcwu/repos/opencv/modules/imgproc/opencv_test_imgproc_pch_dephelp.cxx:1:
In file included from /home/jcwu/repos/opencv/modules/imgproc/test/test_precomp.hpp:12:
In file included from /usr/lib/gcc/x86_64-linux-gnu/4.6/../../../../include/c++/4.6/iostream:40:
In file included from /usr/lib/gcc/x86_64-linux-gnu/4.6/../../../../include/c++/4.6/ostream:40:
In file included from /usr/lib/gcc/x86_64-linux-gnu/4.6/../../../../include/c++/4.6/ios:39:
In file included from /usr/lib/gcc/x86_64-linux-gnu/4.6/../../../../include/c++/4.6/iosfwd:42:
In file included from /usr/lib/gcc/x86_64-linux-gnu/4.6/../../../../include/c++/4.6/bits/postypes.h:42:
In file included from /usr/lib/gcc/x86_64-linux-gnu/4.6/../../../../include/c++/4.6/cwchar:46:
/usr/include/wchar.h:40:11: fatal error: 'stdarg.h' file not found
# include <stdarg.h>

==== Update ====

Read CommonOptionsParser.cpp source code. It uses FixedCompilationDatabase to guess CompilationDatabase by arguments after --, then pass arguments before -- for custom (only -p in CommonOptionParser) options. In my case compile_commands.json is required, so I could skip using CommonOptionsParser.

Therefore my problem reduce to how to pass those options from "clang -###" to LibTooling when I have a compile_commands.json? Should I invoke shell command for every file I want to parse?

==== Update ====

I think modify compile_commands.json might be easier. I'm not sure why the compile_commands.json generated by CMake won't include my system header files folder correctly, since the Makefile generate by that CMakeList.txt could compile correctly, how come the compile_commands.json miss a lot of things.

Answer 1

Please use the matching clang's version for your tools and the first argumnent of the compile command (and use the right path).

For example:

  std::vector<std::string> Args {"/home/theuser/.local/llvm-8.0.0/bin/clang-8", "-c", "a.cpp"};
  llvm::IntrusiveRefCntPtr<clang::FileManager> fileManager(new clang::FileManager(clang::FileSystemOptions()));
  clang::tooling::ToolInvocation invoker(Args, new BlankAction(), fileManager.get());
  invoker.run();

If you use clang 8.0's library to build this tool, it will succeed. If you use the other versions (clang 3.6, etc.), it may cause this error.

Answer 2

I have encountered similar problem, which panics stddef.h not found, when I invoke clang-tidy against a compile_commands.json. In my case, the reason is that, the clang-tidy was based on llvm version 7, and the clang++ version was based on llvm version 4.0, after changing clang-tidy to the right version ,the problems vanished.

Answer 3

In my case I was getting the error having installed clang-tidy but not clang as well on Ubuntu.

Answer 4

I encountered a similar issue using the python bindings.

[<Diagnostic severity 4, location <SourceLocation file '/usr/include/stdio.h', line 33, column 11>, spelling "'stddef.h' file not found">]

In the "Tips" section at

http://clang.llvm.org/docs/LibTooling.html

they mention that the default include path is

$(dirname /path/to/tool)/../lib/clang/3.3/include

The idea here it seems, is that it is expected that your tool is executing from the bin directory that also contains the clang executable itself. Typically, this will be a system directory so going up one from that will have the lib directory which contains the clang/3.4/include directory. So I manually included $(which clang)../lib/clang/3.4/include to the parser. In python, this looks something like

translation_unit = index.parse("test.cc",["-I/home/archgoon/usr/local/lib/clang/3.4/include"])

This resulted in translation_unit.diagnostics being an empty list.

Answer 5

Someone replied me says compilation database should be self contained. First I need to make sure the compile_commands.json is generated with using clang and I could use clang to build opencv.

I set these environment variables

export CC=/home/jcwu/repos/llvm-release/Release/bin/clang 
export CXX=/home/jcwu/repos/llvm-release/Release/bin/clang++ 
export C_INCLUDE_PATH=/usr/local/include:/home/jcwu/repos/llvm-release/Release/lib/clang/3.4/include:/usr/include/x86_64-linux-gnu:/usr/include  # these are from clang -v -c files.cpp 
export CPLUS_INCLUDE_PATH=/usr/local/include:/home/jcwu/repos/llvm-release/Release/lib/clang/3.4/include:/usr/include/x86_64-linux-gnu:/usr/include 

then regenerate compile_commands.json, it could find stddef.h but new issue comes up

[ 31%] Building CXX object modules/ts/CMakeFiles/opencv_ts.dir/src/ts.cpp.o 
In file included from /home/jcwu/repos/opencv/modules/ts/src/ts.cpp:116: 
/usr/include/setjmp.h:60:12: error: conflicting types for '__sigsetjmp' 
extern int __sigsetjmp (struct __jmp_buf_tag __env[1], int __savemask) __THROWNL; 
           ^ 
/usr/include/pthread.h:727:12: note: previous declaration is here 
extern int __sigsetjmp (struct __jmp_buf_tag *__env, int __savemask) __THROW; 
           ^ 
1 error generated. 
make[2]: *** [modules/ts/CMakeFiles/opencv_ts.dir/src/ts.cpp.o] Error 1 
make[1]: *** [modules/ts/CMakeFiles/opencv_ts.dir/all] Error 2 
make: *** [all] Error 2 

I can't use clang to build opencv due to a type conflict or two system header files. Havne't figured out how to solve this.