Deciding where to place a function implementation

Question

Let me first state that I know that inline does not mean that the compiler will always inline a function...

In C++ there really are two places for a non-template non-constexpr function implementation to go:

1. A header, definition should be inline
2. A source file

There are benefits/negatives to placing the implementation in one or the other:

1. inline function definition
   • compiler can inline the function
   • slower compiler times both due to having to parse definitions and include implementation dependencies.
   • multiple copies of a function between multiple users on the same site
2. source file definition
   • compiler can never inline the function (maybe that's not true with LTO?)
   • can avoid recompilation if the file hasn't changed
   • one copy per site

I am in the midst of writing a reusable math library where inlining can offer significant speedups. I only have test code and snippets to work with right now, so profiling isn't an option for helping me decide. Are there any rules - or just rules of thumb - on deciding where to define the function? Are there certain types of functions, like those with exceptions, which are known to always generate large amounts of code that should be relegated to a source file?

Answer 1

The rule of thumb I work by is simple: No function definitions in headers, and all function definitions in a source file, unless I have a specific reason to do otherwise.

Generally speaking, C++ code (like code in many languages) is easier to maintain if there is a clear separation of interface from implementation. Maintenance effort is (quite often) a cost driver in non-trivial programs, because it translates into developer time and salary costs. In C++, interface is represented by declarations of functions (without definition), type declarations, struct and class definition, etc i.e. the things that are typically placed in a header, if the intent is to use them in more than one source file. Changing the interface (e.g. changing a function's argument types or return type, adding a member to a class, etc) means that everything which depends on that interface must be recompiled. In the long run, it often works out that header files need to change less often than source files - as long as interface is kept separate from implementation. Whenever a header changes, all source files which use that header (i.e. that #include it) must be recompiled. If a header doesn't change, but a function definition changes, then only the source file which contains the changed function definition, needs to be recompiled.

In large projects (say, with hundreds of source and header files) this sort of thing can make the difference between incremental builds taking a few seconds or a few minutes to recompile a few changed source files versus significantly longer to recompile a large number of source files because a header they all depend on has changed.

Then the focus can be on getting the code working correctly - in the sense of producing the same observable output given a set of inputs, meeting its functional requirements, and passing suitable test cases.

Once the code is working correctly enough, attention can turn to other program characteristics, such as performance. If profiling shows that a function is called many times and represents a performance hot-spot, then you can look at options for improving performance. One option that MIGHT be relevant for improving performance of a program that is otherwise correct is to selectively inline functions. But, every time this is done, it amounts to deciding to accept a greater maintenance burden in order to get performance. But it is necessary to have evidence of the need (e.g. by profiling).

Like most rules of thumb, there are exceptions. For example, templated functions or classes in C++ do generally need to be inlined since, more often than not, the compiler needs to see their definition in order to instantiate the template. However, that is not an justification to inlining everything (and it is not a justification for turning every class or function into a template).

Without profiling or other evidence, I would rarely bother to inline functions. Inlining is a hint to the compiler, which the compiler may well ignore, so the effort of inlining may not even be worth it. Doing such a thing without evidence may achieve nothing - in which case it is simply premature optimisation.

Answer 2

If you have no data, keep it simple.

Libraries that suck to develop don't get finished, and those that suck to use don't get used. So split h/cpp by default; that makes build times slower and development faster.

---

Then get data. Write tests and see if you get significant speedups from inlining. Then go and learn how to profile and realize your speedups where spurious, and write better tests.

How to profile and determine what is spurious and what is microbenchmark noise is between a chapter of a book and a book in length. Read SO questions about performance in C++ and you'll at least learn the 10 most common ways to microbenchmark are not accurate.

---

For general rules, smallish bits of code in tight loops benefit from inlining, as do cases where external vectorization is plausible, and where false aliasing could block compiler optimizations.

Often you can hoist the benefits of inlining into your library by offering vector operations.

Answer 3

Generally speaking, if you are statically linking (as opposed to DLL/DSO methods), then the compiler/linker will basically ignore inline and do what's sensible.

The old rule of thumb (which everyone seems to ignore) is that inline should only be used for small functions. The one problem with inlining is that all do often I see people doing some timed test, e.g.

auto startTime = getTime();
for(int i = 0; i < BIG_NUM; ++i)
{
  doThing();
}
auto endTime = getTime();

The immediate conclusion from that test is that inline is good for performance everywhere. But that isn't the case.

inlining also increases the size of your compiled exe. This has a nasty side effect in that it increases the burden placed on the instruction and uop caches, which can cause a performance loss. So in the case of a large scale app, more often than not you'll find that removing inline from commonly used functions can actually be a performance win.

One of the nastiest problems with inline is that if it's applied to the wrong method, it's very hard to get a profiler to point out a hot spot - It's just a little warmer than needed in multiple points in the codebase.

My rule of thumb - if the code for a method can fit on one line, inline it. If the code doesn't fit on one line, put it in the cpp file until a profiler indicates moving it to the header would be beneficial.