JIT compiler vs offline compilers

Question

Are there scenarios where JIT compiler is faster than other compilers like C++?

Do you think in the future JIT compiler will just see minor optimizations, features but follow a similar performance, or will there be breakthroughs that will make it infinitely superior to other compilers?

It looks like the multi core paradigm has some promise but it's not universal magic.

Any insights?

Answer 1

Are there scenarios where JIT compiler is faster than other compilers like C++?

Definitely. For example, if you find a very bad AOT compiler and an extremely good JIT compiler then you can naturally expect that JIT will be faster for those 2 implementations.

Do you think in the future JIT compiler will just see minor optimizations, features but follow a similar performance, or will there be breakthroughs that will make it infinitely superior to other compilers?

No. The reverse is significantly more likely.

Historically, most implementations of AOT have been used by developers (and not end users) causing them to optimize for a generic target (e.g. "all 64-bit 80x86 with who knows-how much RAM of what type") and not the end user's specific hardware (e.g. "AMD Ryzen model 2345 with 16 GiB of DDR3-2400 RAM"); and software is broken up into "compilation units" that are optimized separately (to create object files) and then linked with no further optimization. This created major optimization barriers that prevented AOT from achieving the performance it's capable of.

In recent years there's been a push towards whole program optimization (in the form of link time optimization and/or link time code generation) to break one of those optimization barriers.

To break the other optimization barrier (not knowing the specific target at compile time) some compilers (e.g. Intel's ICC) generate multiple versions of some parts of the code and select (at run-time) which version to use. There are also some cases where "install time AOT" occurs (e.g. Gentoo Linux); and some cases where the developer provides many separate binaries (optimized for many different targets) and the installer chooses which binary to download/install.

The other barrier to optimization comes from usage - e.g. changing the code to better suit the data it's given. Nothing prevents AOT from generating different code for different scenarios and selecting which version to use based on run-time data. The simplest and most common case is "memcpy()" where you can expect multiple versions of the code that does the copy where the version used is selected based on the amount of data being copied. With a sufficiently advanced AOT (possibly in conjunction with profiler guided optimization) this same technique can become significantly more sophisticated.

Essentially; these "historical barriers" to (AOT) optimization don't exist for JIT, and are the reason why JIT can come close to the performance of AOT; and it's far more likely that AOT will continue to find ways to avoid/fix these barriers and that AOT will increase its performance advantage over JIT.

On the other side; JIT can't do things like whole program optimization (without becoming a form of AOT) and therefore can never become as good as AOT regardless of how advanced JIT compilers become.

In addition; modifying or generating code at run-time has performance costs on modern CPUs (due to breaking speculative execution and needing special serialization, in addition to polluting things like trace caches and branch prediction data); and for modern multi-threaded software it's worse because the JIT compiler also needs to ensure code is consistent across all CPUs; and making sure that code can be modified also has a "persistent cost" (e.g. use of indirect calls instead of direct calls so that you have a single place that can be atomically updated to point to a different piece of code after JITting has been done) even when the code is not being modified and all JITting has finished.

For (virtual) memory consumption, JIT is also much worse - rather than having the program's code and the program's data; you have the program's original code and the program's data, plus the program's JITted code, the JIT compiler's code and the JIT compiler's data. As JIT compilers get more advanced they consume more memory and memory consumption becomes worse. Higher memory consumption also reduces performance a little (due to "not O(1)" overhead in managing memory at lower layers - page tables, etc), but also pushes the program closer to "out of memory" conditions (e.g. swap space use, memory allocation failures and "OOM killer" in some operating systems).

Of course most systems are multi-tasking, and global resources are shared by multiple processes; which means that if one process uses more CPU time and more memory then other completely unrelated processes (and unrelated parts of the OS - e.g. file data caches) have less resources to work with. Even when the inefficiency of JIT doesn't matter for one process (that one process is still "fast enough" and doesn't run out of memory itself) it impacts everything else and can matter for all other processes.

In other words; if you're comparing specific implementations JIT might be better (or similar, or worse), but this is an an implementation detail and not because JIT actually is better. Any fair benchmarks comparing JIT and AOT that show JIT being as good as AOT must be dependent on specific implementations; and imply that the implementation of the AOT compiler used can and should be improved (and do not imply that JIT is as good as AOT can be or should be).

However...

or will there be breakthroughs that will make it infinitely superior to other compilers?

This actually depends on what you mean by "superior". JIT has one huge advantage over AOT - the amount of time developers spend waiting for code to compile (where a desire for profit implies that higher development costs means higher cost of the final product).

Over the years/decades, for many fields, we've seen a trend towards sacrificing the quality/efficiency of the end result to reduce development time/costs (e.g. the shift to "web apps"). We've also seen the same in unrelated markets (for fun, see how hard it is to find simple clothes pegs that aren't cheap & nasty plastic that perishes in sunlight after a few years).

In the end; "low quality, low cost" wins unless there are quality standards enforced by law (which do exist, mostly for safety reasons - e.g. cars, medical equipment, electrical, flammable gases, ..); and (because there are no quality standards of any kind for almost all software) it would be reasonable to assume that JIT will become more dominant because it's cheaper (despite being lower quality/worse performance).

Answer 2

A point not yet mentioned which favors "off-line" compilers is that such compilers can usefully target platforms with small amounts of RAM--even as few as sixteen BYTES. To be sure, anything that's even remotely PC compatible is apt to have (literally) millions of times more RAM than that, but I think it will be awhile before one can find a machine with many megs of RAM that costs less than $0.50 and consumes less than a milliwatt of power during operation.

Note that the 16 bytes of RAM isn't quite so feeble as it sounds, since the chips with such small RAM don't store code in RAM--have a separate non-volatile area of memory to hold code (384 bytes is the smallest I know of). That's not a lot, of course, but it's enough to allow a $0.25 processor to perform functions that would otherwise require $1.00 worth of discrete components.

Answer 3

One advantage of JITing which has not yet been mentioned is that it's possible for a program to define an infinite number of generic types. For example:

interface IGenericAction { bool Act<T>(); }

struct Blah<T>
{
  public static void ActUpon(IGenericAction action)
  {
     if (action.Act<T>())
       Blah<Blah<T>>.ActUpon(action);
  }
}

Calling Blah<Int32>.ActUpon(act) will call act.Act<Int32>(). If that method returns true, it will call Blah<Blah<Int32>>.ActUpon(act), which will in turn call act.Act<Blah<Int32>>(). If that returns true, more calls will be performed with an even more-deeply-nested type. Generating code for all the ActUpon methods that could be called would be impossible, but fortunately it's not necessary. Types don't need to be generated until they are used. If action<Blah<...50 levels deep...>>.Act() returns false, then Blah<Blah<...50 levels deep...>>.ActUpon won't call Blah<Blah<...51 levels deep...>>.ActUpon and the latter type won't need to be created.

Answer 4

JIT compilers know more of the systems then static compilers. Adding multitheading on the fly specific to the machine can be a gigantic speed improvement once they get it working.

JIT compilers in general have a bit of startup latency where the first run of the program/code can be much slower then precompiled code. The cold start disadvantage.

Another big advantage of JIT compilation is that the compiler can be updated after your program has been build and gain new compiler tricks without needing a complete new program deployment.

Answer 5

Lots of people answered maybe I am skimming (maybe I have have the wrong end of the stick) but for me they are two different things:

AFAIK there is nothing stopping you JIT'ing compiled c++ for example project Dynamo JIT'ed machine code:

http://arstechnica.com/reviews/1q00/dynamo/dynamo-1.html

And it did actually provide speed improvements under certain circumstances.

Compiling code, in the sense of a c++ compiler means taking code written in a language and turning it into a set of instructions (or in some cases another language to then be compiled again) that can be executed by some kind of logical device.

e.g. c++ compiling to assembly (I think ;-) or c# compiling to IL or Java compiling to byte code

JIT is a process that happens during execution. The machine executing the code analyses it to see if it can improve it. Java and C# are able to make use of both as the compiler prepares commands for the VM and then the VM has the opportunity at least to have another go at optimising it.

Some programs are not compiled they are interpreted, meaning the machine that runs them reads the exact code you wrote. These machines have the oppourtunity to do some JIT, but remember they can also be statically compiled, potentially be a third party vendor in ways that the original designer of the language never intended.

So to answer your question I don't think JIT will replace static compilers. I think there will always (as long as there is programming at least) be a place for taking a representation of a program and converting that into an set of instructions for a machine of some type. (potentially optimising while it does it)

HOWEVER, I do think that JIT may become a bigger part of the story, as the Java run time and .net run time evolve I am sure JIT will get better and given things like project Dynamo I guess there is scope for hardware taking up JIT too, so that everything your processor does is re-optimised based on run time environment.

Answer 6

Another thing that was skipped in this conversation is that when you JIT a piece of code it can be compiled to a free spot in memory. In a language like C++ if the DLL is based such that that piece of memory is not available it will have to go through the expensive process of rebasing. It is faster to JIT code into an unused address then rebase a compiled DLL into a free memory space. To make things worse, a rebased DLL can no longer be shared. (see http://msdn.microsoft.com/en-us/magazine/cc163610.aspx)

I haven't been very impressed with some of the optimizations in C# 3.5 JIT code. Simple things like bit twiddling that is necessary for compression are horribly inefficient (it refused to cache values in a CPU register and instead went to memory for every operation). I don't know why it would do this but it makes a huge difference and there is nothing I can do about it.

Personally I think a good solution would be an Optimization Level (1-100) that you could set to tell JIT compiler how much time you think it ought to spend optimizing your code. The only other solution would be an AOT (Ahead of Time) compiler and then you lose many of the advantages of JIT code.

Answer 7

JIT has advantages, but I don't see it taking over completely. Conventional compilers can spend more time optimizing, while a JIT needs to strike a balance between too much optimization (taking more time than is saved by optimization) and too little (taking too much time in straight execution).

The obvious answer is to use each where it's superior. JITs can take advantage of run-time profiling more easily than conventional optimizers (although there are compilers that can take run-time profiles as input to guide optimization), and can generally afford to do more CPU-specific optimizations (again, lots of conventional compilers do this, but if you expect to run the executable on different systems they can't take full advantage of it). Conventional compilers can spend more time, and do it in different ways.

Therefore, the language system of the future will have good optimizing compilers that will emit executable code designed for use by good optimizing JIT compilers. (This is also, for many people, the language system of the present.) (The language system of the future will also support everything from modern Python/VB scripting to the ugliest high-speed number crunching.)

As with many things, this was foreshadowed by Lisp. Quite some time ago, some Lisp systems (can't really say many, there haven't been all that many Common Lisp implementations) interpreted Lisp functions by compiling them on the fly. Lisp S-expressions (what code is written in) are fairly straightforward descriptions of parse trees, so compilation could go pretty fast. In the meantime, an optimizing Lisp compiler could crunch the code where performance was really important ahead of time.

Answer 8

Yes, JIT compilers can produce faster Machine Code optimized for the current environment. But practically VM programs are slower than Native programs because JITing itself consumes time (more Optimization == more time), and for many methods JITing them may consume more time than executing them. And that's why GAC is introduced in .NET

A side effect for JITing is large memory consumption. However that's not related to computation speed, it may slow down the whole program execution, because large memory consumption increases the probability that your code will be paged out to the secondary storage.

Excuse me for my bad English.

Answer 9

Yes, there certainly are such scenarios.

• JIT compilation can use runtime profiling to optimize specific cases based on measurement of the characteristics of what the code is actually doing at the moment, and can recompile "hot" code as necessary. That's not theoretical; Java's HotSpot actually does this.
• JITters can optimize for the specific CPU and memory configuration in use on the actual hardware where the program happens to be executing. For example, many .NET applications will run in either 32-bit or 64-bit code, depending upon where they are JITted. On 64 bit hardware they will use more registers, memory, and a better instruction set.
• Virtual method calls inside of a tight loop can be replaced with static calls based on runtime knowledge of the type of the reference.

I think there will be breakthroughs in the future. In particular, I think that the combination of JIT compilation and dynamic typing will be significantly improved. We are already seeing this in the JavaScript space with Chrome's V8 and TraceMonkey. I expect to see other improvements of similar magnitude in the not-too-distant future. This is important because even so-called "statically typed" languages tend to have a number of dynamic features.

Answer 10

Basically, JIT compilers has a chance to actually profile the application being run, and do some hinting based on that information. "offline" compilers will not be able to determine how often a branch jumps and how often it falls through, without inserting special code, ask the dev to run the program, put it through its paces, and recompile.

Why does this matter?

//code before
if(errorCondition)
{
  //error handling
}
//code after

Gets converted into something like:

//code before
Branch if not error to Code After
//error handling
Code After:
//Code After

And x86 processors would not predict a conditional jump ahead in the absence of information from the branch prediction unit. That means that it predicts the error handling code to run, and the processor's going to have to flush the pipeline when it figures out that the error condition didn't occur.

A JIT compiler could see that, and insert a hint for the branch, so that the CPU would predict in the correct direction. Granted, offline compilers can structure the code in a way that would avoid the mispredict, but if you ever need to look at the assembly, you might not like it jumping around everywhere....

Answer 11

JIT compilers have more data they can use to influence optimizations. Of course, someone actually has to write code to use that data, so it's not a simple as that.