CODEWITHSUNDEEP
c++openglvulkan
Antoine Morrier
Antoine Morrier

Reputation: 4078

Rendering and testing, does the driver matter?

I am developping a rendering engine (a tiny one ahah)

I want to write regression test, to be sure that I don't introduce bug when I add a new feature.

If I develop a compute shader, or something that transfer data from a buffer to a buffer, test is easy. We fill the first buffer, and we check if, after transfering, the second buffer has corresponding data.

However for rendering, it seems difficult... I ended up with several solutions

  1. No test at all
  2. Take a screenshot, and when the test begins display the screenshot and the rendering, and ask the user (the tester) to check if the result seems good, or not
  3. Take a screenshot, and when the test begins compare the rendering against the screenshot.

The first solution is... not a solution. The second solution seems good, however I think it is sad that we lose the the automatic thing...

The third solution seems to be the best one. However, to be truly effective, it means that one algorithm, if it is executed on a Nvidia card, or an AMD, or intel or ... must produce exactly the same result.

So here is the question, can we rely on a screenshot and bit by bit comparison to have regression tests? Or will it depends on the driver and or the manufacturer?

Upvotes: 1

Views: 74

Answers (2)

stridecolossus
stridecolossus

Reputation: 1561

To be honest the inherent differences between hardware implementations is a relatively minor problem, the bigger issue is the time taken to create test data for a bit-by-bit regression test.

The real power of a test suite is (as you say) to quickly check for regression when you add new functionality or resolve bugs. Using your example, if you changed the way you've implemented buffers it's easy to modify the unit-test accordingly if needed, e.g. perhaps the memory type of the buffer changes, refactoring the unit-test for that is trivial.

Applying the same approach to the visual output of your software requires a lot more effort, especially if you make a change that requires you to re-generate the expected output, i.e. changing an expected value in a unit-test is orders or magnitude easier than changing an entire scene.

Even for web most developers don't bother trying to 'test' the visuals and focus on the functionality, it's just not worth the effort of maintaining the test.

Upvotes: 1

Nicol Bolas
Nicol Bolas

Reputation: 474116

Neither Vulkan nor OpenGL provides any guarantees about binary identical results across different implementations. Even a driver update for the same card can produce different results.

The best you can do for something like that is get a result that is "close" to the expected result.

Upvotes: 3

Related Questions