CODEWITHSUNDEEP
swiftmetalrounding-errorpixelformatcametallayer
Marcel Tesch
Marcel Tesch

Reputation: 152

Using MTLPixelFormat.rgba16Float results in random round-off errors

I'm using a Metal compute pipeline to render into a CAMetalLayer's drawable and keep running into problems with non-integer pixel formats resulting in a slightly noisy image, due to some apparent round-off errors during compositing.

Specifically I'd like to configure the layer to use .rgba16Float, but the noise is also noticable when using the fixed-point .bgr[a]10_xr[_srgb] pixel formats as well. Using integer formats such as the default .bgra8Unorm always gives the expected result.

Here is a small demo project with a side-by-side comparison to illustrate what I mean. Both boxes are backed by a CAMetalLayer that gets filled with float4(0.5f, 0.5f, 0.5f, 1.0f) for each pixel, with the left box using .bgra8Unorm and the right box using .rgba16Float. Inspecting the resulting UI with e.g. the Digital Color Meter reveals that only the left box gets filled with a solid color, while the right box exhibits some small fluctuations.

I find this rather suprising. How can such a spatial non-uniformity come to be, even if the shader only ever writes a constant value? Am I holding it wrong? Is this an Apple bug? Is it at all possible to guarantee spatially consistent rounding behavior with the current APIs?

I'd be grateful to anyone who can give some background on this phenomenon. It's obviously not a massive problem, but it still irks me.

Edit No. 1

Metal allows for specifying an explicit rounding mode for texture writes via the -ftexture-write-rounding-mode compiler flag, with possible options being native (default), rte (round to nearest even), and rtz (round toward zero). I tried out all three in the demo app, but unfortunately it didn't make any difference. (Which makes sense I suppose, as the problem seems to lie with whatever logic is applied to the texture after the shader has finished writing to it, I think.)

Edit No. 2

Setting the layer's wantsExtendedDynamicRangeContent to true seems to resolve the issue. Explicitly specifying EDR apparently causes compositing to be done with a higher precision conversion algorithm, which is interesting. As far as I'm aware, this property is only documented to influence clamping behavior.

Upvotes: 2

Views: 294

Answers (1)

Jeshua Lacock
Jeshua Lacock

Reputation: 6668

For the system to display the image, it ultimately can only display .bgra8Unorm and .bgra8Unorm_srgb (10-bit on HDR displays). So you are forcing the system to convert from .rgba16Float to .bgra8Unorm or .bgra10_xr which is relinquishing any control for the conversion.

You can still have Metal work with .rgba16Float, but the display will ultimately be 8-bit Unorm (10-bit on HDR displays). If you want full control, then provide a format that will be displayable.

Upvotes: 1

Related Questions