Load HDR 10-bit AVIF image into a rgba16float texture in WebGPU

Question

The Problem

I'm trying to load a 10-bit AVIF into a rgba16float texture and it's not being loaded as a HDR image (it doesn't render any values bigger than 1.0). I will post screenshots below, but this is what I'm doing:

First, I fetch the HDR Image and convert it to a blob:

const assetRequest = await fetch(assetUrl);
const assetBlob = await assetRequest.blob();

Then I create a bitmap from the blob:

const bitmap = await createImageBitmap(
    assetBlob,
    {
        colorSpaceConversion: 'none',
        imageOrientation: 'flipY',
        resizeQuality: 'high'
    }
);

And finally I create a texture and upload the bitmap to it:

const tex = device.createTexture({
    size: [bitmap.width, bitmap.height],
    format: 'rgba16float',
    usage: GPUTextureUsage.TEXTURE_BINDING | GPUTextureUsage.COPY_DST | GPUTextureUsage.RENDER_ATTACHMENT
});

device.queue.copyExternalImageToTexture(
    { source: bitmap },
    { texture: tex, colorSpace: 'display-p3' },
    { width: bitmap.width, height: bitmap.height }
);

As the image is an equirectangular projection, I convert it to a cubemap using the following shader:

struct VSCommonUniforms {
    camera: mat4x4f,
    projection: mat4x4f
};
@group(0) @binding(0) var vsCommonUniforms: VSCommonUniforms;

struct VSOutput {
    @builtin(position) position: vec4f,
    @location(0) localPos: vec3f
};

@vertex
fn vertex(@builtin(vertex_index) vertexIndex : u32) -> VSOutput {
    var output: VSOutput;

    var vertPositions = array(
        vec3f(1.0, -1.0, 1.0),
        vec3f(-1.0, -1.0, 1.0),
        vec3f(-1.0, -1.0, -1.0),
        vec3f(1.0, -1.0, -1.0),
        vec3f(1.0, -1.0, 1.0),
        vec3f(-1.0, -1.0, -1.0),

        vec3f(1.0, 1.0, 1.0),
        vec3f(1.0, -1.0, 1.0),
        vec3f(1.0, -1.0, -1.0),
        vec3f(1.0, 1.0, -1.0),
        vec3f(1.0, 1.0, 1.0),
        vec3f(1.0, -1.0, -1.0),

        vec3f(-1.0, 1.0, 1.0),
        vec3f(1.0, 1.0, 1.0),
        vec3f(1.0, 1.0, -1.0),
        vec3f(-1.0, 1.0, -1.0),
        vec3f(-1.0, 1.0, 1.0),
        vec3f(1.0, 1.0, -1.0),

        vec3f(-1.0, -1.0, 1.0),
        vec3f(-1.0, 1.0, 1.0),
        vec3f(-1.0, 1.0, -1.0),
        vec3f(-1.0, -1.0, -1.0),
        vec3f(-1.0, -1.0, 1.0),
        vec3f(-1.0, 1.0, -1.0),

        vec3f(1.0, 1.0, 1.0),
        vec3f(-1.0, 1.0, 1.0),
        vec3f(-1.0, -1.0, 1.0),
        vec3f(-1.0, -1.0, 1.0),
        vec3f(1.0, -1.0, 1.0),
        vec3f(1.0, 1.0, 1.0),

        vec3f(1.0, -1.0, -1.0),
        vec3f(-1.0, -1.0, -1.0),
        vec3f(-1.0, 1.0, -1.0),
        vec3f(1.0, 1.0, -1.0),
        vec3f(1.0, -1.0, -1.0),
        vec3f(-1.0, 1.0, -1.0)
    );

    let pos = vertPositions[vertexIndex];

    output.position = vsCommonUniforms.projection * vsCommonUniforms.camera * vec4f(pos, 1.0);
    output.localPos = pos;

    return output;
}

@group(1) @binding(0) var mapSampler: sampler;
@group(1) @binding(1) var mapTexture: texture_2d;


const invAtan = vec2f(0.1591, 0.3183);
fn SampleSphericalMap(v: vec3f) -> vec2f {
    var uv = vec2f(atan2(v.z, v.x), asin(v.y));
    uv *= invAtan;
    uv += 0.5;
    return uv;
}

@fragment
fn fragment(v: VSOutput) -> @location(0) vec4f {

    var uv = SampleSphericalMap(normalize(v.localPos.xyz)); // make sure to normalize localPos
    var color = textureSample(mapTexture, mapSampler, uv).rgb;
    
    return vec4f(color, 1.0);
    
}

This shader renders into another rgba16float, but at this point there are no values greater than 1.0. (By changing return vec4f(color, 1.0) to return vec4f(color - 1.0, 1.0) I get a black image).

What I've tried

By running exiftool on my avif image, I get the following output:

ExifTool Version Number         : 11.88
File Name                       : thatch_chapel_4k.AVIF
Directory                       : game/public/assets/img
File Size                       : 793 kB
File Modification Date/Time     : 2023:09:03 09:59:03-03:00
File Access Date/Time           : 2023:09:03 09:59:03-03:00
File Inode Change Date/Time     : 2023:09:03 09:59:03-03:00
File Permissions                : rw-r--r--
File Type                       : AVIF
File Type Extension             : avif
MIME Type                       : image/avif
Major Brand                     : AV1 Image File Format (.AVIF)
Minor Version                   : 0.0.0
Compatible Brands               : avif, mif1, miaf, MA1A
Handler Type                    : Picture
Handler Description             : AVIF Still Picture
Primary Item Reference          : 1
Exif Byte Order                 : Little-endian (Intel, II)
Software                        : Adobe Photoshop Camera Raw 15.4 (Windows)
Exif Version                    : 0231
Offset Time                     : -03:00
Offset Time Digitized           : -03:00
XMP Toolkit                     : Adobe XMP Core 7.0-c000 1.000000, 0000/00/00-00:00:00
Creator Tool                    : Adobe Photoshop Camera Raw 15.4 (Windows)
Create Date                     : 2023:08:31 11:06:29-03:00
Modify Date                     : 2023:09:03 09:56:11-03:00
Metadata Date                   : 2023:09:03 09:56:11-03:00
Format                          : image/avif
Raw File Name                   : 
Version                         : 15.4
Compatible Version              : 251658240
Process Version                 : 15.4
White Balance                   : As Shot
Incremental Temperature         : 0
Incremental Tint                : 0
Exposure 2012                   : 0.00
Contrast 2012                   : 0
Highlights 2012                 : 0
Shadows 2012                    : 0
Whites 2012                     : 0
Blacks 2012                     : 0
Texture                         : 0
Clarity 2012                    : 0
Dehaze                          : 0
Vibrance                        : 0
Saturation                      : 0
Parametric Shadows              : 0
Parametric Darks                : 0
Parametric Lights               : 0
Parametric Highlights           : 0
Parametric Shadow Split         : 25
Parametric Midtone Split        : 50
Parametric Highlight Split      : 75
Sharpness                       : 0
Luminance Smoothing             : 0
Color Noise Reduction           : 0
Hue Adjustment Red              : 0
Hue Adjustment Orange           : 0
Hue Adjustment Yellow           : 0
Hue Adjustment Green            : 0
Hue Adjustment Aqua             : 0
Hue Adjustment Blue             : 0
Hue Adjustment Purple           : 0
Hue Adjustment Magenta          : 0
Saturation Adjustment Red       : 0
Saturation Adjustment Orange    : 0
Saturation Adjustment Yellow    : 0
Saturation Adjustment Green     : 0
Saturation Adjustment Aqua      : 0
Saturation Adjustment Blue      : 0
Saturation Adjustment Purple    : 0
Saturation Adjustment Magenta   : 0
Luminance Adjustment Red        : 0
Luminance Adjustment Orange     : 0
Luminance Adjustment Yellow     : 0
Luminance Adjustment Green      : 0
Luminance Adjustment Aqua       : 0
Luminance Adjustment Blue       : 0
Luminance Adjustment Purple     : 0
Luminance Adjustment Magenta    : 0
Split Toning Shadow Hue         : 0
Split Toning Shadow Saturation  : 0
Split Toning Highlight Hue      : 0
Split Toning Highlight Saturation: 0
Split Toning Balance            : 0
Color Grade Midtone Hue         : 0
Color Grade Midtone Sat         : 0
Color Grade Shadow Lum          : 0
Color Grade Midtone Lum         : 0
Color Grade Highlight Lum       : 0
Color Grade Blending            : 50
Color Grade Global Hue          : 0
Color Grade Global Sat          : 0
Color Grade Global Lum          : 0
Auto Lateral CA                 : 0
Lens Profile Enable             : 0
Lens Manual Distortion Amount   : 0
Vignette Amount                 : 0
Defringe Purple Amount          : 0
Defringe Purple Hue Lo          : 30
Defringe Purple Hue Hi          : 70
Defringe Green Amount           : 0
Defringe Green Hue Lo           : 40
Defringe Green Hue Hi           : 60
Perspective Upright             : 0
Perspective Vertical            : 0
Perspective Horizontal          : 0
Perspective Rotate              : 0.0
Perspective Aspect              : 0
Perspective Scale               : 100
Perspective X                   : 0.00
Perspective Y                   : 0.00
Grain Amount                    : 0
Post Crop Vignette Amount       : 0
Shadow Tint                     : 0
Red Hue                         : 0
Red Saturation                  : 0
Green Hue                       : 0
Green Saturation                : 0
Blue Hue                        : 0
Blue Saturation                 : 0
HDR Edit Mode                   : 1
HDR Max Value                   : +4.00
SDR Brightness                  : 0
SDR Contrast                    : 0
SDR Clarity                     : 0
SDR Highlights                  : 0
SDR Shadows                     : 0
SDR Whites                      : 0
SDR Blend                       : 0
Convert To Grayscale            : False
Override Look Vignette          : False
Tone Curve Name 2012            : Linear
Camera Profile                  : Embedded
Camera Profile Digest           : 54650A341B5B5CCAE8442D0B43A92BCE
Auto Tone Digest                : EF29FDA952AD4CE1DB5945CEABCF8B96
Auto Tone Digest No Sat         : 1AC6E8F5D949A1FA79E9A8B651E20598
Toggle Style Digest             : A93C55B0292EADC9C5FC3AE252C3EA76
Toggle Style Amount             : 1
Has Settings                    : True
Crop Top                        : 0
Crop Left                       : 0
Crop Bottom                     : 1
Crop Right                      : 1
Crop Angle                      : 0
Crop Constrain To Warp          : 0
Has Crop                        : False
Already Applied                 : True
Ccv primaries xy                : 0.6799,0.3200,0.2650,0.6900,0.1500,0.0600
Ccv white xy                    : 0.3127,0.3290
Ccv min luminance nits          : 0.486939
Ccv max luminance nits          : 3248
Ccv avg luminance nits          : 49.857499
Scene referred                  : False
Document ID                     : xmp.did:e923d0fd-6876-8849-88f2-5ad4b49e1d7b
Instance ID                     : xmp.iid:e923d0fd-6876-8849-88f2-5ad4b49e1d7b
Original Document ID            : xmp.did:e923d0fd-6876-8849-88f2-5ad4b49e1d7b
Tone Curve PV2012               : 0, 0, 255, 255
Tone Curve PV2012 Red           : 0, 0, 255, 255
Tone Curve PV2012 Green         : 0, 0, 255, 255
Tone Curve PV2012 Blue          : 0, 0, 255, 255
History Action                  : derived, saved
History Parameters              : saved to new location
History Instance ID             : xmp.iid:e923d0fd-6876-8849-88f2-5ad4b49e1d7b
History When                    : 2023:09:03 09:56:11-03:00
History Software Agent          : Adobe Photoshop Camera Raw 15.4 (Windows)
History Changed                 : /
Derived From                    : 
Image Width                     : 4096
Image Height                    : 2048
Image Spatial Extent            : 4096x2048
Image Pixel Depth               : 10 10 10
AV1 Configuration Version       : 1
Chroma Format                   : YUV 4:4:4
Chroma Sample Position          : Unknown
Color Representation            : nclx 12 16 6
Image Size                      : 4096x2048
Megapixels                      : 8.4
Create Date                     : 2023:08:31 11:06:29-03:00
Modify Date                     : 2023:09:03 09:56:11-03:00

Given the lines: HDR Max Value : +4.00 and Image Pixel Depth : 10 10 10 I assume the image is in fact a HDR image.

The image I'm using is a conversion I did of this asset from Poly Haven inside Photoshop's Camera Raw and exported it without any changes as a 10-bit AVIF file (I selected color as P3 when exporting from Photoshop). I uploaded my conversion on Google Drive if anyone wants to see it, and I guess it's fine since the image was released under CC0.

This is the result I get with the code as described above: Please ignore the sideways rotation. The cubemap generation is fine, it's just my camera that is pointing downwards and I turned it sideways to show this part of the cubemap.

If, when generating the bitmap, I change colorSpaceConversion: 'none' to colorSpaceConversion: 'default', I get this result: The browser clearly knows it's a HDR image and applied its own tone mapping to it. I'm using Brave as my browser and the results are the same on Chrome (as expected since Brave uses Chromium). Edge just doesn't load the site for some reason and Firefox doesn't completely support WebGPU as of now, so I cannot test on it (Firefox doesn't even work with some of the WebGPU Samples).

Changing from colorSpace: 'display-p3' to colorSpace: 'srgb' when calling copyExternalImageToTexture doesn't produce any noticeable changes for me.

Since, when creating the bitmap, I'm specifying no color space conversion I'd assume it just lets the values be and doesn't do anything to them. I don't know if createImageBitmap is clamping the values behind the scenes since both the createImageBitmap and the ImageBitmap MDN pages don't specify this, they only say that colorSpaceConversion:

Specifies whether the image should be decoded using color space conversion. Either none or default (default). The value default indicates that implementation-specific behavior is used.

Now, regarding copyExternalImageToTexture, the MDN Page, on the colorSpace part, notes the following:

Note: The encoding may result in values outside of the range [0, 1] being written to the target texture, if its format can represent them. Otherwise, the results are clamped to the target texture format's range. Conversion may not be necessary if colorSpace matches the source image color space.

In this part: "The encoding may result in values outside of the range [0, 1] being written to the target texture, if its format can represent them" I would assume it can, since it's rendering to a rgba16float texture.

What I think is happening

What I'm guessing is either that:

createImageBitmap is clamping the values, and if that's the case, what else could I use to pass the texture to copyExternalImageToTexture? A VideoFrame or an OffscreenCanvas? Won't these also clamp the image?
copyExternalImageToTexture is doing some trickery and clamping the values when it shouldn't.

I don't know how I would test these options, it may even be something else.

Extra files

The code presented on this question was simplified from my project. The logic and execution order are still the same, however in my project the files are separated. If you'd like to see the actual relevant files, they are the following:

AssetManager - fetches the images;
IMGAsset - bitmap creation;
TextureUtils - creates the rgba16float texture from the bitmap on the function createRGBA16fFromHDRBitmap;
EquirecToCubemapRenderer - renders the equirectangular rgba16float texture into a rgba16float cubemap (width x height x 6) rgba16float texture;
BoardSkybox puts everything together: gets the IMGAsset and calls TextureUtils and EquirecToCubemapRenderer.

There are more files down the line (like the skybox renderer and the tone mapper) but these are the most relevant ones, as the problem happens on or before the ones I linked. The equirec to cubemap shader is already above, I only removed the comments to post it here.

Update 1

I found this (ImageBitmap: Do not clamp to 8-bit sRGB) Chromium patch that was submitted August 7th 2023. The status says "merged", however I couldn't see any difference in neither Chrome Canary (Version 118.0.5989.0 (Official Build) canary (64-bit)) nor Chrome Dev (Version 118.0.5979.2 (Official Build) dev (64-bit)).

Update 2

Instead of going Blob > ImageBitmap > copyExternalImageToTexture. I tried going HTMLImageElement > 2D P3 Canvas > copyExternalImageToTexture. The result is the same as if I set colorSpaceConversion to default, that is, the browser automatically tonemaps the image.

This is what I tried:

const canvas2d =

Update 3

I tried using ImageDecoder from the WebCodecs API (MDN) and the VideoFrame it returned was still in LDR:

const a = new ImageDecoder({ data: await this._blob.arrayBuffer(), type: 'image/avif', colorSpaceConversion: 'none' });
const result = await a.decode();
return result.image;

On top of not leaving my colors alone it also added a red tint to it:

I'm losing my faith in the supported inputs of copyExternalImageToTexture. I'm gonna start looking for some AV1 decoder and try to pass the raw data values to writeTexture.

Update 4

All of the AVIF decoders that I found returned a clamped result one way or another. So I decided to try to load the .hdr RGBE image through this io-rgbe package (GitHub, NPM) into device.queue.writeTexture and it worked.

Below is the image with an exposure of 0.1:

I also used this Float16Array polyfill to load into the texture. This is the code I wrote:

static async createRGBA16fFromRGBEData(data: HDRImageData) {

    const tex = device.createTexture({
        size: [data.width, data.height],
        format: 'rgba16float',
        usage: GPUTextureUsage.TEXTURE_BINDING | GPUTextureUsage.COPY_DST | GPUTextureUsage.RENDER_ATTACHMENT
    });

    const f16Buffer = new Float16Array(data.width * data.height * 4);

    let j = 0;
    for (let i = 0; i < data.data.length; i += 3) {
        f16Buffer[j + 0] = data.data[i + 0];
        f16Buffer[j + 1] = data.data[i + 1];
        f16Buffer[j + 2] = data.data[i + 2];
        f16Buffer[j + 3] = 1;
        j += 4;
    }

    device.queue.writeTexture(
        { texture: tex },
        f16Buffer.buffer,
        { bytesPerRow: data.width * 8 },
        { width: data.width, height: data.height }
    );

    await device.queue.onSubmittedWorkDone();

    return tex;

}

However I'm not too happy with this solution, as I've switched from loading a 793 KB image to a 26 MB one. Also that for to copy the values from the Float32Array that the package returns into a Float16Array with an extra 1.0 is terribly inefficient, hanging the loading for several seconds.

Just using the Float32Array would be tricky, as WebGPU doesn't support a format like rgb32float since the formats have to align to 4 bytes. I'd need to add an extra value to the array and I think it would be just as bad as this for presented earlier.

I'll look into the io-rgbe package source code and make my own adaptation of it to return the values inside a Float16Array with an extra 1.0 as the alpha. However I'm still not happy about the file sizes, maybe I'll try to cache it on the IndexedDB. I don't know if I'll update the question if/when I do that because it's not really related to the question.

I'll leave this question open if there's some update in the future about AVIF loading and HDR bitmaps.