CODEWITHSUNDEEP
c++openglfloating-pointglsl
m. bs
m. bs

Reputation: 93

How to quantize floating point to unsigned byte in GLSL

I used floating point texture as data buffer in GLSL and need to save the data on a normal texture (each pixel's color has 1 byte). In my situation, floating point is [-2048.0, 2048.0] and so I have to quantize [-2048.0, 2048.0] to [0, 255]. I think the C++ code for this problem is like :

//*quantization*
float fvalue = ... ;  // floating point data
fvalue /= 16.0f;  // [-128.0, 128.0]
fvalue = roundf(fvalue);  // [-128, 128]
if(fvalue > 127.0f) fvalue = 127.0f;
else if(fvalue < -128.0f) fvalue = -128.0f;
u_char byte = (int)fvalue + 128;  // [0, 255]

//*inverse quantization*
u_char byte = ...;  // [0, 255]
float fvalue = byte - 128;  // [-128, 127]
fvalue *= 16.0f;  // [-2048, 2032] (it can't be helped?)

I'm not certain this code is good, but moreover I'm not really sure what is great in GLSL (GLSL handles byte value [0, 255] as floating point [0.0, 1.0]). My code is :

//*quantization*
vec3 F = ...;  //F is floating vector [-2048.0, 2048.0]
F /= 16;  // [-128.0, 128.0]
F /= 256;  // [-0.5, 0.5]
F += vec3(0.50f);  // [0.0, 1.0]
gl_FragData[0] = vec4(F, 1.0);

//*inverse quantization*
vec3 F = texture2D(...); //byte data [0.0, 1.0]
F -= vec3(0.50f); //byte data [-0.5, 0.5]
F *= 256; //[-128, 128]
F *= 16; //[-2048, 2048]

This didn't work well. However, if I rewrite codes F += vec3(0.50f); to F += vec3(0.51f); and also F -= vec3(0.50f); to F -= vec3(0.51f);, It seems works well. But I don't think the value 0.51f is reasonable. In fact, this works well in one hardware, while this doesn't work well in another hardware. I want to know the good way to quantize (also inv-quantize) float values.

Upvotes: 0

Views: 2186

Answers (2)

Andon M. Coleman
Andon M. Coleman

Reputation: 43329

First of all, each pixel having 1 byte does not adequately convey what you are trying to describe. This so-called "normal texture" is more accurately referred to as "unsigned normalized" (often shortened to unorm).

You want an 8-bit unorm texture here (ideally with multiple components); these are textures that store fixed-point data and are treated like floating-point (in the range [0.0,1.0]) when sampled by normalizing the data to its intrinsic range (e.g. promoting to floating-point and dividing by 255.0).

Given what was just described, you simply need to transform the original data [-2048.0,2048.0] into [0.0,1.0] and then multiply by 255.

This is rather undesirable though, because you will lose the ability to represent the original range without severe aliasing. Instead, multiply by 4294967295 (2564-1) and pack 8-bits into R, 8-bits into G, 8-bits into B and 8-bits into A. You have made no attempt to pack the components in the shader shown.

Upvotes: 1

m. bs
m. bs

Reputation: 93

I can find the way which works "well". I'm afraid to say I can't explain reasonably why it works and so I don't know whether this is versatile method.

//*quantization*
vec3 F = ...;  //F is floating vector [-2048.0, 2048.0]
F += 2048;
F /= 16;
F /= 255;
gl_FragData[0] = vec4(F, 1.0);

//*inverse quantization*
vec3 F = texture2D(...);  //byte data [0.0, 1.0]
F *= 255.0;
F *= 16.0;
F -= vec3(2048 + 8);  //adding bias -16.0/2.0
F = 2.0 * F * qp * Q / 16.0;

Upvotes: 1

Related Questions