GML High Float Precision Challenge (Buffers and Floats)



buffer prec.PNG

I've been stuck on this problem for weeks now. Read a lot of material but didn't help. Finally I decided to ask for your expertise. May be solved by someone more knowledgeable easily. Here is the issue:

I have a GLSL ES shader that sends some high precision floats to a surface. For the sake of simplicity, the floats are hardcoded like this:

// end of the frag shader:
gl_FragColor = vec4(0.12156862745098039, 0.47843137254902179, 0.77647058823465565, 0.31490958202630281);
These floats create a RGBA color and they are written to a surface. Then I get the surface to a buffer to get the RGBA data. Here is how i read them from the surface:

// buffer
pbuffer = buffer_create(((10*10)*4), buffer_fixed, 8);
buffer_get_surface(pbuffer, sur, 0, 0, 0);
var pixel = buffer_peek(pbuffer, (1+1)*4, buffer_u32);
var r = (pixel >> 16) & 255;
var g = (pixel >> 8) & 255;
var b = pixel & 255;
var a = (pixel >> 24) & 255;
I'm aware of the floats are converted to 4 channel 8-bits color data when they are written on the surface and the loss is inevitable like this. But is there any other way of getting them by using another method? What is the best way to get the data as close as possible to the hard-coded float values?

At the end of the day, I don't need the colors. I just need the high precision floats.

Here is the code as extension file If you wanna tinker with: precision test.gmez?dl=0

Any tiny little information is highly appreciated.

Thanks in advance.

PS: This will be a part of an open-source project which will be published on the marketplace.

IDE: GM:S EA v1.99.549.
Last edited by a moderator:


ᴏɴʟɪɴᴇ ᴍᴜʟᴛɪᴘʟᴀʏᴇʀ
Forum Staff
I think the general things to note are:
  • If you store the value in a 8-bit section, you only have precision of 0.00390625.
    That's not high-precision by most definitions.
    Could be worthwhile to write the values into a buffer as buffer_f32, which would have each float occupy one pixel (split into 4 values).
  • If you are drawing the surface to itself or other surface, you may need an appropriate blend mode (in simplest case, bm_add with surface previously cleared into black transparent color)


@YellowAfterlife Thanks for taking your time to reply.

Currently on GMS, I think 'only way'* to get shader created data to a buffer is "buffer_get_surface". I may be wrong. I hope I am wrong. *Citation needed.

If GMS is converting the shader data to 4 u8 for the color channels when writing to a surface, then we lose precision as you pointed out. I'm hoping it isn't converting to u8s. I don't know how to check tho.

Any help on inspecting is appreciated :)


aka fel666
Each r,g,b or a channels are only 8 bits wide. Thats means you get full 32 bits if you use all 4 channels.
So, i your shader, you have to somehow convert you floating point number into a full 4 bit color.
However wierd things can happen with transparency. So its safer to always set the transparency to 1, and only use rgb channels, giving you 24 bits of depth.


As has been mentioned, just split the value into 4 pieces and store it across the channels. Kind of a pain but not that hard.

In regards to the alpha issue mentioned above me, just disable alpha blending or change the blend mode when performing the render pass.