1Bit Bayer Matrix Ordered Dithering Shader

[Heavybrush]

I’m having a problem trying to fix my 1bit ordered dithering shader

this shader is going to be perfect to be used but there is some detail that I would like to fix

first of all, this shader is not just a shader, I'm going to make it for a videogame, so I don't need it to be smaller, with very few characters etc, but useful

for this reason I used a lot of uniform variables, to let them be like a bridge between the shader and the game

in this way some parameters could be dynamics and not just static, so that I can animate them or change them by pressing a button

this below is my final shader for now
I studied a lot of variants by looking at ordered dithering in wikipedia and studying almost all the dithering shaders in shadertoy, none of them are useful because I'm making this game in game maker studio 2 and some ways to write a shader are not supported, so I had to understand very well how it behaves and try to make my own shader

// Ordered dithering aka Bayer matrix dithering
varying vec2 v_vTexcoord;
varying vec4 v_vColour;

uniform vec2 resolution;

uniform float pixelSize;
uniform float ditherSize;

uniform float gamma;
uniform float contrast;

uniform int invert;

uniform int vignette;

uniform float vsize;
uniform float vouter;
uniform float vfalloff;
uniform float valpha;

uniform sampler2D textureImage;

void main()
{
    vec2 uv = gl_FragCoord.xy / resolution.xy;
    vec2 tuv = gl_FragCoord.xy / 8.0 / ditherSize;
   
    vec2 pseudoPixel = floor( gl_FragCoord.xy / pixelSize );
    vec2 pseudoResolution = floor( resolution.xy / pixelSize );
    vec2 pseudoUv = pseudoPixel / pseudoResolution;
   
    vec4 color = v_vColour * texture2D( gm_BaseTexture, pseudoUv );
    float alpha = 1.0;
   
    tuv = fract(tuv);
   
    vec4 tdither = texture2D( textureImage, tuv );
   
    vec4 lum = vec4(0.299, 0.587, 0.114, 0);
   
    float center = vsize - distance(vec2(0.5),uv);
   
    if(vignette == 1) {
        vec3 vcol = vec3(smoothstep(vouter, vouter + vfalloff, center));
        color.rgb = mix(color.rgb, color.rgb * vcol, valpha);
    }
   
    float dither = dot(tdither, lum);
    float grayscale = dot(color, lum) * gamma;
    grayscale = (grayscale - 0.5) * contrast + 0.5;
   
    vec3 col = vec3(step(dither,grayscale));
   
    if(invert == 0) {
        gl_FragColor = vec4(col, alpha);
    } else {
        gl_FragColor = vec4(1.0 - col, alpha);
    }
}

as you can see here I'm using a texture for the bayer matrix, for different reasons
- is more light to manage for the computer instead of computing a matrix
- is more easy to do, because I don't have to know how to write matrices in glsl, in game maker studio 2 and I don't have to manipulate them to get the final texture

In game maker studio I was not able to write double arrays like this

//1Bit ordered dithering patterns
    vec3 dither_8(vec2 pos, float val) {
   
        const int mat[] = int[](
        0, 48, 12, 60, 3, 51, 15, 63,
        32, 16, 44, 28, 35, 19, 47, 31,
        8, 56, 4, 52, 11, 59, 7, 55,
        40, 24, 36, 20, 43, 27, 39, 23,
        2, 50, 14, 62, 1, 49, 13, 61,
        34, 18, 46, 30, 33, 17, 45, 29,
        10, 58, 6, 54, 9, 57, 5, 53,
        42, 26, 38, 22, 41, 25, 37, 21
        );

or like this

int dither[8][8] = {
            { 0, 32, 8, 40, 2, 34, 10, 42    },
            {48, 16, 56, 24, 50, 18, 58, 26    },
            {12, 44, 4, 36, 14, 46, 6, 38    },
            {60, 28, 52, 20, 62, 30, 54, 22    },
            { 3, 35, 11, 43, 1, 33, 9, 41    },
            {51, 19, 59, 27, 49, 17, 57, 25    },
            {15, 47, 7, 39, 13, 45, 5, 37    },
            {63, 31, 55, 23, 61, 29, 53, 21    }
        };

so I decided to try sample a texture of a bayer matrix 8x8 px
try to put on top of my source image (game canvas)
multiply the texture to cover the whole canvas
and render a step function to have only black or white

the shader works to not just have the dithering but to manage the single parameters to change the image dithered

in this way I can have different effects which make the game more dynamic
some of these are:

- gamma to get the image darker or lighter
- contrast to get the image more vivid
- pixelSize to get some pixelization effects
- invert to just invert the whole game colors

IMPORTANT!
v_vColour is important to be multiplied by the final color, to keep let game maker still use his blending feature,
game maker studio use 2D sprites but could be used for 3D also, so it manage the sprites like them are made by a triangular 3D plan in an orthographic view, so v_vColour is the vertex color of the plan where there are the sprites, in this way you can blend the sprites with a color by gml.
Is very useful to have and I would not leave also if my final render is black and white, the reason is pretty simple, if I use white sprites I can always make them black multiplying black on them, but without using the shader for it, in this way I can use the shader for the whole game canvas, but i can use also some gml to fix here and there.

I got a better shader adding a vignette effect with lots of uniforms to change all parameters dynamically
so:
- in this way I can first have a vignette effect which change a little the edges of the game canvas giving more variants to all the sprites
- I can enable/disable a lamp if I need one
- I can mix the vignette effect and the gamma/contrast to have more image control
- I can animate the lamp if I need it making the lights breath and having this way a more dynamic final effect

for the vignette I decided to make dynamic every parameter:
- vignette on/off
- vignette size
- vignette outer edge
- vignette falloff
- vignette alpha

At a first moment I had the problem of the dithering size but I recently found how to fix it keeping the texture resolution but giving a third paramether to consider

vec2 tuv = gl_FragCoord.xy / 8.0 / ditherSize;

ditherSize like every other parameter is dynamic to let me change the dither size in game

at this point the shader is going to be perfect, but in some points the shader is going to cut in half some "pixels" of the bigger bayer matrix texture, with the step function
the final effect would be with a perfect dithering with squares black or white, not cutted half

I should try to leave this cut

feel free to play with it

my dither size works perfectly on the 426x240 resolution canvas
but changes when I get the fullscren
I found a good dither size of 5.0 at fullscreen but it still cut some dither pixel

I hope to have been very good explaining this issue becasue is not an easy task
I hope this could be useful for you guys to understand more about dithering and in this particular case bayer matrix and ordered dithering
and I really hope you can help me fixing it to make it perfect

thank you

--------------
edit:

I got another thing to fix unfortunately
the bayer matrix texture is just a texture applide to a surface, so
all the time I go left or right with my character everything is moving but the dithering which remain still on top
the final effect is a still dithering pattern on top of my game

maybe I should return back to a version with the matrix instead of the texture?
how can I fix this?