Shaders Coordinate Issue

[Erayd]

I cannot seem to find a solution to my problem no matter how much time and research I put in to this topic. By all sources, my code is correct and I should have no problem, yet I do and now I need some expertise. My goal is to edit a sprite in a shader based on it's current pixel. I can't seem to get the x and y coordinate of the pixel being shaded no matter what I do. Currently I can only raise and rgb value of the entire sprite being shaded.

Vertex Shader:

attribute vec3 in_Position;                  // (x,y,z)
attribute vec4 in_Colour;                    // (r,g,b,a)
attribute vec2 in_TextureCoord;              // (u,v)

varying vec2 v_vTexcoord;
varying vec4 v_vColour;
varying vec2 position;

void main()
{
    vec4 object_space_pos = vec4( in_Position.x, in_Position.y, in_Position.z, 1.0);
    gl_Position = gm_Matrices[MATRIX_WORLD_VIEW_PROJECTION] * object_space_pos;
   
    v_vColour = in_Colour;
    v_vTexcoord = in_TextureCoord;
    position = in_Position.xy;
}

I figured this code snippet would give more red to the top right corner of my sprite, instead, the whole sprite is now simply more red instead of just a few pixels.
Fragment Shader:

varying vec2 v_vTexcoord;
varying vec4 v_vColour;
varying vec2 position;

uniform vec2 resolution;
uniform vec4 UV;

void main() {
    vec4 color = texture2D(gm_BaseTexture, v_vTexcoord);
    vec2 currPoint = vec2(v_vTexcoord.x + position.x, v_vTexcoord.y + position.y);
    vec2 checkPoint = vec2(v_vTexcoord.x + 1.0/(resolution.x/2.0), 0.0);
    if(distance(currPoint, checkPoint) > 0.25){
        color.r = 1.0;
    }
    gl_FragColor = v_vColour * color;
}

Draw Object Code:

/// @description Shading
shader_set(testShade);
UV = sprite_get_uvs(sprite_index, 0);
shader_set_uniform_f(shader_get_uniform(testShade, "UV"), UV[0], UV[1], UV[2], UV[3]);
shader_set_uniform_f(shader_get_uniform(testShade, "resolution"), sprite_width, sprite_height);
draw_self();
shader_reset();

show_debug_message("1: " + string(UV[0]) + " 2: " + string(UV[1]) + " 3: " + string(UV[2]) + " 4: " + string(UV[3]));

The debug message above outputs:

1: 0.73 2: 0.44 3: 0.74 4: 0.44

Edit: Updated with cleaner code

 

[Simon Gust]

You could tick the "used for 3D" in the sprite so you don't have to worry about texcoords.

varying vec4 v_vColour;
varying vec2 v_vTexcoord;

void main() {
    vec4 color1 = texture2D(gm_BaseTexture, v_vTexcoord);
   
    if (v_vTexcoord.x > 0.5 || v_vTexcoord.y > 0.5) {
      color1.r = 0.1;
    }

    gl_FragColor = color1 * texture2D( gm_BaseTexture, v_vTexcoord );
}

 

[Erayd]

Wouldn't that create a seperate texture page for every sprite, thereby creating a ton of texture page swaps and causing a slow down down the line? If I had the coordinates correct based on the offset of the sprite being drawn on the texture page I wouldnt ever have to worry about that issue cropping up.

 

[Simon Gust]

I guess so,

Your code just seems very complicated.
Basically:
- get sprite uvs (x1 and y1)
- get sprite width (normalised)
- get sprite height (normalised)

you can make 2 vec2 uniforms out of those.

check if the current texcoord.x is bigger than the first uv plus half of the width of the sprite.
Same with y

 

[Erayd]

I created a new shader to get this coordinate thing locked down. Currently, for the past few days every time I try to use operators on values I'm getting all kinds of errors that tell me absolutely nothing. I've checked a lot of similar code and I can't see what I'm doing wrong.

Fragment Shader:
varying vec2 v_vTexcoord;
varying vec4 v_vColour;
varying vec3 v_vInPosition;
uniform vec2 resolution;

void main()
{
    vec2 currPoint = vec2(v_vTexcoord.x + v_vInPosition.x, v_vTexcoord.y + v_vInPosition.y);
    vec2 checkPoint = vec2(v_vTexcoord.x + 1/(resolution.x/2), 0.0);
    if(distance(currPoint, checkPoint)){
        v_vColour.r = 1.0;
    }
    gl_FragColor = v_vColour * texture2D( gm_BaseTexture, v_vTexcoord );
}

The above code gives me 3 errors referencing line 14:

Fragment Shader: testShade at line 14 : '/'
Fragment Shader: testShade at line 14 : '/'
Fragment Shader: testShade at line 14 : '+'

What is that trying to say and how do I fix it?

 

[Simon Gust]

I created a new shader to get this coordinate thing locked down. Currently, for the past few days every time I try to use operators on values I'm getting all kinds of errors that tell me absolutely nothing. I've checked a lot of similar code and I can't see what I'm doing wrong.

The above code gives me 3 errors referencing line 14:

Fragment Shader: testShade at line 14 : '/'
Fragment Shader: testShade at line 14 : '/'
Fragment Shader: testShade at line 14 : '+'

What is that trying to say and how do I fix it?

Wow that's a mean one.
It's a float to int conversion problem. Since you can't compare or assign to different data types, it gives you the error.
When you write a number add a decimal point and a zero

Fragment Shader:
varying vec2 v_vTexcoord;
varying vec4 v_vColour;
varying vec3 v_vInPosition;
uniform vec2 resolution;

void main()
{
    vec2 currPoint = vec2(v_vTexcoord.x + v_vInPosition.x, v_vTexcoord.y + v_vInPosition.y);
    vec2 checkPoint = vec2(v_vTexcoord.x + 1.0 / (resolution.x / 2.0), 0.0);
    if(distance(currPoint, checkPoint)){
        v_vColour.r = 1.0;
    }
    gl_FragColor = v_vColour * texture2D( gm_BaseTexture, v_vTexcoord );
}

 

[Erayd]

I added in the point and the zero. I can't believe I missed that, I definitely already knew that. However the errors now changed, there are two but they are different. No more decipherable than the last.

Fragment Shader: testShade at line 16 : 'assign'
Fragment Shader: testShade at line 15 : ''

Line 15 and 16 are:

}
   gl_FragColor = v_vColour * texture2D( gm_BaseTexture, v_vTexcoord );

 

[Simon Gust]

I added in the point and the zero. I can't believe I missed that, I definitely already knew that. However the errors now changed, there are two but they are different. No more decipherable than the last.

Fragment Shader: testShade at line 16 : 'assign'
Fragment Shader: testShade at line 15 : ''

Line 15 and 16 are:

}
   gl_FragColor = v_vColour * texture2D( gm_BaseTexture, v_vTexcoord );

That's weird, are you using a glsl shader or is it an older version or even hlsl?
Try just

gl_FragColor = texture2D( gm_BaseTexture, v_vTexcoord );

 

[Erayd]

I believe its GLSL. I'm using GM Studio 2, all at base default settings. In GM1 I remember seeing an option to change the type of shader make but I don't see it on this one. Whatever the default is, thats what I'm running.

When I assigned the varying vec4 v_vColour to a new temp vec4 and used the new variable in the code to edit and mix with, the assign error went away. No idea why since I've seen many people edit the v_vColour variable just fine. Still don't know what the line 15 error is.

 

[Simon Gust]

I believe its GLSL. I'm using GM Studio 2, all at base default settings. In GM1 I remember seeing an option to change the type of shader make but I don't see it on this one. Whatever the default is, thats what I'm running.

When I assigned the varying vec4 v_vColour to a new temp vec4 and used the new variable in the code to edit and mix with, the assign error went away. No idea why since I've seen many people edit the v_vColour variable just fine. Still don't know what the line 15 error is.

varyings may be just read-only, I don't know, I always replaced them with my own. I like to use "color" or "position" as they are colorcoded.
Also, what's this distance function you have?

 

[Erayd]

Thats just a function inherited from the glsl language itself. It tells me the distance between two points. Which tells me I'm doing that wrong, here's an updated version of the code. I was doing the function wrong. Changing those two things got rid of those two errors. Now I have a blank sprite so I imagine I'm using the wrong number to be greater than.

varying vec2 v_vTexcoord;
varying vec4 v_vColour;
varying vec3 v_vInPosition;
uniform vec2 resolution;

void main()
{
    vec4 color = vec4(0.0);
    vec2 currPoint = vec2(v_vTexcoord.x + v_vInPosition.x, v_vTexcoord.y + v_vInPosition.y);
    vec2 checkPoint = vec2(v_vTexcoord.x + 1.0/(resolution.x/2.0), 0.0);
    if(distance(currPoint, checkPoint) > 0.25){
        color.r = 1.0;
    }
    gl_FragColor = color * texture2D( gm_BaseTexture, v_vTexcoord );
}

 

[Simon Gust]

So errors like to add a line to the actual line of the occuring error. so it might actually be at line 9, though I can't see the error. Have you tried clearing the cache?

 

[Erayd]

I ran a test and these are the UVS that show up:
1: 0.03 2: 0.55 3: 0.53 4: 0.80

All I'm getting currently is no drawing, how can I change the code given above which now runs, to drawing a specific pixel a specific color and the rest the default?

 

[Simon Gust]

I ran a test and these are the UVS that show up:
1: 0.03 2: 0.55 3: 0.53 4: 0.80

All I'm getting currently is no drawing, how can I change the code given above which now runs, to drawing a specific pixel a specific color and the rest the default?

Instead of creating color as vec4(0.0), let it duplicate texture2D( gm_BaseTexture, v_vTexcoord );, then don't forget to read back the v_vColor.

varying vec2 v_vTexcoord;
varying vec4 v_vColour;
varying vec3 v_vInPosition;
uniform vec2 resolution;

void main()
{
    vec4 color = texture2D( gm_BaseTexture, v_vTexcoord );
    vec2 currPoint = vec2(v_vTexcoord.x + v_vInPosition.x, v_vTexcoord.y + v_vInPosition.y);
    vec2 checkPoint = vec2(v_vTexcoord.x + 1.0/(resolution.x/2.0), 0.0);
    if(distance(currPoint, checkPoint) > 0.25){
        color.r = 1.0;
    }
    gl_FragColor = v_vColour * color;
}

 

[Erayd]

Ok, I added that in, still can't get just a part of the sprite to be red instead of the whole thing, something is still just not quite right. I updated the OP with the cleaner code and a better problem description.

 

[Simon Gust]

How much do you want to be red? I currently made this:

varying vec2 v_vTexcoord;
varying vec4 v_vColour;

uniform vec2 resolution;
uniform vec2 UV;

void main() 
{
    vec4 color = texture2D(gm_BaseTexture, v_vTexcoord);
    if (v_vTexcoord.x < UV.x + resolution.x / 2.0)
    {
        color.r = 1.0;
    }
   
    gl_FragColor = v_vColour * color;
}

shader_set(testShade);
var UV = sprite_get_uvs(sprite_index, 0);
shader_set_uniform_f(shader_get_uniform(testShade, "UV"), UV[0], UV[1]); // only x and y, You shouldn't assign coordinates to vec4s anyway.
shader_set_uniform_f(shader_get_uniform(testShade, "resolution"), UV[2] - UV[0], UV[3] - UV[1]); // uv2 - uv0 = width (normalised), uv3 - uv1 = height (normalised)
draw_self();
shader_reset();

Also, the varying position from in_Position.xy in the vertex shader is not useful when comparing inside textures. Those return the actual position of the pixel in the room which is still handy, just not here.

 

[Erayd]

So, correct me if I'm wrong here.

in_Position is the current pixel being shaded in the room.

v_vTexcoord is the current pixel being drawn on that sprite offset by its location on the texture page.

UVs are the x and y normalized coordinates of the sprite on the texture page(top left corner)

The normalized resolution is found using the bottom right corner of the UVs by subtracting the left from the right and the top minus the bottom, being the width and the height of the sprite in the texture page.

0,0 is the top left corner of the texture page and the sprite and 1,1 is the bottom right of the texture page and the sprite

 

[Simon Gust]

So, correct me if I'm wrong here.

in_Position is the current pixel being shaded in the room.

v_vTexcoord is the current pixel being drawn on that sprite offset by its location on the texture page.

UVs are the x and y normalized coordinates of the sprite on the texture page(top left corner)

The normalized resolution is found using the bottom right corner of the UVs by subtracting the left from the right and the top minus the bottom, being the width and the height of the sprite in the texture page.

0,0 is the top left corner of the texture page and the sprite and 1,1 is the bottom right of the texture page and the sprite

That's correct, normalised vectors always range from 0 to 1 no matter how long they are.
A texel is a pixel vector on a normalied vector.
Say the real width = 1000, or 1 normalised.
A texel in that case is 1 / 1000 wide.

 

[Erayd]

So what exactly is being returned in the sprite_get_texel_width? That's the only term I have left to understand. Where does that returned value factor in to what we just did? Oh and it did work, thank you greatly, its been two weeks of hard work at this.

 

[Simon Gust]

So what exactly is being returned in the sprite_get_texel_width? That's the only term I have left to understand. Where does that returned value factor in to what we just did? Oh and it did work, thank you greatly, its been two weeks of hard work at this.

Using the texel width of a sprite let's you know how big your texture page really is, because texture_get_width get's the normalised value which doesn't help for sending the sprite width through the shader. The way I did it is just subtract right from left in the uv which gives me the normalised sprite width.
I actually never used sprite_get_texel_width so I don't really know what it should be used for.

 

[Erayd]

Thank you, I understand perfectly, I really dont think the GM documentation is very good at explaining what each variable and method they supplied actually does but either way I know now.

 

[Erayd]

I don't mean to double post, but I wanted you to resee this, since I actually have a great application for texels. Using texels I can actually get a specific pixel by multiplying the texel width or height by a float, offsetting it by the UV and checking it against the current texture coordinate being processed.

if(v_vTexcoord.x > UV.x + texRes.x*4.0){
        color.r = 1.0;
    }

I can shade a sprite down the the exact specific pixel edge instead of just by dividing from the width of the texture. This is the exact function I needed, with your help, we've made it possible.