GameMaker [Solved] Need help with 3D Shadows using a single directional light

Anixias

Member
Hi all. I'm trying to get real-time dynamic lighting in my prototype right now. The game is 3D in a perspective projection. It will have a view similar to that of an RTS or base-building game, so top-down ish at around 30 degrees down from straight forward (probably going to tweak that later).

Anyway, to get to the point, I found an old GM:S 1.4 project that has dynamic lighting using point lights. It can have an infinite number of them. I studied and studied it, and I understand how it all works. I've also been studying and watching videos relating to Shadow Mapping, so I understand the principle.

Here's where I'm at now. I have a directional light that renders the scene in an orthographic projection (where it renders from is currently fixed for testing, but later it will dynamically move so the whole view of the camera is captured by the light's projection). It renders the scene to a surface, ignoring any objects I flag as not being a shadowcaster. It uses a shader to store depth/distance to each fragment in the color values of the pixels, rather than texture information.



Now I'm stuck. I have a Depth Buffer/Z-Buffer/Shadow Map, whatever you want to call it. It stores the depth/distance to each fragment in the world via an orthographic view (because it's a directional light, so all light rays are parallel). I'm trying to work out how to adapt the old GM:S project's shader to fit a directional light, but I'm not exactly sure how.

I know, in principle, how it works. For each fragment, I multiply it's position by a few different matrices that describe the view from the light's perspective, and use that new position to sample the shadow map the light creates. From this, I compare the fragment's distance from the light to the distance recorded in the shadow map. If it's further away than the distance recorded in the shadow map, this fragment is in shadow.

But, I just don't know how to actually do all of that. I'm still fairly new to using GLSL ES. I think I'm supposed to do the projections in the fragment shader, because doing them in the vertex shader will actually project the pixel, so the game would literally be rendered from the light's perspective. But I'm still unsure of how to do this. Could anyone walk me through creating a shader for this purpose?

Extra information:
I have all of the uniforms in the shader working, including passing the sampler2D shadow map from the light. This game will never need more than one light, so only supporting directional lights (or just a single directional light) is perfectly acceptable.
I have a global ambient color variable as well that is simply a floating point value (0.0 - 1.0), 0.0 meaning that shadows should be solid black, and 1.0 meaning shadows should be invisible (white, when multiplying colors together this means it's basically invisible).

TL;DR Working on 3D lighting with only a single directional light (will never need any other lights of any kind). I need someone to help me create the second-pass shader for actually transforming pixel positions into a directional light's projection and view so I can sample the shadow map and render the amount of shadow it's in from ambient_color to white.
 
Basically you are taking a 3d world position, and transforming it into a texture position. So multiply that world position by the light's projection and view matrix (uv = light_proj_view * world_pos)... and then transform the xy components uv.xy*(0.5,-0.5)+0.5.

I believe you can do the matrix multiplication in the vertex shader.
 

Anixias

Member
Don't I need to take into account where the light is rendering from? Where do I fit that in?

My current understanding is that the shader runs on the whole screen when I go to render it. This is rendered to a surface, which is then multiplied onto the screen.

Within the shader itself, it acts on each fragment of the screen from the camera's point of view. It is multiplied by the light's view and proj mat. This puts it into the light's projection, however it does not take into account where the light actually is rendering from, so I'm not sure how to find the exact UV coordinates.

Also, would all of this automatically shade planes pointing partially away from the light? As in, partially facing the light but looking partially away, would this automatically decide that it is only partially in the shade, or that it isn't in the shade? So do I need to continue using GameMaker's built-in lighting system for this aspect? Wouldn't that darken sides facing away from the light more than they need to be?
 
Last edited:
Don't I need to take into account where the light is rendering from? Where do I fit that in?

My current understanding is that the shader runs on the whole screen when I go to render it. This is rendered to a surface, which is then multiplied onto the screen.

Within the shader itself, it acts on each fragment of the screen from the camera's point of view. It is multiplied by the light's view and proj mat. This puts it into the light's projection, however it does not take into account where the light actually is rendering from, so I'm not sure how to find the exact UV coordinates.
The idea is to use the same view and projection that you used to draw the shadows. Transforming the same world position by the same view and projection will result in the same uv position.

Also, would all of this automatically shade planes pointing partially away from the light? As in, partially facing the light but looking partially away, would this automatically decide that it is only partially in the shade, or that it isn't in the shade? So do I need to continue using GameMaker's built-in lighting system for this aspect? Wouldn't that darken sides facing away from the light more than they need to be?
This is totally different from the shadows problem. You can do basic lighting easily in your shader. Try this... light_level = max(0.0, dot( light_direction, normal_direction ) )
 

Anixias

Member
Ah yeah my bad, I forgot the view matrix is built using the light's position anyway. And I'm assuming I would use the dot product part in conjunction with the shade/notshade thing (max(0.0, dot( light_direction, normal_direction ) ) * (shadowamount) ) to determine the final output color from 0.0 - 1.0, where shadowamount is 0.0 - 1.0 (0 when in shadow, 1 when in light, and between those when near the edge of shadows for anti-aliasing).

Thank you for giving me a better understanding of this!

Edit: I tried working it out but it's just outputting solid black.

Vertex Shader:
Code:
varying vec3 vertPosition;
varying vec3 vertNormal;
varying vec4 vertColour;
varying vec2 vertTexcoord;
varying vec2 uvPosition;

attribute vec3 in_Position;                  // (x,y,z)
attribute vec3 in_Normal;                    // (x,y,z)
attribute vec4 in_Colour;                    // (r,g,b,a)
attribute vec2 in_TextureCoord;              // (u,v)

uniform mat4 proj_view;

void main() {
    vertPosition = (gm_Matrices[MATRIX_WORLD] * vec4(in_Position,1.0)).xyz; // Position in world
    vertNormal = (gm_Matrices[MATRIX_WORLD] * vec4(in_Normal, 0.0)).xyz; // Normal in world
 
    uvPosition = (proj_view * vec4(vertPosition,1.0)).xy;
    uvPosition = (uvPosition * 0.5)+0.5;
 
    vertTexcoord = in_TextureCoord;
    vertColour = in_Colour;
 
    gl_Position = gm_Matrices[MATRIX_WORLD_VIEW_PROJECTION] * vec4(in_Position, 1.0);
}
Fragment Shader:
Code:
const float pi = 3.141592653;
const float tau = pi * 2.0;

uniform vec3 lightPosition;
uniform float lightRange;
uniform vec3 lightColor;
uniform vec3 lightDirection;
uniform float worldAmbient;
uniform float shadowMapSize;
uniform sampler2D shadowMap;

varying vec3 vertPosition;
varying vec3 vertNormal;
varying vec2 vertTexcoord;
varying vec4 vertColour;
varying vec2 uvPosition;

float texelSize = 1.0 / shadowMapSize;

float ColorToFloat(vec4 c) {
    return (c.r + c.g / 255.0 + c.b / (255.0 * 255.0)) * lightRange;
}

float getShadow(sampler2D shadowMap, vec2 uv) {
    // Returns the shadow (1=Full shadow, 0=In light) of the pixel.
    // It convertes the pixel to a point on the given shadow map
    float shadow = 0.0;
 
    float myDepth = distance(vertPosition, lightPosition);
    float detail = 10.0; // Higher detail = more lag
    float sampleDistance = 0.1 + 2.5 * pow(1.0 - clamp(myDepth / lightRange, 0.0, 1.0), 2.0); // Higher distance = smoother shadows, more artifacts
    float bias = 0.02; // Higher bias = less artifacts, shadows seem to be floating ("Peter Panning")
 
    // Percentage Closer Filtering
    for (float i = 0.0; i < tau; i += tau / detail) {
        float sampleDepth = ColorToFloat(texture2D(shadowMap, uv + vec2(cos(i), sin(i)) * texelSize * sampleDistance)); // Grab samples from circle around texture position
        if (sampleDepth / myDepth < 1.0 - bias) shadow += 1.0 / detail;
    }
    shadow = sin(shadow * (pi / 2.0));

    return shadow;
}

void main() {
    vec3 toLight = vertPosition - lightPosition;
    vec4 lookDir = vec4( // Get the direction from the pixel to the light
        (toLight.x) / distance(vertPosition.xy, lightPosition.xy),
        (toLight.y) / distance(vertPosition.xy, lightPosition.xy),
        (toLight.z) / distance(vertPosition.xz, lightPosition.xz),
        (toLight.z) / distance(vertPosition.yz, lightPosition.yz)
    );
    vec3 N = normalize(vertNormal);
    vec3 L = normalize(-toLight);
    float shadow = 1.0; // Shadow factor
    float dif = max(0.0, dot(N, L)); // Diffuse factor
    float att = 1.0 - distance(vertPosition, lightPosition) / lightRange; // Attenuation factor
 
    if (att > 0.0) {
        shadow = getShadow(shadowMap, uvPosition);
    }
 
    float col = ((1.0 - shadow) * dif + worldAmbient) * att;
 
    //gl_FragColor = vec4(lightColor * col, 1.0);
    gl_FragColor = texture2D(shadowMap,uvPosition);
}
As a test at the end of the fragment shader, I was trying to just output the color corresponding to this vertex in the shadow map, with no success (still black everywhere). This is code from an old GM:S 1.4 project someone made that I'm trying to adapt for my game (their game had infinite point lights, mine is a single directional light).

My guess is that the uvPosition is incorrect in the vertex shader.
 
Last edited:
Are you sure that the shadow map is being drawn correclty in the first place?

Are you sure that proj_view is calculated correctly?

How can a direcitonal light have a position? It should have a direction only.

By the way, if this light is supposed to be an overhead sun, there might be easier ways of creating and using the shadow map than view or projeciton matrices.
 

Anixias

Member
I have to render the shadow map from somewhere, so I'm just moving the directional light with the camera in such a way that the whole field of view is covered. I don't know how else to do it with directional lights, as that's all I could find on it online (render the scene from the light's point of view, and if it's a directional light, make it orthographic. One video even showed how to place the directional light so that the whole area in front of the camera is covered).
Code:
//The matrices themselves are built like this:
mproj = matrix_build_projection_ortho(shadowmap_size,shadowmap_size,0.1,ZFAR);
mlook = matrix_build_lookat(x,y,z,xto,yto,zto, 0,0,1);

//This is how I'm setting the shader's uniform:
shader_set_uniform_f_array(other.u_shadows_proj_view, matrix_multiply(mproj,mlook));
The light is indeed supposed to be an overhead sun, and there are no day/night cycles. It is always day and the sun never changes orientation. I calculate the shadow map by using mproj and mlook and rendering the scene with this shader:
Code:
//Vertex
varying vec3 vertPosition;

attribute vec3 in_Position;

void main() {
    vertPosition = (gm_Matrices[MATRIX_WORLD] * vec4(in_Position,1.0)).xyz;
    gl_Position = gm_Matrices[MATRIX_WORLD_VIEW_PROJECTION] * vec4(in_Position, 1.0);
}

//Fragment
uniform vec3 lightPosition;
uniform float lightRange;

varying vec3 vertPosition;

vec4 FloatToColor(float f) {
    return vec4(floor(f * 255.0) / 255.0, fract(f * 255.0), fract(f * 255.0 * 255.0), 1.0);
}

void main() {
    // Store the distance from the pixel to the light (0-1) as a color.
    gl_FragColor = FloatToColor(distance(vertPosition, lightPosition) / lightRange);
}
 
What I can do is show you a very simple working example

https://www.dropbox.com/s/6biuebpgpdqhqeo/simple_shadow_mapping.gmz?dl=0

You know what though, I was never able to be sure that I'm encoding depth into the color channels correctly. Every resource I searched for seemed to say something different about that. If you've found reliable information about that, please do share. It also occurred to me, though I haven't tried it yet, that there might be a way to set the projection on the shadow map in a much more intelligent way.
 

Anixias

Member
From the example I've been using, this is how they encode depth into color:
Code:
vec4 FloatToColor(float f) {
    return vec4(floor(f * 255.0) / 255.0, fract(f * 255.0), fract(f * 255.0 * 255.0), 1.0);
}
Code:
float ColorToFloat(vec4 c) {
    return (c.r + c.g / 255.0 + c.b / (255.0 * 255.0)) * lightRange;
}
Although, in the actual project, he used 256.0. I changed them all to 255.0, but I'm not sure which is correct.

I'll give that link a look.

Edit:
Also, in the example I've been using, the creator wrote a function in the shadows shader that converts a 3D coordinate to a 2D coordinate in a very complex way (but he was using point lights, which use a perspective projection, so I couldn't just use what he wrote). I don't see what's wrong with my shader code, the uv position seems like it should be calculated correctly.


Edit 2:
I discovered that I had my matrix_multiply in the wrong order. Swapping them around, it no longer outputs completely black and I can correctly sample from the shadowMap in the shader. Now my next issue is that at the moment, I'm rendering a tilemap layer made in the room editor, but since it has no normals, trying to access the normals causes the shader to give up and output solid black for those pixels (other objects with normals defined are unaffected). I probably won't end up using tiles in the actual game, as the landscape will be made of blocks, but it's a little inconvenient at the moment.
 
Last edited:
It is very annoying that the matrix_multiply function takes arguments in the opposite order from how you would write it. a*b, becomes b,a. This seems totally bizarre to me, but perhapse there's some not entirely strange reason this is the convention.
 

Anixias

Member
Yeah, I'm just glad I fixed that issue. And actually, I just noticed that the uvPosition is still incorrect. I'm just sampling the shadowMap to get the color for the scene (as a test), and only at some perspectives is it anything other than white, but it doesn't seem to be working correctly.

Code:
uvPosition = (proj_view * vec4(in_Position,1.0)).xy;
uvPosition = (uvPosition * 0.5)+0.5;
Edit:
Nevermind, I fixed it. I didn't realize I needed to include the world matrix in the UV calculation. It samples the shadowMap correctly now.

Edit 2:
Alright, shadows work now, there's just some odd effects like Peter Panning and weird sampling distribution going on.



Edit 3:
I fixed the Peter Panning by lowering the bias in the shader. Now I'm currently trying to tweak the light's projection and shadowMap size to make the shadows look better. Any ideas? Currently, the shadowMap size is the same as the width and height of the orthographic projection. Should one be increased and the other constant to improve shadows?

Is there a way to zoom in the orthographic projection, so that it can be rendered closer up without increasing or decreasing the resolution?

Edit 4:
Nevermind, I figured it out. There is still some slight amount of Peter Panning that I can't fix by lowering the bias, but I guess it's acceptable. I still have to figure out how to correctly position the light now, to cover the whole camera view.
 
Last edited:
What I can do is show you a very simple working example

https://www.dropbox.com/s/6biuebpgpdqhqeo/simple_shadow_mapping.gmz?dl=0

You know what though, I was never able to be sure that I'm encoding depth into the color channels correctly. Every resource I searched for seemed to say something different about that. If you've found reliable information about that, please do share. It also occurred to me, though I haven't tried it yet, that there might be a way to set the projection on the shadow map in a much more intelligent way.

this example is awesome and simple thank you for share it
 
I think I've found better functions than what I was using in my example project, for encoding and decoding depth into 3 color channels.
Code:
vec3 depth_to_col ( float depth ) {
    float R = floor(depth*255.9999847412109375);
    float G = floor(depth*65535.99609375 - R*256.0);
    float B = floor(depth*16777215.0 - R*65536.0 - G*256.0);
    return vec3(R,G,B)/255.0;
}
and
Code:
float color_to_depth( vec3 rgb) {
     return dot(rgb, vec3( 0.996093809371817, 0.00389099144285866, 0.0000151991853236666 ));
}
Using those functions, I doesn't appear I even need to use a bias when comparing against the depth value in the shadow map. It might be a good idea to use one anyway for reasons I haven't encountered yet.
 
Top