Graphics Toon and Outline Shaders

angelwire

Member
First off, here’s a quick shout-out to the people who created the resources/tutorials that helped me out
xygthop3 - https://marketplace.yoyogames.com/assets/4894/hlsl11-basic
DragoniteSpam - https://www.youtube.com/user/2andaHalfStooges
@Xor
@Binsk - https://forum.yoyogames.com/index.php?threads/multiple-render-targets-shaders.4008/
@Juju - Thanks for the tip on using texture_get_texel_width() and texture_get_texel_height()
Screenshot:
screenshot0.png

GM Version: 2.0
Target Platform: Windows and maybe Mac
Download: Download Demo Project
Summary:
This tutorial will show you how to implement an HLSL11 toon shader and outline shader. This tutorial will give you an overview of how the shaders work and how to set them up within GameMaker. I’m going to assume you know how to set up a 3d environment, so I won’t go over how to load models or set up a camera or anything like that (the download link includes a very basic 3d environment). Here's a guide to 3d if you need one

// If you see any errors or potential for improvements let me know. I'll update the tutorial and add you to the credits

Disclaimer:
The tutorial starts by giving an overview of the methods used to achieve the toon effect and outlining. At the end I go over everything step-by step. I wish this tutorial was more beginner friendly, but 3D in GameMaker is hard. If you just want to copy over a couple lines of code then you're going to be very disappointed. The code provided is not "plug-and-play" and will require modifications to work in your project.​

Outline:
  • How the shaders work
    • Standard texture
    • Ramp Texture
    • Drawing the outline
  • The Shader Source Code
  • Step-By-Step
  • GameMaker Source Code
  • More Info
  • Help!
  • Change Log
  • Closing Thoughts

How the shaders work:
This toon effect is broken down into two separate shader: the toon shader and the outline shader. Every single object is drawn straight to the screen using the toon shader, and then the outline shader is applied to the entire screen using an "information surface" generated by the toon shader.

The toon shader uses two textures: the standard UV mapped texture that you’d usually apply to a 3d model, and a “ramp texture”.

- The standard texture is the texture you’d normally apply to a 3d model. For the best results make the texture colors flat. A lot of textures are made to look as realistic and as detailed as possible, but you don’t want that.
- The ramp texture is used to determine the levels of lighting. Here’s the ramp texture I use for the demo: 228d84b4-2068-42bd-82ee-2fe58e9af604.png (yes, it’s only 32x32, ramp textures can -and should be- tiny). The idea is that the amount of light will determine where on the ramp texture to grab our “lighting pixel” that determines how bright each pixel on the 3d model is. This gives us a lot of control over how the lighting is handled without using any complex math. The ramp texture in the demo uses three colors, so there are only three different shades of lighting in the demo. The more colors you include in the ramp texture the more shades of lighting there will be on the model (notice that the dark colors always go on the left and the bright colors on the right). The color of the ramp sprite will be the color of the light, so giving the ramp sprite a red tint will make the model appear more red. Another feature for the ramp sprite is that the bigger a block of color is in the ramp sprite, the more it will cover the model. So if 75% of the ramp sprite is covered in black, then anything that’s lit less than 75% will be black. (Note: the ramp sprite needs to be set up properly in GameMaker to work right, see “GameMaker Implementation” below for details).

- Not only does the toon shader draw the model with the toon shading, but it also draws the information that the outline shader uses to draw the outline. The information is packed into colors and then drawn to an “information surface”. The information surface is then drawn to the screen with the outline shader. The outline shader will use all the information packed into the colors to draw the outline. There are two pieces of information packed into the information surface: The direction each pixel is facing (the “normal”), and how far away each pixel is from the camera (the “depth”). Each color is made up of 4 components ranging from 0 to 1: Red, Green, Blue, Alpha. Without getting too technical, the information is stored like this: Red and Green hold the “normal” and Blue and Alpha hold the “depth” (see the source code for more details). The toon shader packs that information into the information surface and then the outline shader unpacks the information and uses some math to draw an outline (see the source code for more details).

The Shader Source Code:

toon_shader vertex
Code:
//Base HLSL vertex shader with a few modifications
struct VertexShaderInput
{
    float4 vPosition : POSITION;
    float2 vTexcoord : TEXCOORD0;
    float4 vNormal : NORMAL;
    float4 vColor    : COLOR0;
};

//The output is important to note
//This is the information that will be passed into the fragment shader
struct VertexShaderOutput
{
    float4 vPosition : SV_POSITION;
    float4 vColor    : COLOR;
    float2 vTexcoord : TEXCOORD;
    //Pass in the normals
    float4 vNormal : NORMAL0;
    //Pass in the camera direction (as a normal)
    float4 vCamera : NORMAL1;
    //Pass in the depth of the vertex
    float vDepth : DEPTH;
};

VertexShaderOutput main(VertexShaderInput INPUT)
{
    VertexShaderOutput OUTPUT;

    //Set the camera to the proper direction
    OUTPUT.vCamera = -float4(gm_Matrices[MATRIX_WORLD_VIEW][2][0],
                        gm_Matrices[MATRIX_WORLD_VIEW][2][1],
                        gm_Matrices[MATRIX_WORLD_VIEW][2][2],
                        gm_Matrices[MATRIX_WORLD_VIEW][2][3]);

    //Build the world view projection
    float4 matrixWVP = mul(gm_Matrices[MATRIX_WORLD_VIEW_PROJECTION], INPUT.vPosition);
  
    //Get the depth as a value between 0 and 1
    OUTPUT.vDepth = matrixWVP.z / matrixWVP.w;
    //Pass in the rest of the values
    OUTPUT.vPosition = matrixWVP;
    OUTPUT.vColor    = INPUT.vColor;
    OUTPUT.vTexcoord = INPUT.vTexcoord;
    OUTPUT.vNormal = INPUT.vNormal;

    return OUTPUT;
}
toon_shader fragment
Code:
//This needs to match up to the VertexShaderOutput in the vertex shader
struct PixelShaderInput {
    float4 vPosition : SV_POSITION;
    float4 vColor    : COLOR;
    float2 vTexcoord : TEXCOORD;
    float4 vNormal : NORMAL0;
    float4 vCamera : NORMAL1;
    float vDepth : DEPTH;
};

//This is used to write to two separate render targets (surfaces)
struct PixelShaderOutput
{
    //The target to draw the texture with lighting
    float4 outDiffuse  : SV_TARGET0;
    //The target to draw the normal and depth information to
    float4 outNormal   : SV_TARGET1;
};

//This is a texture that will be used for the lighting ramp
Texture2D rampTexture : register(t1);
SamplerState ssRampTexture : register(s1);

//This is the direction for the light
uniform float3 lightDirection;

//Here's where the magic happens
PixelShaderOutput main(PixelShaderInput INPUT)
{
    //Create the output struct to return multiple colors
    PixelShaderOutput OUTPUT;
  
    //--Color
        //Dot values are changed so that they range from 0,1 instead of -1,1
        //Get the dot value between the normal and the light direction
        float lightDot = (dot(INPUT.vNormal.rgb,lightDirection) *.5) + .5;
        //Get the dot value between the normal and the camera
        float cameraDot = (dot(INPUT.vNormal.rgb,INPUT.vCamera.rgb) *.5) + .5;
        //Get the dot value relative to the camera's up (using a swizzle)
        float upDot = (dot(INPUT.vNormal.rgb,INPUT.vCamera.rbg) *.5) + .5;
        //How far along the ramp texture to go
        //Change the .35 to determin how much impact the light has on the shading vs the camera angle
        //(0=only light has an impact and 1=only the camera has an impact)
        float rampAmount = lerp(lightDot,cameraDot,.35);
  
        //Get the actual texture
        float4 diffuseTexture = gm_BaseTextureObject.Sample(gm_BaseTexture, INPUT.vTexcoord); 
        //Get the lighting color based on the ramp amount
        float4 rampColor = rampTexture.Sample(ssRampTexture,float2(rampAmount,rampAmount));
  
        //Cheat lighting
        //Shade is based on the cameraDot. 1.4 is sort of the threshold. Higer=more shade
        float shade = saturate(cameraDot * 1.4);
        //Extra cheat lighting. Anything below .9 gets cut out.
        //Lower .9 for more extra light (the two values should always add up to 1)
        //For example: clamp((lightDot-.7),0,.3) ... clamp((lightDot-.4),0,.6)
        float extraLight = clamp((lightDot-.9),0,.1);
        //(I think adding a little bit of normal shading on top of the toon shading looks good)
        //If you don't want it then remove the "shade" and "extraLight" lines
        //And change the comments on on the "OUTPUT.outDiffuse.rgb = ..." line
  
        //The OUTPUT.outDiffuse will write to the 0th surface supplied in the draw event
        //Return the texture color multiplied by the ramp color and shade value and then adding in the extra light
        OUTPUT.outDiffuse.rgb = (diffuseTexture.rgb * rampColor.rgb * shade) + extraLight;
        //To remove cheat lighting ^Comment this out^
        //And uncomment  v this line v
        //OUTPUT.outDiffuse.rgb = (diffuseTexture.rgb * rampColor.rgb);
      
        //Set the alpha to the normal texture alpha
        OUTPUT.outDiffuse.a = diffuseTexture.a;
    //--End Color
  
    //--Normal and depth
        //The OUTPUT.outNormal will write to the 1th surface supplied in the draw event
      
        //Higher depth resolution means closer pixels will stand apart better
        //But it also caps how far away something can be before the depths become indistinguishable
        float depthResolution = 250.0;
        //Red is set based on the dot value of the camera's pointing direction and the surface normal
        OUTPUT.outNormal.r = cameraDot;
        //Green is set based on the dot value of the camera's up direction and the surface normal
        OUTPUT.outNormal.g = upDot;
        //Blue is set based on the depth of the vertex
        OUTPUT.outNormal.b = INPUT.vDepth;
        //Alpha is set based on the depth of the vertex but it loops based on the depthResolution
        OUTPUT.outNormal.a = (INPUT.vDepth * depthResolution) % 1.00;
    //--End normal and depth

    //Return the OUTPUT struct that contains both colors
    return OUTPUT;
}
outline_shader vertex
Code:
//Standard HLSL vertex shader
struct VertexShaderInput
{
    float4 vPosition : POSITION;
    float2 vTexcoord : TEXCOORD0;
    float4 vColor    : COLOR;
};

struct VertexShaderOutput
{
    float4 vPosition : SV_POSITION;
    float4 vColor    : COLOR;
    float2 vTexcoord : TEXCOORD;
};

VertexShaderOutput main(VertexShaderInput INPUT)
{
    VertexShaderOutput OUTPUT;

    float4 matrixWVP = mul(gm_Matrices[MATRIX_WORLD_VIEW_PROJECTION], INPUT.vPosition);

    OUTPUT.vPosition = matrixWVP;
    OUTPUT.vColor    = INPUT.vColor;
    OUTPUT.vTexcoord = INPUT.vTexcoord;

    return OUTPUT;
}
outline_shader fragment
Code:
//This is to get the input from the vertex shader
//It must match the "VertexShaderOutput" from the vertex shader
struct PixelShaderInput{
    float4 vPosition : SV_POSITION;
    float4 vColor    : COLOR;
    float2 vTexcoord : TEXCOORD;
};

//Returns the depth from the given color
//Only uses the blue and alpha values
float getDepth(float4 color)
{
    float bv = floor(color.b * 256);
    float av = floor(color.a * 256);
    float returnValue = (bv * 256) + av;
    return returnValue;
}

//Returns the "difference" between the two given colors
//This is where a bulk of the math is done
float difference(float4 thisColor, float4 otherColor)
{
    //Get the difference in the red and green values (where the normals are packed)
    float redDifference = thisColor.r - otherColor.r;
    float greenDifference = thisColor.g - otherColor.g;
    //Get the total difference between the two packed normals
    //The two represents how important the normals are to the outline (higher=more important)
    float dotDifference = abs(redDifference * greenDifference) * 2;
    //Get the depth difference
    float dif = (getDepth(thisColor)/65536) - (getDepth(otherColor)/65536);
    //Takes the difference in depth and uses saturate to remove all values below .01
    //Multiplying it by 700 ensures the value is easily above 1 before getting saturated
    dif = saturate(700*(dif-.01));
    //Drawing the outline on a face that is perpendicular to the camera is a bad idea
    //It leaves really ugly artifacts
    //So any face that is perpendicular to the camera will reduce the difference
    dif *= thisColor.r - .4;
    //Increment the total difference by the amount of dot difference
    dif += dotDifference;
    return dif;
}

//Values for one pixel (set within the draw event)
//Set it to 1 divided by the width or height
//If a surface is 1024 by 1024 then set both values to (1/1024)
uniform float oneHorizontal;
uniform float oneVertical;

float4 main(PixelShaderInput INPUT) : SV_TARGET
{
    //The resolution for sampling neighbors. Higher is better but more expensive
    int pixelCount = 8;
    //How far away to sample. Higher is a thicker border and more expensive
    float maxDistance = 2;
    //^ Play with these values if you want ^
    //(the total number of pixel operations will be (pixelCount * maxDistance)
  
    //Get the pixel at the current position
    float4 textureDataOriginal = gm_BaseTextureObject.Sample(gm_BaseTexture,INPUT.vTexcoord);
    //The totalDifference measured the total difference between the pixel and its neighbors
    float totalDifference = 0;
    //The direction currently sampling
    float lookDirection = 0;
    //The amount to turn after each sample
    float turnAmount = 6.283185/pixelCount;
  
    //Loop through the number of max distances
    for (int jj=1; jj <= maxDistance; jj+=1)
    {
        //Pixels further away will have less influence on the "totalDifference"
        float distanceMultiply = (jj * maxDistance) * .6;
        //How far away the pixels will be
        float wMultiply = (oneHorizontal * distanceMultiply);
        float hMultiply = (oneHorizontal * distanceMultiply);
      
        //Perform division outside of loop so we don't have to do it in the loop
        distanceMultiply = 1/distanceMultiply;
      
        for(int ii = 0; ii < pixelCount; ii += 1)
        {
            //Sample the texture at the positions
            //Simple math to look in the lookDirection
            float4 textureDataNew =
            gm_BaseTextureObject.Sample(gm_BaseTexture,
                float2(INPUT.vTexcoord.x + (cos(lookDirection) * wMultiply),
                        INPUT.vTexcoord.y + (sin(lookDirection) * hMultiply)));
            //Increment totalDifference based on the difference multiplied by the distanceMultiplier
            totalDifference += difference(textureDataOriginal, textureDataNew) * distanceMultiply;
            //Turn the look direction
            lookDirection += turnAmount;
        }
    }
  
    //Use the total difference as the alpha
    //Any color outline can be used if you want to try some other effects
    return float4(0,0,0,totalDifference);
}

Step-By-Step:
  1. Set up a 3d environment with a camera, a 3D model, and a texture
  2. Create your own ramp sprite (or copy the one here in the post)
  3. Make sure "Separate Texture Page" is checked for the ramp sprite
  4. Create two shaders: "toon_shader", "outline_shader"
  5. Set both shader types to HLSL 11 (right click on the shader -> Shader Type -> HLSL 11)
  6. In the CREATE event of your 3D model object ramp_sampler = shader_get_sampler_index(toon_shader, "ssRampTexture");
  7. In the DRAWevent of your model you'll need to do the following:
    1. Check if the proper surfaces exist
    2. Disable the blending (so that the Alpha values don't blend together)
    3. Set the color mask surface (or application_surface) as the 0th target
    4. Set the outline mask surface as the 1th target
    5. Reset the camera's projection matrix, view matrix, and apply the changes
    6. Set the ramp_sampler filtering to false. This will make sure the ramp sprite never gets blurred and the shading is always sharp
    7. Set the texture stage to the ramp sprite texture
    8. Set the light direction within the shader
    9. Sumbit the model vertices with the sprite texture (draw the model)
    10. Reset the surface target (only once, even though we used two surface_set_target_ext)
    11. Reset the shader
    12. Re-enable blending
  8. The code used in the demo for the draw event is copied below. This isn't something you can copy and paste into your project, you will have to make changes for it to work.
    1. In the demo, the MAIN_CAMERA object holds the surfaces, the matrices, the camera, and the light direction. Your project probably won't work this way.
  9. In the demo, the camera object handles the outline drawing
  10. In the create event for the camera, you'll want to get a reference to the outline_shader uniforms "oneHorizontal", "oneVertical" and "lightDirection"
  11. In the DRAW GUIevent you want to do the following
    1. Check if the color mask surface exists (if it doesn't exist create it and clear it using draw_clear_alpha(c_white,0))
    2. Draw the color mask surface (without a shader)
    3. Clear the color mask surface using draw_clear_alpha(c_white,0)
    4. Check if the outline mask surface exists (if it doesn't exist create it and clear it using draw_clear_alpha(c_white,0))
    5. Set the shader to the outline_shader
    6. Set the "oneHorizontal" uniform to the size of a "texel" using texture_get_texel_width();
    7. Set the "oneVertical" uniform to the size of a "texel" using texture_get_texel_height();
    8. Draw the outline mask surface
    9. Clear the outline mask surface using draw_clear_alpha(c_white,0)
  12. And you're finished
GameMaker Source Code:
Model object DRAW event
GML:
//Make sure both surfaces exist
if(surface_exists(MAIN_CAMERA.outline_mask))
{
    if (surface_exists(MAIN_CAMERA.color_mask))
    {
        //Disable blending so the alpha values don't change
        gpu_set_blendenable(false);
        //Set the toon shader
        shader_set(toon_shader)
            //Set the targets
            surface_set_target_ext(0, MAIN_CAMERA.color_mask);
            surface_set_target_ext(1, MAIN_CAMERA.outline_mask);
                //Reset the camera matrices
                camera_set_proj_mat(MAIN_CAMERA.camera, MAIN_CAMERA.projection_matrix);
                camera_set_view_mat(MAIN_CAMERA.camera, MAIN_CAMERA.view_matrix);
                //Apply the changes to the camera
                camera_apply(MAIN_CAMERA.camera);  
                //Disable the filtering for the ramp sampler
                gpu_set_tex_filter_ext(ramp_sampler,false);
                //Set the texture stage to the ramp
                texture_set_stage(ramp_sampler,sprite_get_texture(ramp_sprite,0));
                //Set the light direction to up
                shader_set_uniform_f(MAIN_CAMERA.lightDirectionUniform,0,0,-1);
                //Draw the model
                vertex_submit(my_model,pr_trianglelist,sprite_get_texture(model_sprite,0));
            //Reset the target
            surface_reset_target();
        //Reset the shader
        shader_reset();
        //Re-enable blending
        gpu_set_blendenable(true);
    }
    //Debug stuff to know when there's not a surface
    //The camera handles creating the surfaces
    else { show_debug_message("Model says: No color surface"); }
}
else { show_debug_message("Model says: No outline surface"); }
Camera object DRAW GUI event
GML:
//Draw the color mask
if (surface_exists(MAIN_CAMERA.color_mask))
{
    draw_surface(MAIN_CAMERA.color_mask,0,0);
    //Clear it after drawing it
    surface_set_target(MAIN_CAMERA.color_mask);
        draw_clear_alpha(c_white,0);
    surface_reset_target();  
}
else
{
    //Create a new color mask
    show_debug_message("Main Camera says: No color surface");
    MAIN_CAMERA.color_mask = surface_create(render_width,render_height);
    //Clear it
    surface_set_target(MAIN_CAMERA.color_mask);
        draw_clear_alpha(c_white,0);
    surface_reset_target();
}

//Draw the outline mask
if (surface_exists(MAIN_CAMERA.outline_mask))
{
    if (draw_outline)
    {
        //Set the shader
        shader_set(outline_shader);
            //Set the uniforms to know what the size of one pixel is
            shader_set_uniform_f(oneHorizontalUniform,texture_get_texel_width(surface_get_texture(MAIN_CAMERA.outline_mask)));
            shader_set_uniform_f(oneVerticalUniform,texture_get_texel_height(surface_get_texture(MAIN_CAMERA.outline_mask)));
            //Draw the outline mask
            draw_surface(MAIN_CAMERA.outline_mask,0,0);
        shader_reset();
        //Clear the outline mask
        surface_set_target(MAIN_CAMERA.outline_mask);
            draw_clear_alpha(c_white,0);
        surface_reset_target();
    }
  
    if (draw_debug)
    {
        draw_surface(MAIN_CAMERA.outline_mask,0,0);
    }
}
else
{
    //No outline mask! OH NO!
    show_debug_message("Main Camera says: No outline surface");
    //Creat a new outline mask and clear it
    MAIN_CAMERA.outline_mask = surface_create(render_width,render_height);
    surface_set_target(MAIN_CAMERA.outline_mask);
        draw_clear_alpha(c_white,0);
    surface_reset_target();
}

Extra info
Multiple Render Targets (MRTs):
I'm not sure how well known MRTs are among the GameMaker community. Just in case you didn't know, each HLSL11 shader and GLSL shader in GameMaker has the potential to write to 4 different surfaces (from what I understand, GLSL ES cannot use MRTs). This is much better than the alterantive of drawing a model 4 times with different shaders and surface targets each time (I would imagine it's a pretty big speed boost, but even if it's not, it's still much easier to read and write the code for MRT). GameMaker has pretty good documentation on the surface_set_target_ext(index number, surface id) function, but I had trouble implementing it on the shader side. Here's a great guide to getting started: MRT Shader Guide

I've included GLSL shaders in the demo: I'm not sure if they work because GLSL ES doesn't support MRTs and I'm not able to test GLSL

Help!
Nothing is being drawn:
Be sure to reset the camera matrices and use the apply function after setting the surface targets. There also could be issues with the outline and color surface sizes not matching.
The lighting looks weird: The ramp sprite should be in black and white with the dark side on the left and the bright side on the right. Be sure "Separate Texture Page" is checked in the sprite. Be sure you set the ramp texture filtering to false.
"Unbalanced surface stack": Only use one surface_reset_target when you're drawing the model
The outline is drawing weird: Be sure to use gpu_set_blendenable(false); before drawing the model

Change Log
Started using texture_get_texel_width()/texture_get_texel_width() to set the one pixel values

Closing thoughts
Thank you very much for making it to the end of this tutorial! I’ll continue to work on this and provide updates depending on the demand. There are still some things I would like to do such as: use GameMaker’s built-in lights and make the outline sharper, cleaner, faster etc...

No credit needed if you use this tutorial for one of your games, but let me know because I’d love to see it!
 
Last edited:
Top