Normal Fog as d3d_set_fog but in a Shader (GM 1.4)

[Edgamer63]

Hello, i want to know if somebody can help me with this... i don't know how to restore my fog settings... because i drawed 3D Things with a Shader... but, i dont know where my d3d_set_fog gone ... so, i need to know how to draw a fog in a shader...

 

[flyingsaucerinvasion]

You're going to need 5 uniform or constant components. 3 for the colour of the fog. And 2 more to define its start and end distances.

The first thing you need to do in the vertex shader is get the distance from the camera to the vertex, and pass that distance as a varying to the fragment shader. I think that can most easily be accomplished like this:

varying float dist;
//....
dist = length(vec3( gl_Position.xy / vec2(gm_Matrices[MATRIX_PROJECTION][0][0],gm_Matrices[MATRIX_PROJECTION][1][1]), gl_Position.w ));

Some applications only use depth as a basis for calculating fog, but that causes the fog level to change weirdly as you rotate the camera. So why not use the proper distance instead? That division by a couple of matrix elements is to bring the vertex position into view space as opposed to whatever is the name of the space which results from the projection matrix multiplication. You could also just multiply the object space column vector with MATRIX_WORLD_VIEW, but that is a lot of extra math. Note: The above is for when using a perspective projection, it will give incorrect results with an orthographic projection.

Then in the fragment shader, you use the distance value to compute the fog level. Fog level should be (dist-near)/(far-near), and you'll want to clamp that between 0 and 1. It helps if you precalculated 1/(far-near) and pass that in as the second component of your "fog" uniform. So, x component of "fog" would be its near value. And y component would be 1/(far-near).

float fog_level = clamp((dist-fog.x)*fog.y,0.0,1.0);

You can then use fog_level to interpolate between the base colour of the fragment, and the fog colour:

vec4 base_colour = v_vColour * texture2D( gm_BaseTexture, v_vTexcoord );
gl_FragColor = vec4( mix( base_colour.rgb, fog_colour, fog_level ), base_colour.a );

I feel like there could be some possibility of error there when dealing with sem-transparent images. So if you are making use of any, double check that produces correct results.

 

[Edgamer63]

Do you got any idea of how to use: "gl_FogCoord" and "gm_FogStart"? :/

 

[flyingsaucerinvasion]

I've never heard of gl_FogCoord.

As far as I know the gm fog uniforms don't work in cursom shaders. This information might be out of date. You don't really need them though, since you can just make your own uniforms.

 

[Edgamer63]

Does somebody have an example for this? a gamemaker project or somewhat where this work?

 

[Joe Ellis]

Here's a copy of the method I use:

///basic vertex shader with fog

attribute vec3 in_Position;
attribute vec4 in_Color;
attribute vec2 in_TextureCoord;

uniform vec3 view_pos;

varying vec4 v_color;
varying vec2 v_texcoord;
varying float v_depth;

void main()
{
    gl_Position = gm_Matrices[MATRIX_WORLD_VIEW_PROJECTION] * vec4(in_Position, 1.0);
    v_color = in_Color;
    v_texcoord = in_TextureCoord;
    v_depth = distance(view_pos, vec3(gm_Matrices[MATRIX_WORLD] * vec4(in_Position, 1.0)));
}

///basic pixel shader with fog

varying vec4 v_color;
varying vec2 v_texcoord;
varying float v_depth;

uniform vec3 fog_color;
uniform float fog_start;
uniform float fog_length;

void main()
{
gl_FragColor = v_color * texture2D(gm_BaseTexture, v_texcoord);
gl_FragColor.rgb = mix(gl_FragColor.rgb, fog_color, min(1.0, max(0.0, v_depth - fog_start) / fog_length));
}

or a more basic way to get the v_depth: (but has that problem flyingsaucerinvasion was talking about)

v_depth = gl_Position.z;

I don't think there are errors particularly when transparency is used, at least not caused by the fog cus it only affects the rgb,
so errors would be from not handling the transparency properly with the draw order etc.

Also I use fog_start and fog_length instead of fog_start and fog_end, so if you want to use fog_end instead you'd need to calculate it like this:

fog_amount = min(1.0, max(0.0, fog_end - v_depth) / (fog_end - fog_start));

Using fog length is a bit more efficient though cus it's basically pre calculating the length (fog_end - fog_start) so it doesn't need to be done in the shader, but using fog_length is a bit more awkward to configure the fog values in some cases, I guess it depends on your preference. You could use fog_start and fog_end as variables which are then used to calculate the fog_length that ends up being used in the shader. Also if you really want the best performance you can calculate the reciprocal of fog_length (1 / fog_length or 1 / (fog_end - fog_start)) and use that as the uniform in the shader and multiply instead of divide:

fog_amount = min(1.0, max(0.0, v_depth - fog_start) * fog_length_rec);


I've heard a lot of games use exponential fog instead of linear, which is meant to look more realistic, but I haven't tried that yet so I can't really recommend which methods to use.
Using smoothstep could also produce a nice result:

fog_amount = smoothstep(0.0, 1.0, min(1.0, max(0.0, v_depth - fog_start) * fog_length_rec));

 

[Edgamer63]

uniform vec3 view_pos;

How does i can set the uniform view_pos? is with shader_set_uniform_* functions? What is happening here, because i don't know yet.
Also the fog_color is set with shader_set_uniform_*?

I'm asking because i'm not too much experienced with shaders.

 

[Joe Ellis]

Yeah it's with the shader_set_uniform_* functions
If you have coordinates for the view which are used to make the view_matrix, or if you use d3d_set_projection(view_x, view_y, view_z, etc.) it's easiest to just put the 3 values into shader_set_uniform_f.
For the fog color to be used in the shader it needs to be converted into 0 - 1 range instead of 0 - 255 like the make_color_function uses,
which you can do by using color_get_red, green and blue and divide them by 255.
So you can either create 3 variables: fog_r, fog_g, fog_b and put those into the shader with shader_set_uniform_f, or create a 3 value array and call it "fog_color_rgb" or something, and use shader_set_uniform_f_array.

//in create event
uid_view_pos = shader_get_uniform(shader, "view_pos")
uid_fog_start = shader_get_uniform(shader, "fog_start")
uid_fog_length = shader_get_uniform(shader, "fog_length")
uid_fog_color = shader_get_uniform(shader, "fog_color")

fog_start = 2000
fog_end = 10000
fog_color = make_color_hsv(fog_hue, fog_sat, fog_val)

//convert color to 3 component array\vec3:
fog_color_rgb[0] = color_get_red(fog_color) / 255
fog_color_rgb[1] = color_get_green(fog_color) / 255
fog_color_rgb[2] = color_get_blue(fog_color) / 255

//or get 3 separate variables:
fog_color_r = color_get_red(fog_color) / 255
fog_color_g = color_get_green(fog_color) / 255
fog_color_b = color_get_blue(fog_color) / 255

//in draw event

d3d_set_projection(view_x, view_y, view_z, view_x2, view_y2, view_z2, 0, 0, -1)

shader_set(shader)

shader_set_uniform_f(uid_view_pos, view_x, view_y, view_z)
shader_set_uniform_f(uid_fog_start, fog_start)
shader_set_uniform_f(uid_fog_length, fog_length)
shader_set_uniform_f_array(uid_fog_color, fog_color) //set vec3 array
shader_set_uniform_f(uid_fog_color, fog_color_r, fog_color_g, fog_color_b) //or set 3 separate values

//draw stuff

shader_reset()

 

[Edgamer63]

Thank you, this worked a lot, now my 3D game look better with a fog.

Thank you all <3 !