Projection matrix with oblique near clip plane


Heavy metal viking dentist
GMC Elder
Projection matrix with oblique near clip plane
I've been working on a portal system lately, and figured I'd share some of my findings.

When rendering planar world-space reflections, you usually need to create a reflected virtual camera at the other side of the mirror, sea level or portal, and render the world from this point of view to a surface. This surface is then drawn only within the confines of the mirror's surface, resulting in a realistic real-time reflection.

This method is excellent if there is nothing behind the mirror. By enabling counter-clockwise culling, the mirror itself and the wall behind it won't be drawn to the reflection surface, and you're left with a good-looking reflection:

However, what if there was something behind the mirror? Say there's a hallway at the other side of the wall. How would you prevent the walls and contents of the hallway from drawing to the reflection surface?

One way is to use a custom shader for the world in the virtual camera, with the following pseudo-code in the fragment shader:
if (pixel is behind mirror){discard;}
Nice and easy, but this would require an extra version of each shader, or an extra uniform in all shaders. Assuming this is undesirable, what other methods are there?

The graphics pipeline automatically transforms vertices to what's called normalized device coordinates, or NDC space. Once in this space, triangles are cut up so that only the part that is inside the NDC space is kept. Everything outside is culled. This is where you get stuff like the near and far planes, as well as the top, bottom, right and left planes. What if we could utilize this automatic process to cull everything behind the mirror?

In a standard projection matrix, the near and far planes are perpendicular to the looking direction, like so:

Now this isn't particularly useful...
However, there is a way to alter the projection matrix so that the clip planes look like this:

Now this is more like it! With this projection matrix, everything behind the mirror is culled, and we don't even need separate shaders for it! This is incredibly efficient, since it uses a culling process that would already happen anyway. The only problem with it is that the depth buffer gets messed up, and so this kind of projection cannot be used in conjuction with a regular projection. However, this is usually not an issue, since reflections and portals usually are rendered to separate surfaces anyway.

The code for this kind of projection is based on the process outlined on this page:
I haven't found it adapted to GML yet, so I did it myself. So here you go, the script for creating a projection matrix with an oblique near plane:
/// @description projection_make_oblique_near_clip_plane(projMat, clipX, clipY, clipZ, clipZ)
/// @param ProjMat
/// @param clipX
/// @param clipY
/// @param clipZ
/// @param clipW
Modifies the given projection matrix so that it has an oblique near clip plane.
Must supply the camera-space 4D vector of the given plane, as well as the projection matrix.
The resulting projection matrix will mess up your depth buffer and any shaders that use it.
This is useful for virtual cameras like used in portals or reflections, to prevent anything behind the portal/mirror
from being rendered.

Adapted to GML by Sindre Hauge Larsen
var P = argument0;

var clipX, clipY, clipZ, clipW;
clipX = argument1;
clipY = argument2;
clipZ = argument3;
clipW = argument4;

//Calculate the clip-space corner point opposite the clipping plane
//as (sgn(clipPlane.x), sgn(clipPlane.y), 1, 1) and
//transform it into camera space by multiplying it
//by the inverse of the projection matrix
var Qx, Qy, Qz, Qw;
Qx = (sign(clipX) + P[8]) / P[0];
Qy = (sign(clipY) + P[9]) / P[5];
Qz = -1;
Qw = (1 + P[10]) / P[14];

//Replace the third row of the projection matrix
var c = 2 / (clipX * Qx + clipY * Qy + clipZ * Qz + clipW * Qw);
P[2]  = clipX * c;
P[6]  = clipY * c;
P[10] = clipZ * c;
P[14] = clipW * c;

return P;
You need to supply the projection matrix to modify, as well as the camera-space clip plane.
Here's how to convert from world-space to camera-space clip plane:
//Get the current view and projection matrices
var V = matrix_get(matrix_view);
var P = matrix_get(matrix_projection);

//Find a camera-space position on the plane (it does not matter where on the clip plane, as long as it is on it)
//Wx, Wy, Wz is the world-space plane coordinate
var Px, Py, Pz;
Px = V[0] * Wx + V[4] * Wy + V[8] * Wz + V[12];
Py = V[1] * Wx + V[5] * Wy + V[9] * Wz + V[13];
Pz = V[2] * Wx + V[6] * Wy + V[10]* Wz + V[14];

//Find the camera-space 4D reflection plane vector
//Nx, Ny, Nz is the world-space normal of the plane
var Cx, Cy, Cz, Cw;
Cx = V[0] * Nx + V[4] * Ny + V[8] * Nz;
Cy = V[1] * Nx + V[5] * Ny + V[9] * Nz;
Cz = V[2] * Nx + V[6] * Ny + V[10]* Nz;
Cw = - Cx * Px - Cy * Py - Cz * Pz;

//Modify the projection matrix so that it uses an oblique near clip plane
var _P = projection_make_oblique_near_clip_plane(P, Cx, Cy, Cz, Cw);
matrix_set(matrix_projection, _P);
Et voila!

Here's a gif from my portal engine btw:
Last edited:


Heavy metal viking dentist
GMC Elder
Put together another demo, this time is a basic portal effect:
It's the same effect as seen in the gif in the first post. Here it is again:

This demo does not let you walk through the portals. It also has all optimizations removed, like making portals that aren't visible not draw. Things like these are absolutely necessary for an actual project, but it makes it easier to follow the code when it's absent.
In the gif I used a simple trick to allow for "infinite" portals, by simply drawing the surface of the last portal over and over. In the downloadable demo, the world will actually be drawn three times inside the portal.

Joe Ellis

Wowww! I've been wanting to do this for ages for water and mirrors. I didn't even know it was possible to do this. Is it ok if I use this technique in Warp3D? I'm wondering cus I'm selling it, so if you want some money for it just let me know, and I will credit you aswell

Joe Ellis

Btw if you have anything that I use that you want to see the code for, let me know, I have no problem at all sharing my knowledge base.
While I am going to sell my engine it's for the construction as a whole project, I want to share the knowledge of techniques with everyone.
I've worked out alot of complicated things from scratch without looking online at all, which might not be a good thing but can certainly give people a unique perspective and maybe inspire new ways of looking at things.
This goes for anyone else reading this
Last edited: