# Convert a screen coordinate into a 3D value

#### LittleAngel

##### Member
Hi everybody !

I'm looking for a script converts an x,y screen coordinate into 3D coordinates with d3d_set_projection (GM 1.4).
For example the middle of the screen is x=512 y=384 (for a 1024*768 screen). I would like this middle of the screen (the pointer in a fps game) return the x,y,z values of the 3d world.
(my 3d camera has the values for horizontal and vertical direction (z-tilt))

I've searched hard, and I can't find anything convincing (or else only for GM2).

GML:
``````// Convert_2d(x,y,xfrom,yfrom,zfrom,view)
// The script returns the 3d x and y coordinates at z = 0

screenx = 2*argument0/view_wview[argument5]-1;
screeny = 1-2*argument1/view_hview[argument5];
mX = dX + uX*screeny + vX*screenx;
mY = dY + uY*screeny + vY*screenx;
mZ = dZ + uZ*screeny + vZ*screenx;

if mZ != 0
begin
x_3d = argument2 - argument4 * mX / mZ;
y_3d = argument3 - argument4 * mY / mZ;
end;
else
begin
x_3d = argument2 - argument4 * mX;
y_3d = argument3 - argument4 * mY;
end;``````
What are the variables dX, dY, dZ? They are not defined. uX, uY, uZ?

Can someone help me ? Thanks !

#### TheSnidr

##### Heavy metal viking dentist
GMC Elder
Check out my post here:
I don't remember if gms 1 had cameras, but just change the arguments so that you pass it a view and projection matrix instead if that is not the case

#### FrostyCat

##### Member
A single 2D on-screen coordinate corresponds to a ray in the 3D world, not a distinct point. You can't find anything because what you asked for does not exist. The only way for you to find a distinct point is the intersection of that ray against some plane, such as the one defined by z=0.

• TheSnidr

#### Binsk

##### Member
I have a very old extension you can find here that does this for you. It uses a method before matrices were a proper thing in GM.

Now, some knowledge about doing this.

Realize that it is impossible to directly convert a 2D point (your mouse) into a 3D point (location in 3D space). When you convert your 2D point you actually result in an infinite line of 3D points that your 2D point rests on top of.

This makes things a little extra tricky. Once you have this 3D line you have two options:
1. Define some 3D plane and see where the line intersects it and call THAT the 'mouse point'. So, for example, like a game where you can place tiles in 3D space or something similar.

2. Check your line for collisions against meshes in 3D space to see if you mouse is hovering over the 3D object. So, clicking on 3D objects.

So how do you get the line, anyway? You can convert from one coordinate system to another by multiplying a point by a matrix. For example, if you have a model each vertex is multiplied by a "world matrix" to convert from your model's coordinates to your game world coordinates. It then is multiplied by your "view matrix" to get it relative to the game camera. It then gets multiplied by the "projection matrix" to get it relative to your screen.

All you need to do is get your mouse point and go BACKWARDS through these matrices. You want things in your 3D world coordinates so you need the 'inverse matrix' of both the view and projection matrix. What you do is multiply your view and projection matrix together (you can do this with GMs matrix_* functions) then grab the inverse of the result (you need to write a custom script for this, do a search).

Now, when plugging in your mouse you need 3 points, the mouse x and y (of course) in screen coordinates (so between [-1, 1]), but also a mouse z. The mouse z determines which of these infinite 3D points you want to select. The easiest thing to do is to plug in your mouse coordinates twice, once at 0 (which is the point at the camera's near plane) and once at 1 (the camera's far plane). You can calculate your line from these two points.

Sound complicated? It is. 3D is no walk in the park. It actually isn't very hard if you know how all this works already. If you don't it can be a little baffling.

EDIT: Two scripts for you.

Get the inverse of a matrix:
Code:
``````///    @desc    Calculates the inverse of the specified matrix or undefined if it cannot be calculated.
///    @param {matrix} matrix    matrix to find inverse of
///    @returns {matrix}
///    @private

// Determine if it is a proper matrix:
if (!is_array(argument0))
show_error("UBG 3D2D (argument0): Expected type [matrix]!", true);

if (array_length_1d(argument0) != 16)
show_error("UBG 3D2D (argument0): Expected type [matrix]!", true);

var _inverse = 0,
_determinant = matrix_determinant(argument0);

if (debug_mode && _determinant == 0)
{
show_error("UBG 3D2D: Cannot calculate inverse matrix. Determinant = 0!", false);
return undefined;
}

var _a11 = argument0,
_a12 = argument0,
_a13 = argument0,
_a14 = argument0,

_a21 = argument0,
_a22 = argument0,
_a23 = argument0,
_a24 = argument0,

_a31 = argument0,
_a32 = argument0,
_a33 = argument0,
_a34 = argument0,

_a41 = argument0,
_a42 = argument0,
_a43 = argument0,
_a44 = argument0;

_inverse = 0; // Allocate space

// Calculate second base matrix:
// [1, 1]
_inverse = _a22 * _a33 * _a44 +
_a23 * _a34 * _a42 +
_a24 * _a32 * _a43 -
_a22 * _a34 * _a43 -
_a23 * _a32 * _a44 -
_a24 * _a33 * _a42;

//[1, 2]
_inverse = _a12 * _a34 * _a43 +
_a13 * _a32 * _a44 +
_a14 * _a33 * _a42 -
_a12 * _a33 * _a44 -
_a13 * _a34 * _a42 -
_a14 * _a32 * _a43;

//[1, 3]
_inverse = _a12 * _a23 * _a44 +
_a13 * _a24 * _a42 +
_a14 * _a22 * _a43 -
_a12 * _a24 * _a43 -
_a13 * _a22 * _a44 -
_a14 * _a23 * _a42;

//[1, 4]
_inverse =  _a12 * _a24 * _a33 +
_a13 * _a22 * _a34 +
_a14 * _a23 * _a32 -
_a12 * _a23 * _a34 -
_a13 * _a24 * _a32 -
_a14 * _a22 * _a33;

//[2, 1]
_inverse =  _a21 * _a34 * _a43 +
_a23 * _a31 * _a44 +
_a24 * _a33 * _a41 -
_a21 * _a33 * _a44 -
_a23 * _a34 * _a41 -
_a24 * _a31 * _a43;

//[2, 2]
_inverse =  _a11 * _a33 * _a44 +
_a13 * _a34 * _a41 +
_a14 * _a31 * _a43 -
_a11 * _a34 * _a43 -
_a13 * _a31 * _a44 -
_a14 * _a33 * _a41;

//[2, 3]
_inverse =  _a11 * _a24 * _a43 +
_a13 * _a21 * _a44 +
_a14 * _a23 * _a41 -
_a11 * _a23 * _a44 -
_a13 * _a24 * _a41 -
_a14 * _a21 * _a43;

//[2, 4]
_inverse =  _a11 * _a23 * _a34 +
_a13 * _a24 * _a31 +
_a14 * _a21 * _a33 -
_a11 * _a24 * _a33 -
_a13 * _a21 * _a34 -
_a14 * _a23 * _a31;

//[3, 1]
_inverse =  _a21 * _a32 * _a44 +
_a22 * _a34 * _a41 +
_a24 * _a31 * _a42 -
_a21 * _a34 * _a42 -
_a22 * _a31 * _a44 -
_a24 * _a32 * _a41;

//[3, 2]
_inverse =  _a11 * _a34 * _a42 +
_a12 * _a31 * _a44 +
_a14 * _a32 * _a41 -
_a11 * _a32 * _a44 -
_a12 * _a34 * _a41 -
_a14 * _a31 * _a42;

//[3, 3]
_inverse = _a11 * _a22 * _a44 +
_a12 * _a24 * _a41 +
_a14 * _a21 * _a42 -
_a11 * _a24 * _a42 -
_a12 * _a21 * _a44 -
_a14 * _a22 * _a41;

//[3, 4]
_inverse = _a11 * _a24 * _a32 +
_a12 * _a21 * _a34 +
_a14 * _a22 * _a31 -
_a11 * _a22 * _a34 -
_a12 * _a24 * _a31 -
_a14 * _a21 * _a32;

//[4, 1]
_inverse = _a21 * _a33 * _a42 +
_a22 * _a31 * _a43 +
_a23 * _a32 * _a41 -
_a21 * _a32 * _a43 -
_a22 * _a33 * _a41 -
_a23 * _a31 * _a42;

//[4, 2]
_inverse = _a11 * _a32 * _a43 +
_a12 * _a33 * _a41 +
_a13 * _a31 * _a42 -
_a11 * _a33 * _a42 -
_a12 * _a31 * _a43 -
_a13 * _a32 * _a41;

//[4, 3]
_inverse = _a11 * _a23 * _a42 +
_a12 * _a21 * _a43 +
_a13 * _a22 * _a41 -
_a11 * _a22 * _a43 -
_a12 * _a23 * _a41 -
_a13 * _a21 * _a42;

//[4, 4]
_inverse = _a11 * _a22 * _a33 +
_a12 * _a23 * _a31 +
_a13 * _a21 * _a32 -
_a11 * _a23 * _a32 -
_a12 * _a21 * _a33 -
_a13 * _a22 * _a31;

// Multiply by determinant:
_determinant = 1 / _determinant;

for (var i = 0; i < 16; ++i)
_inverse[i] *= _determinant;

return _inverse;``````

Get the determinant of a matrix (required by the above script):
Code:
``````///    @desc    Calculates the determinant of a matrix.
///    @param    {matrix}    matrix    matrix to process
///    @returns    {real}
///    @private

if (!is_array(argument0))
show_error("UBG 3D2D (argument0): Expected type [matrix]!", true);

if (array_length_1d(argument0) != 16)
show_error("UBG 3D2D (argument0): Expected type [matrix]!", true);

var _determinant = 0;

var _a11 = argument0,
_a12 = argument0,
_a13 = argument0,
_a14 = argument0,

_a21 = argument0,
_a22 = argument0,
_a23 = argument0,
_a24 = argument0,

_a31 = argument0,
_a32 = argument0,
_a33 = argument0,
_a34 = argument0,

_a41 = argument0,
_a42 = argument0,
_a43 = argument0,
_a44 = argument0;

_determinant = _a11 * _a22 * _a33 * _a44 +
_a11 * _a23 * _a34 * _a42 +
_a11 * _a24 * _a32 * _a43 +

_a12 * _a21 * _a34 * _a43 +
_a12 * _a23 * _a31 * _a44 +
_a12 * _a24 * _a33 * _a41 +

_a13 * _a21 * _a32 * _a44 +
_a13 * _a22 * _a34 * _a41 +
_a13 * _a24 * _a31 * _a42 +

_a14 * _a21 * _a33 * _a42 +
_a14 * _a22 * _a31 * _a43 +
_a14 * _a23 * _a32 * _a41;

// Part two:
_determinant += -_a11 * _a22 * _a34 * _a43
-_a11 * _a23 * _a32 * _a44
-_a11 * _a24 * _a33 * _a42

-_a12 * _a21 * _a33 * _a44
-_a12 * _a23 * _a34 * _a41
-_a12 * _a24 * _a31 * _a43

-_a13 * _a21 * _a34 * _a42
-_a13 * _a22 * _a31 * _a44
-_a13 * _a24 * _a32 * _a41

-_a14 * _a21 * _a32 * _a43
-_a14 * _a22 * _a33 * _a41
-_a14 * _a23 * _a31 * _a42;

return _determinant;``````

Bloody hell, just noticed 1.4 doesn't have matrix / vector multiplication either. GM fails as a game engine. -.- Here's another script:
Code:
``````///@desc    Multiplies a 4D vector by a matrix and returns the final vector.
///    @param    {array}    vector    4D vector in the form of an array
///    @param    {matrix} matrix    matrix in the form of an array
///    @returns    {array}
///    @private

var _vx = argument0,
_vy = argument0,
_vz = argument0,
_vw = argument0;

var _m = argument1;

var _nx = 0,
_ny = 0,
_nz = 0,
_nw = 0;

_nx =    _vx * _m +
_vy * _m +
_vz * _m +
_vw * _m;

_ny =    _vx * _m +
_vy * _m +
_vz * _m +
_vw * _m;

_nz =    _vx * _m +
_vy * _m +
_vz * _m +
_vw * _m;

_nw =    _vx * _m +
_vy * _m +
_vz * _m +
_vw * _m;

var r;
r = _nx;
r = _ny;
r = _nz;
r = _nw;
return r;``````

Your code for your mouse may look something similar to this (going off the top of my head, excuse typos / incorrect arguments). You'l need to modify for your setup, though:
Code:
``````// Get our mouse coords between [-1 and 1]
var mouse_x_screen = (mouse_x / view_wport) * 2.0 + 1.0;
var mouse_y_screen = (mouse_y / view_hport) * 2.0 + 1.0;

// Grab our matrices:
var mat_view = matrix_get(matrix_view);
var mat_proj = matrix_get(matrix_projection);

// Calculate the 'backwards' versions to go into 3D space:
var mat_combined = matrix_multiply(matrix_view, matrix_projection);
var mat_inverse = matrix_inverse(mat_combined) // The above script

// Put our mouse in a vector to convert into 3D points:
var v;
v = mouse_x_screen
v = mouse_y_screen
v = 1 // Far plane of the cam
v = 1 // Take my word, just have it set to 1

// Convert:
var mouse_point_1 = matrix_vector(v, mat_inverse); // Above script
v = 0 // Near plane of the cam
var mouse_point_2 = matrix_vector(v, mat_inverse); // Above script

// Now you have two points, the mouse's 'position' at the camera's near plane and far plane. You can make your line from these points.``````

Last edited:

#### TheSnidr

##### Heavy metal viking dentist
GMC Elder
Good point FrostyCat, didn't realize he wanted a point and not a vector!
Another method though is to render a depth map to a surface and read from that. You'll still need to create a 3D vector, and the depth map will tell you how far along that vector the intersection point will be. I plan to do this for a level editor some time in the near future. Once your surface has been converted to a buffer, reading from it is fairly efficient. The conversion is slow though.
I just published a 3D collision mesh system that lets you cast rays against meshes, but that is only for 2.3.

But yeah, this is complicated stuff.

#### LittleAngel

##### Member
A huge Thank you to all. Yes it's not easy... I'll have to switch to GM 2.3 soon...

To Binsk:
thank you very much for all these tracks I have already explored a little bit. The line collisions work but lack a lot of precision. It works well for walls, but as soon as you want to test the top of boxes, it's not so good... (for example to calculate the coordinates of the decals for the bullet marks on walls, you need a lot of computing precision )

I also tested the color recognition method that works great, but it's quite slow, and only good for enemies, not for the 3D world.

I will continue to work with your suggestions, thanks

Last edited: