GameMaker Projecting 2D Sprites into 3D space

2021-05-09_19-45-16.gif


I previously had a mode-7 type "special stage" working, but it used the old d3d functions from GMS 8.
I'm redoing it using cameras and projections matrixs for two reasons.

1 - Since d3d functions are not in GMS2, so there was a lot of messy baggage in the code with compat scripts
2 - The old way does not work on Android for me. The new test gif you are seeing here does work on Android.

I'm wanting to draw sprites for objects (rings, spikes, etc) as 2D billboards that ignore the deformation of the projection, and instead "scale" based on handrawn frames, BUT are still drawn in the appropriate location in 3D space. Essentially, translating 3D world space to 2D screen space. Think like the old school SNES Mario Kart effect.
1620670138431.png


We had previously figured it out, but its not working with the new functions, and only works with the old d3d method. So I'm opening a new topic, since its starting from scratch.

Here is the Camera code that achieves the 3D effect.
GML:
/// @description Insert description here

var VW = global.ViewWidth;
var VH = global.ViewHeight;

camera = camera_create();
view_camera[0] = camera;

var viewmat2D = matrix_build_lookat(objSS_Player.x, objSS_Player.y, -100, objSS_Player.x, objSS_Player.y, objSS_Player.z, 0, 1, 0) //matrix_build_lookat(640, 240, -10, 640, 240, 0, 0, 1, 0);
var projmat2D = matrix_build_projection_ortho(VW, VH, 1.0, 32000.0);


var viewmat3D = matrix_build_lookat(x, objSS_Player.y, -50, objSS_Player.x, objSS_Player.y, objSS_Player.z, 0, 0, 1);
var projmat3D = matrix_build_projection_perspective_fov(100, VW/VH, 1, 320)


camera_set_view_mat(camera, viewmat3D);
camera_set_proj_mat(camera, projmat3D);
camera_apply(camera);
 

TheSnidr

Heavy metal viking dentist
GMC Elder
Here's a couple of scripts for converting between worldspace and screen space:
 
Here's a couple of scripts for converting between worldspace and screen space:
Snap! Very cool! This is where I'm at with implementing this. It's not quite working yet. What am I missing here?

GML:
var VW = global.ViewWidth;
var VH = global.ViewHeight;

camera = view_camera[0]

//get 2d view
var viewmat2D = matrix_build_lookat(objSS_Player.x, objSS_Player.y, -100, objSS_Player.x, objSS_Player.y, objSS_Player.z, 0, 1, 0) //matrix_build_lookat(640, 240, -10, 640, 240, 0, 0, 1, 0);
var projmat2D = matrix_build_projection_ortho(VW, VH, 1.0, 32000.0);


//get 3d view
var viewmat3D = matrix_build_lookat(x, objSS_Player.y, -50, objSS_Player.x, objSS_Player.y, objSS_Player.z, 0, 0, 1);
var projmat3D = matrix_build_projection_perspective_fov(100, VW/VH, 1, 320)


//Perform conversion
var array = convert_3d_to_2d(x,y,0,viewmat3D,projmat3D);
var xx = array[0];
var yy = array[1];


//Set to 2D
camera_set_view_mat(camera, viewmat2D);
camera_set_proj_mat(camera, projmat2D);
camera_apply(camera);

//draw flat sprite
draw_sprite(sprite_index,0,xx,yy);
 
Here's a couple of scripts for converting between worldspace and screen space:
Been experimenting and still not getting it.
Since the script, is returning 2D screen coordinates, wouldn't it need to be on the Draw GUI?
Then again, the script has (0, 0) as top left) and (1, 1) as bottom right, so how does that translate into usable coordinates, if it's just a range of 1 px?

Currently, the best I've been able to do is have the sprite float around in the top left corner.
2021-06-05_06-50-02.gif
 
I'm still not getting anywhere with this and its flustering. :confused:
I'm getting weird results as the emerald sprites float around in all the wrong places.
Does anyone see what's missing?
 

TheSnidr

Heavy metal viking dentist
GMC Elder
What if you do this instead:
Code:
var array = convert_3d_to_2d(x,y,0,viewmat3D,projmat3D);
var xx = array[0] * window_get_width();
var yy = array[1] * window_get_height();
 

Yal

🐧 *penguin noises*
GMC Elder
One cool thing with the new rendering system is that 2D and 3D drawing are the same mode, so by changing the world/view/projection matrices you can affect flat drawing as well (and it's easier than ever to overlap 2D and 3D visuals). For instance, when drawing to a surface, you can move the entire world <view position> to the up-left and then trigger a normal draw event of the things you want on the surface, instead of having to convert to surface coordinates yourself.

The flip side is that the matrix affects all drawing, so it's important to make sure to keep track of what you're drawing and what mode you're in. (E.g. to draw 2D sprites you want to have a regular "as is" matrix, but you draw them at 3D coordinates you've converted to screenspace). Also you're gonna need to manually depth-sort the sprites to prevent weird overlap ruining the illusion, a priority queue works well for this (sort by screen Y)

<checks code more closely>
Actually I misread it, seems like you're doing the matrix-setting right. Derp, my bad.

Since the script, is returning 2D screen coordinates, wouldn't it need to be on the Draw GUI?
Then again, the script has (0, 0) as top left) and (1, 1) as bottom right, so how does that translate into usable coordinates, if it's just a range of 1 px?
Screen coordinates are traditionally in the range 0-1, it's a percentage of the viewport. So if your screen is 256x224 pixels wide, coordinates 0.5,0.5 represents the pixel 128,112. (Basically all numbers are handled this way in shaders, colors are 0.0-1.0 floats instead of 0-255 integers for instance)
 
What if you do this instead:
Code:
var array = convert_3d_to_2d(x,y,0,viewmat3D,projmat3D);
var xx = array[0] * window_get_width();
var yy = array[1] * window_get_height();
Trying it now. For reference, this is how the room looks in the room editor.
1625508586017.png


This is the result.
2021-07-05_13-07-28.gif


What if you do this instead:
Screen coordinates are traditionally in the range 0-1, it's a percentage of the viewport
Ah! That makes sense. However, I feel like multiplying by the window size would do the trick if it's a percentage...but that GIF begs to differ.
I also tried multiplying by view width and height, just in case. Same results.

Notice how when the player becomes even with the emerald, it inverses it's vertical orientation.
 

TheSnidr

Heavy metal viking dentist
GMC Elder
Oki, found the problem. It's actually two different problems:
1. You've copied the code from creating the view and projection matrices from the camera object to the emerald object. But you're still referencing the local variable x, so the view matrix of the emerald will be different from the camera.
2. For some reason, GameMaker alters the projection matrix after it has been assigned to a camera. A projection matrix received from matrix_get will be flipped compared to matrix_build_projection_etc. The values returned from convert_3d_to_2d will as such need to be flipped vertically.

So here's the new draw GUI code for objEmerald:
Code:
var VW = global.ViewWidth;
var VH = global.ViewHeight;


camera = camera_create();

var viewmat2D = matrix_build_lookat(objSS_Player.x, objSS_Player.y, -100, objSS_Player.x, objSS_Player.y, objSS_Player.z, 0, 1, 0)
var projmat2D = matrix_build_projection_ortho(VW, VH, 1.0, 32000.0);


var viewmat3D = matrix_build_lookat(Camera.x, objSS_Player.y, -50, objSS_Player.x, objSS_Player.y, objSS_Player.z, 0, 0, 1);
var projmat3D = matrix_build_projection_perspective_fov(100, VW/VH, 1, 320)

//Perform conversion
var array = convert_3d_to_2d(x,y,0,viewmat3D,projmat3D);
var xx = array[0] * window_get_width();
var yy = (1 - array[1]) * window_get_height();

//draw flat sprite
draw_sprite(sprite_index,0,xx,yy);
Also, I'd suggest using the existing view and projection matrices instead of creating them from scratch in the emerald object ;)
 
Oki, found the problem. It's actually two different problems:
1. You've copied the code from creating the view and projection matrices from the camera object to the emerald object. But you're still referencing the local variable x, so the view matrix of the emerald will be different from the camera.
2. For some reason, GameMaker alters the projection matrix after it has been assigned to a camera. A projection matrix received from matrix_get will be flipped compared to matrix_build_projection_etc. The values returned from convert_3d_to_2d will as such need to be flipped vertically.

So here's the new draw GUI code for objEmerald:
Code:
var VW = global.ViewWidth;
var VH = global.ViewHeight;


camera = camera_create();

var viewmat2D = matrix_build_lookat(objSS_Player.x, objSS_Player.y, -100, objSS_Player.x, objSS_Player.y, objSS_Player.z, 0, 1, 0)
var projmat2D = matrix_build_projection_ortho(VW, VH, 1.0, 32000.0);


var viewmat3D = matrix_build_lookat(Camera.x, objSS_Player.y, -50, objSS_Player.x, objSS_Player.y, objSS_Player.z, 0, 0, 1);
var projmat3D = matrix_build_projection_perspective_fov(100, VW/VH, 1, 320)

//Perform conversion
var array = convert_3d_to_2d(x,y,0,viewmat3D,projmat3D);
var xx = array[0] * window_get_width();
var yy = (1 - array[1]) * window_get_height();

//draw flat sprite
draw_sprite(sprite_index,0,xx,yy);
Also, I'd suggest using the existing view and projection matrices instead of creating them from scratch in the emerald object ;)

This is wonderful. Thank you very much! I actually ran into that weird vertical flip issue when I was previously using converted d3d functions on Android. Everything was upside down. Still not sure why that's a thing, but good to know where it comes from.

Ok, I've cleaned up, made the camera obj contain all the matrices so they no longer created by each object.
Put in some basic code for swapping the image_index based on the distance (still need to fine tune it to make it feel right).
2021-07-08_15-49-52.gif

Also confirmed it can run on Android without issues. No weird flipping XD
Next, on to depth sorting 🤔. We'll see how that goes.

Thank you very much for your help @TheSnidr! You don't know much cleaner this is than the mess I had going on before starting over. 😅
 
Top