Hey Mike, first of great post, this is one of those needed topics that as Andrew said gets caught on performance.
Now i have a few comments/questions about the post.
First I think that "tilemap_get" should have a mention, at least in the docs that cell x is not coordinate x(as in pixel), but rather a number that tells you which tile we are talking about. Maybe an OBS at the end of the doc.
I found this confusing when i saw "var px1 = xx*tile_size;", Until i realized that tilemap_get had to be position and not absolute values.
Further down you confused me because it seemed you said that c² = a²+b² would always have c = 1.
Then later on i realized you were actually talking about using pitagoras to get c, then do a/c, b/c to get sin/cos then sin²+cos² = 1.
I think that's what you were referring to.
Later on i didn't really get how the whole thing worked, i mean what makes the projections be draw with the mouse_pos as a point of origin.
For me, looking at the code there is no indication of direction in any moment, so assuming 0,0 as a starting point and positive x,y values going right and down respectively; Looking at adx(for instance), it is the x of a +[ the_shadow* the proportional(sin/cos) length of the respective side of the triangle], or in other words a really big positive number.
So in my world this is some x value(positive) + a really big x value(+) , which would lead exclusively to a bigger x value, meaning something to the right.
And at the very end, in the "cross product" that you made, unless I missed something it will only equal 0, and not fail the if statement, if you compare top left with bottom right, which you never will since it's the diagonal of the square. So i didn't really see how that helps.
Another thing, and this might be a gms way of coding, you did 1.0*SHADOW_LENGTH, is this you way to casting the number to a floating point with a 0.1 precision, or maybe normal f.point precision? Why did you add the 1.0?
If you got the time let me know,
thanks either way, and feel free to also pm me if necessary.