I have a problem:

in my 3d level editor, i am trying to correctly detect the object selected by mouse.

To do this, i first detect the coordinate of the mouse in the 3d space at the ground level (z=0) ;

then i do a kind of raycasting;

- I take the camera's lookat "from" coordinates as origin
- i take the position of the mouse at the ground , as destination
- i compute the distance (sqrt(diffx²+diffy²+diffz²)
- then i do a loop from zero to distance, using a small step (like 0.1)
- i check the poisiton of the point at distance N like :
- px = lerp(fromx,tox,n/distance) (same for py and pz)
- i check if that point is inside an object bounding box,
- if an object is found, i return it

It work well when the camera is at position x= zero , but when the camera is, by example, at position x = 10 ; or when the object is heigher (position z=2) it is not accruate : it doesent intersect the correct object, but it found the object near from the target.

i am sure that my algorithm is not optimal, i have also tried another aproach:

i construct a normalized vector using the camera position as origin, and the mouse position as target

vecLine = [ diffx/distance,diffy/distance,diffz/distance]

then for each object, i also compute 8 vectors (for each corner), using also the camera as origin, but the corner coordinates at target ; this vector is also normalized using the distance between the camera and the corner coordinates

then i get the minX and maxX, minY maxY, minZ maxZ from all 8 vectors, and i check if the first vector (from the camera to the mouse) is inside the bounds vectors.

this method gives me almost the same result than the first one

do you guys, already implemented 3d mouse picking with game maker, how did you that?