An example of what i'm currently working on:
N64 styled graphics are indeed possible. And they don't need to look awful. It all depends on the artstyle (and maybe some shader trickery. Maybe bloom lighting, HDR, Depth of field, color correction, etc... to make the game look even nicer.).
Realistic artstyles age horribly compared to more abstract/cartoony ones.
Heck, it's even possible that the current AAA video games will pale in comparison to "realistic looking" games in 3-4 years.
More abstract artstyles on the other hand age much better. Wind waker is the most notable example. Just increase the resolution to a modern standard (1080p) and maybe slap a post processing filter over it. And you are done.
If you have an engine which has limitations in one way or another, then you have to try to work around them.
Can you make a 3D N64 looking game in GM? Absolutely.
But would i advise in doing so (for commercial projects)? No.
After working on my current project, i wouldn't start a new (large) 3D project in GM again.
There are a lot of reasons for that. But the most notable ones which come to my mind are:
- GML - compared to other languages (Java,C#, etc...) it's very simplistic. (absence of methods, access modifiers, constructors, etc...) It can make code maintenance on larger project tedious as it's harder to enforce software engineering practices.
- lack of multithreading - want to load and process files or game data asynchronously? can't do that. (Which is a bummer as you can't tap into the capabilities of multicore processors.)
- lack of texture filtering (Mipmaps, etc...) - you can of course activate texture filtering by using extensions (DLL on windows) but then multiplatform support goes straight out the window. (Which destroys the whole point of having an easy to use multiplatform engine.)
There are other smaller caveats like Multiple Render Targets (MRTs) which in GMS 1.4 only work by using HLSL shaders (which makes your project windows only.)
Of course, this is my personal opinion. If you still want to try things out, experiment with shaders and/or vertex buffers in a fammiliar enviroment then go for it.
(That's how i learned the majority of stuff in 3D programming. Helped me to get started with OpenGL.)