If you really want to be able to add meaningful and true modding, then you will need some sort of scripting interface that allows people to create custom logic. Not only that, but you also need your games interface/systems to be well-designed so that other mods can directly make use of your engine. There are a few scripting marketplace assets, though I imagine it would take a larger amount of work if you wanted to achieve them yourselves.
Personally, I always have mixed opinions on mods. I feel that asset replacement is generally pointless for the most part and that people will often get bored of that very quickly, as even if you do support some amounts of behaviour modification, it will always be limited and bound by content rules that already exist within your game, content rules that most mod creators would strive to extend upon or change.
Secondly, I personally that if modding is added early on to a game, then controversially, it shows a lack of foresight on the developers part. Whilst it might seem like a great idea to allow people to extend the game, I think it is somewhat missing the point if a game necessitates mods from the get go. I always feel that if a modding system is necessary, then you haven't done enough as a developer to make the base game compelling. Otherwise, a modding system can be a reasonable addition to a game, but I don't think its worth most developers implementing it early on. it's something that I believe should only be focused on once the game has been released and you have slowed down your progress on the game.
If however, you are purely making one out of interest, in order to achieve a FULL modding system, you need to be able to do the following:
- Have some form of scripting system. Allow players to write a script in a high level language like lua, and have those scripts executed within your game.
- Have a means of bundling scripts and assets together into a data format that your game can recognise and load.
- Create a mod-manager that keeps track of which mods the player wants to load, and is also responsible for firing off scripts that are associated with that mod.
- Structure your games system code in a way that allows external scripts to make use of your gameplay systems, much like existing objects in your game already would. This means having clean systems for things like the creation of inventories, the firing of projectiles, the application of damage to another instanec via a function. These functions will need to be exposed to the scripting language you are using so that the scripting language back-system can make subsequent calls to your own game functions.
- Have a number of set events in-game that external scripts can hook into to modify/add content throughout specific stages of the game. This may for example be finding the place where all of your weapon stats are defined, and allowing an external script to run a few functions to add weapons of its own.
- Have a resource management system (similar to what others have talked about in this thread) which can allow for the loading and use of external game assets.
- Use YYC as you'll need every ounce of performance you can get when using a scripting language on top of another engine
Naturally, all of this is rather complicated though it is definitely possible. Though unfortunately it requires rather in-depth knowledge from a number of different areas, most importantly, having a really clean and well-documented project which is designed in a modular fashion. Even if you have the building blocks to make modding possible, it is still a very involved process. No one could create a general modding architecture that could work for all games as each game will have different demands as to what level of access users may have to different aspects of the game.
What would be very neat however, is if somehow, GameMaker could allow developers to publish mod builds of their game. What that would be is a partially open-source version of the project which exposes all of the games internal scripts and allows users to add new objects to that game (However would not show object code, unless specified for specific objects). It would then export those objects as standalone compiled units which could be imported and merged with the main game. It would be a bit of a one-way relationship though, in that the mod could perform events like any object, and the mod objects could use and access any of the games functions, however the game itself would not be able to interact with the mod object. Someone smart enough could probably somehow cheese this, by making use of function addresses.
In the past, for older versions of GM, there used to be a C++ library called GMAPI, which would allow a DLL to make use of any GameMaker function. In this, you could essentially allow mods to be made in C++, because you could just expose that api to a DLL, and have your game arbitrarily load DLLs which followed a given structure.