Marketplace assets namespacing and structure

[Alice]

One of the recurring problems in developing Marketplace assets is avoiding name collisions.
Before GM:S 2.3+, all defined functions were global. However, now we can use combinations of global variables and structs to introduce namespacing of code entities with little chance of collisions.
Additionally, GM:S 2.3+ adds more flexibility when it comes to assets groups/directories structure. This in particular allows us to apply consistent grouping.

The question becomes - what kind of namespacing/structures can we apply? Should we try to aim for namespacing/structure conventions in the first place? I believe the widespread the convention becomes, the smoother the experience of Marketplace asset users; provided the convention is well-designed in the first place.

Code namespacing

Spoiler

One idea I have is declaring global structs for publishers, which in turn have items for packages, then for modules or however we call them. So, there could be a module like:
global.marketplace_publisher_Alphish.package_StackFlow.Process

That module would be a struct providing functions and/or variables related to "Process" functionality, whatever the Process is in the context of StackFlow package.

Of course, we need to define a struct for global.marketplace_publisher_Alphish global, then another struct for its package_StackFlow variable, then yet another struct for Process module. And let's not forget the same publisher or package can have multiple packages/modules, yet it only should be defined once.
With that in mind, I came up with three scripts for accessing publishers/packages/modules:

// defines the publisher if needed, returns the publisher; marketplace_publisher_* prefix is added automatically
marketplace_get_publisher(publisher)
// defines the package if needed, returns the package; package_* prefix is added automatically
marketplace_get_package(publisher, package)
// defines the module if needed, returns the module
marketplace_get_module(publisher, package, module)

As long as we use these scripts consistently (using the same prefixes inside), we can use these scripts interchangeably among packages - whether the user ignores, overwrites or keeps the scripts, the result would be the same.

Note: I assume the scripts (as defined with function scriptName(...) rather than scriptName = function(...)) are all available before any global code is run (see also: this thread). If it's not the case, it complicates the use of these scripts a lot.

Code shorthands

Spoiler

Additionally, we could provide macros with shorthands for packages, modules and/or functions. These shorthands would be kept in separate Script assets, so that they can be skipped/removed easily. Something like these:

// script: shorthands_Alphish_StackFlow_package
#macro StackFlow global.marketplace_package_Alphish_StackFlow
global.marketplace_package_Alphish_StackFlow = marketplace_get_package("Alphish", "StackFlow");

// script: shorthands_Alphish_StackFlow_modules
#macro Flow global.marketplace_module_Alphish_StackFlow_Flow
global.marketplace_module_Alphish_StackFlow_Flow = marketplace_get_module("Alphish", "StackFlow", "Flow");

#macro Process global.marketplace_module_Alphish_StackFlow_Process
global.marketplace_module_Alphish_StackFlow_Process = marketplace_get_module("Alphish", "StackFlow", "Process");

#macro AetCetera global.marketplace_module_Alphish_StackFlow_AetCetera
global.marketplace_module_Alphish_StackFlow_AetCetera= marketplace_get_module("Alphish", "StackFlow", "AetCetera");

// script: shorthands_Alphish_StackFlow_scripts
function StackFlow_Flow_push(_controller) {
    return global.marketplace_publisher_Alphish.package_StackFlow.Flow.push(_controller);
}
function StackFlow_Flow_pop() {
    return global.marketplace_publisher_Alphish.package_StackFlow.Flow.pop();
}
function StackFlow_Process_run(_instructions) {
    return global.marketplace_publisher_Alphish.package_StackFlow.Process.run(_instructions);
}

// script: shorthands_Alphish_StackFlow_scripts_short
function Flow_push(_controller) {
    return global.marketplace_publisher_Alphish.package_StackFlow.Flow.push(_controller);
}
function Flow_pop() {
    return global.marketplace_publisher_Alphish.package_StackFlow.Flow.pop();
}
function Process_run(_instructions) {
    return global.marketplace_publisher_Alphish.package_StackFlow.Process.run(_instructions);
}

Assets naming

Spoiler

The simplest option would be something like prefix_Publisher_Package_*. It should work well enough for items like Scripts, Rooms, Sprites, Objects... - as long as we don't expect the user to type out the asset name directly, we should be good.

It might be trickier if we expect the user to e.g. refer to an Object - for example, something like Tween_create(obj_QuadraticTween, 20, 80, 30).
In such case, I'd suggest still sticking with the exhaustive naming convention (i.e. obj_Publisher_Package_QuadraticTween), but also provide a shorthand macro like #macro obj_QuadraticTween obj_Publisher_Package_QuadraticTween in some kind of shorthands script (e.g. shorthands_Publisher_Package_objects).

Directory structure

Spoiler

To easily separate the Marketplace assets from others and from the user's code, I'd suggest putting all the package utilities into Marketplace > Publisher > Package directory in the Asset Browser.
We can do pretty much whatever we want in our package folder, as long as we follow the assets naming properly.

The marketplace_get_publisher/package/module scripts would go straight into the Marketplace folder, so that they can be easily skipped/overwritten when importing multiple packages following the conventions.

If our package comes with a demonstration example, maybe we could add related assets Example > Publisher > Package. I'd keep it at the same level as Marketplace directory rather than something like Marketplace > Publisher > Package > Example, so that you can easily opt-out from importing the non-essential examples.

Not sure about the shorthands, if we were to include them: should they go into Marketplace > Publisher > Package directory structure, or maybe we could have a separate directory like Shorthands > Publisher > Package?
Having it in the separate directory has the advantage of easily choosing shorthands or opting out from them altogether. Considering the shorthands are the most prone to name collisions, and their macros can block out other variable names, we probably want to have them readily available in the import tree.

Summary

Overall, common Marketplace asset conventions might add a little extra work for setting up the Marketplace asset, but also save the time developer needs to figure out the conventions themself.
Additionally, if common conventions become widespread enough, this should lead to more unified experience of Marketplace asset users, with external assets kept in their own section instead of cluttering the main code.
Of course, the conventions outlined above are applicable to GM:S 2.3+ only (though certain aspects - such as shorthand macros - can be applied in earlier versions as well).

Do you think these kinds of conventions are worth pursuing? Would you like to make some changes? Or maybe you'd like to point out some cases that aren't easily covered by the proposed conventions?
I'd love to hear your thoughts. ^^

 

[kburkhart84]

This is interesting. I'm not sure how much of it all is actually worth it.

Namespacing...on paper it isn't bad... but structs still have some performance issues, and may always have some overhead to them even when/if they do make them faster than they are now. It might be better(until GML gets real namespaces) to just use prefixes. In my current project, all functions called by the user start with fhInput, while all internal functions start with double underscore __fhInput, so that the internal functions don't easily find their way to the auto-complete box. If the adding struct stuff would be a thing that automatically got added upon importing the package and was required to use it, then they would need to make sure performance is much better, as I'm sure some functions in some assets are going to get called quite often and adding on a struct access overhead to each call is not going to be pretty.

The code shorthands as you describe them don't seem bad(and is very similar to the discussion we were having in my topic). I think the issue is as I mentioned in my topic, the user will see it as a macro instead of as a function in the auto-complete box, which may be confusing for newer coders that don't understand properly what's going on. Besides that though, at the least in the examples you mention, it leads to the performance issues of forcing structs to be involved. If we were to leave that out and just assume we were talking about shortening long names, macros would work fine except for the possible confusion in the auto-complete box. However, maybe someone could convince Yoyo to add something like a "function alias" that counts like a function calling another function, doing a similar literal replacement with the function call, but would use the alias name in the errors and debugging. If you use #macros, then at run-time the errors will show the original name instead of the cloned name. If you do like I may do on mine, duplicating the functions completely, you get better clarity with errors, but the asset creator has to duplicate the functions totally, or make the clone call the original(which has a slight overhead itself).

As far as shorthand naming of assets, I'm personally following the same prefixing I'm doing for the functions, fh is "Firehammer" as in Firehammer Games, while the system is named Firehammer Input, so I'm prefixing anything external with fhInput. in this case, there is only one object that the user would ever have to create(and its only for making the game load the configuration for the system instead of running the game rooms, you put it in your first room). The other object the system has is created by a function you call so never gets accessed directly, and since its internal, like all internal assets/functions, its getting the prefix __fhInput. In my opinion, I'm not sure if it is worth it for Yoyo to actually implement something for this into the marketplace, or just let us as developers handle it(like I am, or however they prefer).

Directory Structure...the first thing I did with the re-write for my system was rework the directory structure since we finally can do that with 2.3. Any of my assets will be under the FHGames folder, with each asset gettings its own subfolder(I only have an easing system out for 2.3, and this input system re-write is the second thing I'll finish for 2.3, I'm just pre-planning for the future). Similar to the other things, I'm not sure if it would be worth it for them to enforce or handle directories, or if they should just let us devs handle it like we do now. I think they could add at the least an automatic "Marketplace" or "Packages" folder for this, but I'm not sure if its necessary due to how open Gamemaker and GML are in general, how there are few rules compared to other engines.

So basically, overall, the ideas seem good on paper(as far as having YoyoGames do something with it directly, enforcing it in the marketplace). But I'm not sure if it would be worth it for them or us, since as of 2.3 we can easily handle some of it ourselves, and some people I would assume are not going to be happy with those kinds of forced changes. It may be a better option to leave well enough alone as far as marketplace enforcement.

That said, there should be a FAQ or some resource where some "standards" are defined. This would be similar to what I describe above, where I make my own "namespace" using prefixes on all the functions and assets, and double underscrore plus prefixes on all internal assets that the user doesn't directly touch. The same "Standards" FAQ could have a section on GML friendly functions with snake_case, compared to using camelCase, and just in general giving recommendations that make assets easier to use and organize, including for example a publisher using a single asset tree directory for all their assets, and having each one as a subfolder inside that.

Another thing to consider is how we can have "local" packages as well. Those may not be on the marketplace(though they get used often on other marketplaces like itch.io). Would they have to start doing some of the automatic stuff described here with those as well? I don't know if people will like that either, considering they would generally expect the package to import exactly in the same condition it was exported in.

 

[Alice]

Just for the record, none of these convention suggestions is meant to be for YoYo to implement for automatic handling. Every Marketplace publisher still would be able to arrange their assets however they deem appropriate for their project.

These conventions are intended as an opt-in agreement between Marketplace developers of sorts, so that their assets are handled consistently (I say "agreement", but it basically boils down to a developer choosing to adopt this convention). The more asset developers use these, the higher the benefits of consistent handling.
Additionally, having a pre-made convention means not needing to re-invent the conventions yourself. And if multiple people come up with their conventions independently, the project importing multiple assets can get pretty messy.
If the convention doesn't work for your Marketplace asset, just use something else.

Re: Namespacing and structs overhead. It's true that right now structs can be slower than even data structures, and ~3-4 thousands struct readings per a single millisecond aren't all that impressive. Nonetheless, I'm pretty sure most Marketplace assets provide functionality that doesn't need to be called thousands of times every step and/or has so much processing of its own, that the cost of reading structs along the way becomes negligible.
In general, whenever you code something a specific way for the sake of performance, you better run benchmarks to make sure the performance gain is even worth it. Otherwise, you might fall into the trap of premature optimisation, making some extra effort and e.g. repeating code to avoid dreaded function calls, making the implementation harder to maintain in the future. Don't think about the overheads in isolation, but compare them to the overall performance.

Also, while the proposed namespacing convention introduces some structs nesting, proper shorthands definition can mitigate part of it.
Note how along the package/modules macros I defined a package/module variables like global.marketplace_package_Alphish_StackFlow or global.marketplace_module_Alphish_StackFlow_Process. This allows me to skip one or two struct reads, cutting down the module functionality access time a little.
Admittedly, I didn't do that in functions - though I could probably define similar globals for them. What's important to note about functions, though, is that I defined them as actual functions rather than macros - more or less in line with your thinking that having function declared as macro can be confusing. We still have some overhead of a function call, but again - chances are the function is called rare enough the extra cost is negligible, or the core of the function itself has much higher cost than the overhead.

As far as Asset naming goes - I'm not much hung up about whether the name should start with the package name or rather the entity type. However, using a publisher + package naming rather than just package itself gives some extra safety against packages that are named the same. I guess it's unlikely that a FeatherHeart publisher will make a framework called fhInput, but I can totally see multiple people using prefix like autotiles_*. Consistently adding own publisher name to asset names helps mitigate this issue.
(then again, maybe it's not as important of an issue - as long as people name their packages and prefixes descriptively, it's extremely unlikely you'll need to use multiple packages with fhInput_* or autotiles_* prefix at the same time)

Directory Structure: do we at least agree on putting the package code in Marketplace >> Publisher >> Package folder (or at the very least Marketplace >> Package folder, though there's still some risk of name collision)?
If people follow the convention of putting package core in the Marketplace folder, it ensures the users can import dozens of packages into their project without really cluttering their assets tree, which should make the experience with Marketplace packages smoother. Plus, directory structure should be completely neutral as far as performance is concerned.

I agree it would be good to have some easily accessible and discoverable listing of asset packages conventions/standards. Not really meant to be strictly enforced, but rather to give publishers general direction, that they might or might not follow. Not quite sure if we should go as far as promoting specific style of casing (whether camelCase or snake_case).

As for local packages - since they're local, which implies they are only meant for the current user - we should just let the current user arrange them however they want.

Once again - these conventions are meant to be general opt-in guidelines for Marketplace developers, rather than strict rules enforced by IDE/software.