GameMaker [solved] Testing GM code / resources

D

Deleted member 45063

Guest
Hello GMC,

I am struggling with the whole testing topic in GMS. The way I see it any setup for this would need to take into consideration:
  1. Separation of resources just for testing and production resources
  2. Testing code should not be included in the final game
  3. Developer experience should be pleasant
  4. Setup should not be overcomplicated / should be stable across GMS updates
I have managed to create a setup that allows me to perform Unit Tests in GMS but it is not perfect as it violates point 4 and point 3 is not the best either (especially in a computer with a low amount of resources).

In a nutshell the process I have consists of:
  • A "base" project with all of the production resources except rooms
  • A "main" project that inherits from the "base" project and adds the rooms for the final game
  • A "test" project that inherits from the "base" project and adds the room(s) and other resources for running the tests
This solution:
  1. Provides complete separation of production resources and test resources as they live in different projects
  2. All of the testing code is contained in the "test" project
  3. Involves having between two to three IDE instances open if you want to regularly switch between developing production resources and running tests, which is a pain in computers with low resources
  4. Consists of a lot of manual hackery with the project files just to allow this setup to be runnable at all, plus I only tested it on non-YYC Windows targets without even performing the export of the game, so it isn't even guaranteed to work across the board
Therefore I would like to know what alternatives are there / what other developers are doing in this regard.

Thanks in advance!

P.S.: I can explain / share an example of my setup if anyone is interested on the internals of it.
 
C

Catastrophe

Guest
Hey :D I've done unit testing in a corporate job, and my personal opinion is that unit tests aren't worth it for indie game development, by a very wide margin. Unit tests absolutely exploded development time due to maintenance, they don't save it by catching bugs. The one thing they are good at is catching a few bugs in a high risk environment where a single bug can have devastating consequences. The only case I can think of where they would have reasonable use in game development is a multi-million dollar multiplayer game app, with in app purchases and no open beta branches.

I know your question isn't whether unit testing is worth it, but I've got a very strong opinion on this, having done many unit tests.

As for whether it should be stable across updates, I don't think this should be a big concern: In general you shouldn't update GMS while working on a project anyways.
 
D

Deleted member 45063

Guest
Hey :D I've done unit testing in a corporate job, and my personal opinion is that unit tests aren't worth it for indie game development, by a very wide margin. Unit tests absolutely exploded development time due to maintenance, they don't save it by catching bugs. The one thing they are good at is catching a few bugs in a high risk environment where a single bug can have devastating consequences. The only case I can thing of where they would have reasonable use in game development is a multi-million dollar multiplayer game app with no open beta branches.

I know your question isn't whether unit testing is worth it, but I've got a very strong opinion on this.

As for whether it should be stable across updates, I don't think this should be a big concern: In general you shouldn't update GMS while working on a project anyways.
I can agree on the added development time, but in this case Unit Testing is just one example (which would be more useful for components that you'd want to just plug and play in different projects). This could also apply to testing e.g. object behavior of core parts of your game. Or generally as a proof of concept of the complete separation of resources between different "views" of the same project. It's also an exploratory project to see if it would even be feasible in GMS, so I guess the biggest benefit is just a better personal understand on limitations of GMS.
 

Kezarus

Endless Game Maker
I can agree on the added development time, but in this case Unit Testing is just one example (which would be more useful for components that you'd want to just plug and play in different projects). This could also apply to testing e.g. object behavior of core parts of your game. Or generally as a proof of concept of the complete separation of resources between different "views" of the same project. It's also an exploratory project to see if it would even be feasible in GMS, so I guess the biggest benefit is just a better personal understand on limitations of GMS.
Yeah, just like we are talking, I do the same as you do. Setup a test room and run the scripts and objects that I want to test and dump the results in the console.

I too see the usefulness of Unit testing (even better without mocks!). And I bet that someone around the forum had this issue solved in a better way. =]
 
D

Deleted member 45063

Guest
Yeah, just like we are talking, I do the same as you do. Setup a test room and run the scripts and objects that I want to test and dump the results in the console.

I too see the usefulness of Unit testing (even better without mocks!). And I bet that someone around the forum had this issue solved in a better way. =]
Not sure if you are talking about the same, but I set up a test room in a completely separate project that inherits the original project's resources. That way no test code / resources (rooms included) are ever seen from the main project. But yes, this way is extremely hacky so someone probably has a better way to go about it (I hope).
 

Kezarus

Endless Game Maker
Not sure if you are talking about the same, but I set up a test room in a completely separate project that inherits the original project's resources. That way no test code / resources (rooms included) are ever seen from the main project. But yes, this way is extremely hacky so someone probably has a better way to go about it (I hope).
Wops, not at all. I do that in the same project, hehe. It's transparent to the user and it's just code, so it's very small in comparison to the rest of the project.

I am very curious to know how it should be properly done without any hacks or workarounds. =]
 
D

Deleted member 45063

Guest
Wops, not at all. I do that in the same project, hehe. It's transparent to the user and it's just code, so it's very small in comparison to the rest of the project.

I am very curious to know how it should be properly done without any hacks or workarounds. =]
It is just code, but during the tests you might need to change the configuration of the game in different ways or you might even want to test resources other than code in which case it isn't really feasible to include all that in the same project.
 

Ido-f

Member
It is just code, but during the tests you might need to change the configuration of the game in different ways or you might even want to test resources other than code in which case it isn't really feasible to include all that in the same project.
Maybe you can add anything you'd put in the test project in separate "Tests" folders in the resource tree, and work with source control to safely revert any changes you did while testing?
When reverting you should be able to choose to revert only certain files I believe (Source Control->view history->choose a revision->choose files->right click->revert path seems like it should work, but I never tried it, only reverted complete commits).

edit: yea, no, GM's source control is buggy as a whole, I tested it and only reverting to certain commits as a whole seems to work.
 
Last edited:

samspade

Member
I think just having a separate test folders and version control is the easiest way to do it. It meets all your original requirements. Also, I recommend never using GMS built in version control as it has a tendency to crash and be buggy. Although maybe if enough people complain it'll get fixed. Theoretically, using version control with multiple branches would be the best, but again GMS has some issues with that too.

Otherwise, I use GMAssert for unit testing along with a separate object to run the tests (although like Catastrophe I think unit testing has only a limited place in game development).

This concept is also helpful:

 
D

Deleted member 45063

Guest
Maybe you can add anything you'd put in the test project in separate "Tests" folders in the resource tree, and work with source control to safely revert any changes you did while testing?
When reverting you should be able to choose to revert only certain files I believe (Source Control->view history->choose a revision->choose files->right click->revert path seems like it should work, but I never tried it, only reverted complete commits).

edit: yea, no, GM's source control is buggy as a whole, I tested it and only reverting to certain commits as a whole seems to work.
I think just having a separate test folders and version control is the easiest way to do it. It meets all your original requirements. Also, I recommend never using GMS built in version control as it has a tendency to crash and be buggy. Although maybe if enough people complain it'll get fixed. Theoretically, using version control with multiple branches would be the best, but again GMS has some issues with that too.

Otherwise, I use GMAssert for unit testing along with a separate object to run the tests (although like Catastrophe I think unit testing has only a limited place in game development).

This concept is also helpful:

Thanks for your replies!

The thing with source control is that GMS doesn't follow a hierarchical storage of it's resources AFAIK, everything is stored with a UUID in a flat structure. So this makes source-control based solutions not work very cleanly. One could always separate test commits from main commits and automatically revert them but that would be quite hacky as well.

Regarding macro use, I do use it extensively exactly for compile time optimizations of code but I wasn't aware that you could actually scope them to a configuration! This makes my previous attempt of relying on it for tests much more versatile. Although that it still doesn't take care of resource clutter completely.
 

Mool

Member
I didnt read the thread, just a few lines. I use tests too. Small tests, which are fast to edit and create. And it was worth it!. I found like 10 huge bugs. I test things like. Random input, save load stuff, etc and see if it crashes etc
 
D

Deleted member 45063

Guest
Marking this topic as solved since I decided to create a GameMaker Unit Tests framework. Still early in development but should be usable if anyone else is interested in giving it a go.
 

samspade

Member
I took a look at it. It's interesting. It seems significantly more complicated than just creating an object and writing your own tests in it. But I also admittedly did not entirely understand how to use it.
 
D

Deleted member 45063

Guest
I took a look at it. It's interesting. It seems significantly more complicated than just creating an object and writing your own tests in it. But I also admittedly did not entirely understand how to use it.
It's more complex yeah since it tries to mimic features from popular testing frameworks like JUnit (test discovery, ability to do different things at different points of the test's lifecycle, etc). The point is that it works solely based on scripts (which themselves can then do whatever they want - load sounds, create objects, etc) and that they are automatically discovered so you can just put all of your test scripts under the same group in the IDE. When you want to export a Production build you can just delete the group and export the game, then undo the deletion to have your tests back.
 
Top