Performance decreases the more levels are generated/the longer the game is played, but it isn't a memory leak/ or instance count.

Fluury

Member
Heya.

The project I am working on uses procedural generation to generate it's levels. In said levels, NPCs and everything spawn. After the level is done, we wipe the current level setup, and create a new one over it.
Sounds simple enough - this isn't a new concept.

The problem I am running into is that at first launch of the game entering the first level, I am greeted with an FPS of 200. After playing the game, generating levels, creating instances etc. etc. and going to the first level _again_ under the exact same Seed for generation, and unfortunately, the FPS decreases. I simulated several "playthroughs" of the first area over and over again (By essentially just spawning every single enemy you'd normally encounter, and then killing them all, then just finishing the level.), came back to the first level under the same seed, and the FPS keeps decreasing. The FPS decrease isn't insanely drastic, but it roughly looks like this:

1st: 200-190 FPS.
2nd: 170-160 FPS.
3rd: 150-140 FPS.

etc. etc.

Obviously, this is a problem as the performance shouldn't change at all given it's the same setup with the same memory usage, same instances active and everything.

First Idea: There must be a Memory Leak! Apart from (afaik) Memory usage only slowing down your game once it reaches a critical point of memory usage, I did some testing.
No matter the amount of simulated playthroughs, the Memory Usage does not go over 70. From there, it sometimes depending on the area (given they vary in sizes...) it sometimes spikes up to 75, and then quickly drops down to 70 again. A memory leak would probably result in something much, much worse. So, unless I have misunderstood something, the performance loss is not caused by a Memory Leak.

Second Idea: I must not be resetting instances properly! Maybe I had forgotten to destroy certain instances, and it resulted in there being more instances active than before? After doing some simply debugging, and unfortunately, even just looking at the built-in variable instance_count, the instance_count stays the same for each first level generated under the same seed, no matter the simulated playthroughs I did before. The instances for the enemies, the walls, the floors, etc. etc. are all the same. So, it isn't that.

Third Idea: I must not be resetting asset layers properly! The generation and collision is built around instances, which means a wall is an instance. To decorate said walls and given the walls don't exactly move, when each wall spawns, they slap some sprites surrounding the wall on an asset layer. A variety of instances in this game (mostly environmental instances which do not move over the course of the level) do this - they put sprites on-the-fly on a specific asset layer. After the level is done, I'd destroy the entire asset layer (thus destroying it's contents) and also recreate it so the next area can use the empty asset layers. After testing the amount of elements which are active on each asset layer, even after several playthroughs, the number would stay the same. So it isn't this either.

And now - I am here, without a clue as to what is causing this. I use particle systems, I use paths, I use audio emitters... but all three of those things should, if I were to, say, just create them over and over again, result in a memory leak, right?

If anyone has any idea, or could even just tell me how to perhaps deal with this more efficiently, I am all ears. Thank you.
 

Fluury

Member
More Information after extensive debugging:

Having compared a version of the game that has been heavily played vs. an entirely fresh one in the exact same level (same seed etc...), there are the differences in what the Compiler is telling me:
This is the fresh, and well performing version of the game - first level generated.


Below, the heavily played level - as mentioned however, they are both operating under the same seed, and have generated the exact same level.

The big difference is that DoAStep is taking 60% longer than before. Outside of that, strangely enough, Finish_Frame is quite fast in the Played Version. How are these related?

Upon further inspection, the big difference maker is DrawRoom. DrawRoom is 4ms slower in the played version. If you then dive a bit deeper, you see that DrawTheRoom is 3ms slower. This is where the lead ends, because interesting enough, the contents of the Profiler are (roughly) the same for either version. Take a look:

Fresh, non-played:

Heavily played:

It's as if the "hidden" 3ms of delay of the first version just casually doubled themselves. What is happening here?
 
I'll try to help but I'm not sure how helpful I'll be. Do you use buffers? Particle effects? If yes, try clearing them if you havent.

One debugging tip is turn off one feature of your game and see if the lag persists then repeat with every feature until the lag disappears. This should narrow down the problem. I know you have been doing this but just keep on doing it. There must be something that's causing it

Sorry if this isn't helpful enough.
 

Fluury

Member
I'll try to help but I'm not sure how helpful I'll be. Do you use buffers? Particle effects? If yes, try clearing them if you havent.

One debugging tip is turn off one feature of your game and see if the lag persists then repeat with every feature until the lag disappears. This should narrow down the problem. I know you have been doing this but just keep on doing it. There must be something that's causing it

Sorry if this isn't helpful enough.
Buffers are not being used. Particles yes, but only for, well, a single bush prop that sometimes farts out a couple of particle effects with a small amount of lifetime. As far as I know, particle effects being the cause of this should also result in a memory issue, rather than the game just gradually dying.

Any help is appreciated. I am doing that already, but I suppose I'll just strip the game of it's elements until only the bare minimum exists :V
 
Are you destroying the particle system when the it's no longer in use (i.e. when resetting the level)? If you're not, I'd hazard a guess that it is indeed the problem, even if it's not manifesting itself in the typical way. Also, any data structures you are using need to be destroyed. There's a list somewhere in the manual that points out all the structures that need to be manually garbage collected, IIRC.
 

Fluury

Member
Are you destroying the particle system when the it's no longer in use (i.e. when resetting the level)? If you're not, I'd hazard a guess that it is indeed the problem, even if it's not manifesting itself in the typical way. Also, any data structures you are using need to be destroyed. There's a list somewhere in the manual that points out all the structures that need to be manually garbage collected, IIRC.
I am aware of having to destroy data structures and have went over every single one being created - they are all axed accordingly, and if not, you'd clearly see a memory spike/ the memory usage would constantly increase over time, which it does not.

The only data structures which are left active after a level is done are a few global ones which get created once when the game boots up, and that's it. Same goes for the particle system actually - there is only a single one, and it gets created at boot. Said global lists are then cleared, and re-filled whenever a new level is being generated, not destroyed and recreated.
 

TheouAegis

Member
Open the Task Manager and expand it so you can see the GPU load. How does the GPU load compare between fresh and heavily-played?
 

Yal

šŸ§ *penguin noises*
GMC Elder
My guess is that there's some internal bookkeeping data that doesn't get cleared, filling the internal instance data with unused garbage that slows down internal loops through assets. It doesn't sound like you're restarting the room to go to a new level, you delete everything and create new things? Consider restarting the room instead, to make sure all volatile data is cleared properly.
 

ph101

Member
Are you destroying the paths? What about surfaces? You can count some o these aspects using something along these lines and then display the values. You probably already checked all these but just in case, may help. (also, disable particle system to see if problem persists - also are you using any 1.4 legacy tile scripts which use layers, I once ran into a "layer leak" related to using one of these, forget which. Store and destoy all layers manually.) Credit to @Daveinhispants for this approach.

if keyboard_check_released(ord("B"))
{

var __layers = layer_get_all();
numlayers = array_length_1d(__layers); //drawn in gui debug info

countsurfaces = 0;
for (var i = 0; i < 1000; ++i) {
if (surface_exists(i)) {countsurfaces++}
}

countgrids = 0;
for (var i = 0; i < 1000; ++i) {
if (ds_exists(i,ds_type_grid)) {countgrids++}
}

countmaps = 0;
for (var i = 0; i < 1000; ++i) {
if (ds_exists(i,ds_type_map)) {countmaps++}
}

countlists = 0;
for (var i = 0; i < 1000; ++i) {
if (ds_exists(i,ds_type_list)) {countlists++}
}

countstacks = 0;
for (var i = 0; i < 1000; ++i) {
if (ds_exists(i,ds_type_stack)) {countstacks++}
}

countqueues = 0;
for (var i = 0; i < 1000; ++i) {
if (ds_exists(i,ds_type_queue)) {countqueues++}
}

countpriority = 0;
for (var i = 0; i < 1000; ++i) {
if (ds_exists(i,ds_type_priority)) {countpriority++}
}
}
 
Last edited:

Fluury

Member
Open the Task Manager and expand it so you can see the GPU load. How does the GPU load compare between fresh and heavily-played?
I investigated and the GPU load is identical, both sitting at 9%~.

Are you destroying the paths? What about surfaces? You can count some o these aspects using something along these lines and then display the values. You probably already checked all these but just in case, may help. (also, disable particle system to see if problem persists - also are you using any 1.4 legacy tile scripts which use layers, I once ran into a "layer leak" related to using one of these, forget which. Store and destoy all layers manually.) Credit to @Daveinhispants for this approach.
Paths are being destroyed. Surfaces aren't really being used a lot, only for the pause screen. Not using the tile scripts.
I ran the code you suggested either way to double check (given I am still paranoid that it might be list stuff, but in a weird way...) and all these values stay the same. Thanks for linking it though, so I can finally put that to rest.

My guess is that there's some internal bookkeeping data that doesn't get cleared, filling the internal instance data with unused garbage that slows down internal loops through assets. It doesn't sound like you're restarting the room to go to a new level, you delete everything and create new things? Consider restarting the room instead, to make sure all volatile data is cleared properly.
This was a pretty interesting theory to me. It was the only thing that made sense to me really given anything else (logically...) wouldn't make any sense.

So I went ahead and painstakingly reworked the entire level generation a bit, and now the "levels" are being generated in a separate "game specific" room instead of one super-room. Said room starts with a single instance in it, that simply generates the level. When the level is done, some global variables get setup differently and we restart the room just like you suggested (using room_restart). I tried two variants, one where I'd wipe all the instances and then restart it, and one where I'd just "let the room" wipe the room of instances.

I was _really_ optimistic about this but unfortunately, still nope. Both variants result in the exact same problem.

If anyone has the slightest idea about this, please please tell me about it. I don't wanna hit up the pity effect but oh man this is getting frustrating. It HAS to be some weird internal spaghetti that is causing this, right?
 

ph101

Member
This is a weird one then. How many instances being made and destroyed are we talking about here out of interest? And what time period?

Also, is this VM or YYC? Does it happen in both? This is windows right? (also a very unlikely side thought: what sleep settings are you using, could it be related? And are you testing on a laptop wher the battery runs low and it switches to low power settings..? :D)

Did you disblae particles and it reproducs? What if you comment out the path gen entirely (if its not impossible?) as the code I posted doesn't verify number of paths. I'd be interseted in what your are doing with layers and if you can disable whether if it still reproduces. What about audio emmittesr you mentioned. You need to test removing creation of all of those and see if that is culprit if you haven't already (I didnt see from you comment that you have actually done all of that - I saw you have said "paths are detsroyed", "audio is destroyed", but then that's usaully where the worst bugs come from, assuming and being certain something is happening when actually it isn't hence me asking about the tests by removing/commenting out :D).

If it's still reproducable you should submit it to YYG as a bug with the project attached and see what they can see as I suppose if its v high numbers of instances its some sort of trash collection bug along the lines Yal said, that is not reset room change. Let us know what you find.
 
Last edited:

Fluury

Member
This is a weird one then. How many instances being made and destroyed are we talking about here out of interest? And what time period?

Also, is this VM or YYC? Does it happen in both? This is windows right? (also a very unlikely side thought: what sleep settings are you using, could it be related? And are you testing on a laptop wher the battery runs low and it switches to low power settings..? :D)

Did you disblae particles and it reproducs? What if you comment out the path gen entirely (if its not impossible?) as the code I posted doesn't verify number of paths. I'd be interseted in what your are doing with layers and if you can disable whether if it still reproduces. What about audio emmittesr you mentioned. You need to test removing creation of all of those and see if that is culprit if you haven't already (I didnt see from you comment that you have actually done all of that - I saw you have said "paths are detsroyed", "audio is destroyed", but then that's usaully where the worst bugs come from, assuming and being certain something is happening when actually it isn't hence me asking about the tests by removing/commenting out :D).

If it's still reproducable you should submit it to YYG as a bug with the project attached and see what they can see as I suppose if its v high numbers of instances its some sort of trash collection bug along the lines Yal said, that is not reset room change. Let us know what you find.
The first level operating under the seed, without any optimization done on my part, would have 33983 Instances. However (!) a massive portion of these instances get "turned" into sprite assets, and then destroyed. The result is a level with 8000~ instances. The only "global" controller instances which always remain active are 8.

I'm afraid I am not familiar with what you mean with YYC or VM - I'll read up on that later (It's almost 1AM in europe...) and report on my findings tomorrow.

It is not particles of paths.

I spent the last few days breaking a version of the project apart, thinner and thinner, with less and less features. When the FPS values go "really high" given there is way less stuff going on it becomes hard to estimate a performance loss over time I feel like, but it naturally came down to the script that creates a bunch of instances in a single step. I even went back to the ancient versions of the project I had saved up as backups which did not have any of the fancy stuff like audio emitters or particle effects, same deal.

I did a test project in which I essentially created several thousand objects and then deleted them over and over and over again but the original FPS value did not budge.

The logical next step is to, I guess, try to "widen" the timeframe the instances get created (and acted upon), see if that fixes anything, and if all fails and none of you wizards have an idea what's going on, I suppose the last thing I could do is wait for 2.3, pray that fixes something, and otherwise see this as a cosmic sign that maybe instance collisions aint it.

But I won't give up just yet!
 

ph101

Member
errrr whut? 33983 instances? To put it mildly that suggests to me some pretty unsound game pattern/architecture...

massive portion of these instances get "turned" into sprite assets
Perhaps using (I assume) layer_sprite_create 25000 times is creating a previosuly unknown leak? Is that what you are using? if you remove that step and the destroying of them (are you destroying the sriate assets?), does the issue reproduce?

Anyway, I still suggect to comment out the various features until the issue doesn't reproduc. As you havent answereed my other questions that, and about commenting out the various elements, I won't continue. But I have no reason to think 2.3 will affect this, be it instance collisions (I dont think we talked about those anyway?) or not. Ulitmately, I'm not sure how far along in your project you are but have 33k instances seems a really bad idea so I would look into that.
 

Fluury

Member
errrr whut? 33983 instances? To put it mildly that suggests to me some pretty unsound game pattern/architecture...


Perhaps using (I assume) layer_sprite_create 25000 times is creating a previosuly unknown leak? Is that what you are using? if you remove that step and the destroying of them (are you destroying the sriate assets?), does the issue reproduce?

Anyway, I still suggect to comment out the various features until the issue doesn't reproduc. As you havent answereed my other questions that, and about commenting out the various elements, I won't continue. But I have no reason to think 2.3 will affect this, be it instance collisions (I dont think we talked about those anyway?) or not. Ulitmately, I'm not sure how far along in your project you are but have 33k instances seems a really bad idea so I would look into that.
..These instances don't stay active and get axed later on, and are mostly there for generational purposes. Don't really see what's wrong with it as it isn't causing any performance issues (Atleast I'd like to think so? Tested something like that in a test project).

Also, I did answer your questions: I went back to versions which did not have audio emitters and the like. I commented out so much stuff that the only thing left in the game was was quite literally the "skeleton" generational process and the script that creates the massive amount of instances. Yes I am destroying the sprite assets. Yes I tried it without turning them into sprite assets and just axing them as usual. Commenting out the script that creates a bunch of instances from the skeleton seemed(?) to be "fixing" the issue, although without that script there basically isn't a level. It must be bound to the creation process of a bunch of instances in a single step of that script, which is why I said I'd look into that today as that is the only idea I have as to how to deal with this...

I commented out the script that turns said instances into sprite assets, so it isn't that either. It did not matter if I was to simply kill the instances, and then replace them, or instead simply only kill them, either way the performance would slowly deteriorate. That's why I said it "came down to the script that creates the bunch of objects".

I also already tried reducing level size (which would reduce the 33k instances to a meager 8k (?) at worst at level gen, and to like 1500~ instances active at "playtime") and it still resulted in the issue happening, albeit a bit slower.

EDIT: And no... I am not testing on a laptop... nor is it anything power related as far as the computer goes.
 
Last edited:

Nocturne

Friendly Tyrant
Forum Staff
Admin
This is a really interesting issue, and obviously something that should be investigated... which is hard for us to do through the forums - especially as it seems you really have covered all the bases - so I have to ask if you'd be willing to share a link to a test project so that we can test on our own machines and dig around in the code too?
 

Fluury

Member
This is a really interesting issue, and obviously something that should be investigated... which is hard for us to do through the forums - especially as it seems you really have covered all the bases - so I have to ask if you'd be willing to share a link to a test project so that we can test on our own machines and dig around in the code too?
I will investigate the script further today, try implementing alternatives, spreading out the creation over a few frames instead of a single one etc. and if that doesnt fix anything, I'd create a focused test project undergoing the same methods used as in the real project and see if the problem persists.

If yes, I'd probably bunk out a test project for everyone else to screw with, so: Update later today.
 

Fluury

Member
Alright. I'm back after a bunch of investigation, and some updates:
- It is not the script.
- It is, in fact, the basic generation that seems to be causing the performance to become worse and worse.

So I went ahead and essentially ripped the generation code out of the game and awkwardly placed it in a test project. Some stuff is pretty nasty to look at, and there are a bunch of values flying about which arent "needed" but I left them in given getting rid of them would just be unnecessary effort given it won't exactly change anything.

I let it run and perform on seed 0 - on my machine, after the "skeleton track" was generated, I was sitting at 2900~ FPS.

I've put in a small controller object that simply gets rid of the current floors and then spawns a new track generator if auto_generate is turned on. F1 will turn it off/on, while F2 will merely set the seed to random, or 0.

To replicate the issue I am experiencing, do the following:
- Open the game, and let it generate the first track for seed 0. Note down your current FPS.
- Press F1. Now, let it keep generating tracks for 5 minutes or so.
- Press F2, let it generate the track for Seed 0 and before it's done, turn off auto_generate with F1.
- Take a look at your FPS. It should be lower than before.

You can continue this process as much as you want to, have it generate tracks for 5, 10, 20 minutes, the longer you wait/the more tracks are being generated, the worse and worse performance gets.
NOTE: This is not using room restart or something like that, given in the original project I tried that out, and it didn't fix anything.

-> https://www.dropbox.com/s/5jrjb47q14rd4co/GenerationPerfTest.rar?dl=0 <-

I'm basically at the end of my abilities here, and I really hope one of you wizards have any idea whats going on.

EDIT: If you have time, just let it run for 20-15 min, the difference in performance is massive.
 
Last edited:

Fluury

Member
Updates:
- In the main project I ended up having a couple of enums in creation events of objects which get recreated over and over. I ended up slapping all of these enums in a script that I call once, speculating that maybe that was related. Nope.
- Another thought I had was that maybe (?) it was the recursive way the generators are laid out. Nope, still screwed.
 
Top