Feedback GMS:2 Memory Management Inquiry/Suggestion

[kupo15]

I'm not sure if there is a "suggestion" thread for features since we are in beta but I have a big question about what the memory management system will be like in GMS:2. I was hoping to have this quelled by the announcement as well as other tech specs but it wasn't.

From what I understand in 1.4 there is a sprite_add/delete for external loading (as always) but this creates a texture page for each call which inherently is not good.

1.4 added texture atlases which significantly optimizes memory usage and the more recent texture_flush working for windows gave more control to managing VRAM...BUT this does nothing for the actual ram resources take up which is capped at around 1.7GB.

So as good as 1.4 improved memory we are still limited by that 1.7GB cap by keeping resources internal. The only way to get the best of both worlds is to create your texture pages using the "preview texture pages" and moving them into your external folder. Use sprite_add/delete to manage them and painstakingly find the coordinates on those pages to reference when you use draw_sprite_part.

Is GMS: 2 going to improve on this even further and give us proper external loading support for big, serious games? If not I ask that yoyo please include functionality that outputs all the coordinates and references to a text file that point to each sprite packed in the atlases. GM holds all of this pointer information when it compiles it textures otherwise it wouldn't be able to draw the sprites properly so please open this process up to us like you did with the drawing pipeline

@Mike
@ShaunJS
@Nocturne
@rwkay


(please correct me if I'm wrong here)

 

[JeffJ]

Just want to show support for this. This would be a great help, especially for trying to add extensive graphical modding support to a game. It's doable, yes, but with a few helping hands like mentioned above, it would be so much easier.

 

[FrostyCat]

This suggestion isn't new: #0017212

Maybe with the 1.x card off the table, it would stand a chance this time?

 

[kupo15]

The RAM cap is a 32 bit thing.

Your 1.7 GB is really more like 800 MB Max as GM makes copies all over the place.

yeaaaa, about that haha

This suggestion isn't new: #0017212

Maybe with the 1.x card off the table, it would stand a chance this time?

Yep which is why I'm bringing it up again. I've asked this directly in another GMS2 thread and before in 1.4 and feel like its intentionally getting skipped over which is why I'm making it its own post. I hope that wasn't the case and hope to get some sort of answer here (and went crazy on the tags haha) but yea...new studio, new ground up internals I'm hoping this gets the attention it deserves

 

[kupo15]

With external texture loading, I'd expect each texture to naturally be on its own page.

GMS, creates the texture pages (from internal textures) at build time. To calculate this with external textures at runtime would be a very ugly solution.

What's stopping you from creating your own texture pages externally and use the appropriate UV's at runtime?

That is my point...don't you need to know the UVs of each sprite on the page and manually type them in? I know there is a get_uv function..I guess you can use that to streamline things? And you also need to get the origins as well and program that in. I was hoping that all that info can be generated when you hit the "preview" button in the GGS so we can copy paste that information easily.

Now that you mention it, is there a fairly easy way to do this ourselves? Loop through all the sprites, use the get_uvs and the xy_offset functions and write them to an ini file then use those when using draw_part?

 

[kupo15]

yea but texture pages that are 4096x4096 have tons of sprites on them and packed for efficiency...you will never get such clean numbers with the "random" looking order the sprites are placed on there.

 

[Juju]

Use Braffolk's very good sprite loader.

If you absolutely have to have everything loaded from sprite atlases, I recommend taking the output from this sprite loader and bunging it into JSON (it's a shame the core structure isn't a JSON tp begin with, I'll never know exactly why Braffolk uses 1D grids deep down the tree). You'll need to juggle things a bit on import.

 

[Shamanovitch]

The problem is external adds take 4 times more ram than internal resources. What is the point of using external things if you can put 4 times less resources for the same amount of ram ?

 

[kupo15]

I'll have to look at this more closely when things become more dire, I know I can definitely make my own workaround with a little bit of help. I would just like to have it done natively instead of using extensions for example for compatibility reasons. Does anyone know if the consoles work with dlls?

 

[mMcFab]

I'll have to look at this more closely when things become more dire, I know I can definitely make my own workaround with a little bit of help. I would just like to have it done natively instead of using extensions for example for compatibility reasons. Does anyone know if the consoles work with dlls?

Like the debug output console? Yes, DLL's do work with that, for C++ dlls, just call "AttachConsole(ATTACH_PARENT_PROCESS);" at some point in the dll, and printing debug messages will just show in the console in the IDE (or at least, it does in 1.4). It's handy to do this when working with a dll anyway so you can test it with a debug console before integrating it with GameMaker.

 

[JeffJ]

Rubbish

I hope you're joking and I just don't get the joke.

http://gmc.yoyogames.com/index.php?showtopic=669935&page=2#entry4931556

If so, disregard this post. If not, that link confirms that it isn't "rubbish".

 

[Mike]

Internal and external resources take up the same memory, its just that internal resources fit many sprites onto a page, external ones do not. If you pre-pack your sprites onto a TPage, it'll be the same.

Ideally, you'd have something that pre-packs them and creates a PNG along with a small data file, youd then load both and use the data file to draw "sections" of your packed TPage.

 

[JeffJ]

So what about the compression that Russell talks about in the link I posted? Is that no longer relevant?

Here is a direct quote from Russell:

When the asset is present in our WAD file (containing the compiled game assets then we store the RAM version as a PNG so that is compressed) when we load an external image then we keep it as a texture that is full 32bit this is then given to D3D which will keep an in memory image and a VRAM copy as well so the memory usage is much higher for externally loaded images.

This is not a simple thing to change and I would highly encourage you to use pre-compiled images as much as possible.

Sorry - just couldn't help it.

 

[Shamanovitch]

Ok so nothing actually changed from GM1 right ? External adds still eat a lot of ram ? Or am I rubbish ?

And please stop feeding the confusion between ram and vram. We are talking about the ram here, the vram management is already really good, the ram management is very poor. It is not normal that you have to keep 100% of your assets in ram in moment of a game which you only use 10-20%.

Are you planning to make a real texture group system which would work like audio groups ? With a proper complete unload from memory and load when needed. That would be real next-gen 2D.

 

[rwkay]

Loading external resources has not changed between GMS1 -> GMS2 and is unlikely too - we have to keep a copy in main RAM of the original so it can be re-uploaded, we do this in the WAD (though it is kept as a PNG so does take up less RAM) - because of the generalities of sprite_add and what it will accept the source file is discarded and we keep the full 32bit decoded image in RAM (we have no plans to change this behaviour, it is the penalty for using sprite_add)

Russell

 

[Shamanovitch]

Thank you for your answer @rwkay

So I guess we won't have a new texture group loading in ram system either ? (Like, discarding and adding texture groups into the WAD or something)

Now that GM2 is out and the answer is clear, it means we cannot expect this to come out. Thanks

 

[Mike]

They did in fact used to be the same, but yeah...Russ is right (Oooo that hurts...) that we managed to optimised out the WAD one so we didn't need to keep it's copy around.

 

[rwkay]

@Shamanovitch - I am not saying that we will not do it, just that is not how it works at the moment - we have plans for more flexibility in the external loading system but we have not finalised (or scheduled) them at all so we are not in a position to discuss them just now, this will change in the lifetime of 2.x.

As I have said on the forums elsewhere 2.0 is a start on a journey not a destination - we have a lot more flexibility available to us in the IDE -> Compiler -> Runner chain and we aim to expand on that through the lifetime of 2.x

Russell

 

[Shamanovitch]

Ok nice and thanks for the answer guys

I think a cool suggestion would be :
- to have a way to compile external things on add, or maybe put a format limitation if that means it can be compressed.
- loading and unloading texture groups into the WAD, when I started using GM I thought that was how it works, so the texture groups would make a lot of sense with something like this (and the need for external adds would be cut if internal resource weren't all loaded in ram at game start)

I'll try to find another solution, if someone is available to make this system for me as a custom DLL or something, and is reading this, please PM me that will be a paid work. I have to pause my game dev for the moment then.

 

[zbox]

Would be great to see this happen. New HQ DLC assets are relatively impossible as of now (mobile), also can vouch however for Braffolk's thing if you do want to try. Its fantastic.

 

[kupo15]

Yes! So happy to see some good discussions. Thank you @Mike and @rwkay for coming here to speak with us!

Ideally, you'd have something that pre-packs them and creates a PNG along with a small data file, youd then load both and use the data file to draw "sections" of your packed TPage.

Yes, this is exactly what I'm hoping for if there isn't any plans for something better. Some functionality that makes this method easier for us to use

@Shamanovitch - I am not saying that we will not do it, just that is not how it works at the moment - we have plans for more flexibility in the external loading system but we have not finalised (or scheduled) them at all so we are not in a position to discuss them just now, this will change in the lifetime of 2.x.

As I have said on the forums elsewhere 2.0 is a start on a journey not a destination - we have a lot more flexibility available to us in the IDE -> Compiler -> Runner chain and we aim to expand on that through the lifetime of 2.x

Russell

This is all I needed to hear, that there are plans and hopes to improve external loading. I'm not worried that it won't happen right away and understand that GMS2 is a journey instead of an end product. So happy to hear some optimistic news about this!

Ok so nothing actually changed from GM1 right ? External adds still eat a lot of ram ? Or am I rubbish ?

And please stop feeding the confusion between ram and vram. We are talking about the ram here, the vram management is already really good, the ram management is very poor. It is not normal that you have to keep 100% of your assets in ram in moment of a game which you only use 10-20%.

Are you planning to make a real texture group system which would work like audio groups ? With a proper complete unload from memory and load when needed. That would be real next-gen 2D.

I know it sucks but even if it does double the ram value, if you are only using 10%-20% it wouldn't effect things anyway. It would only effect things if you needed to load a LOT of assets haha