SOLVED BAD memory leaks with data structures

Neptune

Member
Alright. There are some serious memory leaks with ds_*_write and ds_*_read...
It's not noticeable in tiny data structures, but with big data (especially nested-WRITTEN data grids & lists) 1 to 25 MB can be leaked from a single in-game action.

It's not just ds_grid_resize (with strings being lost when cells are clipped).

I know others have experienced and reported some of this... Has anyone seen or can confirm that Opera/Yoyo even knows about this problem?

Any information is appreciated :)
 

Roldy

Member
Alright. There are some serious memory leaks with ds_*_write and ds_*_read...
It's not noticeable in tiny data structures, but with big data (especially nested-WRITTEN data grids & lists) 1 to 25 MB can be leaked from a single in-game action.

It's not just ds_grid_resize (where strings are lost when cells are clipped).

I know others have experienced and reported some of this... Has anyone seen or can confirm that Opera/Yoyo even knows about this problem?

Any information is appreciated :)
Create a reproducable example. Report it. They will let you know if they confirm it or not, know about it or not, and add it to the bug tracker.

EDIT: If you want help quickly for ways to remedy your problems, then post your code.
 
Last edited:

Neptune

Member
I've reported without an attached project a long time ago, maybe I'll try to isolate it and send in... It's difficult to create a reproducable example.
Posting code is going to waste everyone's time at this point.
 

Roldy

Member
It's difficult to create a reproducable example.
This should be a red flag for you. It simply means you don't know what the actual problem is and are just assuming. You would be better served actually trying to reproduce it. Most likely one of two outcomes will result
  1. You figure out how to reproduce it and show it is a GMS bug and YoYo can now fix it much quicker
  2. You figure out what is actually a problem with your code and you can now fix it yourself much quicker
Even for the first result. Once you understand the scope of the bug you can probably devise a work around faster than YoYo can fix it.
But you have to understand the problem.
 

Neptune

Member
The problem is there is a engine-side bug for cleaning up after reading/writing massive nested data structures 😂
And before I sacrifice anymore of my time to go through Yoyos cumbersome process I wanted to ask if anyone else has info.
 

Roldy

Member
The problem is there is a engine-side bug for cleaning up after reading/writing massive nested data structures 😂
Prove it and demonstrate it and it will be more likely to be fixed.

And before I sacrifice anymore of my time to go through Yoyos cumbersome process I wanted to ask if anyone else has info.
As stated before, your time will be better served understanding the problem and then fixing it/working around it. Do you understand the problem? Can you explain it? Can you reproduce it?

If you want help, post code that demonstrates the issue on the programming forum and I am sure someone will have good ideas.


EDIT: Not sure which part you think is cumbersome. If it is having to make a clear reproducible example then that is something YOU should be doing regardless. When you encounter problems, it is best to understand them before proceeding.
 
Last edited:

Nocturne

Friendly Tyrant
Forum Staff
Admin
And before I sacrifice anymore of my time to go through Yoyos cumbersome process I wanted to ask if anyone else has info.
We can't give more info if you don't give more info. In my experience, properly handled Data Structures don't leak unless you're not cleaning them up correctly (but I confess top not using them for massive amounts of data, so I could be wrong). You are talking about nested data structures, which automatically makes me ask if you are going through and destroying all the contained structures before destroying the container structures? Or do you use the DS mark functions, which should flag nested structures for deletion when the container is destroyed?

As for the "cumbersome" process, it's more cumbersome for someone at YYG to wade their way through your code, which they don't know, trying to find a possible error than it is for you to make a trimmed down version or to give exact reproducible steps to get the issue and say exactly where it is. ;)
 

Neptune

Member
There are no "active" structures contained in this read (they are all written as strings)
This is just an example spot where the data is large enough to quickly see the leak happening.

GML:
var slot_grid = ds_grid_create(0,0); ds_grid_read(slot_grid,global.DS_slots[# 0,number]);
    //grab some cells
ds_grid_destroy(slot_grid); slot_grid = noone;
//MASSIVE LEAK
For this particular leak case, this runs about 40 times for 1 step (grabbing some data from global.DS_slots)

The data being read looks something like this:
1623182855634.png

Some of the data has more than two nests, but the picture is just giving some visual idea of WTF im talking about.

The read that is leaking is containing nothing but ints, bools, and strings (BIG STRINGS from all the written nested ds)
 
Last edited:

Roldy

Member
There are no "active" structures contained in this read (they are all written as strings)
This is just an example spot where the data is large enough to quickly see the leak happening.

GML:
var slot_grid = ds_grid_create(0,0); ds_grid_read(slot_grid,global.DS_slots[# 0,number]);
    //grab some cells
ds_grid_destroy(slot_grid); slot_grid = noone;
//MASSIVE LEAK
For this particular leak case, this runs about 40 times for 1 step (grabbing some data from global.DS_slots)

The data being read looks something like this:
View attachment 40484

Some of the data has more than two nests, but the picture is just giving some visual idea of WTF im talking about.

The read that is leaking is containing nothing but ints, bools, and strings (BIG STRINGS from all the written nested ds)
If the problem is internal it is most likely to do with STRINGS. I have submitted plenty of bugs in many cases where strings leak.

Here is your reproducible example:

GML:
// Create Event
doWrite = false;

// Spacebar pressed event
var _w = 1000;
var _h = 1000;

var _foo = ds_grid_create(_w, _h);

for (var _i = 0; _i < _w; _i++) {
    for (var _j = 0; _j < _h; _j++) {

        _foo[# _i, _j] = _i * _j;
    }
}

if (doWrite) {
    show_debug_message("WRITING");
    var _string = ds_grid_write(_foo);
}

ds_grid_destroy(_foo);

if (!doWrite) {
    show_debug_message("NO WRITING");
    doWrite = true;
}
When you press spacebar the first time it makes a 1000x1000 grid then destroys it. NO MEMORY LEAK
When you press spacebar the second time it makes a 1000x1000 gird, calls ds_grid_write, then destroy it. THIS SHOWS A LEAK.

However, note that further presses do not increase the leak size.

The problem is with strings and how their memory is managed. ds_grid_write/read demonstrate the problem most likely only because they deal with strings.

Submit this.

For a work around, write you own read/write functions.

On A side note this might technically NOT be a leak, even though it looks like one. Internally how they manage heap data for strings is a bit weird. Most likely some dedicated 'string heap' has just increased in size. But who knows. Regardless it doesn't help if you make a 25mb string and the 25mb never gets freed.

EDIT: This should have been posted in the programming forum asking for help on how to reproduce this issue.
 
Last edited:

Neptune

Member
I was mad and went for a walk, and on the walk I was thinking "Maybe I could try to script my own read/write??" -- I think I'll try!

I appreciate the detailed reproduction! @Roldy

(filed the report!)
 
Last edited:

Roldy

Member
I was mad and went for a walk, and on the walk I was thinking "Maybe I could try to script my own read/write??" -- I think I'll try!

I appreciate the detailed reproduction! @Roldy

(filed the report!)
Ahh Man. If I made you mad it is understandable, I know my way of speaking can hit some people the wrong way, and intent isn't easily conveyed on the internet. But I am honestly trying to help.
 

Neptune

Member
Well I wrote up my own read/write system, my project hasnt really made me feel stretched coding wise like that in a while! 😁

My read/write scripts work and fix the memory leaks, the down side is they are very slow compared to the native functions... @Roldy

I will try to optimize them I suppose (mostly the read?) and then if that is still too slow, I can use mostly the native functions and substitute my version in places where having a leak is unacceptable.
 
Last edited:

Roldy

Member
My read/write scripts work and fix the memory leaks, the down side is they are very slow compared to the native functions... @Roldy
I haven't looked into it, but I am assuming that the ds_grid_read handles all the nested structures' ds_*_reads as well. This is potentially create one giant string. Something to consider is to iterate over the nested structures manually and call their ds_*_read directly to reconstruct them. This would possible keep the string size low.

So basically your own read/write routines to handle nested structures but still calling the built in ds_*_read/write functions.

This may get you better performance and avoid causing a large leak.

But like I said... I haven't look into any of that and just spit balling. That may be a horrible idea.
 

Neptune

Member
My scripts handle nests fine assuming they're written as strings and not "live pointers" data structures.
I'll try to optimize and then share, if you're interested at all or maybe it will help someone else~
 

Neptune

Member
Alright I optimized it a bit, but it still needs work 🤔

It will read and write nested grids (that are already strings and not pointers)

GML:
/// @param ds_pointer
function scr_grid_write()
{
    var grid = argument[0];

    var str = "[";
   
    var ww = ds_grid_width(grid);
    var hh = ds_grid_height(grid);
    var type = 0;
   
    str += string(ww)+","+string(hh)+","; //add dimensions for read-parsing
    for(var a = 0; a < hh; a++)
    {
        for(var i = 0; i < ww; i++)
        {
            var data = grid[# i,a];
           
            type = is_string(data); //0 for real, 1 for string
            str += string(data)+"|"+string(type)+"|";
            if i == ww-1 {str += "}";}

        }
    }
    str += "]";
   
    return(str);  
}

GML:
/// @param ds_pointer
/// @param str
function scr_grid_read()
{
    var grid = argument[0];
    var str = argument[1];
   
    var cc = 1;
    var size = string_length(str);
    var ww = 0;
    var hh = 0;
   
    cc++; // [
   
    #region parse dimensions
        var width_str = "";
        while(string_char_at(str,cc) != ",") {width_str += string_char_at(str,cc); cc++;}
        cc++; // ,
        ww = real(width_str);
   
        var height_str = "";
        while(string_char_at(str,cc) != ",") {height_str += string_char_at(str,cc); cc++;}
        cc++; // ,
        hh = real(height_str);
    #endregion
   
    ds_grid_resize(grid,ww,hh);
   
    #region INJECT DATA
        var col = 0;
        var row = 0;
        var sub_str = "";
        var loop = false;
        var char = "";
        var loop_max = size-1;
        while(cc < loop_max)
        {
            sub_str = "";
            if string_char_at(str,cc) == "["
            {
                #region NESTED DATA
                    var nest_count = 0;
                    sub_str = "[";
                    cc++;
               
                    loop = true;
                    while(loop)
                    {
                        char = string_char_at(str,cc);
                        if char == "[" {nest_count += 1;} //another nested DS
                        if char == "]" {if nest_count > 0 {nest_count -= 1;} else {loop = false;}}
                        sub_str += char;
                        cc++;
                    }
                    cc++;
                #endregion
            }
            else
            {
                #region NORMAL DATA
                    loop = true;
                    while(loop)
                    {
                        char = string_char_at(str,cc);
                        if char != "|" {sub_str += char;} else {loop = false;}
                        cc++;
                    }
                #endregion
            }
           
            if string_char_at(str,cc) == "0" {sub_str = real(sub_str);} // covert data type

            cc += 2;

           
            grid[# col,row] = sub_str;
           
            col += 1;
            if col == ww {col = 0; row += 1; cc++;}
        }
    #endregion

}
 
Top