Behavior of ds_list_clear() with Nested Lists

TheMagician

Member
I read a JSON string into a ds_map using json_decode().

Inside there is a list of lists. Here is an overview over the hierarchy and the indices that GMS assigns to the data structures:

{map id 0
..[list id 3
....[list id 0]
....[list id 1]
....[list id 2]
..]
}


So far so good.

Now I copy list 3 into a new list using

ds_list_copy(list_new, 3);

This works as intended. The original list (id 3) gets copied into a new list (id 4) which now also contains the references/indices of lists 0,1,2.

However, when I now do

ds_list_clear(list_new);

GMS will not just remove the values 0,1,2 from the new list but will also destroy/free the corresponding lists!

Is that intended behavior?

I would have expected such a behavior from ds_list_destroy() because it also destroys any nested data structures but from my understanding ds_list_clear() should only empty the slots in the list?
 
Last edited:

samspade

Member
This sentence can't be true unless you have a typo: "This works as intended. The original list (id 3) gets copied into a new list (id 4) which now also contains the references/indices of lists 0,1,2." If you did it as you wrote, then you would be copying, what is presumably a blank list, into list 3. Instead, it should be:

Code:
ds_list_copy(list_new, 3);
See the manual. That sentence aside, if you created a new blank list and then copied the contents (nothing) of the blank list into list three and then cleared the presumably already empty list you would get two blank lists.

Since list ids are just numbers you could verify whether or not those lists that disappear still exist by using ds_exists(1, ds_type_list) right after the copy and clear.
 

TheMagician

Member
That was indeed a typo here in my pseudo-code. I've corrected my original post.

And I know for a fact that the referenced lists get destroyed/freed because I checked using ds_exists(). My question is: is this a bug or intended behavior?

Also, what is the workaround if I want to free the new list (id 4) from memory without freeing the original nested lists?
 

Ido-f

Member
Maybe you destroyed the upper level ds_map somewhere in between? That also destroys every list and map nested within it.

Alternatively, maybe the resulting lists after json_decode aren't actually id #0,1,2? Even if they were when you Json_encoded them, Json_decode assigns new ds id's for them (as Json_encode doesn't save the original id's, only the ds' contents)
 

kupo15

Member
Yes! I ran into this too and it doesn't make any sense to me either. I think clear should simply remove the pointer references from the list and not destroy the nested lists. I filled a bug report weeks ago and haven't heard back yet
 

TheMagician

Member
@Ido-f: I've done my research and narrowed it down to ds_list_clear() being the culprit as described in the original post.

@kupo15: Glad to hear I'm not the only one with that problem. I think I will also file a bug report to back you up on this. There's nothing in the manual that indicates this behavior.

Did you come up with a walkaround? What seems to work for me is that I first iterate through the list and set all of the entries to undefined and only after that I call ds_list_destroy().
 

kupo15

Member
@Ido-f: I've done my research and narrowed it down to ds_list_clear() being the culprit as described in the original post.

@kupo15: Glad to hear I'm not the only one with that problem. I think I will also file a bug report to back you up on this. There's nothing in the manual that indicates this behavior.

Did you come up with a walkaround? What seems to work for me is that I first iterate through the list and set all of the entries to undefined and only after that I call ds_list_destroy().
I'm happy to hear I'm not alone as well. No I haven't gotten far enough to attempt a workaround, I just went a different direction entirely. I like your workaround for the issue for the moment.
 

samspade

Member
It does seem that ds_list_clear will destroy nested lists, but only if they are marked as nested lists. You can show this with the following code in the debugger:

Code:
main_list = ds_list_create();
list_a = ds_list_create();
list_b = ds_list_create();
list_c = ds_list_create();

ds_list_add(main_list, list_a, list_b, list_c);
ds_list_mark_as_list(main_list, 0);
ds_list_mark_as_list(main_list, 1);
ds_list_mark_as_list(main_list, 2);

copy_of_main_list = ds_list_create();
ds_list_copy(copy_of_main_list, main_list);

ds_list_clear(copy_of_main_list);
list_exists = false;
list_exists = ds_exists(list_a, ds_type_list);
list_exists = ds_exists(list_b, ds_type_list);
list_exists = ds_exists(list_c, ds_type_list);
If you remove the three ds_list_mark_as_list lines, then the three sub lists will still exists. You can also just do:

Code:
main_list = ds_list_create();
list_a = ds_list_create();
list_b = ds_list_create();
list_c = ds_list_create();

ds_list_add(main_list, list_a, list_b, list_c);
ds_list_mark_as_list(main_list, 0);
ds_list_mark_as_list(main_list, 1);
ds_list_mark_as_list(main_list, 2);

ds_list_clear(main_list);
list_exists = false;
list_exists = ds_exists(list_a, ds_type_list);
list_exists = ds_exists(list_b, ds_type_list);
list_exists = ds_exists(list_c, ds_type_list);
And get the same results. In other words, ds_list_clear on nested lists marked as lists will delete the list. This does seem like it should be considered a bug or at least a failure in documentation (I can half understand why you might want this behavior as clearing a list with sub lists might forever lose you the reference to the list and therefore cause a memory leak, but this seems like a bad way to solve that problem since it prevents you from doing things you might actually want or need to do).

When you copy a list with sub lists marked as lists, the marked as list data does carry over, which is why ds_list_clear on the copy list also destroys the sub lists.
 

kupo15

Member
I got a reply from my bug filing of this

Dan said:
There are a few things going on here, so I will have to get this reviewed fully to confirm exactly what is correct.

The short answer is "if you don't want GM to automatically manage the child lists, either don't mark them or remove their entry in the parent list before you clear the parent. Then your code will work fine."

However, as you say, clearing the parent is destroying the marked child lists - it's not clearing them. This is the main bit which will need to be reviewed.

Additionally, I can see that right now the manual implies (pretty much says explicitly) that cleared lists would be filled with "undefined" values, which is not the case - the list is completely cleared and so has a size of 0. I will need to confirm if we simply need to update the manual here, or if an in-game bug is going on.

I have attached a new sample to show all this.

And finally, there is no mention in the manual that the _clear() function will do the same as _destroy() and be applied to "marked" child structures also - that's the confusion here. We do have another report to add a clarification onto those specific pages about "mark" and how this affects memory management, so we will do this anyway.

I will update you further and close the ticket once we have reviewed this functionality and I have entered the required bug reports. For now, however, you can avoid this issue by not marking / removing the entry in the parent list, as I mentioned above, so you don't need to wait for any fix before you can carry on with your game.
Also is there no way to "unmark" marked lists? It seems like that would be a helpful feature
 

TheMagician

Member
Here's my answer from the bug filing:

Please do not knowingly send us duplicate tickets to "back up someone else's report" - this will only slow us down and cause that person to wait longer for a reply.

We do not work off "read this GMC discussion", as you can see others on that forum have already said that this is a manual issue and that they would expect the clear to remove the sub-lists in order to avoid memory leaks and that if you can modify your code right now to get the result you wanted, so that thread is not "proof" of a bug. This is also why we ask for samples to be added onto tickets - it removes any confusion as to what is going on in your code and allows us to determine very quickly if there is a genuine bug or just a coding issue in the sample, which we can then inform you about / make a manual change to stop others having the same confusion.

Please be assured nothing on our bug-reporting form is "unnecessary" - even in this small example, the compiler log would allow us to know you are actually using the correct version of the GMS2 runtime, that your machine built the project correctly, and if your code was causing any output debug messages or errors to be shown. In future, please do not deliberately send us false information - again, this will only slow us down, which causes the situation you're seeking to address by sending us duplicate reports...

Filling in the form correctly allows us to deal with tickets much more quickly and efficiently, so we can get a result back to you sooner.

I will close this report and continue dealing with the original report sent in by the other user, as they have added a sample and more detailed information on the issue directly in the ticket - right now, that clear is affecting all "marked" child lists/maps is intended behaviour and this is just a manual issue, but we will review this in-game behaviour fully.

I saw that you also requested a "manual addition" for extra information on the actual destroy/clear function pages to clarify this situation in a separate ticket - that is entirely correct/agreeable and we will do this, so I have already assigned that ticket to be addressed for the next release.
I don't think that "others on that forum said that they would expect the clear to remove the sub-lists in order to avoid memory leaks" is really the take-away from this thread. Anyway, if the manual gets updated with the information on how to correctly clear a list without freeing the nested lists then I guess I'm happy.
 

kupo15

Member
ouch sounds like you murdered a kitten or something! Yeah i don't see where memory leaks came from either. What should be the expected outcome of this function? I originally thought that only the target list should be cleared with the nested in tack, but it sounds like perhaps this should be clearing everything that is nested as well? In which case, it would seem like your undefined workaround is the way to go
 
Top