OFFICIAL Copy on Write Behaviour for arrays - possible removing it?

[GMWolf]

Totally. We would have to have access to pointers for that.

Not as user-friendly as pointers, but you can use variable_instance_set(instance_id, name, val); if you absolutely need it (alongside the other variable_instance* stuff)

Easier would be to wrap that variable in a struct. Not super clean but that's what you would do in say, java.

 

[RichHopefulComposer]

I agree with everyone else. Get rid of it, yeah.

 

[kburkhart84]

I think a nice addition that would go perfect with this change is the addition of a "&" operator for function arguments. It would work like it does in C++, converting the argument to a pointer to the variable instead of the variable itself. The function would then work like normal.

function myFunc(myArgument)
{
    myArgument = 10;
    return true;
}
//later
var value;
var result = myFunc(&value);

That code would result in "result" being true and "value" being 10, all without having to do workarounds like storing things in structs for temporary usage. Note that this operator should also work on ANY variable types that aren't already references(like structs).

 

[chamaeleon]

I think a nice addition that would go perfect with this change is the addition of a "&" operator for function arguments. It would work like it does in C++, converting the argument to a pointer to the variable instead of the variable itself. The function would then work like normal.

function myFunc(myArgument)
{
    myArgument = 10;
    return true;
}
//later
var value;
var result = myFunc(&value);

That code would result in "result" being true and "value" being 10, all without having to do workarounds like storing things in structs for temporary usage. Note that this operator should also work on ANY variable types that aren't already references(like structs).

I prefer the idea of ref in the function declaration as mentioned by @HalRiyami, personally.

 

[Slow Fingers]

I think a nice addition that would go perfect with this change is the addition of a "&" operator for function arguments. It would work like it does in C++, converting the argument to a pointer to the variable instead of the variable itself. The function would then work like normal.

You are confusing & (adress) and * (pointer), but yep, I also think that solution would be quite elegant. It would stray away a bit from the "for beginners" motto Yoyo seems to stick to, but it would indeed be a really nice addition to the language for the more experienced users.

 

[Slow Fingers]

sorry, double post mistake

 

[kburkhart84]

I prefer the idea of ref in the function declaration as mentioned by @HalRiyami, personally.

I like the way I mention it more because it lets you explicitely see and do it directly where you call the function. I can certainly understand the merit of doing it the way you mention as well as you have to explicitly define the reference. I'd personally be satisfied with either approach if they chose to implement it.

It would stray away a bit from the "for beginners" motto Yoyo seems to stick to, but it would indeed be a really nice addition to the language for the more experienced users.

I agree that it wouldn't exactly be "for beginners." That said, the good part about this specific thing is that it would work for advanced users that understand and want it, but it does nothing to get in the way of beginners, unless they start trying to use it without understanding it first of course(we know that never happens ).

 

[Slow Fingers]

I agree that it wouldn't exactly be "for beginners." That said, the good part about this specific thing is that it would work for advanced users that understand and want it, but it does nothing to get in the way of beginners, unless they start trying to use it without understanding it first of course(we know that never happens ).

Well, it could be as simple as a checkbox in DND like "use referenced value"... and 582 new forum threads

 

[chamaeleon]

I like the way I mention it more because it lets you explicitely see and do it directly where you call the function. I can certainly understand the merit of doing it the way you mention as well as you have to explicitly define the reference. I'd personally be satisfied with either approach if they chose to implement it.

If it's done in the function call, all functions must be compiled in such a way that any argument could potentially be either a value or a reference/pointer, which would possibly mean additional overhead. Additionally, it could restrict your function implementation as you would no longer be able to feel completely free to change the value of a passed variable unless you know it is or is not passed with the reference symbol (which would be impossible for an extension used by others). The ref in the function declaration on the other hand makes it explicit both to the implementor of the function and the user of the function that it may very well be modified.

 

[kburkhart84]

If it's done in the function call, all functions must be compiled in such a way that any argument could potentially be either a value or a reference/pointer, which would possibly mean additional overhead. Additionally, it could restrict your function implementation as you would no longer be able to feel completely free to change the value of a passed variable unless you know it is or is not passed with the reference symbol (which would be impossible for an extension used by others). The ref in the function declaration on the other hand makes it explicit both to the implementor of the function and the user of the function that it may very well be modified.

Those are certainly good reasons as well to do it as part of the function declaration. I guess it is probably better off that way. I'm also guessing Yoyo would lean that way as well considering the GML language.

 

[chamaeleon]

Well, it could be as simple as a checkbox in DND like "use referenced value"... and 582 new forum threads

That seems a bit excessive. Surely... [proceeds to search for "improper-script-usage" for the globalscript runtime error] ... Never mind.

 

[drandula]

Easier would be to wrap that variable in a struct. Not super clean but that's what you would do in say, java.

But even easier is to do instance[$ "variable_name"], as struct accessors should work with instances too

 

[GMWolf]

But even easier is to do instance[$ "variable_name"], as struct accessors should work with instances too

Yeah but that only works for instance variables. With a boxed variable it works with instances, locals, globals, what have you.

f it's done in the function call, all functions must be compiled in such a way that any argument could potentially be either a value or a reference/pointer, which would possibly mean additional overhead

Nah, gml is dynamically typed. It would just add one more variable variant 'ref'.
It would mean every access would need to check the variable type, but that's already the case (accessing a string is different from accessing a real).
Where overhead might appear is on assignment, where it would need to check the type. Though it's entirely possible it's already checking the type on assignment anyways. If bit, I don't think the overhead would be all that significant when compared to the rest of the inefficiencies dynamic typed languages have.

Besides even if it was in the function declaration, I would expect the same checks to be there, as a new variable type would be the simplest implementation. The alternative would be to make refs distinct from variables at compile time, effectively introducing some static typing, which I doubt gnl is equipped to deal with.


Whether it's on the declaration or the call site is interesting.
If on the call site, it's immediately clear what the side effects of the function calls are. however in the function itself, it may seem non-sensical to modify a variable unless you know it is passed in by reference (incidentally, this limits some optimization opportunities for the compiler, unless the function gets inlined or lto got really good).

Ref in the function call means it's clear inside the function why we modify a variable, but it's no longer clear in the call what the side effects would be.


I think ref types would be quite interesting, but probably too complex to deal with. In the meantime boxing solves most of these issues.
It's clear both in the function, out of the function, and even in the original variable déclaration (as it's boxed) that you are using a ref.
However it does limit flexibility as now you can only use variables that have been boxed to start with. Perhaps gnl could offer some utilities around the concept of boxing, rather than introduction refs.


But this is all off topic, and I suspect this warrants a separate thread.
Incidentally I used to implement refs using arrays and the @ accessor.

(Ps, I'm just laying out my views, but since I have no ongoing GM projects, or plans for a new GM project, Im deliberately trying not to say 'keep it' or 'remove it'.)

 

[gnysek]

I know they're not deprecated yet

Probably only because it's just part of syntax which produces same code as new syntax. Same is with begin, end and :=, which easily can be replaced on compile time with new version, so compiler even doesn't need to support them But yeah, if not now, then I think when this "new runtime" after "current lifetime" will be released (later this year?), things like that should be removed completely forever.

So, removing copy on write + deprecating few "old" syntax things which have new equivalents would be welcome.

 

[NightFrost]

I think a nice addition that would go perfect with this change is the addition of a "&" operator for function arguments.

If we're speaking of wishlists, how about destructuring and spread operations.

// Destructuring
function Foo(){
    return [4, 8];
}
[Bar, Baz] = Foo(); // Bar = 4, Baz = 8

// Spread
[Foo, Bar, ...Others] = [1, 2, 3, 4, 5]; // Others = [3, 4, 5]

 

[FlameRooster]

This is just my opinion.

Accessors, ds_x, and things like in-built variables with important variable names like name/score should be removed.

Theses things complicate the user experience. Arrays are much more easier to use and beginner friendly.

 

[drandula]

Accessors, ds_x, and things like in-built variables with important variable names like name/score should be removed.

Lists and maps can be replaced with arrays and structs.
But ds_grids allow easy and fast way of setting large areas at once, copying and multiplying etc. which are much faster than doing as array equilevant.
So removing ds_grids would actually remove something useful.
But grids feel bit limited (only 2D, can't do division etc.), and instead I would like to have maths-library with n-dimensional support

Spoiler

Like you could only use numbers with something like "math.array" and ability to do operations just by "c = a + b", where all they should be equilevently shaped arrays, and operation would do element-wise additions which results are stored in c-array. Without need to do

for(var i = 0; i < array_length(a); i++) {
for(var j = 0; j < array_length(a[i]); j++) {
    c[i][j] = a[i][j] + b[i][j];
}}

But those are things to wish for.

Now as back to topic, I think copy-on-write does generally cause more confusion on how arrays behave than it's being useful. But also usually isn't desired, and have to use @ pretty much always.
With multi-dimensional arrays I think you need to put it in each index. So 4-dimensional array would look like: array[@ i][@ j][@ k][@ l]. Starts getting pretty ugly

 

[8BitWarrior]

With multi-dimensional arrays I think you need to put it in each index. So 4-dimensional array would look like: array[@ i][@ j][@ k][@ l]. Starts getting pretty ugly

If I am correct, I think you could just do:

array[i][j][k][@ l]

with the @ only applied to the last dimension. Regardless... I don't think this is going to help the confusion!

 

[drandula]

If I am correct, I think you could just do:

array[i][j][k][@ l]

I recall trying that, which didn't work the way I wanted, but instead had to use all-accessor But that was in version 2.3.2 or something, I should retry. I think GMS is like "hey you going to write", and starts Copy-on-write behaviour from first array. Not sure.

Anyway, yeah it's not pretty either way. Even if you need only to use accessory at end, it is indeed confusing.

 

[Solidity]

Can you please give an example of the pass-by-reference you are talking about? I wonder if it would be more like the one used in JavaScript, i.e. pass-by-sharing, or something else?

 

[chamaeleon]

Can you please give an example of the pass-by-reference you are talking about? I wonder if it would be more like the one used in JavaScript, i.e. pass-by-sharing, or something else?

Is "pass-by-sharing" an actual proper name for some Javascript behavior? Does not most people say pass by reference when objects are involved for Javascript as well?

 

[Slow Fingers]

Can you please give an example of the pass-by-reference you are talking about? I wonder if it would be more like the one used in JavaScript, i.e. pass-by-sharing, or something else?

Why, is something unclear in the 5+ examples and 12+ explanations on the previous page?

 

[Alice]

Can you please give an example of the pass-by-reference you are talking about? I wonder if it would be more like the one used in JavaScript, i.e. pass-by-sharing, or something else?

Pass-by-reference would be handled just like structs are right now.

Consider the following behaviour with structs:

function some_struct_stuff(_struct) {
    _struct.stuff = "some";
    return _struct;
}

var _hello = { hello: "world" };
var _hello_stuff = some_struct_stuff(_hello);
show_debug_message(_hello == _hello_stuff); // 1, i.e. _hello and _hello_stuff are same references
show_debug_message(_hello); // { stuff : "some", hello : "world" }
show_debug_message(_hello_stuff); // { stuff : "some", hello : "world" }

The struct argument represents the struct itself - that's the "pass-by-reference" behaviour. Changing any property of the struct argument changes the struct passed to the function.

Compare it with the following behaviour of arrays:

function some_array_stuff(_array) {
    _array[0]= "some stuff";
    return _array;
}

var _hello_array = ["hello", "world"];
var _hello_array_stuff = some_array_stuff(_hello_array);
show_debug_message(_hello_array == _hello_array_stuff); // 0, i.e. _hello_array and _hello_arary_stuff are different references
show_debug_message(_hello_array); // [ "hello","world" ]
show_debug_message(_hello_array_stuff); // [ "some stuff","world" ]

Because an array value was directly written in a function, and arrays use "copy-on-write" behaviour, the function creates a copy of the original _array argument, and the resulting array is distinct from the original.

The important part is that an array value was set directly; if it was set via a function or the value was only accessed, the copy-on-write behaviour wouldn't be triggered and _array reference would remain the same.
Like in this example:

function other_array_stuff(_array) {
    // getting array value won't trigger copy-on-write behaviour
    show_debug_message(_array[0]);

    // it seems like a more elaborate to write "_array[0] = "some stuff";"
    // but because the array is used in a function, copy-on-write isn't triggered
    array_set(_array, 0, "some stuff");

    // because no _array write access happened in the function
    // it's the exact same array as passed in argument
    return _array;
}

var _hello_array = ["hello", "world"];
var _other_array_stuff = other_array_stuff(_hello_array); // prints "hello" in the function's show_debug_message
show_debug_message(_hello_array == _other_array_stuff); // 1, i.e. _hello_array and _hello_arary_stuff are the same references
show_debug_message(_hello_array); // [ "some stuff","world" ]
show_debug_message(_other_array_stuff); // [ "some stuff","world" ]

Because array wasn't directly written to, the copy-on-write behaviour isn't triggered and the other_array_stuff functions acts like pass-by-reference instead. As it should.

So basically, because of copy-on-write, array[index] = value" is a more confusing version of array_set(...) that doesn't change the contents of the original array argument. If the array is not a function argument, the copy-on-write thing doesn't get triggered either. Copy-on-write is a tangled mess that needs to go.

 

[Zhanghua]

Just care about whether the array performance can same as or better than ds_list after changing.

 

[Lukan]

I honestly thought this was removed when GML got its overhaul.
All for removing it, as it's been a headache in the past.
I don't actually know if any of my code is being held together by this right now, but I guess I'll find out, lol.

 

[rwkay]

OK so we have decided to add a game option for Copy On Write so that it can be enabled or disabled per project it will be defaulted to disabled for new projects and enabled for current projects - you can of course disable it yourself via the Main Options - this will be in the February build (2022.2).

This will also (within the compiler, it will not change the actual source code) turn all use of the array accessor [@ into just a plain old array accesses - so code should get a speed up (as it will remove function call overhead).

Thanks everyone for the feedback.

Russell

 

[gnysek]

And that's the best solution. Thanks too!

 

[Zhanghua]

And that's the best solution. Thanks too!

There will be two gangs next month.

 

[Mehdi]

Cou

OK so we have decided to add a game option for Copy On Write so that it can be enabled or disabled per project it will be defaulted to disabled for new projects and enabled for current projects - you can of course disable it yourself via the Main Options - this will be in the February build (2022.2).

This will also (within the compiler, it will not change the actual source code) turn all use of the array accessor [@ into just a plain old array accesses - so code should get a speed up (as it will remove function call overhead).

Thanks everyone for the feedback.

Russell

Could you please tell more exactly when is the time for the January stable release? (You have mentioned it as "real soon")

 

[rwkay]

Could you please tell more exactly when is the time for the January stable release? (You have mentioned it as "real soon")

We are always working toward a Stable release on the last Thursday of the month (that would be the 27th this month) - we lock features for the final 2 weeks of the month and QA are currently testing before we declare it stable (so depending on the number of issues found we could slip but that is our target).

Russell

 

[Mehdi]

We are always working toward a Stable release on the last Thursday of the month (that would be the 27th this month) - we lock features for the final 2 weeks of the month and QA are currently testing before we declare it stable (so depending on the number of issues found we could slip but that is our target).

Russell

27th is not too far away.

 

[Kezarus]

Question: are we going to get a console message when an array copy is performed?

That would be good to people that would like to correct their code to the new, more modern and fast, norm.

Cheers!

 

[gnysek]

it will be defaulted to disabled for new projects and enabled for current projects

When I think about it little more, I believe that it should be enabled for all projects, even new, as some people may not notice in release notes that this changed. Also, for new GMS2 users which are starting to learn GML, if they import asset that rely on copy on write behavior to new project, it might break, if "local" array in script have some write operations, but array passed to script shouldn't be changed - and they won't know why their game break. Except there will be a message when importing scripts, that this need double-check.

OR - maybe add selection when creating new project? Same as there's D'n'D / GML selection? So people are aware and can go to some helpdesk page which explains what does that mean.

 

[Kezarus]

When I think about it little more, I believe that it should be enabled for all projects, even new, as some people may not notice in release notes that this changed. Also, for new GMS2 users which are starting to learn GML, if they import asset that rely on copy on write behavior to new project, it might break, if "local" array in script have some write operations, but array passed to script shouldn't be changed - and they won't know why their game break. Except there will be a message when importing scripts, that this need double-check.

Agreed. That would confuse even us when helping others to troubleshoot their code.

 

[gnysek]

GML Programming help tick list before answering:
- do you have GMS 2.3+ or earlier?
- do you have latest GMS version?
- do you have copy-on-write for arrays enabled?

 

[Kezarus]

Yeah, that seems confusing, isn't it?

What I'm asking is a "deprecated" warning. No need to branch the enable/disable behavior. That way, always enable, all code from we game devs would have the same behavior.

I understand that a deprecated on compile will be complicated, but I think a warning or an exception throw on execution can be the best path.

What do you guys think?

 

[rwkay]

We are deprecating the behaviour (it will not be available in the future toolchain) so we are defaulting new projects to OFF.

We will discuss adding a warning message to telegraph to everyone that Copy on Write behaviour is deprecated in the future and I will consider adding in a message when it happens in the code (NOTE: This could get very noisy).

Russell

 

[Stra]

This would mean the arrays are always passed by reference and would eliminate the need for the [@ accessor.

So basically if I never used that accessor I will not notice any changes?

 

[Alice]

So basically if I never used that accessor I will not notice any changes?

Actually, it's sort of the opposite - the code will act as if the accessor is always used. So in a function like this:

function do_array_stuff(_array) {
    _array[0] = "something"; // will create a copy of array in current system, will set the item of the original array in the new system
    // from that point on, _array refers to the copy of the array rather than the original array argument
}

function do_other_array_stuff(_array) {
    _array[@ 0] = "something"; // will set the item of the original array in both the current and new system, at least until the accessor is removed altogether
}

So if you *always* used that accessor, you will not notice any changes.

 

[Stra]

Sorry, my question was poorly worded and without context:
in essence if I always operate on the original array (never make copies of it, such as passing it to a function) I will not notice any changes? For example I only pass to a function the index in the array to operate on, not the array itself.

 

[Alice]

Yes, as long as you never use the syntax of _array[index] = value; with array function parameters, there should be no changes.
In particular, when passing the array to array_set(_array, index, value) function call, the behaviour won't change either, because it's not a direct array writing (unlike above).

 

[Japster]

Attempting to make a Carnac the Magnificent prediction, I foresee the need for the equivalent of the GlobalScript sticky thread to point people to when the switch is made, and they can't understand why their project is not working properly. In contrast to GlobalScript the breakage will quite possibly be much more insidious and non-apparent. Having said that, I'm very much in favor of the change.

Yep, and I can think of one potential real-world game breakage/example off the bat...

TetraLogical uses this copy on write feature I *think*, in the recursive AI logic.

The routine that checks for millions of possible move combinations, effectively places a piece (modifying it's copy of the board, then calls itself, passing in the modified copy of the board, to place further moves on, and recursively call itself. It works perfectly, using various temporary copies of the array, whilst recursing - i.e. it's not modifying a single referenced original array, just copies of it.

Now I've only just checked this thread, so am I jumping the gun, and getting a bit confused - would this still work? - because I'm specifically passing in the board array, and THAT passes in a copy by default, or would this new change break that?

Don't get me wrong - I'm kind of limited to an older runtime anyway atm, because in 2.2.x, performance DRASTICALLY reduces, for recursion - my 3-6 second AI routine (when running under 2.1.x) takes 20-50 seconds, using 2.2.x versions.

But still, I'd like to know if this is an example of something that would break that, or if I'm getting confused...

Cheers!

 

[Alice]

@Japster It depends on how exactly you copy these arrays. If you make a copy explicitly, by using a function like array_copy, then this function obviously doesn't change.

On the other hand, if you copy the array implicitly by doing something like:

function apply_additional_move(_array, _index, _value) {
    _array[_index] = _value; // copying happens here
    // do some other checks and recursive calling
}

...then this behaviour will break.

Keep in mind that at the start of the function, when array argument was just received, it's still the original array. You need to assign a value to the array for the array argument to become a copy.

function something_with_array(_array) {
    array_push(_array, 123); // it affects the original _array here
    _array[0] = 456; // from this point, _array is a copy
    array_push(_array, 789); // it affects the copied _array here, the original array will be unaffected 
}

Hope it clears things up.

 

[FrostyCat]

The routine that checks for millions of possible move combinations, effectively places a piece (modifying it's copy of the board, then calls itself, passing in the modified copy of the board, to place further moves on, and recursively call itself. It works perfectly, using various temporary copies of the array, whilst recursing - i.e. it's not modifying a single referenced original array, just copies of it.

Now I've only just checked this thread, so am I jumping the gun, and getting a bit confused - would this still work? - because I'm specifically passing in the board array, and THAT passes in a copy by default, or would this new change break that?

If this is the kind of code you are talking about, then yes, it will be affected.

function evaluate(state) {
    state[move] = 1;
    ...
    evaluate(state);
    ...
}

When copy-on-write is removed, you would need to make an explicit copy and pass that instead.

function evaluate(_state) {
    var _stateSize = array_length(_state);
    var state = array_create(_stateSize);
    array_copy(state, 0, _state, 0, _stateSize);
    state[move] = 1;
    ...
    evaluate(state);
    ...
}

 

[Japster]

If this is the kind of code you are talking about, then yes, it will be affected.

function evaluate(state) {
    state[move] = 1;
    ...
    evaluate(state);
    ...
}

When copy-on-write is removed, you would need to make an explicit copy and pass that instead.

function evaluate(_state) {
    var _stateSize = array_length(_state);
    var state = array_create(_stateSize);
    array_copy(state, 0, _state, 0, _stateSize);
    state[move] = 1;
    ...
    evaluate(state);
    ...
}

Thanks a lot @FrostyCat / @Alice ...

Darn it, yep.... ...hmmm - the beauty of the old method was that this would just happen, with minimal overhead thanks to the built-in engine-level copying, and I'm assuming would therefore be a lot quicker over the course of the entire recursion, given the added overhead of manual array copying via GML for every call, using the newer proposed comparable / workaround system?

...unless of course, there's not much in it...

 

[peardox]

Does this mean the end of having to return an array you've altered if you want the caller to know about it?

If so we can finally return a meaningful status code from a function.

While you're at it why not have optional [Out] syntax?

 

[8BitWarrior]

Thanks a lot @FrostyCat / @Alice ...

Darn it, yep.... ...hmmm - the beauty of the old method was that this would just happen, with minimal overhead thanks to the built-in engine-level copying, and I'm assuming would therefore be a lot quicker over the course of the entire recursion, given the added overhead of manual array copying via GML for every call, using the newer proposed comparable / workaround system?

...unless of course, there's not much in it...

I'd be a fan of a GM function which directly returned a full array copy, thus preventing the need for temporary variables and additional overhead for setting the range to be copied.

Example:

// Pass in a "clone" of my_array to be processed
ProcessData(array_clone(my_array));

With the existing functions, is there any faster alternative to this:

var _temp_array = [];
array_copy(_temp_array, 0, og_array, 0, array_length(og_array));
ProcessData(_temp_array);

An additional thing could be allowing for optional arguments to set specific elements of the array upon cloning:

// Pass cloned array with updated values for indexes 5, 6, and 9
ProcessData(array_clone(my_array, 5, "John", 6, "Smith", 9, 500));

But perhaps that might work best as an "array_clone_ext()" sort of thing?

 

[Alice]

I'd be a fan of a GM function which directly returned a full array copy, thus preventing the need for temporary variables and additional overhead for setting the range to be copied.

Sounds like a perfect functionality to file a feature request for.
Just make sure you access the Contact page when logged in, or else there will be only "Account issues" in the topic dropdown. ^^

 

[Zhanghua]

Sounds like a perfect functionality to file a feature request for.
Just make sure you access the Contact page when logged in, or else there will be only "Account issues" in the topic dropdown. ^^

so when do we implement struct_clone?

 

[dT_]

Sure, remove it and forget. This "feature" doesn't work as it should and confused. Copy-on-write happens not only with script arguments. Here's example, where it happens too, but docs don't mention it. Why does it happen? Only "creator" can edit array without copying it?

/// @description Create event

origArray = [0, 1, 2];

array = origArray;

show_debug_message("origArray: " + string(origArray) + " array: " + string(array));
// origArray: [ 0,1,2 ] array: [ 0,1,2 ]

array[3] = 3;

show_debug_message("origArray: " + string(origArray) + " array: " + string(array));
// origArray: [ 0,1,2 ] array: [ 0,1,2,3 ]