GameMaker Array initialization slows down loading

salyossy

Member
Found this nice thread. (Split from another topic ~Tsuk)

Sorry about my poor english :confused: , but I really need your help.
I have a problem with time creation of arrays. I use very long arrays, but initialization isn't made with looping.
for example:
Code:
arr=[1,2,6,4,7,
0,3,2,4,5,
8,2,3,9,8,
...
...
]
These arrays has around 50k elements. Therfore, it takes about 6-7 seconds for my android to read it. it really stucking my games.

Is there any workaround to make it faster ?? any tip? PLEASE.
Thank you in advance :)
 

FrostyCat

Redemption Seeker
In a separate project, create a binary version of the array:
Code:
var fname = get_save_filename("Data file|*.dat", "");
var arr = [1, 2, 6, 4, 7, ...];
var n =  array_length_1d(arr);
var buffer = buffer_create(n, buffer_fixed, 1);
for (var i = 0; i < n; i++) {
  buffer_write(buffer, buffer_u8, arr[i]);
}
buffer_save(buffer, fname);
Then add the file as an Included File in your project, and use this to load the data:
Code:
global.values = buffer_load(working_directory + "arraydata.dat");
Then you can reference from it with buffer_peek():
Code:
// Example: Read position 7
buffer_peek(global.values, 7, buffer_u8)
 

salyossy

Member
I don't understand why it takes 6 seconds to initialize an array it that way???

(I'm relating to ANDROID performance)

Creating an array in that size, with for loop, is really fast!! But I CAN'T do that !! because a need specific values, taken as an output from other program.

What am i doing wrong?

Did I make any mistake here? is it possible to fix my situation?

If not... is there a better way to handle 50K elements other than array?

:(
 

TheouAegis

Member
For starters, each element takes up at least 4 bytes of memory (maybe itcs 8 but that's unimportant here), so that's a few megs total. Saving the data as bytes and loading them into a buffer will take up 25% the memory.

Secondly, an array is broken up into 32k element sets. So your 50k elements is really 2 arrays that need to be initialized.
 

Joe Ellis

Member
@FrostyCat buffer_peek is alot slower than referencing an array
"is there a better way to handle 50K elements other than array?" I don't think there is, buffers are at least half the speed of arrays, but there could be other ways to initialize it.
Is every index a specific value, or are alot filled with empty 0's? You could make a file that holds a list of the none zero values and their index
 

salyossy

Member
@FrostyCat buffer_peek is alot slower than referencing an array
"is there a better way to handle 50K elements other than array?" I don't think there is, buffers are at least half the speed of arrays, but there could be other ways to initialize it.
Is every index a specific value, or are alot filled with empty 0's? You could make a file that holds a list of the none zero values and their index
Thank you.

Well, every index is really a specific value, and imporntant. There aren't bunch of zero's to "compress".

Morever, the buffer's method seems to not be the right solution, becasue the buffer seems to have a LIMIT:
When i tried to "conver"t a SMALL array to a buffer, the conversion worked fine. But doing that with a LARGE array, the buffer couldn't take all data, at some point and further. So i had not have the chance even to test the speed of reading it, i'm struggling with the part of filling the buffer with all of the data.

Now, you say its not faster, so maybe the whole thing is pointless. :/
 

salyossy

Member
For starters, each element takes up at least 4 bytes of memory (maybe itcs 8 but that's unimportant here), so that's a few megs total. Saving the data as bytes and loading them into a buffer will take up 25% the memory.

Secondly, an array is broken up into 32k element sets. So your 50k elements is really 2 arrays that need to be initialized.
Very interesting points.
I use buffer_s32 anyway because some of the numbers are high or negative. So buffers might won't save memory anyway.
I wonder if action of split itself consumes time, i'll try to break myself the array into two, see what happens...
 

salyossy

Member
...
I wonder if action of split itself consumes time, i'll try to break myself the array into two, see what happens...
No influence...:(

And the limitation of buffer size is also 32k exactly...

The whole thing seems weird to me. 7 seconds to create an array. wow!
 
Last edited:
P

ParodyKnaveBob

Guest
Howdy, salyossy,

Seven seconds to fill up multiple Megabytes of data one interpreted byte at a time - on Android.
~shrug~

Licensing has changed a few times, I'm out of the current loop, and I don't know which abilities you picked up at what time. Do you have YYC capability? Have you tried that to remove code interpretation and crunch faster?

Otherwise, I can't help but consider this. The thread you started in very likely had your answer. The very last post of that thread:

I verified my findings. Only the highest 32000 range gets preinitialized. So in this case, index 96000 and beyond is set to 0, whilst 95999 and lower are uninitialized. Thus initializing every last element of each range will nearly negate the discrepancy. So set 31999, 63999, and 95999 along with 99999 and the performance hit is gone.
Did you try that? Spot-initialize the array chunks, then fill it with your actual data?

I hope this helps,​
 

GMWolf

aka fel666
50k element array?
What are you storing in that?
This sounds like it should be data, or computed analytically. Not s built in array.

If having a large array is too slow, perhaps we can help you find an alternative solution to a monstrous 50k array.
 

salyossy

Member
** Do you have YYC capability? Have you tried that to remove code interpretation and crunch faster?

**Did you try that? Spot-initialize the array chunks, then fill it with your actual data?​
Unfortunately, the answer is yes to both: i'm compiling with YYC, and i tried to spot-initialize the array:
Code:
arr[31999]=0;
arr[63999]=0;
arr[95999]=0;
arr=[
0 ,-10,-1,-1,322580,
0 ,1,1 ,70 ,49,
1024 ,-10,-1,-1,309278,
0 ,-8,-1,-1,-1,
0 ,0, 1 ,70 ,0
...
...
]
These attempts did not show any improvement. The problem is only time creation of the array. Reading it is fast enough! Its just the 7-seconds waiting ...
50k element array?
What are you storing in that?
This sounds like it should be data, or computed analytically. Not s built in array.

If having a large array is too slow, perhaps we can help you find an alternative solution to a monstrous 50k array.
Well, the array contains information about music, so one song needs about 50k.. yes.
What did you mean by "SHOULD BE DATA"?

Thank you all for trying to help!!
 
H

Homunculus

Guest
That initialization code is useless, you are initializing the array up to position 95999, and then overwriting it with a new one.

Knowing that the array contains information about music does not really answer the question, does it? What exactly is this data? Also, I think that by "should be data" GMWolf meant that it should be stored in external files just like a sprite or a sound is, not created inline (which is not a good idea for many reasons, if you ask me).

I still think frostycat solution is worth another try, I don't really know how you use that data, but an external file loaded as a buffer for data this size that (possibly) needs to be loaded / unloaded on the fly sounds right to me. Not sure if I have to specify this, but you are not supposed to create the buffer in your released app, so the buffer creation should not count as "loading time" in your benchmarks. If it takes 10 minutes, so be it, not a big deal. Create the buffer, save it to file, include it in the included files, and load when you need it. I have no idea if this will take less or more time than the inline array, but it still is a better approach imho. Moreover, buffers can be loaded as an asynchronous event, without stopping your game flow. You can even load only small chunks at a time.

Note that as pointed out before, accessing buffer data is slightly slower than accessing array data, but it's not THAT slow, and it's not related to your current problem.
 
Last edited by a moderator:

TheouAegis

Member
@FrostyCat buffer_peek is alot slower than referencing an array
"is there a better way to handle 50K elements other than array?" I don't think there is, buffers are at least half the speed of arrays, but there could be other ways to initialize it.
Is every index a specific value, or are alot filled with empty 0's? You could make a file that holds a list of the none zero values and their index
I wouldn't say it's "a lot slower". True, buffer_peek() has a noticeable impact in the performance graph. With that said, that's just buffer_peek(). It requires 3 arguments and sets the index an additional time. If you used buffer_read() instead, calling a buffer_seek() one time before starting a series of buffer_read() calls, the speed of an array lookup and buffer_read() are nearly identical. The array still wins out, but by a negligible amount. So if you're going to be doing a lot of "peeking", then it would make sense to load the contents of the buffer file into an array, with all the slowdown that's going to bring -- you'd sacrifice speed at the start of the game/room in order to speed up the Step event. But again, I'd personally only consider that if I was getting noticeable slowdown in the game already. One or two peeks every step isn't going to have a significant impact. In my speed tests and performance tests, I only saw a significant hit when the buffer was $FFFF bytes long; at 512 bytes there was not enough of a performance hit to warrant stressing over it, and at 15 bytes there was no discernible difference even after 30 seconds of running both tests. (When I say $FFFF bytes long, I also mean the array was $FFFF entries and all $FFFF bytes/entries were processed.)

By the way, how do you fill a buffer with signed 32bit values? I had to use an alignment of 8, which just filled it with empty 0s. I don't get why. It's s32, not f32.

@salyossy One of the things they're probably wondering about is why you need 50k entries for the song. What does each element in the array hold? How much of that data is actually necessary and how much of that can be compressed?
 

salyossy

Member
The content of the array is simply order of notes to be played, the pitch of note, the duration, the instrument, the tempo ect, not a big deal to the topic i think.
I just need to loop through the array to pick the correct values to make my music.
What matters here is technically the initialization.

And yes, i will continue to try frostycat's solution. It took hours to understand whats going on with the values of the buffer that where "disappearing" until i found out that the buffer is also limited to 32k :)() so the data was cut in the middle. I guess, with another effort i could try to compress the information here and there, striving to get below 32k in the price loosing some information. Not the best practice for me.

For now i'll try to separate into two buffers..

By the way, an alignment of 4 was fine for _s32, i think it was ok for me, not sure, didn't check all the data.

I'll come up here with update when i will implement the solution. Hoping for good :)
 
H

Homunculus

Guest
Buffers are definitely not capped at 32k. Where did you get that from?
 

salyossy

Member
hmmm.. i made some tests, to find out at which point the buffer_write no longer affects the buffer, and values no more added to it...

When i saw that it was exactly 32k..... i realized that...
 
H

Homunculus

Guest
Well if you created a fixed buffer large exactly 32k you may very well expect that. Can’t really tell you what’s wrong with just the info you provided, only thing i can do is point out that buffers are definitely not capped at 32k
 

GMWolf

aka fel666
not a big deal to the topic i think.
Very big deal!
This means you do not need random access.
Which means you can save your data as some binary blob, and load it in chunks at a time.
You could even make use of the asynchronous load functions (but let's ignore that for now).
 

salyossy

Member
Well if you created a fixed buffer large exactly 32k you may very well expect that. Can’t really tell you what’s wrong with just the info you provided, only thing i can do is point out that buffers are definitely not capped at 32k
OMG!
I actually tried the buffer_grow, and it was "capped to" 32k, but your insistence had me question that, and i checked it again, and what i found out??
Couldn't believe it... the code:
Code:
show_debug_message("length="+string(array_length_1d(arr)));
result the output: length=32000.

So actually the problem was relied in the false returned value of the array size. I was actually looping with for loop, until that value!! so that what caused the limit!!!
 

GMWolf

aka fel666
OMG!
I actually tried the buffer_grow, and it was "capped to" 32k, but your insistence had me question that, and i checked it again, and what i found out??
Couldn't believe it... the code:
Code:
show_debug_message("length="+string(array_length_1d(arr)));
result the output: length=32000.

So actually the problem was relied in the false returned value of the array size. I was actually looping with for loop, until that value!! so that what caused the limit!!!
Catan is talking about buffers. You are talking about arrays.
Try buffers.
 
H

Homunculus

Guest
I think he’s using the array to generate the buffer @GMWolf , which is reasonable since he already has that defined
 

salyossy

Member
Very big deal!
This means you do not need random access.
Which means you can save your data as some binary blob, and load it in chunks at a time.
You could even make use of the asynchronous load functions (but let's ignore that for now).
Yeah, the asynchronous load functions was metioned earlier, and i think its a brilliant idea to try using these. Need to learn how.. i'll definitely explore it soon.

By the way, i wonder if its possible to receive the length of array that is>32k.
Maybe i did mistake here.. i don't know..
 
Last edited:

salyossy

Member
I think he’s using the array to generate the buffer @GMWolf , which is reasonable since he already has that defined
exactly...
I loop in a special project through the array, only to fill the buffer.. and if the array_length_1d function is restricted to 32k..thats a problem
 
H

Homunculus

Guest
I remember gms1 arrays being capped at that size, which could explain your problem if you are using that version. You can easily solve this by storing the values in a text file instead of an array in order to generate the buffer
 

salyossy

Member
In a separate project, create a binary version of the array:
Code:
var fname = get_save_filename("Data file|*.dat", "");
var arr = [1, 2, 6, 4, 7, ...];
var n =  array_length_1d(arr);
var buffer = buffer_create(n, buffer_fixed, 1);
for (var i = 0; i < n; i++) {
  buffer_write(buffer, buffer_u8, arr[i]);
}
buffer_save(buffer, fname);
Then add the file as an Included File in your project, and use this to load the data:
Code:
global.values = buffer_load(working_directory + "arraydata.dat");
Then you can reference from it with buffer_peek():
Code:
// Example: Read position 7
buffer_peek(global.values, 7, buffer_u8)
I'm glad to report that it worked, eventually. That was the right solution in the first place, and combined with all the tips mentioned her, it worked for me.
After including the data file created in the "other" project, i convert it back to array format in my REAL project, using the peek only once.
(not before Spot-initializing the array with: arr[31999]=0; arr[63999]=0;)

That way the array loads really quick. Not even a second... the main remaining problem here is getting the length of the array. other then that, i think the problem is solved.

I really appreciate all your help, all of you! piece by piece solving the puzzle. THANKS
 
Top