Creating a function to replace ds_grid_write/ds_grid_read

Niften

Member
Would it be beneficial if another way of saving/loading grids was created? ds_grid_write is awfully slow. Can it be done faster manually? Maybe something like...

Code:
var save_str = "";
for (var xx = 0; xx < gwidth; xx++) {
for (var yy = 0; yy < gheight; yy++) {

var val = grid[# xx,yy];
save_str=save_str+"_"+string(val);

}
}

return save_str;
and then a script that takes all of the values in between the underscores and places them back into the grid... would it be faster?
 
E

Edmanbosch

Guest
Would it be beneficial if another way of saving/loading grids was created? ds_grid_write is awfully slow. Can it be done faster manually? Maybe something like...

Code:
var save_str = "";
for (var xx = 0; xx < gwidth; xx++) {
for (var yy = 0; yy < gheight; yy++) {

var val = grid[# xx,yy];
save_str=save_str+"_"+string(val);

}
}

return save_str;
and then a script that takes all of the values in between the underscores and places them back into the grid... would it be faster?
Why would you need the performance for it though?
 

Niften

Member
My game involves a chunk system based on ds_grids, and writing them is really slow and creates a very bad lag spike. There are no memory leaks and it's simply because ds_grid_write takes an awfully long time to do its thing. I've even tried optimizing it with buffers with no luck.
 

Neptune

Member
If you're grids aren't fairly large, it may be another issue. I have a save function that creates / writes / destroys about 20+ grids currently, and its so fast that you wont see even a jitter.
Granted the largest of grids might only have 50 or cells.

Otherwise, I'd imagine you could make your own script to store your data in some way, but I havn't dealt with it much.
Wish you luck!
 

Simon Gust

Member
When I did that sorta thing I started looking into buffers. They save and load fast and can even be loaded saved asynchronously.
But you either have to write the chunk into the buffer frist or only use buffers for the chunks which makes individual cells a pain.
-> buffers are 1D, grids are not.
 

Niften

Member
Each grid is 32x32 and they have to be saved every time you exit them. There are three grids of this size that must be saved.
 
E

Edmanbosch

Guest
My game involves a chunk system based on ds_grids, and writing them is really slow and creates a very bad lag spike. There are no memory leaks and it's simply because ds_grid_write takes an awfully long time to do its thing. I've even tried optimizing it with buffers with no luck.
Do you write them every frame?
 

Niften

Member
No. There are no memory leaks, there's nothing obvious causing the lag other than the atrocious writing speed.
 

Niften

Member
I write them when chunks need to be saved. Chunks ATTEMPT to be saved when the player leaves them (the chunks check if they have been edited and if they haven't they skip the save) - trust me when I say that it isn't a problem of when the chunks are saved.
 

Simon Gust

Member
Here is an example of using buffers for saving.
Code:
/// saving buffers
// generate filename
filename = string(baseX) + “_” + string(baseY) + “.map”;

// create buffer
buffer = buffer_create(32*32*4, buffer_fixed, 4);

// get all grid entries in the chunk
for (var i = 0; i < 32; i++)
{
 var xx = i + baseX; // baseX being the x of the chunk
 for (var j = 0; j < 32; j++)
 {
  var yy = j + baseY; // baseY being the y of the chunk

  // get ds grid value
  var val = ds_grid_get(GRID, xx, yy);

  // get offset
  var offset = 4 * (i * 32 + j);

  // write to buffer
  buffer_poke(buffer, offset, buffer_u64, val);
 }
}

// save buffer
buffer_save(buffer, filename);

// destroy buffer
buffer_delete(buffer);


Code:
/// loading buffers
// generate filename
filename = string(baseX) + “_” + string(baseY) + “.map”;

// load buffer
buffer = buffer_load(filename);

// get all grid entries in the chunk
for (var i = 0; i < 32; i++)
{
 var xx = i + baseX; // baseX being the x of the chunk
 for (var j = 0; j < 32; j++)
 {
  var yy = j + baseY; // baseY being the y of the chunk

  // get offset
  var offset = 4 * (i * 32 + j);

  // get buffer value
  var val = buffer_peek(buffer, offset, buffer_u64);

  // write to grid
  ds_grid_set(GRID, xx, yy, val);
 }
}

// destroy buffer
buffer_delete(buffer);
 

Niften

Member
Thanks for your example. I'm a bit confused about what baseX and baseY are - and I know you commented that they are the 'x' of the chunk, but what does that mean? The position of the chunk in relation to the rest of the room? The position of the chunk in relation to the rest of the chunks?
 

Simon Gust

Member
Thanks for your example. I'm a bit confused about what baseX and baseY are - and I know you commented that they are the 'x' of the chunk, but what does that mean? The position of the chunk in relation to the rest of the room? The position of the chunk in relation to the rest of the chunks?
The relation of the chunk to the room (divided by the size of a grid cell).
Say your grid cell size is 16x16 pixel and a chunk is 32x32 cells.
So if you have a chunk at
x = 40960 in room pixel
baseX would be 40960 / 16;
-> baseX = 2560 the baseX is most likey also dividible by 32 as chunks are adjacent.

Unless you have the chunks held differently.
 

Bingdom

Googledom
I have a few concerns about this code.
  1. Why not just spit out the data rather than continuously peeking and poking? It's been read and written the same.
  2. buffer_u64. Shouldn't it be buffer_u8? If not, 32*32*4 is quite small for that.
 

Simon Gust

Member
I have a few concerns about this code.
  1. Why not just spit out the data rather than continuously peeking and poking? It's been read and written the same.
  2. buffer_u64. Shouldn't it be buffer_u8? If not, 32*32*4 is quite small for that.
1. I could, but it doesn't show the 2D to 1D transformation in logic. I find it very informational and I am willing to take that nanosecond hit.
2. Meant buffer_u32. There is a reason it's not buffer_u8, you just can't store infromation at all. I think that if you created lists, it's index would always increment even beyond destruction. Sure u32 has an end but it's unrealistic to reach it.
If you want to bitmask it you have the full u32 and never run into memory problems.
 
H

Homunculus

Guest
About point 2, Maybe I missed something, but shouldn't the buffer alignment and type (u32 etc...) depend on what kind of data is actually held inside the ds_grid? I don't see that mentioned anywhere. Can't really understand what data structure indexes have to do with this, although I agree that I would store those in a u32 buffer...
 

Simon Gust

Member
About point 2, Maybe I missed something, but shouldn't the buffer alignment and type (u32 etc...) depend on what kind of data is actually held inside the ds_grid? I don't see that mentioned anywhere. Can't really understand what data structure indexes have to do with this, although I agree that I would store those in a u32 buffer...
the index of a data structure is a number. Each time you create a data structure that number is different (probably incrementing) someday you have 255, then what?
It is most likely that the ds_grid holds more than 1 information per cell. So a pointer to a ds_list inside memory should help with that. In the ds_list all desired information is then stored. But you are right, we don't know if that is the case, so u8 might be all that is needed.
 
H

Homunculus

Guest
the index of a data structure is a number. Each time you create a data structure that number is different (probably incrementing) someday you have 255, then what?
It is most likely that the ds_grid holds more than 1 information per cell. So a pointer to a ds_list inside memory should help with that. In the ds_list all desired information is then stored. But you are right, we don't know if that is the case, so u8 might be all that is needed.
I get the first part and agree, but this assumes that the grid holds references to data structures. If each cell keeps more than one info it may very well be the case, but it would be still wrong to store the data structure index in the savefile, because when you run the game and load that data there's no guarantee that those references are still valid.
 

Simon Gust

Member
I get the first part and agree, but this assumes that the grid holds references to data structures. If each cell keeps more than one info it may very well be the case, but it would be still wrong to store the data structure index in the savefile, because when you run the game and load that data there's no guarantee that those references are still valid.
I didn't even think of that. Oh well, bitmasking it is.
 

GMWolf

aka fel666
If you are going to nest data structures and save them, Json is a good bet.
Problem is that it won't deal with non tree data structures (so if multiple nodes reference the same node, then you need something smarter).
 

Niften

Member
@Simon Gust I store chunks differently. Instead of baseX and baseY there are chunk IDs. @Fel666 I'll check these videos out. Edit: I pretty much have done exactly what you outlined, except it doesn't use buffers. I found a nifty extension a while back that does: ds_grid_write_buffer and ds_grid_read_buffer...
Code:
/// @description ds_grid_read_buffer(buffer)
/// @function ds_grid_read_buffer
/// @param buffer
var b = argument[0], ww, hh, g, /*header_size = 8 + 8 + 2, */number_format;

buffer_seek(b, buffer_seek_start, 0);

// extract header
ww = buffer_read(b, buffer_u64);
hh = buffer_read(b, buffer_u64);
number_format = buffer_read(b, buffer_u16);

g = ds_grid_create(ww, hh);

var i,j;

for (i=0;i<hh;++i) {
    for (j=0;j<ww;++j){
        g[# j,i] = buffer_read(b, number_format);
    }
}

return g;
Code:
/// @description ds_grid_write_buffer(grid[,decimals])
/// @function ds_grid_write_buffer
/// @param grid[
/// @param decimals]
var g = argument[0], ww = ds_grid_width(g), hh = ds_grid_height(g),
    number_format, dec = false, header_size = 8 + 8 + 2;
 
if (argument_count > 1)
    dec = argument[1];

number_format = buffer_number_constant(ds_grid_get_min(g, 0, 0, ww - 1, hh - 1),
                                   ds_grid_get_max(g, 0, 0, ww - 1, hh - 1),
                                   dec
                );

b = buffer_create(ww * hh * buffer_sizeof(number_format) + header_size, buffer_fixed, 1);
                          
//write header to buffer
//header: ww(64),hh(64),type(u16)
buffer_write(b, buffer_u64, ww);
buffer_write(b, buffer_u64, hh);
buffer_write(b, buffer_u16, number_format);

var i,j;

for (i=0;i<hh;++i) {
    for (j=0;j<ww;++j){
        if (dec)
            buffer_write(b, number_format, g[# j, i]);
        else
            buffer_write(b, number_format, floor(g[# j, i]));
    }
}

return b;
I think I tried to use these scripts at once point but I was getting a weird problem so I gave up on it. I'm going to revert back to these scripts.

Also, a temporary fix that I've had is generating chunks before the player loads in. Because my chunk system only saves when the player leaves the chunk & if it was edited, this makes exploration way less laggy. Of course this is only temporary and I don't want the player getting lag spikes every time they explore newly generated chunks. Thanks for all the help guys and I'll update after I implement these scripts.
 
Last edited:
Top