GameMaker Data Storage Concept?

Neptune

Member
Hey guys, I'm wondering about good ways to manage lots of data.
For example, say I have 150 items in my game, and each item has 25 pieces of data attached to it (a grid or array), like sprite, name, description, cost etc etc.

How could I store this data (outside the game?) in such a way that I could change the data only ONCE, and my game would have the updated data, and a wesbite would have the updated data (like a simple online wiki)?

I understand this question might not be cut and dry, but I need a good place to start...

Any information is appreciated!
 

TsukaYuriko

☄️
Forum Staff
Moderator
Would JSON files be a suitable option? There are built-in functions to load them and there surely are libraries for using them in web development.
 

Neptune

Member
Would JSON files be a suitable option? There are built-in functions to load them and there surely are libraries for using them in web development.
Hey thanks for the reply.

Maybe I should store everything as CSV data, and then convert to JSON files, which could be included in the GM project, and also used for the web dev? Idk, I could be thinking about this totally wrong...

It feels like the extra step of JSON might be unnecessary -- ideally going straight from CSV to both my project and a website would be ideal.
 
Last edited:

Joe Ellis

Member
You could in theory just write a simple text file:

write: number of variables
repeat (number of variables)
{
write: variable_name
write: value
}

And then make a script that reads each line and loads it into the variables.

But you're talking about storing data in a file, that's basically how any rom or game data file is stored, and the game that uses it is programmed to load it,
I'd look into\or practice storing data with buffers, cus that's the most direct way to deal with data, and you have to do everything yourself, rather than using a function that saves a json file for you.. your learning nothing there, most real programmers know how to deal with binary data, and I think its important
 

Neptune

Member
@The Reverend @Joe Ellis
Thanks guys.

I'm wondering though -- is my thinking right about this?
I'm thinking of storing a bunch of my game's item data outside of the game, to then be included files and finally 'read' into the game.
Is this a common thing? How else would you have uniform information in a game and simultaneously on a website?
 

Tsa05

Member
@TsukaYuriko is wise. Json is beautiful, Json is good.

Let's sat you have an object in a game, and the object has data.
Let's give the object a data structure:

myData = ds_map_create();

Done.

Now, the data has some properties:
Code:
myData[?"sprite"] = "nameOfSprite";
myData[?"name"] = "nameOfObject";
myData[?"description"] = "description";
myData[?"cost"] = amount;
Done.
This much, then, is easy with code. You can quickly and easily design level editors and such, and set these properties accordingly and use code to place the data into a map. And this method goes deeper.

Suppose the object's data should include...a list of items.
Code:
myItemList = ds_list_create();
ds_list_add(myItemList, "item_0", "item_1", "item_2");

ds_map_add_list(myData, "itemList", myItemList);
Done. Now, the value of myData[?"itemList"] is a list of data, embedded within the overall data structure. So yea, you can make a structure with names and values....and the values can be regular text or number data but ALSO map and list structures. Maps inside of maps. Lists inside of maps. All the data in one structure. Easily accessible.

Now, you want to store this as text in a file, right? Json. The ds_map and ds_list structures convert to and from json natively. Check this out:
Code:
json_encode(myData);
Tadaaaa! Your whole data structure, with nested lists and all, is now a simple string that you can write to a text file. You can read it out from a text file and convert it back to maps and list like this:

Writing it down? Depends on how you want to store it, but could be this simple:
Code:
var file = file_text_open_write("my_filename.txt");
file_text_write_string( json_encode(myData) );
file_text_close(file);
Just like that, you have a file with all of that object's data. But now, you need to read that data back in, and put the data back into maps and lists for the game, right? So you create your object, and then you do this:
Code:
if( file_exists("my_filename.txt")){
     var file = file_text_open_read("my_filename.txt");
     var data = file_text_read_string(file);
     file_text_close(file);
     var map = json_decode(data);
     if( map!=-1 ){
          myData = map;
     }
}
That's a bit thorough, more code than strictly needed but here's the idea:
  1. You create the object. Since you don't know initially whether there's saved data, you create, in the Create Event, something like a default value for myData...maybe something like -1. (Bear in mind that you'll want ALL of your code to check for whether myData is -1 before reading from it. This is not specific to ds_maps....any circumstance where you load external data into your game *should* have code to check whether the data is ACTUALLY loaded before trying to read from it, and a condition to fail gracefully with an error if not.
  2. The object checks for the existence of a file that should have its data.
  3. If the data is there, read it, and attempt to turn it into a decoded map structure.
  4. If that succeeds, myData is now equal to the map containing all of the needed data and data lists.
  5. If it fails, myData remains -1 or whatever--it doesn't change at all as a result of this process since no valid map was found.
That's the idea, in a nutshell, and GameMaker does all of the heavy lifting in terms of building out your data structures for you. Now, if want to edit the data *externally* instead of building a ds_map in code and using json_encode(), you can do that, too. The data is readable, assuming you know the basic format:

{Curly Brackets} are Maps (objects in json)
[Square brackets] are lists
The rest is data.
Data values are referenced by names, or "keys."
Keys and value pairs are separated by : colon
Data is separated by comma (,)

Everything I described above, then, looks like this in JSON.
Code:
{
     "sprite" : "nameOfSprite",
     "name" : "nameOfObject",
     "description" : "description",
     "cost" : 10,
     "myItems" : [ "item_0", "item_1", "item_2"]
}
Notice that the entire thing is within curly brackets, and then the name of a thing is in quotes followed by a colon, followed by the data stored with that key, followed by a comma.
Most of the key names are followed by just a piece of data, but the last one is followed by a square bracket, which means that a LIST is stored at that key. Then each item appears in the list.

Back in GameMaker, you might have something like this to read and use the sprite info:
Code:
if( myData!= -1){
     spriteName = myData[?"sprite"];
     sprite_index = asset_get_index(smriteName);
}
To get things from the list, it might be this:
Code:
if( myData!=-1 ){
     var myList = myData[?"itemList"];
     first_item = myList [|0];
}
Note that I retrieved the list from myData, then I can use it like a regular ds_list, including getting the size and looping through items, or whatever I need. (And note that I used the | accessor, since it's a ds_list, not an array).

Once you're working with maps/lists in your code and saving them all out to JSON, it becomes pretty quickly apparent how insanely powerful everything gets. Since everything in a map has a name, it's very powerful and easy to modify your data.

Last point to make, by way of comparison, if you haven't drifted off to sleep yet...
When I first started using this stuff about 4 years ago, I was working with data set property stuff like you. I was storing things in "long sets"--basically the CSV approach, where each "row" is an item, and each "column" is a piece of data about the item. Here's a fun problem I kept having:
This row starts at counting index zero and has properties: shape, color, x, y, description. EZ. Need the X and Y coord? Look up item 2 and 3.
WAIT! I need width and height, too. That'll be useful! Ok, I can put those in after x, y....but that means the position of description just changed, so I've got to change that code, too.........OR I can just tack width and height onto the end. Ok. New row data:
shape, color, x, y, description, width, height
Wait, the color should be customize-able! I need r, g, and b! Ummmmm. If I replace "color" with R, G, and B, then ALL of my data shifts over by 2 spaces. Unless I store it as a string and parse it later?
WAIT! All of these custom appearance settings should go into a central style sheet, to be shared by multiple items...um... So...really, I need:
style, x, y, description?
But that shifts all my data around again!!! GRAREORGRGGE!

And then I discovered that I could just use data structures. Need to change color from a single value to a list of 3 numbers? Fine. Instead of "color" : "ff0000", make it "color" : [255, 0, 0]. The code that reads and interprets what to do with color has to be changed, of course, but no other data in the structure is affected in any way at all. Re-structuring things, and inserting lists and even entire additional map structures into things is simple and easy, and quite robust.
 
Last edited:

Neptune

Member
@Tsa05 Thanks for the reply.
The data im concerned with is stored in nested grids, which I'm assuming will not work with the encode/decode functions.

It sounds like I'd have to write a script to make my own encode functionality -- I don't need to decode.
If I can keep the data stored in the game, I can just encode it into json format, and then it sounds like it can easily be handled on the web dev side from there...
 

Tsa05

Member
@Vether Well, strictly speaking, a grid is a list of lists...

Code:
myData = ds_map_create();
var grid = ds_list_create();
ds_map_add_list(myData, "dataGrid", grid);

repeat(10){
     var list = ds_list_create();
     ds_list_add(grid, list);
     ds_list_mark_as_list(grid, ds_list_size(grid)-1);
}
You now have a grid containing 10 rows of any length you please. This grid will be stored as part of json_encode, and will be automatically re-built from a json string using json_decode.

Accessing data would involve getting the grid from your map:
Code:
var myGrid = myData[?"dataGrid"];
That gives you a list. Then, you get the row and column like this:
Code:
var row = 4; // For example
var col = 3; // For example

var myRow = myGrid[|row];
var dataPoint = myRow[|col];
You get the row, you get the column. Pretty much just like a grid, right?
Heck you can write a helper function, like grid_get( grid, row, col) that combines that piece if you want to.

Great thing is.....You don't have to just put numbers and strings into the grid this way. You can put whole maps, lists, and lists/maps containing lists and maps in there too. Whole json structures inside a json structured grid.

Done in that way, the whole thing would have a native way to convert to and from strings via json, for east saving to a file, sending over the web, etc.
 

Neptune

Member
@Tsa05 Interesting, so what does a ds_list look like that has been converted to json?
If my list was like this:
Code:
var list = ds_list_create();
ds_list_add(list,"a");
ds_list_add(list,"b");
ds_list_add(list,"c");
And I encoded it... Would it look like this?
Code:
{
     "" : "a",
     "" : "b",
     "" : "c"
}
I'm thinking a script to convert my grids might be easier than switching the way the data is handled into lists (yes, i understand grids are fundamentally lists, and vice versa).

Could I just create a script like this (assuming the grid is just a 1xN):
Code:
var json_str = "{";
for(var i = 0; i < ds_grid_height(grid); i++)
{
    json_str += "name : "+string(grid[# 0,i])+",";
}
json_str += "}";
I could use a while loop for nested grids, and I could even throw a switch in there to set the names for this data, but in my case it doesn't need to be human readable.

I have no idea if this is a viable solution -- do you think this could work?
 
Last edited:

Tsa05

Member
@Vether
It would not look like that--having a bunch of empty strings as keys would not serve a purpose. Lists in Json are lists. A list containing a, b, c looks like this:
Code:
["a", "b", "c"]
Maps contain named data, lists contain ordered data (no name is required, as the items are in order, thus able to be referenced numerically).

If you desired to create a json data object containing a list, you could do this:
Code:
var gh = ds_grid_height(grid);
jstring = "{ \"list\" : [";
for(var i=0; i<gh; i++){
     jstring += string(grid[# 0,i]) + ",";
}
jstring += "] }";
The resulting data string looks like this:
Code:
{ "list" : [ "grid 0-0", "grid 0-1", "grid 0-2" ] }
This is valid Json, which can be natively picked up by GameMaker or many web-based services.

But just to be clear, you can also do this:
Code:
var map = ds_map_create();
var gridRows = ds_list_create();
ds_map_add_list(map, "list", gridRows);

var gh = ds_grid_height(grid);
var gw = ds_grid_width(grid);

for( var row=0; row<gh; row++ ){
     var rowList = ds_list_create();
     ds_list_add(gridRows, rowList);
     ds_list_mark_as_list(gridRows, ds_list_size(gridRows)-1);
     for( var col=0; col<gw; col++ ){
          ds_list_add(rowList, grid[#row, col]);
     }
}
jstring = json_encode(map);
This converts the entire grid into a list of lists with all of the grid data, rather than converting just one column.

The secret to leveraging GameMaker to do all the string formatting for you lies with the add_map and mark_as_list functions. Once a data structure has been added to another in a way that GameMaker is aware of, it performs all of the formatting. Remember that by default maps, lists, and grids are referenced by number in GameMaker, so merely adding a list to a map would place a number into the map; same with adding a list to a list. You'd just have numbers. But, by using the functions above, you specifically tell GameMaker to establish that the number inside of a map or list corresponds to a data structure, and in this way, everything converts to a json string easily.

In the above looping code, I've created a map with one named data entry called "list" and the entry contains a ds_list data structure.
Then there's the for loop, which has got to place your data into a grid-like list structure.

First, each row must be represented as a list of its own, so a list is created and added to the grid-list. Then, data is inserted, and the process is repeated.
And, you get something like this: A list of "rows" with no data except for the ID of a list... and within each of those lists, the actual column data for each row.

In JSON, the string looks like this (I added linebreaks for readability--they are ignored by the parser):
Code:
{
    "list": [
        ["a", "b", "c", "d"],
        ["e","f", "g", "h"],
        ["i", "j", "k", "l"],
        ["m", "n", "o", "p"],
        ["q", "r", "s", "t"],
        ["u", "v", "w", "x"]
    ]
}
 
Top