GML How to Store Data for Games in GMS Best?

Petrik33

Member
So, hello everybody. My question may sound strange but I really want to explain it more clear to get an answer to this important thing.
First of all, I mean what is the most efficient way to store something like database of the game. Or another example, from my latest project:

Say, I want to have Narrator in the game, that will play different sounds during the whole game. Of course such sounds will be stores in variables and reused later, but what if I want to change the voice of that Narrator? So I mean, the question is how should I hold the data about the Narrator in the most efficient way?

This is my question in short.
But if you need another example, when I said about databases, I meant something like in my previous project:

Say, I want to make a football simulator, with players that have different stats, photo images, names, nations and else. And then when the match starts this data will be used to create players instances.

So how should I hold my data, that I can use it in GMS and edit easily probably outside GMS?
 

curato

Member
especially if you are doing a big multiplayer deal a cloud database is the best way to store that data so all the clients can see it. In the game itself DS_Map works pretty well to be able to grab information and save it to a file for later use.

Best advice I can give you is start simple. If you are thinking of doing something massive you should know the best way to go about things first hand to save a lot of pain on the back end. There are many ways to store data and get it back. Heck one of the first complete games I every wrote was a basic tile game had only 27 tiles and each had a unique value. I just used a basic array to hold the game data. You learn the strengths of each data type and the real trick is to lean in to theirs strengths so it makes your project so much easier.
 

kburkhart84

Firehammer Games
I generally see 3 issues to consider with this.

1. How do I store it during gameplay? The upcoming beta 2.3 has these things called structs which could be used to great effect for information organization, and if this is a new project it may be worth starting one there(despite the beta status). Alternatives are arrays, lists, grids, and the like. With arrays and lists, if you know for sure what array index gets what point of data, then that's all it takes to store and access the data assuming you document what you are doing so you don't forget later what is what.Structs are generally better in every way however, as you can directly name the variables inside them, and then create an array or list of them for all the different players for your sim/game.

2. How do I store it outside of gameplay? This will generally involve files that you store on the disk somewhere. Since this is something you are making before the game's release, putting the files in the "included files" seems to be the best bet. And with version 2.3, you can just put the files in the right folder and it automatically gets put in the IDE(instead of having to add them like sprites, etc...). It also means that if you update them, the update automatically is integrated instead of you having to re-add it like before version 2.3. As far as what the file format itself is, that depends on how you have it stored in step 1, and how you choose to convert it to file data(text or binary). There are data structure strings(which some people don't like, but work fine for me), there are JSON strings(which have integration with compliant software and is a good known standard if that matters to you). You could put the data in strings with some character separator between values(like "2|3" storing 2 and 3 in order with the '|' separator), which means you need to create and parse that string pulling the data from your runtime storage. Then there are binary formats however you want to do it(you can look up formats for images to get some simpler ways to handle that).

3. How do I edit this data in an easy way? If this is something I'm going to be constantly tweaking/balancing, I like to make an actual nice editor for the data. This editor would read the files the same way your game itself would, and then what the editor needs will depend on the data itself. This means opening the file, seeing what all players are in it(and showing the list of players). Then you would want a way to select one of the players in order to edit the details/stats for it. You also need to display and allow modification of those. Finally, you will need to be saving that data either automatically or on demand clicking a button(or exiting, whatever you want).
 

samspade

Member
It is hard to know for sure, but based on your description it sounds like you could use either a csv file or JSON. If you image storing data in a table, then csv. If you imagine storing it in a more complicated way (e.g. nested structures and so on) then JSON (in the football scenario, JSON makes the most sense). Both are human readable and writable and have built in support inside of GameMaker.

In a shamless plug, I have a short JSON tutorial for GameMaker that will be out soon (I've recorded about half of it, but need to edit it still). It will cover what JSON is and how to write it, import it, and export it. And there will also be a simple project using it after that.
 

Petrik33

Member
Thank you all for your reply, actually I have tried JSONs for making database and I even used Google Spreadsheet and ExportToJson Function in it and it was pretty good, but I think some kind of editor would be much nicer and more convenient, as for the other example, I guess now I understand what eo I have to do:
As I develop the game in the new GMS 2.3 I can really try heading to included files with JSON containing the data about the Narrator Audio: but here comes the question: how do I store the external data in JSON? Is there a better way then storing the name of the asset in GMS and then recieving it using asset get index?
And also, someone has mentioned the CSV files, and how can I use them, does anybody know a good tutorial on it in GMS?
 

kburkhart84

Firehammer Games
Thank you all for your reply, actually I have tried JSONs for making database and I even used Google Spreadsheet and ExportToJson Function in it and it was pretty good, but I think some kind of editor would be much nicer and more convenient, as for the other example, I guess now I understand what eo I have to do:
As I develop the game in the new GMS 2.3 I can really try heading to included files with JSON containing the data about the Narrator Audio: but here comes the question: how do I store the external data in JSON? Is there a better way then storing the name of the asset in GMS and then recieving it using asset get index?
And also, someone has mentioned the CSV files, and how can I use them, does anybody know a good tutorial on it in GMS?
As far as JSON usage in GMS2, there are actually dedicated functions for it. You need to look in the manual on it(or find a tutorial). I still recommend you create a local editor for it too, instead of counting on cloud or google sheets(unless you actually need the data in the cloud, which wasn't part of the OP).

I'm not sure where you get the idea of using asset_get_index()....that function is about getting the id of an asset using a string(not extremely useful but if you were adding strings together to decide what asset to use than it would work). But this function has nothing to do with JSON or even reading/writing files at all.

About CSV files...it literally means Comma Separated Values...so it involves the same process I mentioned above, using commas for the separating character in a string of values. I think there is also something about height and width of things as part of the format, but I don't know it that well. I'm sure you could google it and figure it out, and there may even be a tutorial for GMS2 on it out there as well.
 

Petrik33

Member
As far as JSON usage in GMS2, there are actually dedicated functions for it. You need to look in the manual on it(or find a tutorial). I still recommend you create a local editor for it too, instead of counting on cloud or google sheets(unless you actually need the data in the cloud, which wasn't part of the OP).

I'm not sure where you get the idea of using asset_get_index()....that function is about getting the id of an asset using a string(not extremely useful but if you were adding strings together to decide what asset to use than it would work). But this function has nothing to do with JSON or even reading/writing files at all.

About CSV files...it literally means Comma Separated Values...so it involves the same process I mentioned above, using commas for the separating character in a string of values. I think there is also something about height and width of things as part of the format, but I don't know it that well. I'm sure you could google it and figure it out, and there may even be a tutorial for GMS2 on it out there as well.
As for the JSONs I already have a self-made editor in my latest game which works properly. I use it for map-editing and it saves and loads successfully so it's ok.

And as for the assets, I use function asset_get_name, to save the name of the asset in GMS as a string, and then load it using that string, it is much safer than just saving index, because if you save the asset in JSON by it's index, and then change the order of assets in the tree of assets, you will then load a wrong one.

But I still believe there should be some other way, to store assets, may be there is even a way, of just storing the external files in code of JSON's but I don't know it.

And lastly I just want to ask, whether I need to find this way, I mean, is it inefficient to have ALL the sounds for ALL narrators as sound assets inside GMS? I mean does it make the game compile longer or just slowing it?

And also, will it be more efficient to have sounds as included files, and then load them when you choose the appropriate narrator???
 

kburkhart84

Firehammer Games
And as for the assets, I use function asset_get_name, to save the name of the asset in GMS as a string, and then load it using that string, it is much safer than just saving index, because if you save the asset in JSON by it's index, and then change the order of assets in the tree of assets, you will then load a wrong one.
Gotcha, that makes sense. I was like WTF you doing with asset functions in JSON, but I've seen what you are doing before. Some people just use indices but of their own array to handle that. I personally haven't ever needed it(I don't ever use JSON myself).

As for the JSONs I already have a self-made editor in my latest game which works properly. I use it for map-editing and it saves and loads successfully so it's ok.
The JSON and a custom editor is very possibly the best solution for you personally then, for this new game sports sim whatever since you already understand how to use it and even made a map editor for it.

But I still believe there should be some other way, to store assets, may be there is even a way, of just storing the external files in code of JSON's but I don't know it.
I mentioned multiple ways, including text files and binary files, both of which can be of whatever format you want, not just JSON. As far as something internal to GMS2 instead of being external, storing things in written code is the only thing I can think of(unless you made a bunch of objects and used the object editor to create variables for it, which would not be very nice IMO).

And lastly I just want to ask, whether I need to find this way, I mean, is it inefficient to have ALL the sounds for ALL narrators as sound assets inside GMS? I mean does it make the game compile longer or just slowing it?
I don't think having tons of sounds makes much difference with the compile/test time(but I could be wrong). I know that having sprites does, but then it caches the work of creating texture pages so it only happens once, until you make changes to the sprites. Sound effects can be done externally, whether as included files or just somewhere on the hard disk(assuming you disable the sandbox), but I don't recommend it, since there doesn't seem to be any disadvantage to having them be part of the IDE, unless you were making a system for modding it later.

Sounds don't have to be loaded at all times. You can set them to stream from disk, and that's what I would do for longer clips for something like a narrator, especially since you would only need one at a time going. If you look in the manual about settings on sounds, you will see what each one does, and when best to use each setting.
 

Yal

🐧 *penguin noises*
GMC Elder
I'm a big fan of arrays indiced by macro constants, which are initialized using helper scripts that lets you pass in all the data on a single line:
1596646318418.png

Efficient, readable and easy to deal with. Now when GM lets you store arrays inside arrays you can do some really neat stuff.
 

Petrik33

Member
Gotcha, that makes sense. I was like WTF you doing with asset functions in JSON, but I've seen what you are doing before. Some people just use indices but of their own array to handle that. I personally haven't ever needed it(I don't ever use JSON myself).



The JSON and a custom editor is very possibly the best solution for you personally then, for this new game sports sim whatever since you already understand how to use it and even made a map editor for it.



I mentioned multiple ways, including text files and binary files, both of which can be of whatever format you want, not just JSON. As far as something internal to GMS2 instead of being external, storing things in written code is the only thing I can think of(unless you made a bunch of objects and used the object editor to create variables for it, which would not be very nice IMO).



I don't think having tons of sounds makes much difference with the compile/test time(but I could be wrong). I know that having sprites does, but then it caches the work of creating texture pages so it only happens once, until you make changes to the sprites. Sound effects can be done externally, whether as included files or just somewhere on the hard disk(assuming you disable the sandbox), but I don't recommend it, since there doesn't seem to be any disadvantage to having them be part of the IDE, unless you were making a system for modding it later.

Sounds don't have to be loaded at all times. You can set them to stream from disk, and that's what I would do for longer clips for something like a narrator, especially since you would only need one at a time going. If you look in the manual about settings on sounds, you will see what each one does, and when best to use each setting.
Thank you very much,it wasa very helpful message, I guess the solution for narrators can be a few different containing names of sounds in different variables, and then those names can be used to stream External sounds, or probably the sounds included in IDE, I guess the question whether it affects efficiency remains opened. But anyway, thank you!
 

Petrik33

Member
I'm a big fan of arrays indiced by macro constants, which are initialized using helper scripts that lets you pass in all the data on a single line:
View attachment 33172

Efficient, readable and easy to deal with. Now when GM lets you store arrays inside arrays you can do some really neat stuff.
Oh my god, this looks crazy, could you please explain it a little?
 

Yal

🐧 *penguin noises*
GMC Elder
Oh my god, this looks crazy, could you please explain it a little?
The cliff notes version is:
  • I have all the item data in a 2D array
  • The first axis is the item ID, the second axis is the different types of data an item needs (e.g. guns need a sprite, ammo capacity, shoot script etc)
  • Instead of using magic numbers I've defined a million constants that are easier to remember.
  • I set up all the item data at game start by calling an item_init script with arrays of data for each item in the game, which are then put into the global.item_data array. (I actually have helper scripts for each item type, that pass on the appropriate data for that item type and fills in the defaults for unused data automatically, so things look cleaner)
 

Petrik33

Member
The cliff notes version is:
  • I have all the item data in a 2D array
  • The first axis is the item ID, the second axis is the different types of data an item needs (e.g. guns need a sprite, ammo capacity, shoot script etc)
  • Instead of using magic numbers I've defined a million constants that are easier to remember.
  • I set up all the item data at game start by calling an item_init script with arrays of data for each item in the game, which are then put into the global.item_data array. (I actually have helper scripts for each item type, that pass on the appropriate data for that item type and fills in the defaults for unused data automatically, so things look cleaner)
Thank you, guess I understand how it works, but I don't understand the reason for doing that, and why do you fill up the data using script, and not create event of manager object? That can be my lack of experience, so don't blame on me please)
 

Kezarus

Member
I'm a big fan of arrays indiced by macro constants, which are initialized using helper scripts that lets you pass in all the data on a single line
Good Lord, @Yal! Hahaha. Thank heavens that YoYo finally made structs available! XD

I don't really like that array workaround, but used it from time to time when something better wasn't yet developed.
 

kburkhart84

Firehammer Games
I'm a big fan of arrays indiced by macro constants, which are initialized using helper scripts that lets you pass in all the data on a single line:
View attachment 33172

Efficient, readable and easy to deal with. Now when GM lets you store arrays inside arrays you can do some really neat stuff.
If we weren't getting structs, this is 100% the way I would handle things(and have in the past). And using macros instead of magic numbers is also 100% good. It goes along with always using variables instead of magic numbers in any code(so you can change the values at the top, things like speeds, and similar).

Thank you, guess I understand how it works, but I don't understand the reason for doing that, and why do you fill up the data using script, and not create event of manager object? That can be my lack of experience, so don't blame on me please)
There are plenty of reasons for putting things in scripts instead of directly in events. It means that it can be called from anywhere for example, so if needed you could change which object's create event calls it, or stick it in room creation event, or have something else call it(like for a temp level test or something strange). It also means that it can be made portable and re-usable. Its really a good habit to get into, especially considering how you can re-use code in future projects.

The only glitch in this case is that it is using the technique I mentioned above to store the data in code, meaning there is no easy way to edit it externally, so all the tweaking/balancing has to be done directly in that code(unless you create something to directly modify the gml). For some people that is fine, but not for all.
 

Yal

🐧 *penguin noises*
GMC Elder
Good Lord, @Yal! Hahaha. Thank heavens that YoYo finally made structs available! XD
Still waiting for the stable version of 2.3 :p And when I get it, I'm going to have arrays of structs. The best of both worlds, right?
(I'd probably stick with N-dimensional arrays instead for a lot of "raw data" data, tbh... arrays still are much easier to serialize/deserialize since you just need to loop over them, so stuff like savefiles get much easier to handle if you have array-friendly data. But for things that are only ever gonna be in memory and get thrown around between objects a lot, structs is gonna be revolutionary)
 

Kezarus

Member
@Yal, at first I was against the idea of using arrays as an workaround to non-existent lightweight objects. But if we combine them with enums it's not so bad. ¯\_(ツ)_/¯

Talking about serialization, I was quite happy with the functions to transform Jsons into maps and back. Works perfectly. =]
 

Yal

🐧 *penguin noises*
GMC Elder
Talking about serialization, I was quite happy with the functions to transform Jsons into maps and back. Works perfectly. =]
In my games, there's a lot of serial batch operations even outside of savefiles (e.g. read all the customizable input keys into key status variables, or apply separate attack-vs-defense checks for every stat to compute total damage) so arrays just sort of works naturally... I hate repeating myself more than necessary if I can rewrite it to a loop-friendly form instead :p
1596721599719.png
(k_[] is the current key status, p_[] is how many frames it's been pressed prior to this frame - useful for things like detecting key presses/releases)

Contrast that to the much more verbose version I used earlier:
1596721735985.png
(this is from a simple jam game without gamepad support - my bigger projects would have more than twice as many lines)

Though I can't tell if I'm using arrays because I see loop-folding potential everywhere or if I'm trying to squeeze in loops everywhere because I use arrays for everything, it's a bit of a chicken-or-egg problem.
 

Kezarus

Member
I hate repeating myself
Yeah, me too. One thing that I teach begginers is "if you are copy-pasting too much, something is wrong".

apply separate attack-vs-defense checks for every stat to compute total damage
Ok, ok, hold on! That is one thing that I am VERY interested in! How did you do this? If need be I create a topic just for this. =]

And, to stay on topic, Jsons are pretty much market-standard. But one thing that I miss from GM lists/maps are filters. =/
 

woods

Member
sorry to derail the topic...
apply separate attack-vs-defense checks for every stat to compute total damage
also very interested in this ;o)

im thinking something along these lines is rather brute force -ish...

atk.strength+atk.wep+atk.lvl=atk.total_damage;
def.dex+def.armor+def.lvl=def.total_defence;

atk.total_attack - def.total_defence = damage dealt;
 

Yal

🐧 *penguin noises*
GMC Elder
Ok, ok, hold on! That is one thing that I am VERY interested in! How did you do this? If need be I create a topic just for this. =]
Simple:
GML:
var totaldamage = 0;
for(var c = 0;  c < ELEMENT_MAX; c++){
  totaldamage += max(0, other.atk[c] - def[c]);
}
 

Selek

Member
Still waiting for the stable version of 2.3 :p And when I get it, I'm going to have arrays of structs. The best of both worlds, right?
I've been tooling around with 2.3 for the past month, and I really enjoy using structs. I'm pretty new at this, but they sure seem to make things easier. I have no idea how efficient they are compared to other data structures.

In one of my current projects, I have a simple hex field for a turn-based wargame. (Yes, I've used Amit's fantastic website for cube coordinates, pathfinding, and other hex-related goodness.) I'm wondering whether to represent the hex grid as an array of structs. (Right now I just have a simple two-dimensional array of numbers.) With structs, each hex could have fields like terrain, movement cost, combat modifier, roads, and unit-occupants. Would that be an efficient way to do things?
 
With structs, each hex could have fields like terrain, movement cost, combat modifier, roads, and unit-occupants. Would that be an efficient way to do things?
I mean, this is totally doable without structs. Just hold a map or a list in each array position. Then the map or list can hold all the data you could ever want for each hex. If you want to know something about the tile, you can already find it's array position, so then just read the DS stored there and pull out the stuff you need to check (or don't even pull it out with 2.3, chain accessors babyyyyy).
 

Selek

Member
I mean, this is totally doable without structs. Just hold a map or a list in each array position. ...
chain accessors babyyyyy ...
Thanks for your reply! I haven't played with chain accessors yet, but they look great!

For now, I'm a bit reluctant to rely too heavily on maps or lists. Aren't structs supposed to be lighter-weight? And if I experiment with fancy AI algorithms, I don't want to be passing maps or lists around too much; I tried that with chess and had memory leaks. I don't suppose I'll be able to minmax with even a small wargame (14 x 6 hexes right now), but a guy can dream. :)

Also, I don't really need the flexibility of a map or list, at least not so far. A hex will always have certain characteristics like movement cost, terrain, etc.

But above all, structs are just fun to use, lol. I keep thinking up new fields to stick in 'em. :)
 
I haven't played around with structs yet, as I'm waiting till Spaceslingers is finished to upgrade to 2.3, but I'd be surprised if structs outweighed performance a lot in comparison to DS'. Also, you shouldn't really be passing the actual DS' themselves around, they're literally just there to be a "table" that you read from for each hex so that you can hold multiple data points. You'd only create each DS once when you generate the map and then destroy each DS at the end when you "delete" the map (as in, you've completed the game, the player has discarded the map in some way, etc). If you're more comfortable with arrays, you could just hold another array in each hex and do essentially the same thing.

Before I put Floramancer on the backburner, I was holding the "type" of cell for pathfinding purposes, whether it held a "hidden" bonus when stepped on, whether the cell had been cleared of "rot" or not (a mechanic for the game) and maybe a few other things. All the data was static and the DS' itself was never passed back and forth between anything, it's just there to be read when necessary and then destroyed when I no longer needed the map.

Having said all that structs would be very useful in the sense that you could apply transforms and stuff like hex.DoThis() or whatever the nomenclature is. It's definitely more flexible than the above method, I was just pointing out that you don't need structs to do such a thing. Once I'm done with Spaceslingers, I'm thinking of picking Floramancer back up, simply because of the added benefits of structs will make turn-based games and data heavy games a lot easier to work with.
 

Selek

Member
Thanks for your reply. Yeah, I was mishandling DS_lists in my chess engine. I was passing them back and forth between minmax functions, using the lists to hold different possible moves. I thought I was destroying them when done but I don't think I really was. I switched to arrays and performance improved -- and the leaks abated. I'm still a newb at this.

Your Spaceslingers looks great fun! I'm not up on my opera, but is that Bizet's Carmen in the trailer? And I'm glad to hear of your interest in turn-based games, my favorite genre. I don't see a lot of people working on turn-based stuff in GMS, but I may not be looking in the right places.
 
Indeed it is Carmen, hahaha, I thought it was hilariously appropriate myself, but pretty sure I'm going to have to redo the trailer as the Steam gamers don't seem to like it too much.

Turn-based games (and strategy games in general) are not often done in GMS precisely because of the dirty workarounds that you needed to do before 2.3, combined with the fact that pretty much every tutorial ever has been about "run and jump games" and they don't tend to delve into proper data management and how to work with more complex systems (meaning that people learn only the "action" side of making games and then stick with what they are comfortable with). I'm hoping to see at least a little bit of a change in that now we've got access to the new features.

Data structures can certainly be a pain in the ass when you're learning them (there's some quirks that, while they make sense from a global perspective, can really bite you when you're only learning), but they're ever so helpful once you get a handle on them. I don't think I've made a game, since I figured them out properly, that doesn't heavily involve them. I think @samspade has done some good tutorials about them recently (if I'm remembering correctly).
 

Selek

Member
Heh, well I like the Carmen soundtrack, so here's one vote for keeping it. :)

I will look for that data-structure tutorial you mentioned. I certainly see their value. I have relied on ds_lists in every project I've made so far, and for the pathfinding algorithm I just finished, I used a ds_map to link a hex to its parent.

And thanks for the explanation on why data-heavy games have not been a big thing here til now. I share your hope that 2.3 will help change that!

OK, off to bed for now. Thanks again for all your comments; very helpful.
 
Top