Asset - Scripts GMSDB 3.3 - Database system

P

PvdH

Guest
Hello Catan, what do you think, how big can be data i can keep in your extension? Is it possible to store there more than 30k fields?
I've read its hard to keep there a big data because everything is saved inside memory...
I would love to buy it, just want to be sure
 

Homunculus

Member
30k should be ok imho, but it really depends on the kind of data you store and the platform you are exporting to (mobile devices have less memory). I'd say under 50k should be considered ok in any situation.
 

Homunculus

Member
Almost a year has passed since the last update, and I want to let you guys know that the project is still alive, and a new major update is coming in the next few weeks.
This update will be GMS2 exclusive, since almost everything is being rebuilt from the ground up to take advantage of the new features that have been added to Game Maker and GML in general. This includes a lot of minor changes, but a few very relevant ones:

Array literals: they play a huge role in this update, mainly as as script arguments, and allowed me to get rid of script arguments passed as strings of comma separated values. This means no more string parsing and overhead introduced by type casting.

New persistence model: you will be able to manage the program memory taken by the database by loading and unloading the data from and to the file system, on a per table basis. You will be able for example keep a few lightweight tables always loaded and ready to be queried into the memory, and load/unload the heavier ones on demand from/to the file system at any time, in a very simple way.

Column defaults: you will be able to specify a default value table columns, simplifying the process of inserting new records


If you have any question or request before the actual release, feel free to ask.
 

Homunculus

Member
Version 3.3 is available! Check out the changes in the open post.

As announced in my previous post, this is a GMS2 only update. This update is the result of a major rework on the whole codebase in order to take advantage of the features that have been introduced to Game Maker over time, especially in the transition to GMS2.

If you have already used a previous version of GMSDB, you will recognize most of the scripts and see that the workflow is essentially the same on the surface, but a lot of minor things may be different, so be sure to check out the new updated documentation. The database format has also been slightly changed to reflect the new table structure, and this means that you can't import old databases into version 3.3 (should you need to do this, it is still possible to export / import the data as CSV).

Feedback is welcome, and as always if you run into any problem do not hesitate to let me know.
 

kinfox

Member
HI,
I am a Chinese gamemaker,and purchased your software for database。
I have some trouble when I want importing CSV.
Since I used gms2, I updated it to version 3.3.
But I found in 3.3 version,
db_table_import_csv does not support utf8
the previous version is supported.
my CSV data must be utf8 format for Chinese
so what can I do?
how to deal with it
Need your help

Look forward to your reply
 

Homunculus

Member
Hi kinfox. In version 3.3 i switched from my own csv importer to the native one, load_csv(). That’s probably why you are experiencing this problem.

If it is in fact a problem with load_csv, I’ll try to port the old csv importer to version 3.3 to solve this issue. I’ll let you know
 

Homunculus

Member
Do you have any demo we can try before buying? I can't seem to find it anywhere.
Not really, but this isn't an asset like an inventory system or a graphics pack where you can expect a very specific output. It's just a generic database system, I can't imagine how a demo could help understanding how the system works or if it fits your criteria.

I could send you a compiled demo of the database viewer (included as demo project) if you'd like, contact me by pm or email in this case.
 
I purchased this for a CLIENT / SERVER game I am making, I am using it to store players data and last positions, bags, bank and admin information. Its really easy to use and i haven't had any problems! Thanks
 

mjadev

Member
I think that the last update of GMS broke GMSDG (certainly related to string-handling updates)

for info, I'm using IDE v2.2.2.413 / runtime v2.2.2.325 (and not 2.2.2.326 because too many crashes with this one when trying to import resources...)

Here is the error I have when I select pokemons table in db_demo:
___________________________________________
############################################################################################
FATAL ERROR in
action number 1
of Other Event: User Defined 2
for object obj_db_explorer:

unable to convert string "" to number
at gml_Object_obj_db_explorer_Other_12 (line 23) - var _query_where_value = is_string(table_defaults[? query_where_column]) ? string(query_where_value) : real(query_where_value);
############################################################################################
--------------------------------------------------------------------------------------------
stack frame is
gml_Object_obj_db_explorer_Other_12 (line 23)
called from - gml_Object_obj_db_explorer_Other_11 (line 27) - event_user(OBJ_DB_EXPLORER_EVENTS.query_run);
called from - gml_Object_obj_db_item_Other_12 (line 15) - event_user(OBJ_DB_EXPLORER_EVENTS.table_load);
called from - gml_Object_obj_db_item_Other_10 (line 24) - event_user(OBJ_DB_ITEM_EVENTS.perform_action);
called from - gml_Object_obj_db_toolbar_Step_0 (line 17) - event_user(OBJ_DB_ITEM_EVENTS.pressed);
 

Homunculus

Member
@mjadev I'm aware of this problem, I'm working on a fix that will be uploaded shortly. As you said, it's related to the new runtime and its changes to string handling.
Please note that this only affects the provided demo project (db explorer) and not the core asset resources, as far as I know there are no problems with those.

Thank you for the feedback!
 

jashe369

Member
I am looking for a little help using your code. Here is what I have , I have verified it is saving the data correctly, but accessing it is just creating trash .

So my goal is to display data as a list. on the screen here is the code i am using.


This seems to be what is need to get a DS_list of database based off the documentation.
Code:
    if db_table_exists(filelocation)
                {
                    db_table_load(filelocation)
                    var dirlist = db_query_select_ids(filelocation);
                    db_cl_where(dirlist,db_op_eq,["UserName"],[username]);
                    db_cl_where(dirlist,db_op_eq,["PassWord"],[password]);
                    var dir = db_query_exec(dirlist);

                }
Then here is what I am doing to print out the data line by line in my code. This should print the date but I don't get that at all.

Code:
            for(var i=0; i < ds_list_size(dir); i++)
            {
                  
                    var dir_map = ds_list_find_value(dir,i)
                  
                    global.oldtext[global.currenttotallines] = ds_map_find_value(dir_map,"Date")
                    global.currenttotallines += 1
            }
Here is the stored file for the records
Code:
db_record_create("basedrive",[ ["FileName",+string(third)], ["Date","2/12/1992"], ["Location",filelocation], ["FileOrDir",1], ["FileSize", ""], ["UserName",username],["PassWord",password] ])
As you can see date should be date should be "2/12/1992" it isn't

Other things to note filelocation is a var that is used to dynamically place tables

Here is what the out put is




So what am I doing wrong ... sadly I don't use DS_list or maps often ... so I am just going off how they should be set. Since I need to just spit out lines of table data like in your database viewer I thought about copying that code, but how you do it is so convoluted and not well commented at all so I couldn't do it. from what i saw you draw the rows in the draw even of one thing and use a script to draw the columns of each row. Then you use the values from other objects to do that... it was very much all over the place. I get that it was designed to be efficient, sadly it just isn't something I could more or less copy and modify to do what I want to do.


Again if you have any ideas on how to help me that would be great.
 

jashe369

Member
So over time I got a solution, I just end up using db_query_select because I can treat it like a ds_grid and ds grids can be treated like a sudo 3d array in code.

Where the first value is the grid id the second value is the row and the last is the column of the db.

It took me a little work but it is simple enough and the ds_grid _destroy works fine for it so its good.
 

Homunculus

Member
I currently can’t test the code you posted (i’m on holiday) but the problem is probably related to the query function you chose: db_query_select_ids returns a ds_list of record ids (real numbers). You are trying to use those as a ds_map, while in fact you should do one of the following:

A. Use db_record_fetch / db_record_fetch_all on the resulting ids

B. Use db_query_select_select_records instead, to return a list of ds_maps

Using db_query_select works as well, but you don’t get the benefits of having your records in a ds_map
 

jashe369

Member
To be honest the ds_grid works fine.

When I first started work on my project, I was just going to make a 3d array for my data and store it in a file with some obfuscation so it wasn't easy to read. I haven't used game maker in a while and forgot that game maker doesn't use 3d arrays. Then I thought maybe I could just use a DS_map in a DS_List blah blah blah. I saw your DB on the store and figured may as well use this, sadly there are some issues with it for my perpose and it is also way more powerful than I need in other areas.

I had forgotten all about DS_grid ... It actually works perfectly for my code as I can still write everything as if it were a 3d array. The only hard part is remembering to dumping the results DS_grid variable so I don't cause memory leaks, and your DB system works great for what I need it for with the DS_grid.

Here is what I am trying to do, So basically I have a fake computer console with fake file system and fake username /passwords So each DB is a drive on the system and each folder is a table ... The only issue is I have to use another character for / or \ and a few other characters for folders. but I understand way. I just wish I could make an easy work around. right now I am using ) and ( in plays of / or \

I assume that you are using separate files for each table to make the data retrieval process much faster. I wish there was a option to use a single file so I could use all characters to name tables so I could have tables match folder names in parent directories of the fake file system. That is just a thought though.
 

Homunculus

Member
The idea of having a table per folder doesn’t sound right to me, but I don’t know the specifics of your project.

I’d probably have a single table where every record is a folder, and use a column to store the id of the “parent” folder. This way the folder names are not tied to the filesystem either.
 

jashe369

Member
I had each folder as a table to store the files and directories in said folder as a list, but also had it so I could use a table to go to each folder to make it easy to list the folder....

Then it occurred to me , and I rewrote my code today. To just make each drive a table in the database for the "System" it is running on. Then I can use the file location value to insure that files are in different locations. and set path via a path value and use that to move around. sure its a lot to have a whole file system in one data base but for my purposes it will not be larger than the example database so it shouldn't be an issue in storing all files and folder names in a single table that will be the drive name table.

The game is like one of those text adventures, but you are using a pretend computer console to do the adventure and you have to explore the fake file system to get clues.

I think the new way I am handling things will work out better since I can now use any values which should be great.
 
@Catan , Just bought this Asset and your exInv one too. I am new to GMS2 but I've watched several vids on data, load and save. As far as I can see I can use your Asset as a save load system and to store runtime info as well on say different players and inventories for say a 4 character RPG, whatever? How does using your Asset/solution work when compared to using a combination of ds_maps, ds_lists and JSON? Just interested in that. How would you approach an ARPG and storing and saving out base/template data on skills, stats, inventory and then also saving at runtime custom items, stats,skills, inventory etc? (also - can I persuade you to author an Asset that properly handles stats, buffs, debuffs, timed buffs, debuffs, progression of stats/skills? ;) Just asking on that...)
 

Homunculus

Member
Hi @Niall Edwards , you are right about GMSDB having an automatic save feature, so once setup properly it can definitely handle any kind of data that needs to be persisted to file without having to worry about data formats and file handling. Being a general purpose database you can save inventories, player information and stats, or whatever you want in there.

Compared to data structures serialized to JSON, GMSDB is heavier and definitely more complex, but this is because it also offers much more features like custom queries, sorting, lazy loading / saving from disk etc..., therefore if you don't need any of those features, just ds_stuff and JSON are probably a better choice.

For inventory management, you don't really need GMSDB if you are using EXINV, since the latter has its own inventory specific database and save format. You can however use both, limiting GMSDB to stuff that isn't related to the inventory, like skills and stats.
Unfortunately words like "stats" and "buffs" can mean A LOT of different things depending on the game you are trying to make, I really can't see any meaningful way to create an asset that covers all the possible or even the basic uses cases for those, it's too project specific imho.

Cheers!
 
@Catan , thanks for the v quick response. I think I will start by working on creating the 'static'/template stuff in GMSDB and thereafter consider how much in runtime/customised/changed in runtime should be shunted in to GMSDB and how much not so much.... [edit] so long as I know the names of objects and sprites, or maybe just objects, I presume it's easy enough then to create tables at least in csv (Excel csv) and import etc? Then ultimately save out the big database?
 

ze1

Member
Hi, I have a doubt about this new update. Since the database is stored in a text file in the player's computer, does it mean it can be exploited by editing that file?
 
Last edited:

DarK_SaCoR

Member
Hi @Homunculus, I want export my database as .csv but my system uses ";" as separator and I can't change it when I want import it.

As solution I can change in Windows "," as "." and ";" as "," but in Spain, we use "," as decimal:

I wait your answer and thanks in advance.
 

Homunculus

Member
Hi @DarK_SaCoR ,

when exporting you can specify the separator character, so it should not be an issue. The import process uses Game Maker's own load_csv function, which doesn't allow you to specify this option. I'm aware this can be a problem and I'm already planning to provide an alternative in the future, but I'm holding off non essential updates until the new GML updates are out.

You can definitely convert the separator manually though using excel or another program meanwhile, the fact that "," is used as decimal separator is not a problem at all in the CSV format, even when using "," as value separator as well.

Hope this helps!
 
Last edited:

DarK_SaCoR

Member
Since I had seen that in the documentation the import function did have the separator as an argument, I thought that it could. Until then I will use some of these solutions. I will be watching for new updates.

Thank you again, @Homunculus!
 

Homunculus

Member
Since I had seen that in the documentation the import function did have the separator as an argument, I thought that it could. Until then I will use some of these solutions.
Yeah that used to be the case in a previous version of the asset, but my own CSV parsing script was less than ideal, and when load_csv became available natively, I decided to switch instead of improving what I already had.
 
Top