saving/loading nested ds_maps

W

Woochi

Guest
Hi, i have been trying to save and load nested ds_maps, partially without succes. I need nested maps for saving player data, as i save the data of all players in one single file. So the highest map of the file contains the account names, where each is a map on its own, containing data like password, level, name, but also containing another map representing the inventory of that player.

How i have been doing it:

saving:
1. have ds_map A and ds_map B.
2. add map B to map A under key K, using ds_map_add_map.
3. save map A, using ds_map_secure_save("filename.json").

loading:
1. load map A from the previously created file into variable X, using ds_map_secure_load("filename.json").
2. assign map B to variable Y, using ds_map_find_value(A,K).

then typically, when i check with ds_exists wether Y exists as a map, it returns false. But when check X, it returns true.

exact code (creating account/character):
Code:
        with (obj_textbox) {
            if (self.name == "reg_acc") {
                var acc = self.txt;
            }
        }
        with (obj_textbox_crypt) {
            if (self.name == "reg_pass") {
                var pass = self.txt;
            }
        }
        with (obj_textbox) {
            if (self.name == "reg_name") {
                var name = self.txt;
            }
        }
      

        var create_data_all = ds_map_secure_load("playerdata.json");
      
        if (ds_map_exists(create_data_all,acc)) {
            scr_sysmsg("Account already exists.","reg",c_white);
        } else {
            if (string_length(acc)<1) {
                scr_sysmsg("Enter account name.","reg",c_white);
            } else if (string_length(pass)<1) {
                scr_sysmsg("No password given.","reg",c_white);
            } else if (obj_controller_reg.pick == "") {
                scr_sysmsg("Decide the character's race.","reg",c_white);
            } else if (name == "") {
                scr_sysmsg("Name the character","reg",c_white);
            } else {
                var create_data_player = ds_map_create();
                ds_map_add(create_data_player,"user",acc);
                ds_map_add(create_data_player,"name",name);
                ds_map_add(create_data_player,"password",pass);
                ds_map_add(create_data_player,"race",obj_controller_reg.pick);
                ds_map_add(create_data_player,"lvl",1);
                ds_map_add(create_data_player,"Exp",0);
                ds_map_add(create_data_player,"class","");
                ds_map_add(create_data_player,"loc","");
                var create_inv = ds_map_create();
                ds_map_add(create_inv,"potion_hp",2);
                ds_map_add_map(create_data_player,"inv",create_inv);
              
                ds_map_add_map(create_data_all,acc,create_data_player);
              
                ds_map_secure_save(create_data_all,"playerdata.json");
                ds_map_destroy(create_data_player);
                ds_map_destroy(create_inv);
              
                scr_hyper("reg_back");
                scr_sysmsg("Account created succesfully!","",c_black);
            }
        }
        ds_map_destroy(create_data_all);
loading players data (at login):
Code:
var acc;
var pass;

with (obj_textbox) {
    if (self.name == "user_acc") {
        acc = self.txt;
    }
}
with (obj_textbox_crypt) {
    if (self.name == "user_pass") {
        pass = self.txt;
    }
}


var login_data_all = ds_map_secure_load("playerdata.json");
var login_data_player = ds_map_find_value(login_data_all,acc);

if (ds_map_exists(login_data_all,acc)) {
    if (pass == login_data_player[? "password"]) {
        global.playerdata = ds_map_create();
        ds_map_copy(global.playerdata,login_data_player);
        ds_map_destroy(login_data_all);
        ds_map_destroy(login_data_player);
      
        instance_create(0,0,obj_controller_postlogin);
        with (all) {
            if (self.group == "login") {
                instance_destroy();
            }
        }
    } else {
        scr_sysmsg("Password is incorrect.","login",c_black);
    }
} else {
    scr_sysmsg(string("Account ("+acc+") does not exist."),"login",c_black);

}
(the file is already created in a prior check, containing an empty ds_map)

then the player object is created, which runs the following code:
Code:
name = ds_map_find_value(global.playerdata,"name");
race = ds_map_find_value(global.playerdata,"race");
class = ds_map_find_value(global.playerdata,"class");
lvl = ds_map_find_value(global.playerdata,"lvl");
Exp = ds_map_find_value(global.playerdata,"Exp");
loc = ds_map_find_value(global.playerdata,"loc");

inv = global.playerdata[? "inv"];
(ds_exists(inv,ds_type_map) returns false, other data like race, class, lvl etc seems to load fine)

Saving playerdata:
Code:
var user = ds_map_find_value(global.playerdata,"user");

var data_all = ds_map_secure_load("playerdata.json");
var data_self = ds_map_find_value(data_all,user);

ds_map_replace_map(data_all,user,global.playerdata);

ds_map_secure_save(data_all,"playerdata.json");
ds_map_destroy(data_all);
scr_sysmsg("Saved succesfully","",c_black);
(error at ds_map_replace_map)

Does anyone know what im doing wrong? Also, a clean example of creating, saving and loading nested ds maps would be appreciated a lot!
 
Hi!

So far I haven't found anything suspicious in your code, so I'm contributing with the clean example you mentioned. You're already working with these functions so you probably know most of this, but here it goes anyway:

Code:
character = ds_map_create();
character[?"name"] = "Boaty McBoatface";
character[?"level"] = 8;

stats = ds_map_create();
stats[?"strength"] = 2;
stats[?"intelligence"] = 4;
stats[?"agility"] = 3;
stats[?"boat-factor"] = 32;

items = ds_list_create();
ds_list_add(items, "Holy Steerwheel", "Scared passenger", "Hostage Iceberg");

// IMPORTANT: Mark new entries as nested data
ds_map_add_map(character, "stats", stats);
ds_map_add_list(character, "items", items);

json_text = json_encode(character);

// Freeing is done recursively; no need to free stats and items individually
ds_map_destroy(character);

// Load again from string
character = json_decode(json_text);

if(!ds_map_exists(character, "stats"))
    show_error("Invalid JSON!", true);

if(!ds_map_exists(character, "items"))
    show_error("Invalid JSON!", true);

if(!ds_exists(character[?"stats"], ds_type_map))
    show_error("Invalid JSON!", true);

if(!ds_exists(character[?"items"], ds_type_list))
    show_error("Invalid JSON!", true);

// Running just fine
A wild guess, are you sure you're querying the right keys from the top-level map? Also, are the indices of the nested maps always the same? I suspect that ds_map_secure_save does not do recursive save. If that's the case, you can still just encrypt the string json_encode gives you.
 

icuurd12b42

TMC Founder
GMC Elder
tl;dr

you need to ds_map_add_map for adding maps and ds_map_add_list for adding list... for lists you need to ds_list_add/set then call ds_list_set_map ds_list_set_list to flag the element as a sub map or list.

the map/list parent will become the owner of the list so do not destroy the added list or map. it will be destroyed when you destroy the owner list/map... or the root list/map

I suggest you do the json_encode of the root map and save that and do a json_decode to get it back.

I also suggest you keep the structure throughout the game and initialise an empty structure using a simple json text if you you dont have a file loaded...

root = json_decode('
"Submap1": {
"Val1":1,
"Val2":1,
"Val1":1
},
"Submap2": {
"Val1":1,
"Val2":1,
"Val1":1
}
}');


PS +1 above post
 
W

Woochi

Guest
Hi!

So far I haven't found anything suspicious in your code, so I'm contributing with the clean example you mentioned. You're already working with these functions so you probably know most of this, but here it goes anyway:

Code:
character = ds_map_create();
character[?"name"] = "Boaty McBoatface";
character[?"level"] = 8;

stats = ds_map_create();
stats[?"strength"] = 2;
stats[?"intelligence"] = 4;
stats[?"agility"] = 3;
stats[?"boat-factor"] = 32;

items = ds_list_create();
ds_list_add(items, "Holy Steerwheel", "Scared passenger", "Hostage Iceberg");

// IMPORTANT: Mark new entries as nested data
ds_map_add_map(character, "stats", stats);
ds_map_add_list(character, "items", items);

json_text = json_encode(character);

// Freeing is done recursively; no need to free stats and items individually
ds_map_destroy(character);

// Load again from string
character = json_decode(json_text);

if(!ds_map_exists(character, "stats"))
    show_error("Invalid JSON!", true);

if(!ds_map_exists(character, "items"))
    show_error("Invalid JSON!", true);

if(!ds_exists(character[?"stats"], ds_type_map))
    show_error("Invalid JSON!", true);

if(!ds_exists(character[?"items"], ds_type_list))
    show_error("Invalid JSON!", true);

// Running just fine
A wild guess, are you sure you're querying the right keys from the top-level map? Also, are the indices of the nested maps always the same? I suspect that ds_map_secure_save does not do recursive save. If that's the case, you can still just encrypt the string json_encode gives you.
Hello elementbound, thanks a lot for the example. I was afraid my code was not optimally efficient, yours made things clearer. I have been searching for about half a day for answers/guides/ tutorials or anything at all about ds submaps and saving, but theres amazingly little to be found about it.

Yes, i am pretty sure i am using the right keys there, since other player data than inventory seems to load fine. What do you mean, indices the same? I do recall some error telling me about an index of the list (i dont get that part of the error, since it is a map and not a list).

As for the function, it does i think, because i can acces the previously saved submap containing the other playerdata.

Also, i couldnt find much about the use of json_enc/dec, i mean, after i encode the map into a string variable, whats the best way to save it (function/ file extension) ,in your opinion? I might give the enc/dec method a try instead anyway.

Sorry for the long posts
 
W

Woochi

Guest
tl;dr

you need to ds_map_add_map for adding maps and ds_map_add_list for adding list... for lists you need to ds_list_add/set then call ds_list_set_map ds_list_set_list to flag the element as a sub map or list.

the map/list parent will become the owner of the list so do not destroy the added list or map. it will be destroyed when you destroy the owner list/map... or the root list/map

I suggest you do the json_encode of the root map and save that and do a json_decode to get it back.

I also suggest you keep the structure throughout the game and initialise an empty structure using a simple json text if you you dont have a file loaded...

root = json_decode('
"Submap1": {
"Val1":1,
"Val2":1,
"Val1":1
},
"Submap2": {
"Val1":1,
"Val2":1,
"Val1":1
}
}');


PS +1 above post
Hey icuurd,
About the flagging of data types, i have not worked with list as root type, or list as sub map (yet), so that cannot be the problem.

I was indeed redundantly destroying submaps seperately :D i see that now, thanks for clearing that up.

Here too, about json enc/dec, with what function and as what file extension would you suggest me to save the string?

Also, why initialise the data within the json file? Since different starting-data will be added anyway, overwriting that. And with json text, you mean a .txt file extension or .json file extension?
 
I might have found the error, it's in your login code:
Code:
// ...

var login_data_all = ds_map_secure_load("playerdata.json");
var login_data_player = ds_map_find_value(login_data_all,acc);

if (ds_map_exists(login_data_all,acc)) {
    if (pass == login_data_player[? "password"]) {
        global.playerdata = ds_map_create();

        // Copy the map
        ds_map_copy(global.playerdata,login_data_player);

        // Destroy stuff loaded from JSON
        ds_map_destroy(login_data_all);
        ds_map_destroy(login_data_player);
   
        // ...
    } else {
        scr_sysmsg("Password is incorrect.","login",c_black);
    }
} else {
    scr_sysmsg(string("Account ("+acc+") does not exist."),"login",c_black);

}
ds_map_copy doesn't do a deep ( recursive ) copy. So, the nested maps in your global.playedata still refer to those contained by login_data_player, which is destroyed by freeing login_data_all, then by freeing login_data_player.

I tested it by adding this at the end of my previous example code:
Code:
character_copy = ds_map_create();
ds_map_copy(character_copy, character);

if(character[?"stats"] == character_copy[?"stats"])
    show_error("Ugh!", true);
About encryption: I opened a map I just saved with ds_secure_save. It looked like a base64 encoded string, and it indeed was. Decoding it gave me some gibberish and the contents of the map as JSON.

The easiest thing you can do to manually encode your stuff is to bitwise xor its contents with some arbitrary password. I'll post an example some time later.

If you want to go the extra mile, check out TEA. Pretty simple and easy to implement even in GMS. Just make sure you emulate the unsigned data type's overflows.

UPDATE: Actual XOR example.So, first we need a script to produce a xor'd buffer:
Code:
///buffer_xor(src_buffer, password)
var src_buffer, dst_buffer, password;
src_buffer = argument0;
dst_buffer = buffer_create(buffer_get_size(src_buffer), buffer_fixed, 1);
password = argument1;

buffer_seek(src_buffer, buffer_seek_start, 0);

for(var i = 0; i < buffer_get_size(src_buffer); i++) {
  var src_byte = buffer_read(src_buffer, buffer_u8);
  var pwd_byte = ord(string_char_at(password, (i mod string_length(password))+1));
  
  buffer_write(dst_buffer, buffer_u8, src_byte ^ pwd_byte);
}

buffer_seek(dst_buffer, buffer_seek_start, 0);
return dst_buffer;
The function takes a buffer with your data in it ( in this case, the json_encode'd ds_map string ), and a password. It just loops the password so it's the same length as the buffer. Then, each character is bitwise xor'd and put into a destination buffer. The script returns this buffer. This is most of what we need. XOR'ing is symmetric, so if you xor your encrypted data with the same password, you get your original back.

However, to make it a drop-in solution, here's two scripts to make it convenient:
Code:
///xor_encode(string, password)
var src_buffer = buffer_create(string_length(argument0)+1, buffer_fixed, 1);
buffer_write(src_buffer, buffer_string, argument0);

var xor_buffer = buffer_xor(src_buffer, argument1);
return buffer_base64_encode(xor_buffer, 0, buffer_get_size(xor_buffer));
This function takes a string, a password, and returns an encoded string.

Code:
///xor_decode(xor_string, password)
var crypt_buffer = buffer_base64_decode(argument0);
var decrypt_buffer = buffer_xor(crypt_buffer, argument1);

return buffer_read(decrypt_buffer, buffer_string);
And this one does the reverse.

Looking at the encrypted string, it's pretty obvious that it's base64 encoded. However, if you were to decode it, you'd still get seemingly random gibberish.

Then again, no encryption is totally safe. But giving it two layers of security ( base64 + xor ) will eliminate the majority of threats.

Also note that I did not bother freeing the random buffers I create :p
 
Last edited:
W

Woochi

Guest
I might have found the error, it's in your login code:
Code:
// ...

var login_data_all = ds_map_secure_load("playerdata.json");
var login_data_player = ds_map_find_value(login_data_all,acc);

if (ds_map_exists(login_data_all,acc)) {
    if (pass == login_data_player[? "password"]) {
        global.playerdata = ds_map_create();

        // Copy the map
        ds_map_copy(global.playerdata,login_data_player);

        // Destroy stuff loaded from JSON
        ds_map_destroy(login_data_all);
        ds_map_destroy(login_data_player);
  
        // ...
    } else {
        scr_sysmsg("Password is incorrect.","login",c_black);
    }
} else {
    scr_sysmsg(string("Account ("+acc+") does not exist."),"login",c_black);

}
ds_map_copy doesn't do a deep ( recursive ) copy. So, the nested maps in your global.playedata still refer to those contained by login_data_player, which is destroyed by freeing login_data_all, then by freeing login_data_player.

I tested it by adding this at the end of my previous example code:
Code:
character_copy = ds_map_create();
ds_map_copy(character_copy, character);

if(character[?"stats"] == character_copy[?"stats"])
    show_error("Ugh!", true);
About encryption: I opened a map I just saved with ds_secure_save. It looked like a base64 encoded string, and it indeed was. Decoding it gave me some gibberish and the contents of the map as JSON.

The easiest thing you can do to manually encode your stuff is to bitwise xor its contents with some arbitrary password. I'll post an example some time later.

If you want to go the extra mile, check out TEA. Pretty simple and easy to implement even in GMS. Just make sure you emulate the unsigned data type's overflows.

UPDATE: Actual XOR example.So, first we need a script to produce a xor'd buffer:
Code:
///buffer_xor(src_buffer, password)
var src_buffer, dst_buffer, password;
src_buffer = argument0;
dst_buffer = buffer_create(buffer_get_size(src_buffer), buffer_fixed, 1);
password = argument1;

buffer_seek(src_buffer, buffer_seek_start, 0);

for(var i = 0; i < buffer_get_size(src_buffer); i++) {
  var src_byte = buffer_read(src_buffer, buffer_u8);
  var pwd_byte = ord(string_char_at(password, (i mod string_length(password))+1));
 
  buffer_write(dst_buffer, buffer_u8, src_byte ^ pwd_byte);
}

buffer_seek(dst_buffer, buffer_seek_start, 0);
return dst_buffer;
The function takes a buffer with your data in it ( in this case, the json_encode'd ds_map string ), and a password. It just loops the password so it's the same length as the buffer. Then, each character is bitwise xor'd and put into a destination buffer. The script returns this buffer. This is most of what we need. XOR'ing is symmetric, so if you xor your encrypted data with the same password, you get your original back.

However, to make it a drop-in solution, here's two scripts to make it convenient:
Code:
///xor_encode(string, password)
var src_buffer = buffer_create(string_length(argument0)+1, buffer_fixed, 1);
buffer_write(src_buffer, buffer_string, argument0);

var xor_buffer = buffer_xor(src_buffer, argument1);
return buffer_base64_encode(xor_buffer, 0, buffer_get_size(xor_buffer));
This function takes a string, a password, and returns an encoded string.

Code:
///xor_decode(xor_string, password)
var crypt_buffer = buffer_base64_decode(argument0);
var decrypt_buffer = buffer_xor(crypt_buffer, argument1);

return buffer_read(decrypt_buffer, buffer_string);
And this one does the reverse.

Looking at the encrypted string, it's pretty obvious that it's base64 encoded. However, if you were to decode it, you'd still get seemingly random gibberish.

Then again, no encryption is totally safe. But giving it two layers of security ( base64 + xor ) will eliminate the majority of threats.

Also note that I did not bother freeing the random buffers I create :p
Wow... I dont think i would have ever found it myself, that the copy function does not "really" copy the submaps, since i thought problems with submaps were at a file(-save) level (only.json keeping submaps).

Its funny how the secure_save uses base_64 encryption, did not see that coming xD.

Also, interesting encryption suggestion. But then what keeps people from encrypting data in 100 layers of xor? Like, repeating xor_encrypt X number of times, and keeping track of the number of layers as some sort of key? Would that be possible?

Thank you for helping me out, this has really had me clueless, so in the end i made a forum account to try and ask help here.
 
I'm glad I could help! :)

The trick with xor'ing is that if you want to crack it, you don't know the password. I advise against encrypting multiple times, it is rarely worth the hassle. For example, if you crypt multiple times with xor, it only protects against people who can't crack xor. Because if somebody figures out a method to crack your encryption, they can just use the same algorithm again and again. If you're going for more protection, I suggest looking for more sophisticated encryption algorithms. You are never fully safe, it's just a tradeoff between speed and security ( and possibly hours to implement your algorithm of choice ).

Oh, also, since xor is symmetrical, for a multilayer approach you'd have to always switch passwords. Using the same passwords an even number of times gives your original data, using it an odd number of times gives you a singly-encrypted version.
 
W

Woochi

Guest
oh, yes of course xD thats right. didnt fully understand xor.

I will redo the code and give an update right after, though im very sure it will work after fixing the map copy thing.
 
W

Woochi

Guest
it WORKS LIKE A CHARM ;D :banana::banana::banana:

though still using the secure_save encryption only, but that wont be a big deal, since i saved your encryption method code.

modified part of creation:
Code:
var create_data_player = ds_map_create();
                var create_inv = ds_map_create();
                ds_map_add(create_data_player,"user",acc);
                ds_map_add(create_data_player,"name",name);
                ds_map_add(create_data_player,"password",pass);
                ds_map_add(create_data_player,"race",obj_controller_reg.pick);
                ds_map_add(create_data_player,"lvl",1);
                ds_map_add(create_data_player,"Exp",0);
                ds_map_add(create_data_player,"class","");
                ds_map_add(create_data_player,"loc","");
            
                ds_map_add(create_inv,"potion_hp",2);
                ds_map_add_map(create_data_player,"inv",create_inv);
            
                ds_map_add_map(create_data_all,acc,create_data_player);
            
                ds_map_secure_save(create_data_all,"playerdata.json");
modified loading:
Code:
var login_data_all = ds_map_secure_load("playerdata.json");
var login_data_player = ds_map_find_value(login_data_all,acc);
var login_data_inv = ds_map_find_value(login_data_player,"inv");

if (ds_map_exists(login_data_all,acc)) {
    if (pass == login_data_player[? "password"]) {
        global.playerdata = ds_map_create();
        ds_map_copy(global.playerdata,login_data_player);
        ds_map_delete(global.playerdata,"inv");
        var new_inv = ds_map_create();
        ds_map_copy(new_inv,login_data_inv);
        ds_map_add_map(global.playerdata,"inv",new_inv);
        ds_map_destroy(login_data_all);
and saving:
Code:
var user = ds_map_find_value(global.playerdata,"user");
var data_all = ds_map_secure_load("playerdata.json");

var save_player = ds_map_create();
var save_inv = ds_map_create();
ds_map_copy(save_player,global.playerdata);
ds_map_copy(save_inv,obj_player.inv);

ds_map_delete(save_player,"inv");
ds_map_add_map(save_player,"inv",save_inv);

ds_map_delete(data_all,user);
ds_map_add_map(data_all,user,save_player);

ds_map_secure_save(data_all,"playerdata.json");
ds_map_destroy(data_all);
scr_sysmsg("Saved succesfully","",c_black);
The trick is indeed to keep good track of pointers and the actual data (in memory), and know which functions are sub-map friendly and which are not.

to sum it up:
1. ds_map_copy does NOT copy sub-maps along,just as u said
2. ds_map_replace_map not only requires the value to add to be a map, but the existing KEY as well (see deletion of key, followed by same key but then as a map, in the code. This is how i will do it, since i know not better)
3. ds_map_add_map DOES add the sub-maps, therefore i used this function as much as possible in combination with ds_map_delete, instead of ds_map_replace which gives errors for list index of value

Why does game maker do this to us? :bash: it is not logical whatsoever

EDIT: the next step is to keep everything clean, as u did in your examples. im going to do that, and of course mark this as solved :bunny:
 
Last edited by a moderator:

icuurd12b42

TMC Founder
GMC Elder
Hey icuurd,
About the flagging of data types, i have not worked with list as root type, or list as sub map (yet), so that cannot be the problem.

I was indeed redundantly destroying submaps seperately :D i see that now, thanks for clearing that up.

Here too, about json enc/dec, with what function and as what file extension would you suggest me to save the string?

Also, why initialise the data within the json file? Since different starting-data will be added anyway, overwriting that. And with json text, you mean a .txt file extension or .json file extension?
What I meant with "setting up a startup structure" is, that way, you have the initial structures your game requires with far less code or convolution
say for example you game has a player and the player has an inventory

rootmap=json_decode('
"Player": {
"x":0,
"y":0,
"Inventory":{
}
}
');
global.playermap = ds_map_find_value(rootmap,"Player")
global.inventory = ds_map_find_value(global.playermap,"Inventory")

as opposed to
rootmap = ds_map_create()
playemap = ds_map_create()
inventorymap = ds_map_create()
ds_map_add_map(playermap,"Inventory",inventorymap);
ds_map_add_map(rootmap,playermap);

You can see I also placed x,y in the player structure. forwardly declaring the content of the player tells me at a glance what sort of data is stored in the json for the player. to sort of self documents what is available at a glance...

Here's a tip. Create a function that iterates through the structure for you to find content safely
like
GetValue(map,keys,default)
value = GetValue(root,"Player.Inventory",-1)
or
GetValue(map,key1,key2,....,default)
value = GetValue(root,"Player","Inventory",-1)

that way you can simplify your code a LOT and work with one single huge structure for your game

as for saving, it's a string so. a ZILLION OPTIONS are available to you. but I believe the right way has been mentioned

write the string to a buffer
xor the buffer
bas64 the buffer and save the result.

the base 64 result being single line string... you can save that in an ini... or to a file of it's own...

if you write to a file of it's own, base64 encoded or not AGAIN use a buffer. make a buffer, save the string in it, do a buffer save to file
to read, do a buffer load from file, read the string from the buffer...

reading a simple plain as day json file is simple
buffer load from file. this read the entire text file in one shot
read string from buffer, reads the entire string since the buffer load was done on a simple text file
json decode the string, converts the json text to a structured map
delete the buffer.
 
Last edited:
W

Woochi

Guest
I see, i get it now. Im gonna give that a try.

Also, a function to get data from a file is a great idea :D thanks for elaborating.
 

icuurd12b42

TMC Founder
GMC Elder
I see, i get it now. Im gonna give that a try.

Also, a function to get data from a file is a great idea :D thanks for elaborating.
Yes, the buffer system basically replaces binary and text file io as well as enable quite a few feats in other departments
 
Top