JSON Toolkit: Companion scripts for json_encode() and json_decode()

FrostyCat · Feb 22, 2019

JSON Toolkit v1.1.0
Companion scripts for json_encode() and json_decode()

Overview
JSON Toolkit is a set of companion scripts facilitating the use of GML's built-in json_encode() and json_decode() functions. It contains utilities for visually building JSON structure, accessing and manipulating deeply nested values, iterating through a JSON structure, and saving/loading JSON data in files. With JSON Toolkit, many common JSON operations that would otherwise take several lines, intermediate values and repetitions in conventional GML can be shortened into concise, easy-to-read one-liners.

Downloads
YoYo Marketplace: Link
GMS 2.x: Link | Repository
GMS 1.4: Link | Repository

Examples

Creating Nested JSON Structures

Conventional GML:

Code:

global.stats = ds_map_create();
ds_map_add_map(global.stats, "Alice", ds_map_create());
ds_map_add(global.stats[? "Alice"], "HP", 5);
ds_map_add(global.stats[? "Alice"], "ATK", 5);
ds_map_add(global.stats[? "Alice"], "DEF", 4);
ds_map_add_map(global.stats, "Bob", ds_map_create());
ds_map_add(global.stats[? "Bob"], "HP", 7);
ds_map_add(global.stats[? "Bob"], "ATK", 6);
ds_map_add(global.stats[? "Bob"], "DEF", 2);

With JSON Toolkit:

Code:

global.stats = JsonStruct(JsonMap(
   "Alice", JsonMap(
       "HP", 5,
       "ATK", 5,
       "DEF", 4
   ),
   "Bob", JsonMap(
       "HP", 7,
       "ATK", 6,
       "DEF", 2
   )
));

Accessing Nested JSON Structures

JSON String to Decode:

Code:

[
   {
       "name": "Alice",
       "HP": 5,
       "ATK": 5,
       "DEF": 4
   },
   {
       "name": "Bob",
       "HP": 7,
       "ATK": 6,
       "DEF", 2
   }
]

Conventional GML:

Code:

// Access Bob's HP
var json_data = json_decode(json_str);
var bob_hp = json_data[? "default"];
bob_hp = bob_hp[| 1];
bob_hp = bob_hp[? "HP"];

With JSON Toolkit:

Code:

// Access Bob's HP
var json_data = json_decode(json_str);
var bob_hp = json_get(json_data, 1, "HP");

Loading from a JSON File

Conventional GML:

Code:

// Load from save.json
var f = file_text_open_read(working_directory + "save.json"),
   json_str = "";
while (!file_text_eof(f)) {
   json_str += file_text_read_string(f);
   file_text_readln(f);
}
file_text_close();
var json_data = json_decode(json_str);

With JSON Toolkit:

Code:

// Load from save.json
var json_data = json_load(working_directory + "save.json");

Feedback Welcome!
If you have any suggestions for new constructors/use cases or bug reports, please open an issue or contribute on GitHub.

Dmi7ry · Feb 22, 2019

Marketplace?

chamaeleon · Feb 22, 2019

That looks incredibly useful. I need to remember this when I get to some serious serialization coding.

FrostyCat · Mar 20, 2019

Update: JSON Toolkit is now available on the YoYo Marketplace!

Pfap · Mar 20, 2019

Hello, I was going over the documentation you link to from the marketplace and I have a question about this portion.

JsonData(toplevel_or_fname): Denotes a JSON data structure, headed by the given top structure (should be JsonMap(...) or JsonList(...). This represents an actual data structure ID and should be the final value stored in variables. If the top-level structure is JsonMap(...), a map will be returned. If the top-level structure is JsonList(...), a map with a "default" key containing the list will be returned, as per json_decode() convention.

The function JsonData() does not seem to be in the extension I downloaded from the marketplace.

And for some reason bulgogi sounds pretty good right now...

Edit:
Is there a way to add a new key and map to an existing structure?

Code:

global.stats = JsonStruct(JsonMap(
   "Alice", JsonMap(
       "HP", 5,
       "ATK", 5,
       "DEF", 4
   ),
   "Bob", JsonMap(
       "HP", 7,
       "ATK", 6,
       "DEF", 2
   )
));

json_insert(global.stats,"Pfap",
 JsonMap("HP",25)
);
show_debug_message(json_get(global.stats,"Pfap","HP"));

FrostyCat · Mar 20, 2019

Pfap said:
The function JsonData() does not seem to be in the extension I downloaded from the marketplace.

My apologies, it should have been JsonStruct(). That name came from an earlier prototype and isn't there anymore.

Pfap said:
Is there a way to add a new key and map to an existing structure?

Yes, you can use json_set_nested() or json_insert_nested().

Pfap · Mar 20, 2019

FrostyCat said:
My apologies, it should have been JsonStruct(). That name came from an earlier prototype and isn't there anymore.

Yes, you can use json_set_nested() or json_insert_nested().

Thanks, I'm kind of just working through the examples you provide on the GitHub page and I ran into an issue with the json iterator.

Here is the error message:

Code:

ERROR!!! :: ############################################################################################
FATAL ERROR in
action number 1
of Create Event
for object json_tests:

Expected at least 2 arguments, got 1.
 at gml_Script_json_iterate (line 6) -     show_error("Expected at least 2 arguments, got " + string(argument_count) + ".", true);
############################################################################################
--------------------------------------------------------------------------------------------
stack frame is
gml_Script_json_iterate (line 6)
called from - gml_Object_json_tests_Create_0 (line 35) - for (var i = json_iterate(json_data); json_has_next(json_data); json_next(json_data)) {

The json_iterate() function has the wrong number of arguments and I'm guessing I need to supply a path, but how can I iterate through the top level structure?

Here is all the code I'm working with.

Code:

global.stats = JsonStruct(JsonMap(
   "Alice", JsonMap(
       "HP", 5,
       "ATK", 5,
       "DEF", 4
   ),
   "Bob", JsonMap(
       "HP", 7,
       "ATK", 6,
       "DEF", 2
   )
));

//var new_map = JsonStruct(JsonMap("HP",25));
json_insert_nested(global.stats,"Pfap",
 JsonStruct(JsonMap("HP",25))
);
show_debug_message(json_get(global.stats,"Pfap","HP"));


// Show a message for each name
var json_data = json_decode(global.stats);
for (var i = json_iterate(json_data); json_has_next(json_data); json_next(json_data)) {
 show_message("Hello from " + json_get(i[JSONITER.VALUE]) + "!");
}

Also, for a structure like global.stats where the top level structure is a map do the same rules apply as with gamemaker maps?
For instance, the manual cautions that the functions ds_map_find_first() and ds_map_find_next() may be slow and that the keys are not stored linearly.
Which wouldn't be an issue as I could just use the JsonList() function, I'm just curious about how iterating over things works.

FrostyCat · Mar 21, 2019

Pfap said:
The json_iterate() function has the wrong number of arguments and I'm guessing I need to supply a path, but how can I iterate through the top level structure?

My apologies again, there were a series of typos both the examples and the Wiki documentation for the iterators, which have just been edited and should now match the extension's actual implementation. Please try the new examples. I have run the edited examples on a blank project to confirm that they work properly.

Here is the loop that should work with your example:

Code:

global.stats = JsonStruct(JsonMap(
  "Alice", JsonMap(
    "HP", 5,
    "ATK", 5,
    "DEF", 4
  ),
  "Bob", JsonMap(
    "HP", 7,
    "ATK", 6,
    "DEF", 2
  )
));

json_insert_nested(global.stats,"Pfap",
  JsonStruct(JsonMap("HP",25))
);
show_debug_message(json_get(global.stats,"Pfap","HP"));

for (var i = json_iterate(global.stats, ds_type_map); json_has_next(i); json_next(i)) {
  show_message("Hello from " + i[JSONITER.KEY] + "!");
}

Notice that json_has_next() and json_next() both take the iterator i as their argument. Also, because the names are in keys, you would be looking for i[JSONITER.KEY] inside the loop.

Pfap said:
Also, for a structure like global.stats where the top level structure is a map do the same rules apply as with gamemaker maps?
For instance, the manual cautions that the functions ds_map_find_first() and ds_map_find_next() may be slow and that the keys are not stored linearly.
Which wouldn't be an issue as I could just use the JsonList() function, I'm just curious about how iterating over things works.

Yes, if the subject of iteration is a map, then it will use ds_map_find_first() with ds_map_find_next(). As you said, if the subject of iteration is a list, then ds_map_find_first() with ds_map_find_next() wouldn't have a hand in this.

Thank you so much for your feedback! I'm sorry that the library still had some residual prototype content in its documentation. Please let me know if you see anything else odd in your exploration.

Pfap · Mar 21, 2019

Thanks, this is really nice and clean. Just did some extensive testing as far as my use cases go; with lists and maps being sent to my server and objects and arrays being sent from my server and I'm feeling good enough to start using this in my main project and definitely see it saving me time.

Just an idea and one I have not thought of how to implement, but it would be nice if the extension could automatically strip the { "default": } when a list is supplied to json_encode().

FrostyCat · Mar 21, 2019

Pfap said:
Just an idea and one I have not thought of how to implement, but it would be nice if the extension could automatically strip the { "default": } when a list is supplied to json_encode().

The extension actually comes with a function called json_encode_as_list() that does what you described. In a nutshell, it simply looks for the first [ and the last ], and returns those plus everything in between.

Pfap · Mar 21, 2019

FrostyCat said:
The extension actually comes with a function called json_encode_as_list() that does what you described. In a nutshell, it simply looks for the first [ and the last ], and returns those plus everything in between.

Yea I wasn't very clear, but I meant that it would be nice if the function json_encode_as_list() could be excluded and when a list is passed to json_encode() it behaves the same as json_encode_list(). It might be a non issue, I'm trying to think of how dynamic I would use the extension that it would not be clear what I am encoding... in any case, as soon as it caused an issue it would be easy to just add _as_list to a json_encode call on a list.

Edit:
Just wrote this script.

Code:

/// @function json_tool_encode(data_structure)
/// @description Encode based off of type.
/// @param ds_index
var structure = argument0;
var encoded = noone;
if ds_map_find_first(structure) == "default"{
 show_debug_message("is a list");
 encoded = json_encode_as_list(structure);
}
else{
 show_debug_message("is not a list");
 encoded = json_encode(structure);
}
return encoded;

Pfap · Mar 23, 2019

Hey @FrostyCat, is this allowed behavior or am I not thinking correctly?

I want to be able to "pull" the internal lists out of the top level map and into the global.new_items variable or rather pull the index of that list into the global.new_items variable.

Code:

items_add = JsonStruct(JsonMap(
 "new_list", JsonList(
 "this",
 "is",
 "working!"
 ),
 
 "unlocked_list", JsonList(
 "this",
 "is",
 "also",
 "working!"
 )
));


show_debug_message(json_tool_encode(items_add));
//get the new items
global.new_items = json_get(items_add,"new_list");

show_debug_message(global.new_items);
show_debug_message(json_encode(global.new_items));
if json_exists(global.new_items){
 for (var i = json_iterate(global.new_items, ds_type_list); json_has_next(i); json_next(i)) {
         show_debug_message(i[JSONITER.VALUE]);
 }
}
else{
 show_debug_message("this is not working");
}

The output is quite unexpected, so I'm thinking this is not allowed and the 3 that the show_debug_message(global.new_items) is returning is pointing to a different index.
Here is the relevant console output:

Code:

{ "new_list": [ "this", "is", "working!" ], "unlocked_list": [ "this", "is", "also", "working!" ] }
3
{ "connectFailed": [ 12.000000 ], "cleared": [ 10.000000 ], "incomingFriendReq": [ 21.000000 ], "declined": [ 20.000000 ], "disconnected": [ 13.000000 ], "connected": [ 15.000000 ], "setUp": [ 11.000000 ], "requesterCancel": [ 19.000000 ], "clearedF": [ 18.000000 ], "theyAccepted": [ 22.000000 ], "fromList": [ 14.000000 ], "incomingResult": [ 17.000000 ], "matchResult": [ 16.000000 ] }

Also, when I switch the json_iterate()'s 2nd argument to ds_type_map it returns the values from the console output.

Edit:
Is the json_insert() function used to add values to a list?

This does not seem to be working:

Code:

items_add = JsonStruct(JsonMap(
 "new_list", JsonList(
 "this",
 "is",
 "working!"
 ),

 "unlocked_list", JsonList(
1,
2,
3,
4,
5
 )
));

json_insert(items_add,"unlocked_list",0);

The above code sets the list located at "unlocked_list" to only hold the value of 0.

FrostyCat · Mar 23, 2019

For the first question, json_exists() is falsely reporting values because maps and lists assign IDs independently (a historical problem with GM). json_iterate() has to always start on a map, so giving it a list ID results in undefined behaviour. Here's what would have worked:

Code:

for (var i = json_iterate(global.new_items, "new_list", ds_type_list); json_has_next(i); json_next(i)) {
  show_debug_message(i[JSONITER.VALUE]);
}

For the second question, json_insert() can insert values into a list if that's the final target, but if the final target is a map then it works the same way as json_set(). So your code would set the key "unlocked_list" to hold a scalar 0. If you want to insert into position 0 of the list, then you have to do one further:

Code:

json_insert(items_add, "unlocked_list", 0, "new first value for unlocked list");

Pfap · Mar 24, 2019

FrostyCat said:
For the first question, json_exists() is falsely reporting values because maps and lists assign IDs independently (a historical problem with GM). json_iterate() has to always start on a map, so giving it a list ID results in undefined behaviour. Here's what would have worked:

Code:

for (var i = json_iterate(global.new_items, "new_list", ds_type_list); json_has_next(i); json_next(i)) { show_debug_message(i[JSONITER.VALUE]); }

For the second question, json_insert() can insert values into a list if that's the final target, but if the final target is a map then it works the same way as json_set(). So your code would set the key "unlocked_list" to hold a scalar 0. If you want to insert into position 0 of the list, then you have to do one further:

Code:

json_insert(items_add, "unlocked_list", 0, "new first value for unlocked list");

Thanks! That works, except then the behavior of ds_list_add() is different.

This sets:

Code:

json_insert(global.items_json,"unlocked_list","This will now be the only value in the list.");

While this will insert into the list at position 4:

Code:

json_insert(global.items_json,"unlocked_list",4,"Adding at pos 4.");

I want to be able to add iteratively:

Code:

repeat(20){
 json_insert(global.items_json,"unlocked_list",0);
}

But the above just sets the list to only have the value of 0, 20 times.

So, I got it to work by doing this:

Code:

repeat(20){
 json_insert(global.items_json,"unlocked_list",0,0);
}

Now the 20 extra zeroes are inserted at the head of the list and not the tail. Which makes sense, and might actually work good for what I'm doing, but then I tried using -1 to indicate the last element of the list and was thinking it would be nice if you could use negative based indexes; -2 for penultimate etc...

So, I looked at your code on GitHub and was wondering if something like this would work?

Code:

// Set the last layer and go
if (is_string(k)) {
    if (_json_not_ds(current, ds_type_map)) return -argument_count+2;
    current[? k] = argument[argument_count-1];
} else {
    if (_json_not_ds(current, ds_type_list)) return -argument_count+2;
    if (is_real(k)) {
 




//this is my thought I did not test it
  if (k > 0){
        ds_list_insert(current, k, argument[argument_count-1]);
  }
  else{
   var length = ds_list_size(current);
   var offset = length-abs(k);
   ds_list_insert(current, offset, argument[argument_count-1]);
  }




    } else if (is_undefined(k)) {
 
        ds_list_add(current, argument[argument_count-1]);

    } else {
        return -argument_count+2;
    }
}

Then you could do stuff like this:

Code:

repeat(20){
 json_insert(global.items_json,"unlocked_list",-1,"This will now be repeated 20 times from the end of the original list");
}

I hope I'm not being a pain and thanks for sharing.

FrostyCat · Mar 24, 2019

@Pfap:

You can actually insert at the end of a list by using undefined as the final index (source):

Code:

json_insert(global.items_json, "unlocked_list", undefined, 0);

Currently JSON Toolkit doesn't support negative indexing yet (planned for future expansion), but even that wouldn't have helped as "inserting at -1" would have meant just before the last entry (i.e. inserted entry becomes second last), instead of strictly dead last.

Thank you very much for your extensive feedback, it has given me lots of insight into how end users would use the library and what difficulties they may encounter while learning it.

❤️×1 · Mar 26, 2019

Hello o/

I've been playing with JSON Toolkit, but I'm kinda lost :

Code:

    ItemList = JsonStruct(JsonList(
        JsonMap(
            "TEXT", "Something!",
            "SCRIPT", "Lorem ipsum dolor sit amet"
        ),JsonMap(
            "TEXT", "Another thing!",
            "SCRIPT", "consectetur adipiscing elit"
        )
    ));
    show_debug_message(json_get(ItemList,0,"TEXT"))
    show_debug_message(json_get(ItemList,1,"TEXT"))
    show_debug_message(json_encode(ItemList))

Give me :

Something!
Another thing!
{ "default": [ { "SCRIPT": "Lorem ipsum dolor sit amet", "TEXT": "Something!" }, { "SCRIPT": "consectetur adipiscing elit", "TEXT": "Another thing!" } ] }

Perfect.

Now,

Code:

    ItemList = JsonStruct(JsonList(
        JsonMap(
            "TEXT", "Something!",
            "SCRIPT", "Lorem ipsum dolor sit amet"
        )
    ));
 
    json_insert(ItemList,1,JsonStruct(
        JsonMap(
            "TEXT", "Another thing!",
            "SCRIPT", "consectetur adipiscing elit"
        )
    ));
 
    show_debug_message(json_get(ItemList,0,"TEXT"))
    show_debug_message(json_get(ItemList,1,"TEXT"))
    show_debug_message(json_encode(ItemList))

Is giving me

Something!
Another thing!
{ "default": [ { "SCRIPT": "Lorem ipsum dolor sit amet", "TEXT": "Something!" }, 38.000000 ] }

What do I do wrong?
I'm guessing I shouldn't need to use JsonStruct() in my json_insert() but, if I don't, I get undefined when accessing the new item in the list.

/!\ Edit /!\ I know what I'm doing wrong : I can't read... Should have used json_insert_nested().
Sorry for this unnecessary message T_T

FrostyCat · May 12, 2019

Update: v1.1.0 released! This update adds support for paths in array form (entirely or partially) and negative indices for counting from the back of a list.

Code:

json_set(json_data, "data", [3, "name"], "David");

Code:

json_set(json_data, "data", -1, "name", "Zachary");

You can download the updated version from the YoYo Marketplace.

JSON Toolkit: Companion scripts for json_encode() and json_decode()

FrostyCat

Redemption Seeker

Dmi7ry

Member

chamaeleon

Member

FrostyCat

Redemption Seeker

Pfap

Member

FrostyCat

Redemption Seeker

Pfap

Member

FrostyCat

Redemption Seeker

Pfap

Member

FrostyCat

Redemption Seeker

Pfap

Member

Pfap

Member

FrostyCat

Redemption Seeker

Pfap

Member

FrostyCat

Redemption Seeker

❤️×1

Member

FrostyCat

Redemption Seeker