• Hey! Guest! The 40th (!!!) GMC Jam will take place between February 25th, 12:00 UTC to March 1st 12:00 UTC. Why not join in this very special anniversary jam! Click here to find out more!

Fastest way to load a JSON/text file with tens of thousands of lines?

Startoz

Member
Hi,
I want to load a large JSON file (more than 40k lines when formatted) and decode it.
The system that I'm using at the moment works perfectly fine if all the data is contained in one line (loading the text into a string takes less than 1 second).
However, if the JSON is formatted, it takes more than 25 seconds to load all the data into the string, since the script has to iterate over 40k+ lines.

This is the code that I'm using:
Code:
/// @function decodeJsonFile(_fileName)
/// @param _fileName
var _file = argument0;
if (file_exists(_file)) {
    var _json = "";
    var _filePointer = file_text_open_read(_file);
   
    while (!file_text_eof(_filePointer)) {
        _json += file_text_read_string(_filePointer);
        file_text_readln(_filePointer);
    }
    file_text_close(_filePointer);
   
    var _decodedJson = json_decode(_json);
    return _decodedJson;
} else {
    return -1;
}
Is there a more efficient way of loading a big text/json file into a string?

Thanks!
 

TailBit

Member
Open it, and read X lines every step instead of all at once?

So the game can continue at least, maybe block of some features that the data is required for?
 

NightFrost

Member
Is there a need to have the data as external json file after the game has been compiled? If not, you could process it for the final build. Write some code that reads the json and turns it into GML declarations, copypaste the result into your project and comment out json handling.
 

Startoz

Member
Thanks for the replies!

I've managed to find a much faster way, using buffers:

Code:
/// @function decodeJsonFile(_fileName)
/// @param _fileName
var _file = argument0;
if (file_exists(_file)) {
    var _textBuffer        = buffer_load(_file);
    buffer_seek(_textBuffer, buffer_seek_start, 0);
    var _json            = buffer_read(_textBuffer, buffer_text);
    buffer_delete(_textBuffer);
    var _decodedJson    = json_decode(_json);
    return _decodedJson;
} else {
    return -1;
}
 
Top