Fastest way to load a JSON/text file with tens of thousands of lines?

[Startoz]

Hi,
I want to load a large JSON file (more than 40k lines when formatted) and decode it.
The system that I'm using at the moment works perfectly fine if all the data is contained in one line (loading the text into a string takes less than 1 second).
However, if the JSON is formatted, it takes more than 25 seconds to load all the data into the string, since the script has to iterate over 40k+ lines.

This is the code that I'm using:

/// @function decodeJsonFile(_fileName)
/// @param _fileName
var _file = argument0;
if (file_exists(_file)) {
    var _json = "";
    var _filePointer = file_text_open_read(_file);
   
    while (!file_text_eof(_filePointer)) {
        _json += file_text_read_string(_filePointer);
        file_text_readln(_filePointer);
    }
    file_text_close(_filePointer);
   
    var _decodedJson = json_decode(_json);
    return _decodedJson;
} else {
    return -1;
}

Is there a more efficient way of loading a big text/json file into a string?

Thanks!

 

[TailBit]

Open it, and read X lines every step instead of all at once?

So the game can continue at least, maybe block of some features that the data is required for?

 

[NightFrost]

Is there a need to have the data as external json file after the game has been compiled? If not, you could process it for the final build. Write some code that reads the json and turns it into GML declarations, copypaste the result into your project and comment out json handling.

 

[Startoz]

Thanks for the replies!

I've managed to find a much faster way, using buffers:

/// @function decodeJsonFile(_fileName)
/// @param _fileName
var _file = argument0;
if (file_exists(_file)) {
    var _textBuffer        = buffer_load(_file);
    buffer_seek(_textBuffer, buffer_seek_start, 0);
    var _json            = buffer_read(_textBuffer, buffer_text);
    buffer_delete(_textBuffer);
    var _decodedJson    = json_decode(_json);
    return _decodedJson;
} else {
    return -1;
}