GameMaker Text files & safe reading/writing

Neptune

Member
Consider that code in a loop, reading in a bunch of data -- And then a memory allocation error happens... Is there a way to catch something like that, and retry the read process?
If there IS a memory allocation error in the first place, is that always my code's fault, or is it just something that can happen?

GML:
var file = file_text_open_read(file_name);  
var load_data = file_text_read_string(file);
file_text_close(file);
An allocation error may look like this: Memory allocation failed: Attempting to allocate 17922559 bytes (~17 mb)

And one more question, is there a function to obtain a file's size, before reading it in?
I see the file_bin_X functions, but I don't think that will work for my text files.


Any information appreciated :)
 
Last edited:

chamaeleon

Member
And one more question, is there a function to obtain a file's size, before reading it in?
I see the file_bin_X functions, but I don't think that will work for my text files.
Just open the file as a binary and call file_bin_size() to get the size in bytes, then close the binary file if you do not need/want to actually use any other binary file functions. You can then proceed with whatever you wish to do depending on the size. There's no difference between a binary file and a text file, it's just a set of bytes. What matters is what functions you call to make the program interpret the bytes in a desired way (raw bytes vs strings per line, etc.)
 

Neptune

Member
Do you think it's a reasonable thing to do @chamaeleon -- To obtain the size, and section out a short wait time based on its size, between loading the files?
 

chamaeleon

Member
Do you think it's a reasonable thing to do @chamaeleon -- To obtain the size, and section out a short wait time based on its size, between loading the files?
Wait time shouldn't be required, either it will work or it won't. I think it would be useful to show a little more code that shows what you're actually doing with the file content, what the looping structure looks like, etc. The one read call in your first post won't read in very much (unless you have a multi-megabyte file without newlines). On the other hand, perhaps it is not this particular code that is responsible for your problem, but is only the trigger. But without more code, I don't think it's possibly to say either way.
 
Last edited:

Neptune

Member
Ok this code loads about 60 files varying from 2kb to 30mb each.
Each file contains a string formed by ds_grid_write (or ds_list_write etc), and I just added the "bin" shenanigans.
GML:
for(var i = 0; i < GAME_ROOM.enum_count; i++)
{
    var directory = working_directory + "/save"+string(global.current_file_number)+"/";
    var directory_rooms = directory +"maps/";
    var file_name = directory_rooms + string(i) +".data";
            
    #region ASSESS SIZE
        var bin_file = file_bin_open(file_name,0);
            var file_size = file_bin_size(bin_file);
        file_bin_close(bin_file);
                                                    
        global.read_cushion = 5 + (file_size / 1000000)*2;                                        
    #endregion
                    
    var file = file_text_open_read(file_name);
        var load_data = file_text_read_string(file);
    file_text_close(file);

    global.DS_rooms[# 0,i] = load_data;
}
I was just going to scrap the for loop, and use an instance variable that increments based on "global.read_cushion"... But sounds like that may be futile.
I should also say, this allocation issue is intermittent, dare I say infrequent... And seems to be on lower-end PCs.
 
Last edited:

chamaeleon

Member
Ok this code loads about 60 files varying from 2kb to 30mb each.
Each file contains a string formed by ds_grid_write (or ds_list_write etc), and I just added the "bin" shenanigans.
GML:
for(var i = 0; i < GAME_ROOM.enum_count; i++)
{
    var directory = working_directory + "/save"+string(global.current_file_number)+"/";
    var directory_rooms = directory +"maps/";
    var file_name = directory_rooms + string(i) +".data";
          
    #region ASSESS SIZE
        var bin_file = file_bin_open(file_name,0);
            var file_size = file_bin_size(bin_file);
        file_bin_close(bin_file);
                                                  
        global.read_cushion = 5 + (file_size / 1000000)*2;                                      
    #endregion
                  
    var file = file_text_open_read(file_name);
        var load_data = file_text_read_string(file);
    file_text_close(file);

    global.DS_rooms[# 0,i] = load_data;
}
I was just going to scrap the for loop, and use an instance variable that increments based on "global.read_cushion"... But sounds like that may be futile.
I should also say, this allocation issue is intermittent, dare I say infrequent... And seems to be on lower-end PCs.
Ignore this .... "So, as written in your code snippet, you're only interested in the first line in each file_name file, as that is what file_text_read_string() returns?"

Clearly I should have paid more attention to the ds_grid_write()/ds_list_write() part.
 

chamaeleon

Member
Just what is the sum total of the serialized data structure files if you check it using Windows Explorer on Windows, Finder on Mac, etc. (asking because you're storing the serialized strings in the grid, and the serialized form of data structures are quite a bit larger than the data they represent)?
 

chamaeleon

Member
Silly question, but these files are not generated by a possibly older runtime or anything, are they? One thing that worries me (perhaps needlessly) is having to rely on the format not changing without having any means to verify whether it has or not, or if any update to GMS might suddenly break backwards compatibility.
 

chamaeleon

Member
Files are current. But I'm not really sure about the potential formatting issues.
I find it very unlikely it would be the issue, but figured it was worth asking. At this point, if it was my code, I'd simply take those files, copy them to some other location, create a sandbox project (if you don't have one already), disable the sandbox and read the files from the directory where you copied them to, to ensure you can load them all if you don't do anything else.
 

Neptune

Member
Bah, yes I want to be able to do that. Problem is, I can't get the issue to happen as is... Only some people with seemingly low end desktops/laptops.
Anyways, thanks for looking into it :)
 

chamaeleon

Member
Bah, yes I want to be able to do that. Problem is, I can't get the issue to happen as is... Only some people with seemingly low end desktops/laptops.
Anyways, thanks for looking into it :)
Since you're storing the string in that ds_grid, once you use the corresponding data structure read function for a string, do you clear out that grid as you recreate each data structure by setting the grid cell to an empty string or something like that? If the files add up to ~100MB or so, that's 100MB of RAM just sitting there without any other purpose once you have the data structures. Or are the strings important to keep around for regeneration as needed?
 

Neptune

Member
Basically as you enter each room, it pulls the corresponding string from that global.DS_room grid - Reads it into a grid for that room; Changes are made; Player leaves, and it is written back to where it came from, grabs the next.
So the global.DS_room storing written strings is always around, and then a smaller grid that is being used in the current with accessible data.
 
Last edited:

chamaeleon

Member
Basically as you enter each room, it pulls the corresponding string from that global.DS_room grid - Reads it into a grid for that room; Changes are made; Player leaves, and it is written back to where it came from, grabs the next.
So the global.DS_room storing written strings is always around, and then a smaller grid that is being used in the current with accessible data.
Have you thought about skipping the idea of using the grid holding the strings altogether and just use the filesystem for the same purpose? I can't imagine it takes that long to read and write this information, at least not in a very noticeable manner if it happens in conjunction with room changes.
 

Neptune

Member
Thats an interesting idea, hmm... I could I think, but with my current experience with the file issues, the last thing I want are more read/writes to text files o_O
I'm thinking this is what RAM is for too, 200 mb or so doesn't seem too bad! 🙂
 
Top