Windows Big data store

Hi, i use a ds_map to save players with their cups number, and i created a ds_list to store their ranking position, the problem is when i try to store more than 400 000 names
in that ds_map the program doesnt execute:
GML:
var i = 0;
repeat(500000){
    var name = string(i);
    ds_players[? name] = 3000;
    i++;
}
What would be the best way to save 10 million players names with their cups? and what would be the best way to sort their cups in a ranking list?
 
What are these names exactly? Is it an online high score list? Or something built into the game itself? I haven't stored this many names before, but I would start by looking if I can use a database to store it. SQL / SQL Lite perhaps.

You said you are using a ds_list, but your code above shows you are using a ds_map.

I tested adding 900,000 names to a ds_map just now. I had to wait approx. 2 minutes for the game to start, but it worked fine other than the short wait to add all the names at the start of the game.
 
EDIT: This might be a load of old guff given that IndianaBones had 900 000 entries (though that's still a way off from 10 million) but I'll post it anyway :) I would suggest using a ds priority as your data structure though, as it can hold names and a value (number of cups), and be ordered for later purposes.

I know that there are limits to the size of data structures, so if 400 000 entries is the limit you'd need more than 20 to store 10 million names. Which would require a fair bit of effort to sort through and order, as you'd have to order them all and then compare the results before ordering those results.

Using ds priorities:

1)Check that the priority exists
2) Check that the priority has any content
3)Check it against the others values (and that they exist, and have content etc)
4) place the lowest, find the new lowest value for that structure, and then repeat the comparisons


Or using strings:

To save on space / effort (?) you could consider putting multiple names and cups into one string, and have them divided by certain characters denoting the end of the name / end of the cup. If 400 000 entries is the limit for a ds map then that's 20 or so pieces of data per key, with the naming of the key not being relevant anymore. It could just be a number.

That's maybe better than 20 ds maps / priorities with a singular piece of data for each key.....??

My guess at doing this would be:

You'd have to parse the string in each key to get the details, place those details into a ds_priority (priority is the number, value is the name) and order it, and then rewrite the string so it's ordered. This would give you the lowest and highest values.

Store that as two variables for each key and have a third variable for what key it is.

That's one loop through the keys. The string for each key is now ordered, and you know the range of values for each key.

Then you'd have to rearrange those strings so that the one with the lowest starting value is first.

Do a second loop through the keys, and as you're parsing through the string you need to be checking the value against what you have for the other keys.

If they're less you can simply insert the parsed bit of text into the beginning of the relevant key. If they're more, but less than the other values, you insert it at the end. Update the value of the two held variables for that key to reflect any changes to lowest / highest number.

If they're inbetween you'd have to read the whole string from the relevant key until you find where to insert them.

I'm not sure how long a string can be for each entry, so maybe you'd have a limit? If you have to enter the data into a position inbetween the beginning and the end of an already "full" line, then you shift the end of that string into the next key with a repeat of beginning / end / inbetween.

To be honest: As I'm typing this I'm thinking "blimey! that sounds like a lot of work" and it no doubt is. But using a long string might be better than more structures with lesser info, and if you did use 20 + data structures (maybe a ds priority instead of a ds map, since it could hold the name and order them by the value) you'd still have to do 20+ comparisons between them each time to then order the end result.

Either way it seems like it won't be easy, though the ds priority way is probably simpler. I'm just not sure which is best: Relative ease of use (ds priority, but maybe has issues with the amount you'd need) OR using strings which is not so easy but maybe doesn't have issues with the means of storage (as it could all be in one structure)
 

Lukan

Gay Wizard Freak
I understand wanting to be able to store 10 million player's data, very optimistic... However, 10 million people are NOT going to play your game.
Why do you need such an absurd amount of data stored? Why not just the top 1000?
 

sylvain_l

Member
I understand wanting to be able to store 10 million player's data, very optimistic... However, 10 million people are NOT going to play your game.
Why do you need such an absurd amount of data stored? Why not just the top 1000?
because I'm pretty bad at gaming and I care about my 9'999'999th spot and those around me and don't care at all for whom in the 999th.
In term of incentive to replay the game if you only store the top 100 or 1000 it can be pretty a killer if at first play you end up too far while if you keep it all, it's easy to see a progression between plays even for the worst player.

p.s.
as IndianaBones suggest using a DB (sqlite or any other) to store all those data seems a much better idea, and just recover the data you need to show with GML
 
EDIT: This might be a load of old guff given that IndianaBones had 900 000 entries (though that's still a way off from 10 million) but I'll post it anyway :) I would suggest using a ds priority as your data structure though, as it can hold names and a value (number of cups), and be ordered for later purposes.

I know that there are limits to the size of data structures, so if 400 000 entries is the limit you'd need more than 20 to store 10 million names. Which would require a fair bit of effort to sort through and order, as you'd have to order them all and then compare the results before ordering those results.

Using ds priorities:

1)Check that the priority exists
2) Check that the priority has any content
3)Check it against the others values (and that they exist, and have content etc)
4) place the lowest, find the new lowest value for that structure, and then repeat the comparisons


Or using strings:

To save on space / effort (?) you could consider putting multiple names and cups into one string, and have them divided by certain characters denoting the end of the name / end of the cup. If 400 000 entries is the limit for a ds map then that's 20 or so pieces of data per key, with the naming of the key not being relevant anymore. It could just be a number.

That's maybe better than 20 ds maps / priorities with a singular piece of data for each key.....??

My guess at doing this would be:

You'd have to parse the string in each key to get the details, place those details into a ds_priority (priority is the number, value is the name) and order it, and then rewrite the string so it's ordered. This would give you the lowest and highest values.

Store that as two variables for each key and have a third variable for what key it is.

That's one loop through the keys. The string for each key is now ordered, and you know the range of values for each key.

Then you'd have to rearrange those strings so that the one with the lowest starting value is first.

Do a second loop through the keys, and as you're parsing through the string you need to be checking the value against what you have for the other keys.

If they're less you can simply insert the parsed bit of text into the beginning of the relevant key. If they're more, but less than the other values, you insert it at the end. Update the value of the two held variables for that key to reflect any changes to lowest / highest number.

If they're inbetween you'd have to read the whole string from the relevant key until you find where to insert them.

I'm not sure how long a string can be for each entry, so maybe you'd have a limit? If you have to enter the data into a position inbetween the beginning and the end of an already "full" line, then you shift the end of that string into the next key with a repeat of beginning / end / inbetween.

To be honest: As I'm typing this I'm thinking "blimey! that sounds like a lot of work" and it no doubt is. But using a long string might be better than more structures with lesser info, and if you did use 20 + data structures (maybe a ds priority instead of a ds map, since it could hold the name and order them by the value) you'd still have to do 20+ comparisons between them each time to then order the end result.

Either way it seems like it won't be easy, though the ds priority way is probably simpler. I'm just not sure which is best: Relative ease of use (ds priority, but maybe has issues with the amount you'd need) OR using strings which is not so easy but maybe doesn't have issues with the means of storage (as it could all be in one structure)
Thank you for your response :D, my first though was exactly that: Create 100 ds_maps and save 100 000 players inside each one, but players cups is not
the only data i will store, i need to store their country, id and rank positions, after i will receive new cups info every second beacuse its an onlyne game,
so every time i receive an async i will have to go all the way trough the ds_maps to rewrite this new info, so basically i have to create a very well performed script lag proof. The other way as you said is with strings or buffers, the problem is that every time i need to write data i will have to disassemble the string or buffer to insert the new info, and that script has to be perfect too. So before i go trough all of this, i decided to learn SQL as
IndianaBones said:

What are these names exactly? Is it an online high score list? Or something built into the game itself? I haven't stored this many names before, but I would start by looking if I can use a database to store it. SQL / SQL Lite perhaps.

You said you are using a ds_list, but your code above shows you are using a ds_map.

I tested adding 900,000 names to a ds_map just now. I had to wait approx. 2 minutes for the game to start, but it worked fine other than the short wait to add all the names at the start of the game.
I found this tutorial
and php looks easy, i hope so. I will test SQL limits and if its buggy i think i will back to what the_dude_abides suggest.

I understand wanting to be able to store 10 million player's data, very optimistic... However, 10 million people are NOT going to play your game.
Why do you need such an absurd amount of data stored? Why not just the top 1000?
Im working over an old game that i released in 2014 that has 8 million downlodas on android, and i hope this big actualization will have a lot of new players too, so i think at least i will have 1 million new players account in 2 or 3 months, maybe not or maybe a lot more, well i want to be prepared
anyways in wathever be the case

Thank you all for your responses, i will keep learning SQL today and see what can i do :D
 
Top