The fastest way to compress a string

B

BabyDukaCPH

Guest
Hi community

TL;DR: I need a way to compress a long string without the game going in sleep mode while compressing. Does anyone have a link to a nice tool or a clever method to do this without the wait and without too many extensions?

I'm making an online platformer where, when you beat the world record, your player's ghost run, among other things, is saved on a server. The way this is saved is that I keep the x,y,sprite_index,image_index for each step in an array which in is ultimately a very long string I'm writing to a file which later is read when player wishes to upload a custom level. The data sent contains an array of data, one of them is the player ghost. I'm also compressing large user level files (0-400 tiles each containing their vars) to keep level data files below 10 kb each (even on very large levels) so that player can download and keep thousands of levels and player ghost replays without a problem.

I'm currently compressing the string w Taylor Dale's string_compress from the market place which is working fine and even large levels with a long run is 5-15 kb, but when the player does a run over 1 minute or saves a level with hundreds of tiles, the string_compress script takes 1-2 minutes to compress the string. During that time the game 'waits', basically is dead (sleep) which always is doomed to make players think something is wrong. So far the community consists of nice alpha testers who understand and can bare with the waiting (old school C64 days are back!) but obviously I need to implement something fast to make sure new players who are not supported by the community, and just everyone in general, don't think the game has crashed in an annoying frozen way.

Has anyone had any good experience with a super fast string compressor besides from the one mentioned above?

- Jake, the DASH Team
 
A

appymedia

Guest
I know you mention online but assuming your not targeting HTML5 and using a native build of some sort have you tried using the YYC instead of the default VM? Compression is quite a CPU bound task if I'm right and it might help quite a bit.
 
H

Homunculus

Guest
If the goal is to avoid the freeze during the processing of the string, why not compress it server side? HTTP events are asynchronous, so while the data is being sent your game will not be blocked, and therefore even if the uncompressed data is a bit larger then the compressed counterpart, it shouldn't be a huge problem. You can even add a progress bar this way.

Server side, you could zip the data, and return back the zip file when the level is downloaded, which game maker has the ability to expand (and is also better compressed probably than anything a string utility can produce).

EDIT: sylvain_l answer is probably better, forgot about the existence of buffer_compress
 

sylvain_l

Member
x,y,sprite_index_,image_index that's only numbers
not sure how and why you transform that into a long string... than you compress with an external ext.

just try to compress directly that list of number (put into a buffer and use the GML buffer_compress() function on it
if require break the buffer in chunk so you can give regulary a feed back to the player (a progessing bar)

edit: also the idea of Catan is good alternative (depends of the size of the data before compression; because if you have too much data those with a bad connection aren't going to be happy about that alternative - perhaps let player decide in last resort in settings what thex prefer )
 
B

BabyDukaCPH

Guest
Thanks a lot for your replies. I'm not too experienced in this subject so bear w me and this is super helpful.

The reason I need to also be able to compress a string is because the level tiles' data is a json string written to a file when the player presses Confirm in the Save As dialogue box. I have already made an event to show progress yet I don't want the player to wait for more than 5 seconds for the level to save. That's my goal at least. Even if it means breaking the process into chunks to save as the player continues.

Re buffer_compress: I'm using GM 1.4 and I think buffer_compress is only in GM2

Letting the player determine the download process is nice. This could come in handy if the process has to take more than some seconds

We would love to not have to include a zip-compression tool w the program

Currently I am trying out a couple of scripts I've downloaded to see if the compression is somewhat less effective but shorter compression time

Thanks!
 
B

BabyDukaCPH

Guest
I know you mention online but assuming your not targeting HTML5 and using a native build of some sort have you tried using the YYC instead of the default VM? Compression is quite a CPU bound task if I'm right and it might help quite a bit.
Sorry, I mean online as in online interactive single player game. The game is not html5, but it connects players through our server and Steam on Linux and Windows platforms
 
H

Homunculus

Guest
Actually I suggested compressing the levels server side, not to ship the program with a zip compression tool (although it could also be a reasonable solution).

By the way, I also have a string compression tool too in the marketplace (free). https://marketplace.yoyogames.com/assets/4560/strict
It does work better with natural language, but you can give it a try.

Also, it is known that json generated by json_encode is definitely inefficient in terms of space (you can clearly see that if you look at the numbers stored in the json file). I think YellowAfterlife had some kind of script or utility that helps a lot optimizing the output
 
Last edited by a moderator:
B

BabyDukaCPH

Guest
Actually I suggested compressing the levels server side, not to ship the program with a zip compression tool (although it could also be a reasonable solution).

By the way, I also have a string compression tool too in the marketplace (free). https://marketplace.yoyogames.com/assets/4560/strict
It does work better with natural language, but you can give it a try.

Also, it is known that json generated by json_encode is definitely inefficient in terms of space (you can clearly see that if you look at the numbers stored in the json file). I think YellowAfterlife had some kind of script or utility that helps a lot optimizing the output
Thanks a lot! Will check it them out
 
Top