Text compression ... for web requests

clee2005

Member
Hi gang,

I was wondering if anyone had any bright ideas for text compression (it's actually a ds_map that has been encoded to string). I'm doing cloud save on our web service and at it's peak the save file can be 100k. Sending this back and forth from iOS and Android devices is not going to be appreciated. I can zip it on the server and send the ZIP file back to the device which is like 2k! Awesome! I can't seem to ZIP it on the device before sending though. GM doesn't support zipping out of the box.

Any ideas?

Thanks,
Chris
 
H

Homunculus

Guest
How do you encode the ds_map? If using ds_map_write, you may try using json instead and compare the resulting file size. It's not compressed in any case but you may save a few kb.

Your save file is large though, are you sure you need all that data saved? maybe you can keep some unchanging data local, like an item database, and only store a reference to the database
 

clee2005

Member
How do you encode the ds_map? If using ds_map_write, you may try using json instead and compare the resulting file size. It's not compressed in any case but you may save a few kb.

Your save file is large though, are you sure you need all that data saved? maybe you can keep some unchanging data local, like an item database, and only store a reference to the database
Thanks for the reply. I should have stated it was ds_map_write that is giving me the text file. It's a bunch of level progress data, and yes I could restructure the whole thing and save just pieces, but it would be a LOT of effort and I believe this to be the shortest path to victory if I can just find a way to compress the payload going out of the device.

I did just come across this and am wondering if it's as simple as including this header. I'm just investigating more now : ds_map_add(map, "Accept-Encoding", "gzip,deflate,sdch");

It seems that GM will receive gzip encoded responses from a web server, so I'm wondering if it will also encode requests going out if specified?
 

GMWolf

aka fel666
you could use a buffer to minimize your file size. then send the buffer, or encode as a base 64 string.
 

clee2005

Member
you could use a buffer to minimize your file size. then send the buffer, or encode as a base 64 string.
How would a buffer change the content size? Isn't it just a storage space in memory? Base64ing is usually larger than the original... 37% bigger approx.
 
H

Homunculus

Guest
The reason we are suggesting a buffer or a json string is that ds_map_write generates (at least until the latest update, dunno now) a suboptimal string representation of your data, which is probably much larger than what you could achieve with other encodings. It's easy enough to do to just try it and compare file size.

Gzip encoding would be great, but GM can't read nor write in this format natively
 

GMWolf

aka fel666
How would a buffer change the content size? Isn't it just a storage space in memory? Base64ing is usually larger than the original... 37% bigger approx.
Maps can be quite wasteful. the most obvious way is that they have to store the type of data before storing the data. (is it a string?, a real?, an array?).
they also store numbers as double presision floats and all that.
if you can incorporate a header that defines what data is comming, you can write the data to the buffer with no bloat.

you could have, for example, a series of interger numbers, where each bits represents if a piece of data is present or not. followed by all the values.
for instance, you may have 8 possible keys, (a,b,c,d,e,f,g,h). if the data you are sending only contains key a, b and g, then you could have a header that contains the u8 value 194. (11000010 in binary).
then you would write the value for a, b then g.

you can even use some compression techniques, depending on the data you send. If the data you send will often have strings of numbers that are the same, using something like the targa compression system will work nicely. Though most of the time, this isnt always possible.

I mentioned the base 64 bit as i dont know how you send your data. (http post? or just sockets?).
the best would be to send your buffer raw, as it will be the most compressed, but sometimes you may need to send strings.
 

clee2005

Member
Maps can be quite wasteful. the most obvious way is that they have to store the type of data before storing the data. (is it a string?, a real?, an array?).
they also store numbers as double presision floats and all that.
if you can incorporate a header that defines what data is comming, you can write the data to the buffer with no bloat.

you could have, for example, a series of interger numbers, where each bits represents if a piece of data is present or not. followed by all the values.
for instance, you may have 8 possible keys, (a,b,c,d,e,f,g,h). if the data you are sending only contains key a, b and g, then you could have a header that contains the u8 value 194. (11000010 in binary).
then you would write the value for a, b then g.

you can even use some compression techniques, depending on the data you send. If the data you send will often have strings of numbers that are the same, using something like the targa compression system will work nicely. Though most of the time, this isnt always possible.

I mentioned the base 64 bit as i dont know how you send your data. (http post? or just sockets?).
the best would be to send your buffer raw, as it will be the most compressed, but sometimes you may need to send strings.
Thanks @Catan and @Fel666 for your replies. I'll take a closer look at the buffers to see what the gain is. I'm kinda working off the 100k to 2k gain of zip compression, so if it's a gain of 5-10% or something, then it's likely not worth the effort. I'll play around though and report back if I find any magic!

Cheers!
 
R

renex

Guest
There is a very simple trick to reduce json strings...

Code:
str=string_replace_all(str,'.000000','')
str=string_replace_all(str,', ',',')
str=string_replace_all(str,': ',':')
str=string_replace_all(str,'[ ','[')
str=string_replace_all(str,' ]',']')
str=string_replace_all(str,'{ ','{')
str=string_replace_all(str,' }','}')
Compare these two json files.

{ "frames": [ [ "dust1", [ -26.000000, -101.000000, 0.000000, 0.000000, 1.000000, 0.827638, 0.703470, 336.447723, "", 15.000000 ], [ -37.000000, -107.000000, 0.000000, 0.000000, 1.000000, 1.000000, 1.000000, 0.000000, "", 14.000000 ], [ 34.000000, -105.000000, 0.000000, 0.000000, 1.000000, 0.761905, 0.840000, 32.347443, "", 0.000000 ], [ 10.000000, -95.000000, 0.000000, 0.000000, 1.000000, 1.000000, 1.000000, 0.000000, "", 1.000000 ], [ 15.000000, -126.000000, 0.000000, 0.000000, 1.000000, 1.000000, 1.000000, 0.000000, "", 11.000000 ], [ -6.000000, -138.000000, 0.000000, 0.000000, 1.000000, -1.444444, 1.260870, 0.000000, "", 10.000000 ], [ -22.000000, -124.000000, 0.000000, 0.000000, 1.000000, 1.000000, 1.000000, 0.000000, "", 9.000000 ], [ 25.000000, -62.000000, 0.000000, 0.000000, 1.000000, 0.999011, 1.087901, 239.656754, "", 8.000000 ], [ 36.000000, -89.000000, 0.000000, 0.000000, 1.000000, 1.000000, 1.000000, 0.000000, "", 7.000000 ], [ -6.000000, -78.000000, 0.000000, 0.000000, 1.000000, 1.000000, 1.000000, 0.000000, "", 6.000000 ], [ -39.000000, -32.000000, 0.000000, 0.000000, 1.000000, 1.057143, 1.111111, 270.000000, "", 5.000000 ], [ -23.000000, -56.000000, 0.000000, 0.000000, 1.000000, 1.000000, 1.000000, 0.000000, "", 4.000000 ], [ -15.000000, -81.000000, 0.000000, 0.000000, 1.000000, 1.000000, 1.000000, 0.000000, "", 3.000000 ], [ -5.000000, -103.000000, 0.000000, 0.000000, 1.000000, 1.000000, 1.000000, 0.000000, "", 2.000000 ] ], [ "", [ -78.000000, -135.000000, 0.000000, 0.000000, 1.000000, 0.827638, 0.703470, 254.538788, "", 15.000000 ], [ -56.000000, -143.000000, 0.000000, 0.000000, 1.000000, 1.000000, 1.000000, 0.000000, "", 14.000000 ], [ 57.000000, -128.000000, 0.000000, 0.000000, 1.000000, 0.761905, 0.840000, 32.347443, "", 0.000000 ], [ 33.000000, -118.000000, 0.000000, 0.000000, 1.000000, 1.000000, 1.000000, 0.000000, "", 1.000000 ], [ 12.000000, -130.000000, 0.000000, 0.000000, 1.000000, 1.000000, 1.000000, 0.000000, "", 11.000000 ], [ -6.000000, -148.000000, 0.000000, 0.000000, 1.000000, -1.444444, 1.260870, 0.000000, "", 10.000000 ], [ -22.000000, -134.000000, 0.000000, 0.000000, 1.000000, 1.000000, 1.000000, 0.000000, "", 9.000000 ], [ 31.000000, -76.000000, 0.000000, 0.000000, 1.000000, 1.031693, 1.160916, 246.801422, "", 8.000000 ], [ 43.000000, -97.000000, 0.000000, 0.000000, 1.000000, 1.000000, 1.000000, 0.000000, "", 7.000000 ], [ 2.000000, -89.000000, 0.000000, 0.000000, 1.000000, 1.000000, 1.000000, 0.000000, "", 6.000000 ], [ -53.000000, -57.000000, 0.000000, 0.000000, 1.000000, 1.000000, 1.000000, 233.806793, "", 5.000000 ], [ -24.000000, -72.000000, 0.000000, 0.000000, 1.000000, 1.000000, 1.000000, 0.000000, "", 4.000000 ], [ -13.000000, -95.000000, 0.000000, 0.000000, 1.000000, 1.000000, 1.000000, 0.000000, "", 3.000000 ], [ -5.000000, -113.000000, 0.000000, 0.000000, 1.000000, 1.000000, 1.000000, 0.000000, "", 2.000000 ] ], [ "", [ -81.000000, -143.000000, 0.000000, 0.000000, 1.000000, 0.827638, 0.703470, 210.784149, "", 14.000000 ], [ -48.000000, -151.000000, 0.000000, 0.000000, 1.000000, 1.000000, 1.000000, 0.000000, "", 13.000000 ], [ 77.000000, -144.000000, 0.000000, 0.000000, 1.000000, 0.761905, 0.840000, 32.347443, "", 12.000000 ], [ 48.000000, -129.000000, 0.000000, 0.000000, 1.000000, 1.000000, 1.000000, 0.000000, "", 0.000000 ], [ 15.000000, -138.000000, 0.000000, 0.000000, 1.000000, 1.000000, 1.000000, 0.000000, "", 10.000000 ], [ -3.000000, -156.000000, 0.000000, 0.000000, 1.000000, -1.444444, 1.260870, 0.000000, "", 9.000000 ], [ -19.000000, -142.000000, 0.000000, 0.000000, 1.000000, 1.000000, 1.000000, 0.000000, "", 8.000000 ], [ 73.000000, -86.000000, 0.000000, 0.000000, 1.000000, 1.102137, 0.931245, 317.353302, "", 7.000000 ], [ 51.000000, -107.000000, 0.000000, 0.000000, 1.000000, 1.000000, 1.000000, 0.000000, "", 6.000000 ], [ 14.000000, -93.000000, 0.000000, 0.000000, 1.000000, 1.000000, 1.000000, 0.000000, "", 5.000000 ], [ -78.000000, -69.000000, 0.000000, 0.000000, 1.000000, 1.000000, 1.000000, 213.690063, "", 4.000000 ], [ -43.000000, -77.000000, 0.000000, 0.000000, 1.000000, 1.000000, 1.000000, 0.000000, "", 3.000000 ], [ -12.000000, -96.000000, 0.000000, 0.000000, 1.000000, 1.000000, 1.000000, 0.000000, "", 2.000000 ], [ -2.000000, -121.000000, 0.000000, 0.000000, 1.000000, 1.000000, 1.000000, 0.000000, "", 1.000000 ] ], [ "", [ -48.000000, -116.000000, 0.000000, 0.000000, 1.000000, 0.733730, 0.860929, 309.472473, "", 14.000000 ], [ -40.000000, -141.000000, 0.000000, 0.000000, 1.000000, 1.000000, 1.000000, 0.000000, "", 13.000000 ], [ 52.000000, -119.000000, 0.000000, 0.000000, 1.000000, 0.761905, 0.840000, 11.768289, "", 12.000000 ], [ 35.000000, -114.000000, 0.000000, 0.000000, 1.000000, 1.000000, 1.000000, 0.000000, "", 0.000000 ], [ 16.000000, -132.000000, 0.000000, 0.000000, 1.000000, 1.000000, 1.000000, 0.000000, "", 10.000000 ], [ -2.000000, -150.000000, 0.000000, 0.000000, 1.000000, -1.444444, 1.260870, 0.000000, "", 9.000000 ], [ -18.000000, -136.000000, 0.000000, 0.000000, 1.000000, 1.000000, 1.000000, 0.000000, "", 8.000000 ], [ 64.000000, -54.000000, 0.000000, 0.000000, 1.000000, 1.127546, 1.058099, 316.468811, "", 7.000000 ], [ 45.000000, -84.000000, 0.000000, 0.000000, 1.000000, 1.000000, 1.000000, 0.000000, "", 6.000000 ], [ 10.000000, -92.000000, 0.000000, 0.000000, 1.000000, 1.000000, 1.000000, 0.000000, "", 5.000000 ], [ -61.000000, -84.000000, 0.000000, 0.000000, 1.000000, 1.000000, 1.000000, 160.253159, "", 4.000000 ], [ -24.000000, -67.000000, 0.000000, 0.000000, 1.000000, 1.000000, 1.000000, 0.000000, "", 3.000000 ], [ -13.000000, -93.000000, 0.000000, 0.000000, 1.000000, 1.000000, 1.000000, 0.000000, "", 2.000000 ], [ -1.000000, -115.000000, 0.000000, 0.000000, 1.000000, 1.000000, 1.000000, 0.000000, "", 1.000000 ] ], [ "dust2", [ -27.000000, -102.000000, 0.000000, 0.000000, 1.000000, 0.827638, 0.703470, 327.600159, "", 15.000000 ], [ -31.000000, -125.000000, 0.000000, 0.000000, 1.000000, 1.000000, 1.000000, 0.000000, "", 14.000000 ], [ 28.000000, -102.000000, 0.000000, 0.000000, 1.000000, 0.761905, 0.840000, 37.116859, "", 0.000000 ], [ 19.000000, -99.000000, 0.000000, 0.000000, 1.000000, 1.000000, 1.000000, 0.000000, "", 1.000000 ], [ 17.000000, -125.000000, 0.000000, 0.000000, 1.000000, 1.000000, 1.000000, 0.000000, "", 11.000000 ], [ 2.000000, -138.000000, 0.000000, 0.000000, 1.000000, -1.444444, 1.260870, 0.000000, "", 10.000000 ], [ -17.000000, -129.000000, 0.000000, 0.000000, 1.000000, 1.000000, 1.000000, 0.000000, "", 9.000000 ], [ -38.000000, -37.000000, 0.000000, 0.000000, 1.000000, 1.127546, 1.058099, 270.000000, "", 8.000000 ], [ -22.000000, -60.000000, 0.000000, 0.000000, 1.000000, 1.000000, 1.000000, 0.000000, "", 7.000000 ], [ -13.000000, -83.000000, 0.000000, 0.000000, 1.000000, 1.000000, 1.000000, 0.000000, "", 6.000000 ], [ 13.000000, -46.000000, 0.000000, 0.000000, 1.000000, 1.000000, 1.000000, 209.931519, "", 5.000000 ], [ 34.000000, -69.000000, 0.000000, 0.000000, 1.000000, 1.000000, 1.000000, 0.000000, "", 4.000000 ], [ 15.000000, -84.000000, 0.000000, 0.000000, 1.000000, 1.000000, 1.000000, 0.000000, "", 3.000000 ], [ 0.000000, -108.000000, 0.000000, 0.000000, 1.000000, 1.000000, 1.000000, 0.000000, "", 2.000000 ] ], [ "", [ 35.000000, -115.000000, 0.000000, 0.000000, 1.000000, 0.855574, 0.862244, 20.409882, "", 15.000000 ], [ 3.000000, -111.000000, 0.000000, 0.000000, 1.000000, 1.000000, 1.000000, 0.000000, "", 14.000000 ], [ 18.000000, -87.000000, 0.000000, 0.000000, 1.000000, 0.761905, 0.840000, 311.729523, "", 1.000000 ], [ 20.000000, -101.000000, 0.000000, 0.000000, 1.000000, 1.000000, 1.000000, 356.566376, "", 0.000000 ], [ 18.000000, -127.000000, 0.000000, 0.000000, 1.000000, 1.000000, 1.000000, 0.000000, "", 11.000000 ], [ -6.000000, -141.000000, 0.000000, 0.000000, 1.000000, -1.444444, 1.260870, 0.000000, "", 10.000000 ], [ -17.000000, -131.000000, 0.000000, 0.000000, 1.000000, 1.000000, 1.000000, 0.000000, "", 9.000000 ], [ -49.000000, -52.000000, 0.000000, 0.000000, 1.000000, 1.127546, 1.058099, 241.049011, "", 8.000000 ], [ -27.000000, -67.000000, 0.000000, 0.000000, 1.000000, 1.000000, 1.000000, 0.000000, "", 7.000000 ], [ -12.000000, -85.000000, 0.000000, 0.000000, 1.000000, 1.000000, 1.000000, 0.000000, "", 6.000000 ], [ 42.000000, -80.000000, 0.000000, 0.000000, 1.000000, 1.012986, 1.106906, 239.420776, "", 5.000000 ], [ 61.000000, -97.000000, 0.000000, 0.000000, 1.000000, 1.000000, 1.000000, 0.000000, "", 4.000000 ], [ 16.000000, -86.000000, 0.000000, 0.000000, 1.000000, 1.000000, 1.000000, 0.000000, "", 3.000000 ], [ 1.000000, -110.000000, 0.000000, 0.000000, 1.000000, 1.000000, 1.000000, 0.000000, "", 2.000000 ] ], [ "", [ 71.000000, -144.000000, 0.000000, 0.000000, 1.000000, 0.855574, 0.862244, 36.869896, "", 15.000000 ], [ 32.000000, -128.000000, 0.000000, 0.000000, 1.000000, 1.000000, 1.000000, 0.000000, "", 14.000000 ], [ -22.000000, -115.000000, 0.000000, 0.000000, 1.000000, 0.761905, 0.840000, 281.513824, "", 1.000000 ], [ -4.000000, -123.000000, 0.000000, 0.000000, 1.000000, 1.000000, 1.000000, 0.000000, "", 0.000000 ], [ 16.000000, -137.000000, 0.000000, 0.000000, 1.000000, 1.000000, 1.000000, 0.000000, "", 11.000000 ], [ -13.000000, -151.000000, 0.000000, 0.000000, 1.000000, -1.444444, 1.260870, 0.000000, "", 10.000000 ], [ -18.000000, -141.000000, 0.000000, 0.000000, 1.000000, 1.000000, 1.000000, 0.000000, "", 9.000000 ], [ -65.000000, -70.000000, 0.000000, 0.000000, 1.000000, 1.127546, 1.058099, 162.824097, "", 8.000000 ], [ -34.000000, -78.000000, 0.000000, 0.000000, 1.000000, 1.000000, 1.000000, 0.000000, "", 7.000000 ], [ -14.000000, -95.000000, 0.000000, 0.000000, 1.000000, 1.000000, 1.000000, 0.000000, "", 6.000000 ], [ 70.000000, -82.000000, 0.000000, 0.000000, 1.000000, 1.012986, 1.106906, 327.094757, "", 5.000000 ], [ 49.000000, -102.000000, 0.000000, 0.000000, 1.000000, 1.000000, 1.000000, 0.000000, "", 4.000000 ], [ 14.000000, -96.000000, 0.000000, 0.000000, 1.000000, 1.000000, 1.000000, 0.000000, "", 3.000000 ], [ -1.000000, -120.000000, 0.000000, 0.000000, 1.000000, 1.000000, 1.000000, 0.000000, "", 2.000000 ] ], [ "", [ 34.000000, -102.000000, 0.000000, 0.000000, 1.000000, 0.855574, 0.862244, 36.869896, "", 15.000000 ], [ 10.000000, -103.000000, 0.000000, 0.000000, 1.000000, 1.000000, 1.000000, 0.000000, "", 14.000000 ], [ 6.000000, -104.000000, 0.000000, 0.000000, 1.000000, 0.761905, 0.840000, 320.477356, "", 1.000000 ], [ -5.000000, -118.000000, 0.000000, 0.000000, 1.000000, 1.000000, 1.000000, 0.000000, "", 0.000000 ], [ 15.000000, -132.000000, 0.000000, 0.000000, 1.000000, 1.000000, 1.000000, 0.000000, "", 11.000000 ], [ -14.000000, -146.000000, 0.000000, 0.000000, 1.000000, -1.444444, 1.260870, 0.000000, "", 10.000000 ], [ -19.000000, -136.000000, 0.000000, 0.000000, 1.000000, 1.000000, 1.000000, 0.000000, "", 9.000000 ], [ -53.000000, -61.000000, 0.000000, 0.000000, 1.000000, 1.127546, 1.058099, 162.824097, "", 8.000000 ], [ -32.000000, -74.000000, 0.000000, 0.000000, 1.000000, 1.000000, 1.000000, 0.000000, "", 7.000000 ], [ -15.000000, -90.000000, 0.000000, 0.000000, 1.000000, 1.000000, 1.000000, 0.000000, "", 6.000000 ], [ 49.000000, -41.000000, 0.000000, 0.000000, 1.000000, 1.012986, 1.106906, 298.810791, "", 5.000000 ], [ 32.000000, -71.000000, 0.000000, 0.000000, 1.000000, 1.000000, 1.000000, 0.000000, "", 4.000000 ], [ 13.000000, -91.000000, 0.000000, 0.000000, 1.000000, 1.000000, 1.000000, 0.000000, "", 3.000000 ], [ -2.000000, -115.000000, 0.000000, 0.000000, 1.000000, 1.000000, 1.000000, 0.000000, "", 2.000000 ] ] ], "skeleton": [ [ 1.000000, -104.000000, -138.000000, "a1_arm", "a1_hand", 1.000000, 0.827638, 0.703470, "" ], [ 6.000000, -62.000000, -138.000000, "a1_arm", "a1_elbow", 1.000000, 1.000000, 1.000000, "" ], [ 3.000000, 105.000000, -139.000000, "a1_arm", "a1_hand", 1.000000, 0.761905, 0.840000, "" ], [ 4.000000, 60.000000, -139.000000, "a1_arm", "a1_elbow", 1.000000, 1.000000, 1.000000, "" ], [ 13.000000, 21.000000, -138.000000, "<undefined>", "<undefined>", 1.000000, 1.000000, 1.000000, "" ], [ 13.000000, 0.000000, -150.000000, "<undefined>", "a1_head", 1.000000, 1.481481, 1.250000, "" ], [ 13.000000, -20.000000, -138.000000, "<undefined>", "<undefined>", 1.000000, 1.000000, 1.000000, "" ], [ 8.000000, -56.000000, -28.000000, "a1_leg", "a1_foot", 1.000000, 1.065181, -0.980935, "foot2" ], [ 9.000000, -33.000000, -61.000000, "a1_leg", "a1_knee", 1.000000, 1.000000, 1.000000, "" ], [ 13.000000, -12.000000, -90.000000, "<undefined>", "<undefined>", 1.000000, 1.000000, 1.000000, "" ], [ 11.000000, 58.000000, -29.000000, "a1_leg", "a1_foot", 1.000000, 1.000000, 1.000000, "foot1" ], [ 12.000000, 36.000000, -60.000000, "a1_leg", "a1_knee", 1.000000, 1.000000, 1.000000, "" ], [ 13.000000, 16.000000, -91.000000, "<undefined>", "<undefined>", 1.000000, 1.000000, 1.000000, "" ], [ -1.000000, 1.000000, -115.000000, "<undefined>", "a1_body", 1.000000, 1.000000, 1.000000, "" ] ] }
{"frames":[["dust1",[-26,-101,0,0,1,0.827638,0.703470,336.447723,"",15],[-37,-107,0,0,1,1,1,0,"",14],[34,-105,0,0,1,0.761905,0.840000,32.347443,"",0],[10,-95,0,0,1,1,1,0,"",1],[15,-126,0,0,1,1,1,0,"",11],[-6,-138,0,0,1,-1.444444,1.260870,0,"",10],[-22,-124,0,0,1,1,1,0,"",9],[25,-62,0,0,1,0.999011,1.087901,239.656754,"",8],[36,-89,0,0,1,1,1,0,"",7],[-6,-78,0,0,1,1,1,0,"",6],[-39,-32,0,0,1,1.057143,1.111111,270,"",5],[-23,-56,0,0,1,1,1,0,"",4],[-15,-81,0,0,1,1,1,0,"",3],[-5,-103,0,0,1,1,1,0,"",2]],["",[-78,-135,0,0,1,0.827638,0.703470,254.538788,"",15],[-56,-143,0,0,1,1,1,0,"",14],[57,-128,0,0,1,0.761905,0.840000,32.347443,"",0],[33,-118,0,0,1,1,1,0,"",1],[12,-130,0,0,1,1,1,0,"",11],[-6,-148,0,0,1,-1.444444,1.260870,0,"",10],[-22,-134,0,0,1,1,1,0,"",9],[31,-76,0,0,1,1.031693,1.160916,246.801422,"",8],[43,-97,0,0,1,1,1,0,"",7],[2,-89,0,0,1,1,1,0,"",6],[-53,-57,0,0,1,1,1,233.806793,"",5],[-24,-72,0,0,1,1,1,0,"",4],[-13,-95,0,0,1,1,1,0,"",3],[-5,-113,0,0,1,1,1,0,"",2]],["",[-81,-143,0,0,1,0.827638,0.703470,210.784149,"",14],[-48,-151,0,0,1,1,1,0,"",13],[77,-144,0,0,1,0.761905,0.840000,32.347443,"",12],[48,-129,0,0,1,1,1,0,"",0],[15,-138,0,0,1,1,1,0,"",10],[-3,-156,0,0,1,-1.444444,1.260870,0,"",9],[-19,-142,0,0,1,1,1,0,"",8],[73,-86,0,0,1,1.102137,0.931245,317.353302,"",7],[51,-107,0,0,1,1,1,0,"",6],[14,-93,0,0,1,1,1,0,"",5],[-78,-69,0,0,1,1,1,213.690063,"",4],[-43,-77,0,0,1,1,1,0,"",3],[-12,-96,0,0,1,1,1,0,"",2],[-2,-121,0,0,1,1,1,0,"",1]],["",[-48,-116,0,0,1,0.733730,0.860929,309.472473,"",14],[-40,-141,0,0,1,1,1,0,"",13],[52,-119,0,0,1,0.761905,0.840000,11.768289,"",12],[35,-114,0,0,1,1,1,0,"",0],[16,-132,0,0,1,1,1,0,"",10],[-2,-150,0,0,1,-1.444444,1.260870,0,"",9],[-18,-136,0,0,1,1,1,0,"",8],[64,-54,0,0,1,1.127546,1.058099,316.468811,"",7],[45,-84,0,0,1,1,1,0,"",6],[10,-92,0,0,1,1,1,0,"",5],[-61,-84,0,0,1,1,1,160.253159,"",4],[-24,-67,0,0,1,1,1,0,"",3],[-13,-93,0,0,1,1,1,0,"",2],[-1,-115,0,0,1,1,1,0,"",1]],["dust2",[-27,-102,0,0,1,0.827638,0.703470,327.600159,"",15],[-31,-125,0,0,1,1,1,0,"",14],[28,-102,0,0,1,0.761905,0.840000,37.116859,"",0],[19,-99,0,0,1,1,1,0,"",1],[17,-125,0,0,1,1,1,0,"",11],[2,-138,0,0,1,-1.444444,1.260870,0,"",10],[-17,-129,0,0,1,1,1,0,"",9],[-38,-37,0,0,1,1.127546,1.058099,270,"",8],[-22,-60,0,0,1,1,1,0,"",7],[-13,-83,0,0,1,1,1,0,"",6],[13,-46,0,0,1,1,1,209.931519,"",5],[34,-69,0,0,1,1,1,0,"",4],[15,-84,0,0,1,1,1,0,"",3],[0,-108,0,0,1,1,1,0,"",2]],["",[35,-115,0,0,1,0.855574,0.862244,20.409882,"",15],[3,-111,0,0,1,1,1,0,"",14],[18,-87,0,0,1,0.761905,0.840000,311.729523,"",1],[20,-101,0,0,1,1,1,356.566376,"",0],[18,-127,0,0,1,1,1,0,"",11],[-6,-141,0,0,1,-1.444444,1.260870,0,"",10],[-17,-131,0,0,1,1,1,0,"",9],[-49,-52,0,0,1,1.127546,1.058099,241.049011,"",8],[-27,-67,0,0,1,1,1,0,"",7],[-12,-85,0,0,1,1,1,0,"",6],[42,-80,0,0,1,1.012986,1.106906,239.420776,"",5],[61,-97,0,0,1,1,1,0,"",4],[16,-86,0,0,1,1,1,0,"",3],[1,-110,0,0,1,1,1,0,"",2]],["",[71,-144,0,0,1,0.855574,0.862244,36.869896,"",15],[32,-128,0,0,1,1,1,0,"",14],[-22,-115,0,0,1,0.761905,0.840000,281.513824,"",1],[-4,-123,0,0,1,1,1,0,"",0],[16,-137,0,0,1,1,1,0,"",11],[-13,-151,0,0,1,-1.444444,1.260870,0,"",10],[-18,-141,0,0,1,1,1,0,"",9],[-65,-70,0,0,1,1.127546,1.058099,162.824097,"",8],[-34,-78,0,0,1,1,1,0,"",7],[-14,-95,0,0,1,1,1,0,"",6],[70,-82,0,0,1,1.012986,1.106906,327.094757,"",5],[49,-102,0,0,1,1,1,0,"",4],[14,-96,0,0,1,1,1,0,"",3],[-1,-120,0,0,1,1,1,0,"",2]],["",[34,-102,0,0,1,0.855574,0.862244,36.869896,"",15],[10,-103,0,0,1,1,1,0,"",14],[6,-104,0,0,1,0.761905,0.840000,320.477356,"",1],[-5,-118,0,0,1,1,1,0,"",0],[15,-132,0,0,1,1,1,0,"",11],[-14,-146,0,0,1,-1.444444,1.260870,0,"",10],[-19,-136,0,0,1,1,1,0,"",9],[-53,-61,0,0,1,1.127546,1.058099,162.824097,"",8],[-32,-74,0,0,1,1,1,0,"",7],[-15,-90,0,0,1,1,1,0,"",6],[49,-41,0,0,1,1.012986,1.106906,298.810791,"",5],[32,-71,0,0,1,1,1,0,"",4],[13,-91,0,0,1,1,1,0,"",3],[-2,-115,0,0,1,1,1,0,"",2]]],"skeleton":[[1,-104,-138,"a1_arm","a1_hand",1,0.827638,0.703470,""],[6,-62,-138,"a1_arm","a1_elbow",1,1,1,""],[3,105,-139,"a1_arm","a1_hand",1,0.761905,0.840000,""],[4,60,-139,"a1_arm","a1_elbow",1,1,1,""],[13,21,-138,"<undefined>","<undefined>",1,1,1,""],[13,0,-150,"<undefined>","a1_head",1,1.481481,1.250000,""],[13,-20,-138,"<undefined>","<undefined>",1,1,1,""],[8,-56,-28,"a1_leg","a1_foot",1,1.065181,-0.980935,"foot2"],[9,-33,-61,"a1_leg","a1_knee",1,1,1,""],[13,-12,-90,"<undefined>","<undefined>",1,1,1,""],[11,58,-29,"a1_leg","a1_foot",1,1,1,"foot1"],[12,36,-60,"a1_leg","a1_knee",1,1,1,""],[13,16,-91,"<undefined>","<undefined>",1,1,1,""],[-1,1,-115,"<undefined>","a1_body",1,1,1,""]]}
They are exactly the same data, but the first is 12988 bytes and the second one is 4592 bytes. That's 36% of the original size ;)
 

clee2005

Member
Thanks @renex! That's a nice trick. I just tried converting the ds_map to json rather than ds_map_write and it's 79k instead of the 105k of ds_map_write. If I use your json compression technique I might get another few % savings (my content doesn't have a lot of .00000's in it).
 

GMWolf

aka fel666
just as a note, if you are using floating point numbers, and dont need 64bit precision, you can use 32bit instead and get 50% compression on your data if you do end up using my solution. simmilarly a lot of data can be saved if you use integers. infact, if your integers are all between 0 and 255, you can reduce down that data to 12.5% (so 87.5% off)!
 
Top