1. Hey! Guest! The 36th GMC Jam will take place between February 27th, 12:00 UTC - March 2nd, 12:00 UTC. Why not join in! Click here to find out more!
    Dismiss Notice
  2. NOTICE: We will be applying a Xenforo update on Tuesday 25th of February. This means that from approximately 10:00 to 14:00 BST the forums will be offline (or possibly longer). Sorry for the inconvenience! Official Announcement here.

Is there an upgrade ( by GMS? ) for the JSON file format to handle 64bit values?

Discussion in 'Advanced Programming Discussion' started by Lord KJWilliams, Jan 10, 2020.

  1. Lord KJWilliams

    Lord KJWilliams Member

    Joined:
    Jul 2, 2018
    Posts:
    152
    I was just reading the online manual and it says that the JSON file format can not handle 64bit values. I was wondering if there are any plans in the standard ( see : https://www.json.org/json-en.html ), to upgrade it to handle 64bit values eventually ? I was reading via skimming what I found on that website link and it does not mention in the standard or explanation anything about 64bit values in the context of explaining how the format works.

    Thanks
     
  2. Homunculus

    Homunculus Member

    Joined:
    Jun 20, 2016
    Posts:
    856
    As far as I know that limitation is due mainly to javascript not being able to handle 64bit integers, but it is possible that the json parser on non-js platforms will work just fine. Note that we are talking about integers specifically though.
     
    Lord KJWilliams likes this.
  3. YellowAfterlife

    YellowAfterlife ᴏɴʟɪɴᴇ ᴍᴜʟᴛɪᴘʟᴀʏᴇʀ Forum Staff Moderator

    Joined:
    Apr 21, 2016
    Posts:
    2,507
    JSON numbers are strictly 64-bit floats, in and out of GM. Here's a small test to verify:
    Code:
    var m = json_decode(@'{"k":9007199254740993}');
    show_debug_message(m[?"k"]);
    
    this prints 9007199254740992 because 9007199254740993 cannot be represented with a 53-bit mantissa.
     
    Japster, Kezarus, Yal and 5 others like this.
  4. Lord KJWilliams

    Lord KJWilliams Member

    Joined:
    Jul 2, 2018
    Posts:
    152
    So that goes back to my first question if there is any plans to upgrade .JSON file format standard to handle 64bit integers, which I do not know of, in the programming community that is involved with that standard. I want to use the .JSON file format offline and online, and stick to one data format (.JSON ) , if I can. There is no mention of that limitation that I read on that link I posted ( as far as I could understand ).
     
  5. Lord KJWilliams

    Lord KJWilliams Member

    Joined:
    Jul 2, 2018
    Posts:
    152
    In terms of bit-fiddling , do you have to convert a 64-bit float to a 64int, to bit-fiddle ( using bit operators mentioned in the GML online manual ) and when your done, then assign the 64int back to a 64-bit float var ( a double ) to save in a .JSON file ?
     
  6. FrostyCat

    FrostyCat Member

    Joined:
    Jun 26, 2016
    Posts:
    4,881
    The only JSON-safe way is to store as a string and use int64() when reading back. 64-bit floats have only a 52-bit mantissa, so casting an int64 to one will irreversibly lose 12 significant bits.
     
    Japster and Lord KJWilliams like this.
  7. YellowAfterlife

    YellowAfterlife ᴏɴʟɪɴᴇ ᴍᴜʟᴛɪᴘʟᴀʏᴇʀ Forum Staff Moderator

    Joined:
    Apr 21, 2016
    Posts:
    2,507
    The standard does not explicitly say that numbers should be floats, but the fact that they allow for scientific notation, fractions, and mention Inf/NaN does imply that they are, and thus vast majority of implementations treat them as 64-bit floats.
    You can, but not every bit-fiddled value has an unique string representation (e.g. there are multiple NaNs), and you might conjure a Inf/NaN, which would cause an encoding or decoding error. Encoding 64-bit integers as strings is pretty standard for this reason.
     
    Japster and Lord KJWilliams like this.
  8. Lord KJWilliams

    Lord KJWilliams Member

    Joined:
    Jul 2, 2018
    Posts:
    152
    That means I have to convert a double to int64, then bit fiddle with the int64, then convert the bit fiddled int64 into a string. Then I write the string to a .JSON. When I read that string back from the .JSON, I convert that string back as a double ( via string_digits(), then real() ), and then to a int64 to read the same value that I created from the original bit fiddled int64 to read the individual binary bits as 1s or 0s.

    Would that work?
     
    Last edited: Jan 12, 2020
  9. GMWolf

    GMWolf aka fel666

    Joined:
    Jun 21, 2016
    Posts:
    3,523
    Another option is to write that 64bit int to a buffer then base64 encode it.
    However now it won't be human readable.

    Encoding it a string is probably the most sensible.
    You can just use the string() and real() int64() functions to go between int64 and string. No need for bit fiddling. (AFAIK)
     
    Last edited: Jan 12, 2020
  10. FrostyCat

    FrostyCat Member

    Joined:
    Jun 26, 2016
    Posts:
    4,881
    What the hell is so difficult to understand about NOT converting 64-bit integers through doubles? You are going to lose 12 significant bits, and two different people have told you that by now. The only conversion we have been talking about are conversions between 64-bit integers and strings.

    Suppose you have a 64-bit integer that you want to save, say it is in the variable global.int64val. When you save into the JSON map, you convert from int64 directly to string:
    Code:
    jsonmap[? "int64val"] = string(global.int64val);
    
    When you load from the JSON map, you convert from string directly back to int64:
    Code:
    global.int64val = int64(jsonmap[? "int64val"]);
    
    It doesn't get any more straight-forward than this.
     
    Lord KJWilliams likes this.
  11. Lord KJWilliams

    Lord KJWilliams Member

    Joined:
    Jul 2, 2018
    Posts:
    152
    I need the binary use of the 64-bit field for bit fiddling in the first place.
     
  12. Lord KJWilliams

    Lord KJWilliams Member

    Joined:
    Jul 2, 2018
    Posts:
    152
    Because I am used to cast typing in programming C or using functions like atoi() in C. Thats the problem with switching from a strong typed langauge like C ( they say it isnt strong typed but it is in my opinion ) to GML which has only TWO variable types, double and string.

    I didnt make that connection, because I thought there was a implied cast typing that I had to do, from a string to a int64. There is no mention in the online manual ( nor any index titled ) that mentions cast typing in general for anything in GML . I found instances of functions like atoi(), like string() or real() in GML. Thats why I am asking in the first place.
     
    Last edited: Jan 12, 2020
  13. GMWolf

    GMWolf aka fel666

    Joined:
    Jun 21, 2016
    Posts:
    3,523
    GML is both (mostly) weakly and dynamically typed but that here isn't the issue, since we are dealing with explicit casting.
    GM actually has more types.than string and real, it has bool, int64, array, and more.

    (Fun fact 'true' and 'false' are of type real, so is_bool(true) return false).

    The issue here is with type precision.
    It doesn't matter what language you are using, be it GML or C. If you cast a 64 bit int to a float, you will lose precision.

    I seeped here. It should be int64() not real. Real casts to a floating point value so you will loose precision.
    Strictly speaking it isn't. Since you can assign ints to floats, or floats to chars etc.
    Perhaps you are getting confused between strong typing and static typing?
    There is too. Here is for casting to a 64 bit int:
    https://docs.yoyogames.com/source/dadiospice/002_reference/maths/real valued functions/int64.html

    What I meant is you don't need to bit fiddle to read the value from the JSON. Of course it doesn't matter what you do with it afterwards.
     
  14. Lord KJWilliams

    Lord KJWilliams Member

    Joined:
    Jul 2, 2018
    Posts:
    152
    Sorry for the late reply, but I had to create a bunch of images for a idea that I want to post on the GMS forums, which is related to this post. Using gimp has a limitation that slows me down when I draw my pictures on it.

    I read your post , but I am going to re-explain myself and what I am trying to do, as best as I can. This way you understand where my train of thought is going. I will use pictures to explain what I mean further. I hope other members, can understand this too.

    Im currently planning and designing a complicated game, which is actually a collection of almost 40+ games for offline play. One of these games will be off-line and on-line ( depending on the design ). Because I want to have it online, I want to use .JSON file format offline before I can implement the online version, since using .JSON files saves me from having to reinvent or learn a new data format to use ( since I am going to be using Javascript for the online version ). One of the data types of files that I want to use .JSON files for, is for a extremly complicated mapping system that not just one game can use from my collection off-line.

    The basic mapping system of mine, is based on organization is like a deck of playing cards is stacked. The basic unit that I am talking about is an array or a grid field of 64 x 64 , where there 4096 values that are binary. Well 4096 bytes ( 64 rows x 64 bytes ) is how long a text string holding the binary field would be to write my grid field in to a .JSON file. My plan for my game is to have an area of memory that uses organized binary array of 64 bits, which why I need to use int64 (). An int64 uses 8 bytes. 64 x 8 bytes uses 512 bytes ( there are 4096 positions in binary bits ). For the text stream format for .JSON its 4096 bytes, plus what ever extra data a .JSON file has to add on to it to make it a valid .JSON file.

    So each grid field of 64 x 64 is going to represent two different types of values depending on its use and interpretation. The first interpretation using the location determine by the bit ( 1 to 64 ) or the individual bits of each of the 8 bytes representing 1,2,4,8,16,32,64, and 128, represents a location or a void.

    So what I wanted to know, and I did not know , which I understand now through the replies. Please understand, GML is a hurdle for me compared to my experience with C. They are two different languages.

    @GMWolf ( et al ) I really dont consider Boolean variables as anything special ( it's not a valid variable type ), because any type of variable can be boolean, its not hard to store a 1 or 0. But is it a waste of space, to use 8 bytes for a double or a int64 to serve as boolean? I think so. Is it a waste of space to use text strings to store binary bit fields, but if thats what I have to do to use .JSON file formats, then thats what I have to do. I will have to pass the cost of storage to the end user for this. Because heres where the cost of storage binary data as text data comes in ( as I further explain ).

    I thought that I had to convert a double ( initialized to 0 , e.g. var mybitfield, myint64; mybitfield = 0; myint64 = int64(mybitfield); ) then I do bit fiddling , this for one row only. So really using an array of 64 mybitfields ( var mybitfield[64], myint64[64] ). Next, I got to get rid of the garbage in variables that I instantiate ( for (x=0;x<64;x++) { mybitfield[x] = 0; myint64[x] = 0; } . Now I need to create the binary version of the double floating type ( myint64[x] = int64(mybitifield[x]); then I do bit fiddling with 4096 bits or 64-bit fields by 64 rows.

    @FrostyCat, your last reply makes more sense to me now - again this is a new langauge to me. And what I'm used to thinking from C programming does not work all the time in GML. So thats why I posted the message. Thanks.

    There is a second part to this , but I will reveal it in the next post of mine as a new topic
     
  15. Desert Dog

    Desert Dog Member

    Joined:
    Jul 30, 2019
    Posts:
    71
    If you're doing when I think you're doing, then just a word of caution against over-engineering a solution to be too clever. Remember: Good code is easy-to-maintain code. Speaking of engineering, you have to ask yourself: Does this solution scale well? ( I.e. If requirements change, and you now need to store a 96x96 grid, but you specifically designed to use an int64 for a 64x64 grid, suddenly it doesn't work.) So if you really must bit-fiddle, I'd recommend think about doing it with an ordinary/smaller data-type that you can scale if you need to. (Does.. GML have Byte type?)

    It's kind of interesting, because you normally only see this kind of bit-manipulation on low-spec micro systems these days. And on those systems you'd *never* see an int64..! More like int16 or 32.

    But perhaps saving data as flipped bits in a 64-bit string might be an interesting academic exercise. (Sounds like an interesting challenge!)
     
  16. GMWolf

    GMWolf aka fel666

    Joined:
    Jun 21, 2016
    Posts:
    3,523
    If you are worried about storage space, don't use JSON?
    Or write your bitfield to a buffer and base64 encode it.
     
  17. Lord KJWilliams

    Lord KJWilliams Member

    Joined:
    Jul 2, 2018
    Posts:
    152
    @GMWolf Let me repeat :

    In other words , I am creating a mapping system that is shared by all of my games , should they involve an environment that uses maps.

    @Desert Dog I plan to explain my idea better in a brand new post, but it is related to this one since it uses my idea ( in a different forum area ).


    Since I am talking about also bitwise operations, including the environment of Javascript...
    I found this on W3Schools website about doing bitwise operations:
    https://www.w3schools.com/js/js_bitwise.asp

    This is related to my 64-bit idea, so is W3Schools correct on this, or is this a mistake? Because this will be a challenge....
     
    Last edited: Jan 16, 2020
  18. GMWolf

    GMWolf aka fel666

    Joined:
    Jun 21, 2016
    Posts:
    3,523
    Let me repeat:
    If you need more explanation:
    Base64 encode your bitfield and store that as a JSON value string.
     
    Lord KJWilliams likes this.
  19. Lord KJWilliams

    Lord KJWilliams Member

    Joined:
    Jul 2, 2018
    Posts:
    152
    Sorry, I was tunnel visioned. The online manual is not telling me enough information to impress me with what GML can do with its language. Thats why I have to post messages like my original post of this subject to the forum, to drag the truth out. If I sound stupid, its because of the manual. Again , I am new to this language, but not new to programming and so far the only thing I have done with my game, is planing it out so I know where to go with my programming. Which is a good thing, because I learned the hard way what happens if I dont.
     
  20. GMWolf

    GMWolf aka fel666

    Joined:
    Jun 21, 2016
    Posts:
    3,523
    Perhaps it's time to write code and see how it goes.

    The other way around is worse: you keep worrying about the wrong issues, and never start writing any code down :)

    Also planning code when you don't know the framework well is often counter productive.
    Just make sure each system isn't coupled too tightly so you can re architect any part that doesn't perform as required.
     
    Desert Dog likes this.
  21. Lord KJWilliams

    Lord KJWilliams Member

    Joined:
    Jul 2, 2018
    Posts:
    152
    Its not the question of worrying, its the question of that I dont know. I did not know that you could use a bit pattern created with Encode64 and use it with string() , to write to a .JSON file - and I did not make that connection. Again, the online manual does not make that impression. I thought it was a encrpytion method for sending data from a source to a destination online. In my train of thought even though I forget that a number using Encode64 is a real number, but I am regarding it as a bit pattern to store information about only 1 thing. This is where my tunnel vision is. I want to use a bit field to represent two interpretations, the locations on a 64 x and 64 y grid ( Ive already mention this ) and as a bit plane. In this bit field anything bit that is 1 represents a location, a thing, a rule, or a set of conditions and rules that exist when the player goes to that location on the map. I might as well give a small sample of what my next post will look like

    This is a sample of a bit pattern ( about one fourth from the 64 (x) * 64 (y) grid ). So the location of the bit in row represents the bit in 8 bytes,

    32x32_grid_sample3.png

    In the computer memory its 512 bytes because its 64 rows by 8 bytes. In storage space its 4096+ bytes as a .JSON, since we have to use text string as you and others have suggested. It would be this size (4096) if I wrote it as a text file instead of a binary. To distinguish each byte and certain bits in a 64-bit pattern , I created this pic to illustrate a color coded distinction of all the 64 bits that are used in that bit field ( pay attention to the color arrangement ) :

    64x64_8_byte_grid4_2.PNG

    The grid block which is white - is bit 1, the grid block which is red with a white frame around it ( the last left red block ) is bit 64. Thats my organization of using the bit pattern as also the X coordinate system which has 64 rows for Y. There second use of this bit field that I want to it use for, which is to take the entire bit field and turn it into a bit plane to represent only one value, when one bit is on. This bit plane is part of memory segment that has 64 bit planes (z) in depth which uses the same value as the bits of the bit field, in my bit plane exactly. It looks like this :

    64bitplane_stack2.png

    The black colors represent the same value ( in the order of where they are ) of the bit field in the bit plane. So the white bit plane represents 4096 bits in the organization of the map, and if any the bits have a value of 1, then they equal 1 , like the white block in the picture above showing the 8 bytes. The other 63 bit planes hold the same 4096 bits that represent the same bit value in the bit field in the 8 bytes. The next plane, is bit plane 2 which represents the value 2 when a bit value is 1, the next plane is bit plane 3 which represents the value 4 when the bit value is 1, and so fourth as my bit field represents.

    This is what I want to do, I want to create a mapping system that creates initially bit plane 1, and the games create the rest of the values based on bit plane 1 for bit planes 2 to 64 ( or as many as needed ). Thats 4096 * 64 bit planes. For each location of the map, a 64-bit rod which is sample that is copied to another int64, or 64-bit variable. For instance, x = 1, y = 1, and z from 1 to 64 ( written as 1 - 64 ) is copied bit wise to the 64-bit variable. Thats the 64-bit rod for that location. this is how the rest of the evaluations for the other 4095 64-bit rods are done. Again, an evaluation of whether a bit 1 or 0 determines the details of that location of its x,y position, depending on the game that creates those bit planes after bit plane 1. And thats just part of what my game is going to use for its mapping system that is shared by many other games.

    64 bit planes which are stacked together look like a cube. 64 x 64 x 64, this is how my memory in my game is going to be allocated. Its a memory cube I as I call it. ( see my next post after this one )


    I wrote this on the fly, so there are typos and details I might not have completely told....
     
    Last edited: Jan 17, 2020
  22. Desert Dog

    Desert Dog Member

    Joined:
    Jul 30, 2019
    Posts:
    71
    I would recommend MDN as a reference rather than W3Schools. (W3S is great for a quick intro, is it's not very technically deep)

    But yeah, you'll have to use something like BigInt, which is only new enough to be supported in the latest browsers. (Chrome, Firefox, yes, Safari, Edge, No).
    I'm going to assume that GMS2 won't be trans-piling into a more custom data-types like BigInt.

    If the backward compatibility for older browsers is necessary, you'll need to check the other workarounds they've been doing for the last little bit.
    (Or design using a smaller/more standard bit-sized type and save yourself the headache.)
     
    Lord KJWilliams likes this.
  23. Lord KJWilliams

    Lord KJWilliams Member

    Joined:
    Jul 2, 2018
    Posts:
    152
    @GMWolf Ok here is what I meant by a 64 x 64 x 64 cube - I drew this to illustrate the idea of bit planes stacked together better than my previous picture .

    memory_cube3.PNG


    This is what 1 unit of 4096 units of 64-bit rods looks like. This is a model of what I want to use for part of the memory mapping system of the game that I am designing.
     
    Last edited: Jan 17, 2020
  24. Lord KJWilliams

    Lord KJWilliams Member

    Joined:
    Jul 2, 2018
    Posts:
    152
    I am going to have to go with what everyone is using to be compatible with all browsers ...
     
  25. chamaeleon

    chamaeleon Member

    Joined:
    Jun 21, 2016
    Posts:
    1,108
    At this point, why not simply use a buffer for your binary and base64 encode it to a string and store that in your json? You will get 1/3 overhead over raw binary.
     
    Last edited: Jan 17, 2020
  26. Lord KJWilliams

    Lord KJWilliams Member

    Joined:
    Jul 2, 2018
    Posts:
    152
    I am using a special mapping system, where the game creates the data that becomes bit plane 1 then writes it to a .JSON ( using the discussed conversions ), and the invidividual games create the rest of the bit planes based on bit plane 1 from a algorithm, which means every time I use that data from .JSON the algorithm used by each game will create the same result I want every time. Which saves me the trouble of saving the rest of the bit planes in a .JSON. But there are exceptions, based on each game that uses the mapping data. This is what I want to do , if possible, otherwise - I would have to save bit planes 2 to 64 ( or the highest bit plane the game is using ) to one .JSON file the same way that bit plane 1 is written to a .JSON file. So there would be at least two files not one. This invention is a buffer already, because it holds temporary data for each game that uses the mapping system.
     

Share This Page