Feedback Request: Allow changing endianness of buffers

[K3fka]

So in working on my game, I'm copying a binary file into a buffer. The file can be of a varying size, so I'm doing it one byte at a time until I hit the end. The issue I'm running into now is reading values back from the buffer. The data is all there correctly, but buffer_read() is expecting numbers to be in little-endian and my file is in big-endian. This would presumably be an issue if I wanted to modify the buffer with buffer_write() as well. For now the only workaround I can see if reading the bytes individually and then combining them to get a final value. Is it possible to add a way to read in big-endian?

Edit: In the meantime in case anyone else runs into this issue, here's a quick script I wrote to work around this and switch the endianness of a number

/// @func switchEndian(number)
/// @desc Converts a number between big- and little-endian
/// @param {int} number

var num = argument0;
var new = 0;
var i;

// Find the max power of 2 to start with
for (i = 0; power(256, i) < num; i++) {
}

// i ends up 1 higher than it needs to be
i--;

// Reverse the process
for (var j = i; j >= 0; j--) {
    var temp = num div power(256, j);
    num = num mod power(256, j);
  
    new += temp * power(256, i - j);
}

return new;

Probably not the most optimal way to do so, but it works.

 

[icuurd12b42]

you can write the value then seek and read write, one byte at a time, the data, swapping the end of the sequence with the start, a byte at a time instead... it could be faster and supports a few data types that use endianness...

 

[Mike]

First thing I'd ask.... is why they are big-endian to start with? All modern machines are little endian these days.

Second thing I'd say, is flip them prior to use, that way you're not constantly dealing with flipped byte order, this means you do it only once, prior to even compiling.

Lastly.... if you're only reading "ints" (8,16, or 32 bit INT values), then flipping via your own function is very simple...

  var a = buffer_read(buff,buffer_u8)<<24;
  a |= buffer_read(buff,buffer_u8)<<16;
  a |= buffer_read(buff,buffer_u8)<<8;
  a |= buffer_read(buff,buffer_u8);

.........more or less.

 

[FrostyCat]

First thing I'd ask.... is why they are big-endian to start with? All modern machines are little endian these days.

Second thing I'd say, is flip them prior to use, that way you're not constantly dealing with flipped byte order, this means you do it only once, prior to even compiling.

Because the format of the input file is big-endian, and the same is true for a lot of network and file protocols that GMS may deal with. It's unreasonable to expect that it's always possible to contort the format to suit GMS's tunnel vision. YoYo should have learned that from the debacle that was the HTTP functions, how support for headers were unimplemented for a whole year because you think we can always reprogram the back-end.

 

[Dog Slobber]

First thing I'd ask.... is why they are big-endian to start with? All modern machines are little endian these days.

TCP/IP's standard for representing multi-byte numbers within RFC 1700, also known as Network Byte Order is big endian.

 

[Mike]

Because the format of the input file is big-endian. This is likely a backport of an existing game, so twisting the format to fit GMS is NOT an option. It can be pre-processed to produce an internal little-endian version, but the pre-processing code has to come from somewhere, which is the point of the topic.

Of course it's an option. Porting games does frequently require manipulation of the original data. I've ported quite a few games in the past, and this is almost always a requirement. Very rarely do you keep the original data "as is", in fact... if you ever do, you count it as lucky and move on.

As to "how to pre-process it", I'd do a separate GML program to import, process/modify - slowly, and then export the updated version. Then the game doesn't have to worry about it, it can just read data as needed - more quickly. Using the code snippet above, you can easily add aa custom "read" go through and modify the endianness of things. Again, I've done this in the past, it's not an issue. This could can either stay IN the game - if it's not too slow, or you can do a pre-process on it to get it into a faster format.

TCP/IP's standard for multi-byte numbers, RFC 1700, also known as Network Byte Order is big endian.

This is the header information, not the user "data block" that will follow. User data can be whatever format you want it to be.

 

[K3fka]

To give more info, this is a custom file format used to store maps for my game. I could flip things to little-endian, but big-endian is a bit easier for me to read and parse while I'm creating the files by hand in a hex editor (just for now, a real editor will be made later). Thanks for your input on the bit shifting method - that's a lot faster and simpler than my script.