[SOLVED] Sending surface over network

[DonMaklesso]

Hi, I've been trying to send an application_surface in a buffer (I'm using a server-client system) and displaying it in the client's game so that he would "see what the server sees" but this doesn't seem to work. It doesn't crash or anything just nothing happens.

Here's the code:

// Server - Step Event

buffer_seek(buff,buffer_seek_start,0);
buffer_write(buff,MESSAGE_ID,1);
buffer_get_surface(buff,application_surface,0);
network_send_packet(socket,buff,buffer_tell(buff));

// Script that handles received buffers from the server

function ClientReceive(buffer)
{
    var message_id = buffer_read(buffer,MESSAGE_ID);

    switch (message_id)
    {
        case 1:
            buffer_set_surface(buffer,application_surface,0);
            break;
    }
}

I've searched that buffer_set_surface() doesn't always work but I don't know how to apply some reasonable fix to my specific situation.
Does anyone have some idea that might help?
(I'm using the latest version of GMS 2)

EDIT: If there's better solution to "stream" view over network than using surfaces I'll gladly hear it

 

[Binsk]

Sending over an entire surface is insanely wasteful, that is so much data. A simple 1280x720 surface amounts to roughly 3.5MB that you are trying to send every frame (assuming uncompressed data).

If both people are running the same piece of software then that means they have the same code and resources to construct the same 'world' on their screen. You just need the host to send the data required to construct the world rather than the pixel-by-pixel representation of it.

E.g., If you had a gray room with a moving red circle, don't send the whole surface, just send the data for the circle (such as its position and size) and let the receiving end reconstruct the scene and render it. This would change a 3.5MB data transfer per frame to maybe a 12B data transfer.

 

[gnysek]

Still, locally it should work (even if with a big lag).

The issue here might be, that application surface gets re-drawn in next frame. Can you try separate surface?

 

[Binsk]

Still, locally it should work (even if with a big lag).

The issue here might be, that application surface gets re-drawn in next frame. Can you try separate surface?

I don't think it would be lag in this case. Moreso it would just be a massive flood of data that couldn't be handled.

Assuming the same surface size above, that requires a roughly 1.7gbps connection (at 60fps). While not my expertise, even the best 802.11n wireless connection tops out at 450mbps, I believe.

I suppose if connected through a more recent ethernet connection you would be fine (and that just for local IP connections), but the fact that you even have to think about connection type to make this work already should prove it is a really bad idea.

I suppose you could try to stagger frame updates and compress the buffer but unless the goal is to transfer something that literally requires per-pixel data (such as webcam data or something) this method shouldn't even be considered.

 

[HalRiyami]

Ignoring that uncompressed 1280x720 surface buffer is 211MB per second at 60fps like @Binsk said, you could try sending a single frame first. Once you receive the data, create a new surface (so you don't lose it as opposed to the application_surface) and set the buffer to that then draw it. If that works then try sending data every frame like @gnysek suggested. If that doesn't work then the problem is very likely how much data is being sent and you have to figure out how to compress the data enough to send it realistically.

 

[kburkhart84]

I do IT for a living(though I'm not an expert at networking as far as the programming/gamedev side). I DO understand how networking works however.

And I can tell you....unless you are using the very latest and greatest in ethernet technology, or have fiber network ran even between your local machines somehow, this isn't going to happen. CAT5e cabling is capable of Gigabit speeds, as are most of the modern switches, routers, and net cards you will probably have. But that is only 1.0Gbps....assuming Binsk is correct about needing 1.7Gbps(I didn't calculate but it seems close enough), that just isn't going to happen, even on the extremely fast speeds we can get locally. If you use CAT6 cabling(and equipment, etc...), the cabling itself can handle up to 10Gbps, but there isn't really much equipment that is rated at that speed just yet. You would have to have fiber cabling, as fiber equipment can handle that...but nobody has a PC with a fiber based NIC sitting around that I know of.

So, basically, as they mentioned above, you are simply going to have to either recreate the surface/image on the client itself(really, this is the best bet, easiest to handle, saves bandwidth, and is what ALL the big kids do ), or you are going to have to figure out how to massively compress the video...yes, I didn't say single image, I said video. The amount of compression you are needing is going to take compression types that are meant for video, not for single images. Video has a much higher percentage rate of compression than single images simply because the compression can handle things like pixels that don't change between frames. So something like RLE compression(google it, or PM me for an simple explanation) for single images can be exponentially done between frames as well.

 

[DonMaklesso]

Thanks to everyone for their replies

I know there are obviously better ways to make a multiplayer game (and I've been using them).
The thing is I was just wondering how to "stream" your viewport to other players.
But turns out it won't be happening using surfaces any time soon

Again, thanks for the quick replies!

 

[Juju]

This is a really interesting question. Clearly sending a 60FPS stream of 720p gameplay (uncompressed!) is going to be out of the question, but there are a few straight-forward strategies for reducing the bandwidth of images in GameMaker. Here's what immediately springs to mind:

1. Scale down your surface to 1/2 width & 1/2 height (which is 1/4 area and therefore 1/4 bytes)
2. Update the surface every third frame, dropping you down to 20FPS
3. Use the buffer compression functions (which are lossless)

As people have mentioned, 720p at 60FPS is 211mb/s. With the first two strategies, which reduce amount of data being transmitted by 12x, that drops down to 18mb/s which is... more reasonable. With compression in there, you can probably get that down to around 15mb/s which might be doable. This does mean people will see the game blurry as hell and at a 20FPS in 360p which... might not be what you're after! I also don't know if buffer compression is going to be fast enough to keep up - ideally we'd use a thread for that but no dice in GM at this time.

There's a bit of annoyance here insofar that for what you're trying to do you probably have no interest in the alpha channel of your surface. GM doesn't offer a way to render to 24-bit image buffers otherwise you could shave off 25% of your transmission size simply by ignoring the alpha channel. I'd imagine buffer compression would recognise the alpha channel and mostly compress that away, but it's wasted time and space regardless.

If you're making a pixel art game with scaling then you can use other compression strategies. If you're natively rendering at 360p and scaling up to 720p then you can skip the scaling down part of the process and maintain a nice clear image. If your game has a limited colour palette, up to 256 colours all at 100% alpha, then you can use the 32-bit nature of GM's surfaces to your advantage. You can make a compression system that turns a 2x2 block of input pixels into a single 4x8-bit output pixel, and you can also write that in a shader which is parallelised and faster than a CPU-side compression algo. With a cleverly designed LUT texture, you can achieve a 4:1 compression ratio with a modest performance penalty if you accept the aforementioned 256 colour limitation. If you have a 360p pixel art game and you're compressing it in this way, you can run at 30FPS with a bandwidth of 6.6mb/s which is within reach of some (but by no means all) consumer connections.

At the end of the day, this is mostly academic. You shouldn't really be trying to send packets substantially larger than 1kb (Ethernet has a maximum transmission unit of 1518 bytes which usually acts as an upper limit on the realistic packet size) so a 6.6mb/s stream will require thousands upon thousands of packets to be handled every frame. Additionally, although a connection may be capable of a given bandwidth, that doesn't mean those packets are going to arrive smoothly or consistently. Running a local simulation is always going to achieve a better experience because the client can fill in the blanks if packets arrive in fits and starts. This is where the bulk of multiplayer programming is, and it's for good reason.

As a closing remark, if you don't want to run an actual simulation on the client then, instead of sending the entire surface, you can send information on what the draw calls are. This will rather severely limit what kind of draw calls you can make and will incur a substantial performance penalty on the server, but with some imaginative thinking I reckon you could achieve some impressive results. This would mean caching all the relevant draw calls for a single frame and sending those over as a set of instructions to be executed on the client e.g. draw_sprite_ext() will get boiled down to a few integers and floats which represent a draw_sprite_ext() call that the client can then execute. This is effectively a one-way remote procedure call, which is a topic all of its own that I encourage you to look into.

 

[DonMaklesso]

Thanks for that @Juju
I'll give the remote procedure call method a try.