GML sprite_create_from_surface async

[JeffJ]

Creating a sprite from a surface using sprite_create_from_surface can potentially be slow, depending on the size of the surface and the hardware being utilized, creating a noticeable stall.

I was thinking if it would be feasible to make a sort of async version of sprite_create_from_surface, perhaps using buffers to read the surface in bits at a time, and likewise, create the sprite in the same fashion?
Alternatively you could do the operation on another thread, but to my knowledge that isn't something GMS2 offers any support for.

The idea is simply to let the function take its time in the background (however long needed) and, in turn, avoid the stall / freeze. But I've very little experience with buffers, so I'm not even sure if this would be something you could accomplish with those - both the reading of the surface and the creation of the sprite from that content.

Any thoughts?

 

[GMWolf]

On DX11 and opengl, this is already taken care of for you (and yoyo) by the driver.

As long as you don't touch either the surface or the sprite, the driver will do the actual copy in its own time.

As soon as you touch the surface or sprite, the driver will block and wait for the rendering/copy queue to catch up and then let you change things.
A good driver will let you touch a resource and just do some cross queue synchronisation.

Nor DX11 or OpenGL have copy queues etc so there isn't that much YYG can do to streamline it. You just have to hope you have a good driver implementation.

If you want to avoid any stalls, try not using either resource for a frame to make sure the copy went through before you change it. Though if you aren't accessing anything on the CPU (buffer get surface etc) then you shouldn't need to.
If you are accessing them on the CPU each frame then consider double buffering your surface.

I'm basing all this entirely on my knowledge of OpenGL, graphics drivers and gpus. I don't know what GM may do and it's entirely possible the way they manage resources causes stalls. But I'm sure they know what they're doing and aren't trying to access resources on the CPu immediately after copying.

 

[JeffJ]

But all this assumes desktop, right? What about Android, for example?

It seems that it is definitely sprite_create_from_surface that is causing the stall, so I can't really "not" use it for a frame - using it altogether causes the stall.

 

[GMWolf]

By not use it, I mean, don't use the sprite after you called sprite_create_from_surface.
So don't draw the sprite etc. This should stop any GPU queues waiting on each other.
To avoid the CPU blocking, don't read pixel data from the sprite etc.


Or are you saying the the game is blocking on sprite_create_from_surface? That seems like a bug. It should 100% hand that operation over to the driver.
And that operation shouldn't block the main thread. What android device are you testing on?

I don't have much experience and on mobile, but openglES should be the same.
Maybe on iOS if they use metal, they could be copying the data around using the CPU instead of using a copy queue. Idk.

If sprite_create_from_surface itself is blocking, then that sounds like a bug to me.

 

[GMWolf]

Oh. Oh no....
So reading the description in the manual:

By default all new sprites have their bounding boxes calculated automatically (the exact bbox will depend on the size and transparency of the sprite).

What this tells me is that GM will actually download the surface pixel data to determine the bbox.
So now you are waiting for the graphics queue to complete! Horrible stuff!

Two options here:
Option 1: Just use the surface directly. Forget about sprites. A surface shouldn't be any less efficient than using a sprite created via sprite_create_from_surface.

Option 2: do sprite_create_from_surface right at the beginning of the frame. At that point, the graphics pipeline should be empty so there shouldn't be much of a stall.
This means: on frame 1 draw to surface. On frame 2, create the sprite.

An async version could be possible. But tbh YOYO should just not fetch the pixel.data at all... What were they thinking?!?

 

[JeffJ]

I just did a quick test. Before, I had this:

newSurfSprite = sprite_create_from_surface(global.drawLetterSurf,0,0,global.letterSurfMaxW,global.letterSurfMaxH,true,true,0,0);
objCollider.sprite_index=newSurfSprite;

I changed that into:

newSurfSprite = sprite_create_from_surface(global.drawLetterSurf,0,0,global.letterSurfMaxW,global.letterSurfMaxH,true,true,0,0);
alarm[0]=90;

And then I do objCollider.sprite_index=newSurfSprite; in alarm[0].

This makes me clearly see that alarm[0]'s code isn't creating any stalls - it is only sprite_create_from_surface that is doing the stall. The stall comes as soon as that line fires off - and once it's over, I can clearly see it's still at least a frame or so before the using of the sprite comes, which itself doesn't cause any lag at all.

On option 2, wouldn't that create the same problem as soon as I need to update the surface/sprite? It needs to be updated continually. I have other means in place for temporary collisions, hence why it doesn't matter too much if it takes a while, as long as it does it without a stall.

EDIT
Oh, ninja'd.

Hmm.. Not sure if option 1 would be viable for me - I need it to be a sprite, since I need it for collisions. Am I forgetting that there's a way to do that strictly with a surface?

On option 2, wouldn't that create the same problem as soon as I update the surface/sprite? It needs to be continually updated based on player input. I have temporary means of collisions, which is why it's not a problem if it takes a while to kick in, just as long as it doesn't create a stall.

 

[GMWolf]

Hmm.. Not sure if option 1 would be viable for me - I need it to be a sprite, since I need it for collisions. Am I forgetting that there's a way to do that strictly with a surface?

Use other collision functions or something.

On option 2, wouldn't that create the same problem as soon as I need to update the surface/sprite? It needs to be updated continually. I have other means in place for temporary collisions, hence why it doesn't matter too much if it takes a while, as long as it does it without a stall.

It should lessen the problem.
If you do the sprite create when the pipe is empty, you are just waiting on the pixel data to be downloaded. You don't have to wait on all the rendering to happen.

After you created the sprite, you can draw to the surface again. Just wait until the next frame to actually create a sprite from what you drew.
(And then it's probably best to again wait a frame before using that sprite.).

If you want the best chances, o would try with three surfaces. On frame one, draw to surface 1. On frame two, draw to surface 2 and create sprite 1 from surface 1.
On frame 3 draw to surface 3, create sprite 2 from surface 2, and draw sprite 1.
On frame 4, draw to surface 1, create sprite 3 from surface 3, and draw sprite 2.
Etc etc.
And do the sprite creation before any other rendering anywhere (a good driver shouldn't care but a bad driver might not do look ahead)

 

[JeffJ]

It should lessen the problem.
If you do the sprite create when the pipe is empty, you are just waiting on the pixel data to be downloaded. You don't have to wait on all the rendering to happen.

The thing is, the stall seems to be the same amount no matter how many times I update the surface and recreate the sprite; by this logic, shouldn't it be the biggest the first time I do it, and less after that? That's not what's happening.

 

[GMWolf]

The thing is, the stall seems to be the same amount no matter how many times I update the surface and recreate the sprite; by this logic, shouldn't it be the biggest the first time I do it, and less after that? That's not what's happening.

No: if you are rendering to the surface, you need to wait for the GPU to finish drawing to the surface before GM can read the pixel data (smh yoyo).
But if you create the sprite when the GPU already finished drawing to the surface, you don't have to wait on anything.

So, if you make sure to create the sprite a frame or two after you finished drawing to the surface, you don't need to wait on the drawing to finish

There will still be a small stall from downloading the pixel data but that's unavoidable (unless yoyo fix that, smh).

An async version won't work I think: the driver doesn't like being called from other threads. They could do it with fences and pixel buffer objects I suppose... Yeah, not impossible but not the right solution.
It shouldn't have to calculate the mask in the first place. I would file that alone as a bug.

 

[JeffJ]

Alright, did another test where, when I am done drawing to the surface, I set alarm0 to 180 (room speed 60).
In alarm 0, all that happens is:

newSurfSprite = sprite_create_from_surface(global.drawLetterSurf,0,0,global.letterSurfMaxW,global.letterSurfMaxH,true,true,0,0);
alarm[1]=180;

Then, in alarm1:

objCollider.sprite_index=newSurfSprite;

Again, as soon as alarm0 runs, major stall (same as before - no shorter or longer).
Alarm1 is perfectly fine, no stalls or anything.
Doesn't seem to matter how long I wait between finishing drawing to the surface and doing sprite_create_from_surface(); just doing it, at any point, stalls the game for a good second or two.
EDIT: Also doesn't matter if it's the first time or the tenth time, stall is the same. And always in alarm0 (creating the sprite from surface). Note, at this point it has been 180 frames since the surface was updated.

 

[GMWolf]

[edit] nvm.. I misread your latest post.

Ok, so my only explanation is that you are stalling on fetching the pixel data from the GPU. Fun, not much else but for YYG to use PBOs here.
Kinda surprising tbh, its slow but shouldn't be that slow... Well, unless YYG are doing something else to make it terribly slow.


But yeah, really you shouldn't need to create a sprite from surface:
You could set the collision mask for your object to a separate sprite, and just draw the surface directly.
Unless you need precise collisions. Then... Idk... An async version is indeed what you need.

 

[JeffJ]

That's just the thing; the collisions need to be based on the surface, and since it's completely dynamic, I can't know in advance what type of mask to give it. Right now I'm faking it with a bunch of smaller collision masks that follows along with the surface, but ideally, I'd like for them to be a temporary solution, and just have one collision entity instead - hence why I was trying to do this. It works perfectly (and also does wonders for performance in certain cases on certain hardware), but the stall is unacceptable to the experience, so right now I'm caught between a fake method that is smooth but not very scaleable, or a very quick and scaleable method that then instead causes a huge hiccup in the middle of gameplay. Both are suboptimal, to put it mildly.

Damn, thanks for trying though!

 

[GMWolf]

Hmmm.
Well it seems like you are trying to confound graphics and gameplay data. Never a good idea. As a general rule you never want to be relying on data that was generated on the GPU. It'll be slow and generally crap.

What sort of collisions are we looking at? Could you just use a tilemap, or a quadtree or something to represent what was "drawn"?

I'm very curious about just what GM is doing. I might do a capture and find out.

 

[JeffJ]

I would love to, but the type of games I do very rarely lend themselves to tilemaps. It's extremely dynamic and "fluid", about the farthest you can get from something you can put into any sort of grid or square system. You essentially freeform draw what the player walks on (think along the lines of linerider or similar).

Yeah, I would love to know more about this as well. What I have is generally working, I would just love to be able to support a wider range of hardware and more aggressive playstyles with more confidence in the performance. This would actually let me do that, provided I can remedy the stall.

 

[Bart]

Interesting discussion this is.

Could an alternative for the collisions be to use the built-in physics combined with vertex buffers or primitives?
Bind all fixtures to a single object to make it one static collision shape. Box2D can handle quite a bit from what I've seen.

Or, as you mention "fluid", it might be possible to use softbody particles, of which you can copy the color data to a buffer and then copy that buffer's contents to a surface using buffer_set_surface. Then draw that surface to visualize the collision shapes.
With that, you'd separate the collisions from the drawing and use a buffer "in-between" to get from one to the other.
Not sure if that'll actually create a drawing. I haven't tried that myself yet.

Just some ideas, though

 

[JeffJ]

Trust me, I went back and forth quite thoroughly before diving in, considering whether to go with physics or custom system; physics would definitely have its advantages, but it would also create a whole host of other issues, so way back I decided to not use physics - to switch it on now would be completely out of the question. The game is around 90% done, and at the moment I am mostly doing optimizations where I can (this hopefully being one of them).

I assume softbody particles is also a physics related thing?

 

[Bart]

Ah yeah, okay. I wasn't aware of the design decisions you made. And at 90% completion already. Hmm... Not that obvious indeed.

The softbody particles are indeed built on top of the physics engine, as far as I know.

Not too many options then, it seems.
If the bounding box is determined by checking the pixels of the surface would there be anything you can do to speed that up indirectly, like some basic image processing (the most basic being blend modes).
This is just some more or less educated guessing, though. It may be a bit too far-fetched. Also not sure if it'd have any effect.

 

[JeffJ]

Of course you couldn't know any of that - I really appreciate you taking the time all the same! Yes, not an easy issue at all.

Well, the main problem here isn't so much the speed of the collision itself, rather that the sprite_create_from_surface routine is so slow; that's not in itself an issue, if only I could move it to another thread, or have it load in bits at a time rather than all in one go. That's why I considered if something like that was achievable with buffers; writing the surface and sprite data manually to recreate the same effect of that function manually, but slower. Basically, I want to turn that surface into a sprite - or be able to collide with the surface. The former is possible, I just... Need to be able to avoid that damn stall somehow.

 

[GMWolf]

. It's extremely dynamic and "fluid", about the farthest you can get from something you can put into any sort of grid or square system. You essentially freeform draw what the player walks on (think along the lines of linerider or similar).

You could store the collision data a a bunch of lines, for something like Linerider.
Or for something like worms you could use a coarse grid or quadtree if the worlds are large.

I'm sure there is a good representation for your collisions that is fast, robust, memory efficient and most importantly, doesn't rely on you fetching rendered data.
Unless you need pixel perfect collisions and render potentially complex shapes with shaders etc...

Oh: triangle meshes. You could also collide against triangles.

 

[JeffJ]

You could store the collision data a a bunch of lines, for something like Linerider.

This allows for a lot more curved and dynamic shapes though, so I don't think that technique would work.

Or for something like worms you could use a coarse grid or quadtree if the worlds are large

From the looks of it, this would be a far too fundamental change, unfortunately (like I said, game is 90% done)

Unless you need pixel perfect collisions and render potentially complex shapes with shaders etc

Almost pretty much exactly this.

Oh: triangle meshes. You could also collide against triangles.

This sounds interesting. Do you know where I could find some material on this for this type of need? Not sure if it would also be too much a fundamental change, but it's worth looking into.

 

[GMWolf]

Do you know where I could find some material on this for this type of need?

I'm not sure. But googling "triangle" collisions might help.
Also GJK, SAT etc are good keywords.
Then you will want spatial partitioning too.


If you want to get pixel data from the GPU then you definitely need an async version of that function. (Something implemented using PBOs and fences).
I suppose you should file a bug report stating that in its current form the function is basically unusable.
Also how large is your surface?