Tthecreator
Your Creator!
About these benchmarks:
I'm currently in the process of testing and benchmarking the new game maker. What I want to do is to get as much scripts, functions, examples and features to get benchmarked to get a solid comparison how much faster gms2 will be. If you have anything to say about these benchmarks, then please do so as I want them to be as valid as possible. I will redo the benchmarks whenever a new version of game maker comes out or whenever someone suggest that my testing methods should be different. These values don’t have to completely represent everything about how game maker will run in the future, also because the gms2yyc isn't available yet for testing.
General results
So far, result have been as low as x0.1 times as fast and up to 203.9x(in a very specific case) as fast. Generally the numbers were somewhere between 0.5 and 2.0x. The biggest improvements have been with handling instances (2.4x gms1->gms2) and the lowest seen benchmark difference is for the for loop test (31) which scored a low score of 0.1x compared between the gms1 yoyo compiler and gms2.
The new game maker is faster in every benchmark compared to game maker studio 1, not using the yyc. Lowest performance multiplier was 0.9x. If we compare gms2 with the yoyo compiler from gms1 then results differ. They can be as low as 0.1 and as high 2.3. If we take the average of these numbers we still get a speed improvement of 1.1x. So yes project Zeus is faster.
Here is a quick screenshot of the excel sheet:
Interesting results and conclusions taken
The benchmarks performed differently on certain areas.
drawing(Marked with red on the sheet)
On benchmarks that had to do with drawing, there often was a performance decrease compared to the yyc on gms1. This might be due to the fact of game maker now using dx11 instead of the older dx9/opengl. Drawing circles has a significant performance increase for some reason while normal primitive drawing was only at 0.7x. All drawing tests did perform better on non-yyc gms2 than it performed on non-yyc gms1.
data(marked with blue on the sheet)
Here I did a lot of tests with all sorts of data structures like arrays, ds lists, ds maps ect... What I found was that there was a great performance increase on these structures for both gm1 and gm1yyc. One odd thing was that both gm1 and gm2 performed poorly in the ds_grid test compared to gm1yyc. gm2yyc's performance is still unknown.
Logic
I tried testing some logic based stuff, and the results are that the yyc is still the king when in comes to logic. Logic includes both loops and simple +-*/ operations. I accidentally did the first few test in the draw event without thinking so you can see that I redid them in the sheet. If you want to know whether there was a difference between speeds in the draw and step events than the answer is yes. About 1.05x on gms1, about 1.10x on gms1yyc and about 1.03x on the new gms2. The for loop test was absolutely crushed by the yyc, gms1(nonyyc) and gms2 are like snails in that regard.
Other
Once change that you can all be happy about is the significant performance boost for handling instances. Seems to be optimized for an increase of 2.3 to 2.5x. The random function also seems to be faster. One noticeable thing is that the yyc performs very, very badly on recursive functions. It would even just crash on test 28 so I had to step things down a notch on test 29 to get any results. gms2 is 5.7x faster on handling this than gms1yyc. the normal gms1 also seems 5.9x faster compared tot he gms1yyc.
Real games
Sorry, to little data :/ I'm planning on expanding this.
Performance instability
I had to rerun some tests because I got some instability when running the tests. It would sometimes give a graph looking like this:
This graph is supposed to have the all the red dots centered around one line.(yes those are dozens of dots each representing one step. There are multiple red dots in one pixel line.)
This occurred on all platforms inconsistently and restarting my computer didn't solve the problem(only tried this once though). Another weird thing is that this can suddenly happen in the middle of a test:
The graph for this test on gms1, which was done right after gms2's test looked like:
And the yyc had about the same issues:
It could also be OS/computer related issue, I really don't know exactly.
How these benchmarks were taken
These benchmarks were taken using version 2.0.1.32 of gms2 and version 1.4.1763 of gms1. A minimum of programs were opened (gms1, gms2, excel, paint.net). Especially programs that can give performance spikes (like a music player going to the next track) are avoided. All benchmarks were just ran from game maker, so no executables were made. When running your game in game maker the window automatically gets focus. This window focus was maintained throughout the entire benchmark and the mouse or keyboard wasn’t touched. But on the end of benchmarks a screenshot is still made. The numbers that are put in the excel sheet are taken from these screenshots. The benchmarks were ran on my pc with:
The code that I wrote would calculate the time it took for game maker to go from the begin step event up to the draw gui end event. Code in the benchmarks was either put in the step event or normal draw event. A surface is created on which a graph is created holding the following values:
A graph would look like:
View the benchmarks:
The file contains the complete excel file with notes, all project files, and all screenshots taken after the benchmarks(those are the complete performance graphs).
Benchmark link(4 November): https://drive.google.com/file/d/0B55RKkBYApL1bHVXcDlaeGx6WTg/view?usp=sharing
Up next
What I plan on testing is more loop logic and also some actual games. I really want to look into how 3D works in the new game maker since it looks completely changed. (I even thought they ditched it at first be apparently there are some matrix projection function.)
Also the yyc of gms2 will be tested once that comes out. If I look at the differences I can tell we're in for a treat regarding speed
Help me
You can help me by coming up with scripts, functions, games, ideas to benchmark. You can also help me by criticizing my work to make sure these benchmarks can be as accurate as possible. Of course I will be working on these myself, this is just a little begin I made. All code is out there for any closed beta tester to view. Also it would be nice if others could run this benchmark as well. Your computer will most likely give different numbers but I’d like to know if it changes the performance increase factors (like 1.5x faster would be 1.6x faster on your computer).
I'm currently in the process of testing and benchmarking the new game maker. What I want to do is to get as much scripts, functions, examples and features to get benchmarked to get a solid comparison how much faster gms2 will be. If you have anything to say about these benchmarks, then please do so as I want them to be as valid as possible. I will redo the benchmarks whenever a new version of game maker comes out or whenever someone suggest that my testing methods should be different. These values don’t have to completely represent everything about how game maker will run in the future, also because the gms2yyc isn't available yet for testing.
General results
So far, result have been as low as x0.1 times as fast and up to 203.9x(in a very specific case) as fast. Generally the numbers were somewhere between 0.5 and 2.0x. The biggest improvements have been with handling instances (2.4x gms1->gms2) and the lowest seen benchmark difference is for the for loop test (31) which scored a low score of 0.1x compared between the gms1 yoyo compiler and gms2.
The new game maker is faster in every benchmark compared to game maker studio 1, not using the yyc. Lowest performance multiplier was 0.9x. If we compare gms2 with the yoyo compiler from gms1 then results differ. They can be as low as 0.1 and as high 2.3. If we take the average of these numbers we still get a speed improvement of 1.1x. So yes project Zeus is faster.
Here is a quick screenshot of the excel sheet:
Interesting results and conclusions taken
The benchmarks performed differently on certain areas.
drawing(Marked with red on the sheet)
On benchmarks that had to do with drawing, there often was a performance decrease compared to the yyc on gms1. This might be due to the fact of game maker now using dx11 instead of the older dx9/opengl. Drawing circles has a significant performance increase for some reason while normal primitive drawing was only at 0.7x. All drawing tests did perform better on non-yyc gms2 than it performed on non-yyc gms1.
data(marked with blue on the sheet)
Here I did a lot of tests with all sorts of data structures like arrays, ds lists, ds maps ect... What I found was that there was a great performance increase on these structures for both gm1 and gm1yyc. One odd thing was that both gm1 and gm2 performed poorly in the ds_grid test compared to gm1yyc. gm2yyc's performance is still unknown.
Logic
I tried testing some logic based stuff, and the results are that the yyc is still the king when in comes to logic. Logic includes both loops and simple +-*/ operations. I accidentally did the first few test in the draw event without thinking so you can see that I redid them in the sheet. If you want to know whether there was a difference between speeds in the draw and step events than the answer is yes. About 1.05x on gms1, about 1.10x on gms1yyc and about 1.03x on the new gms2. The for loop test was absolutely crushed by the yyc, gms1(nonyyc) and gms2 are like snails in that regard.
Other
Once change that you can all be happy about is the significant performance boost for handling instances. Seems to be optimized for an increase of 2.3 to 2.5x. The random function also seems to be faster. One noticeable thing is that the yyc performs very, very badly on recursive functions. It would even just crash on test 28 so I had to step things down a notch on test 29 to get any results. gms2 is 5.7x faster on handling this than gms1yyc. the normal gms1 also seems 5.9x faster compared tot he gms1yyc.
Real games
Sorry, to little data :/ I'm planning on expanding this.
Performance instability
I had to rerun some tests because I got some instability when running the tests. It would sometimes give a graph looking like this:
This graph is supposed to have the all the red dots centered around one line.(yes those are dozens of dots each representing one step. There are multiple red dots in one pixel line.)
This occurred on all platforms inconsistently and restarting my computer didn't solve the problem(only tried this once though). Another weird thing is that this can suddenly happen in the middle of a test:
It could also be OS/computer related issue, I really don't know exactly.
How these benchmarks were taken
These benchmarks were taken using version 2.0.1.32 of gms2 and version 1.4.1763 of gms1. A minimum of programs were opened (gms1, gms2, excel, paint.net). Especially programs that can give performance spikes (like a music player going to the next track) are avoided. All benchmarks were just ran from game maker, so no executables were made. When running your game in game maker the window automatically gets focus. This window focus was maintained throughout the entire benchmark and the mouse or keyboard wasn’t touched. But on the end of benchmarks a screenshot is still made. The numbers that are put in the excel sheet are taken from these screenshots. The benchmarks were ran on my pc with:
- Intel Core i5-2380P
- 2x GTX 660 not running SLI
- 8 GB of ram
- SSD 850 evo 240GB (both IDE’s were on the ssd which caps at sata 6g speeds)
The code that I wrote would calculate the time it took for game maker to go from the begin step event up to the draw gui end event. Code in the benchmarks was either put in the step event or normal draw event. A surface is created on which a graph is created holding the following values:
- Fps (green line)
- Average of fps (blue line)
- The time it takes to get from begin step to draw gui end (all red dots)
- The average of the time it takes to get from begin step to draw gui end. (the yellow line)
- Average time the specific code being tested took to run.(fushia, pink/purpelish)
- Average time the specific code being tested took to run.(lime,greenish)
A graph would look like:
View the benchmarks:
The file contains the complete excel file with notes, all project files, and all screenshots taken after the benchmarks(those are the complete performance graphs).
Benchmark link(4 November): https://drive.google.com/file/d/0B55RKkBYApL1bHVXcDlaeGx6WTg/view?usp=sharing
Up next
What I plan on testing is more loop logic and also some actual games. I really want to look into how 3D works in the new game maker since it looks completely changed. (I even thought they ditched it at first be apparently there are some matrix projection function.)
Also the yyc of gms2 will be tested once that comes out. If I look at the differences I can tell we're in for a treat regarding speed
Help me
You can help me by coming up with scripts, functions, games, ideas to benchmark. You can also help me by criticizing my work to make sure these benchmarks can be as accurate as possible. Of course I will be working on these myself, this is just a little begin I made. All code is out there for any closed beta tester to view. Also it would be nice if others could run this benchmark as well. Your computer will most likely give different numbers but I’d like to know if it changes the performance increase factors (like 1.5x faster would be 1.6x faster on your computer).