GameMaker Custom C++ in YYC without DLLs

kraifpatrik

(edited)
GameMaker Dev.
THIS POST IS NOT MAINTAINED AND IT DOES NOT CONTAIN INFO ACCURATE FOR THE LATEST VERSION OF GAMEMAKER. THIS IS ALSO FULLY EXPERIMENTAL, I DO NOT RECOMMEND ANYONE MESSING WITH THE INTERNALS OF GM!

Hello everyone,

recently I got into going through C++ source files generated by YYC, trying to get some understanding of how GameMaker works behind the hood. I'm putting this together and sharing with you in hopes to pontentially enable the community to make custom tools that could improve the generated C++ code, add extra functionality to it or spot incorrectly generated malfunctioning code and file bug reports. This document is still being developed and it's structure and content can change drastically. I don't have access to the full source code, I only go through files that are available on disk in plaintext form, so some information may be on the spot and some may be just a guess based on what I have seen so far and how I understand it.

Cheers!

Existing tools
If you come to this thread not as a tools developer but a user, here is a list of finished/WIP tools which allow you to do modifications to the C++ files.
If you would like me to add a tool to the list, please let me know through a reply to this thread or a PM.

build.bff
Code:
C:\Users\{username}\AppData\Local\GameMakerStudio2\GMS2TEMP\build.bff
This file exists only when a project is opened. It is passed through command line arguments to Igor. It is a JSON file containing info about the project, like it's name, location on the disk, configuration etc. If you have multiple projects open, this will containg info about the project that was open/run last.

Libraries folder
Code:
C:\ProgramData\GameMakerStudio2\Cache\runtimes\runtime-{version}\yyc\Win32\lib
This path (or similar for a different target platform) contains static libraries. It is not possible to add any custom libraries to the compilation process, as clang is triggered by Igor with given set of command line parameters.

Includes folder
Code:
C:\ProgramData\GameMakerStudio2\Cache\runtimes\runtime-{version}\yyc\include
Contains header files. It is possible to add custom ones here and include them in your project through some of the default headers.

YYC cache folder
Code:
C:\Users\{username}\AppData\Roaming\GameMakerStudio2\Cache\GMS2CACHE\{project}_{suffix}\{project}\default\Scripts
This is the place where GM puts C++ files for compilation. The files are overwritten every time you make changes in corresponding GML files. EDIT: Using gml_pragma("forceinline") anywhere in your project causes automatical overwriting of all C++ files, disabling their modification! (seems to no longer apply!) When cleaning cash with the brush icon, this folder gets erased completely. It is not possible to compile any custom file that's not generated by GM.

Source files for object events are named gml_Object_<objectname>_<event>.gml.cpp , gml_Script_<scriptname>.gml.cpp for scripts and gml_Room<roomname>.gml.cpp for rooms.

Through the file names and the project path contained in build.bff it is possible to reconstuct path to the original GML file. This could be used to inject C++ code from comment blocks, perform some optimizations etc.

YYGML.h
This header can be found in the includes folder. It contains includes of standard libraries, class & struct definitions (for some only forward declarations, so we can't see their implementation), declarations of external functions etc., so it's going to be the most important point of reference to us. I wouldn't recommend modifying anything that's already there, but since this header is included in every YYC source file, it is a good place to put your own includes of other libraries (if you want them to be included everywhere of course).

Note: The code is C++, but all included libraries are standard C libraries. I tried to include C++ libs like iostream and thread, but that only resulted in compile time errors. Normal C headers like Windows.h or math.h do work.

Following are a few structs/classes and functions that I find important.

class YYObjectBase
This class (as the name suggests) seems to be the base class for GM objects. The header file contains only a forward declaration of the class, so we can't see its implementation. It is going to be important for us, because instance pointers are casted to it when reading & writing built-in variables.

classes CInstance and CInstanceBase
As for CInstance, the header contains only its forward declaration, so there is not much info for it, except that every script and object event takes a pointer to it as first two arguments (one for 'self', one for 'other'), but when actually dealing with instance variables, the pointers are always casted to CInstanceBase pointers. The definition of CInstanceBase is guarded by #if defined(...) checks without an else branch, so the real CInstanceBase class could be hidden somewhere else where we can't see it, but this one can still give us certain clues on how GM works on the inside.

YYRValue &GetYYVarRef()
C++:
YYRValue &GetYYVarRef(int index)
This is CInstanceBase's method that is used to retrieve a reference to a YYRValue structure containing value and type of a variable with given index. An index of a variable can be found in the gmlids.h file.

Example
C++:
#include <YYGML.h>
#include "gmlids.h"

void gml_Object_MyObject_Create_0(CInstance* pSelf, CInstance* pOther)
{
    // Get object variable myVar and set it to 10
    CInstanceBase* instBase = (CInstanceBase*) pSelf;
    YYRValue* myVar = &instBase->GetYYVarRef(kVARID_self_myVar);
    (*myVar) = 10;

    // Get the global instance
    CInstanceBase* global = (CInstanceBase*) g_pGlobal;

    // Get global variable myGlobal and set it to 50
    YYRValue& global_myGlobal = global->GetYYVarRef(kVARID_global_myGlobal);
    global_myGlobal = 50;
}
struct RValue
Since GML allows us to do stuff like store values of different types into an array/list, change type of a variable throughout its lifetime, perform operations between different variable types etc. and C++ does not, it is important to wrap GM variables into a structure that holds its value and type. This is the base structure that handles just that. It also contains methods for retrieving the value as a different type.

struct YYRValue
This structure inherits from RValue and it is the structure that is actually used for storing variables. It contains overloaded operators for creating a new instance of the structure from different data types as well as performing operations between them. The definition of this structure is also guarded by checks, but since the else branch contains a definition as well (though with an empty body), I think we can take it as the real one.

bool Variable_GetValue_Direct()
C++:
bool Variable_GetValue_Direct(YYObjectBase *inst, int var_ind, int array_ind, RValue *val)
Used for getting a built-in variable.

Arguments:
  • inst - The instance of which we want to read the variable.
  • var_ind - The the id of the variable. Will be defined in the .vars.cpp file as g_VAR_varname.
  • array_ind - Used when the built-in varriable is an array (eg. alarm), otherwise ARRAY_INDEX_NO_INDEX is passed.
  • val - Where to save the variable.

Example:
C++:
#include <YYGML.h>
#include "gmlids.h"

// Defined in .vars.cpp
extern YYVAR g_VAR_x;
extern YYVAR g_VAR_alarm;

void gml_Object_MyObject_Create_0(CInstance* pSelf, CInstance* pOther)
{
    YYObjectBase* objBase = (YYObjectBase*) pSelf;

    // Get the built-in x variable and store it into local variable x
    YYRValue x;
    Variable_GetValue_Direct(objBase, g_VAR_x.val, (int) ARRAY_INDEX_NO_INDEX, &x);

    // Get the built-in alarm[0] variable and store it into local variable alarm0
    YYRValue alarm0;
    Variable_GetValue_Direct(objBase, g_VAR_alarm.val, 0, &alarm0);
}
bool Variable_SetValue_Direct()
C++:
bool Variable_SetValue_Direct(YYObjectBase *inst, int var_ind, int array_ind, RValue *val)
Used for setting a built-in variable. Has the same arguments as Variable_GetValue_Direct, except val is now the value we want to save.

Example:
C++:
#include <YYGML.h>
#include "gmlids.h"

// Defined in .vars.cpp
extern YYVAR g_VAR_x;
extern YYVAR g_VAR_alarm;

void gml_Object_MyObject_Create_0(CInstance* pSelf, CInstance* pOther)
{
    YYObjectBase* objBase = (YYObjectBase*) pSelf;

    // Get x, set it to 20 and save
    YYRValue x;
    Variable_GetValue_Direct(objBase, g_VAR_x.val, (int) ARRAY_INDEX_NO_INDEX, &x);
    x = 20; // Set x to 20
    Variable_SetValue_Direct(objBase, g_VAR_x.val, (int) ARRAY_INDEX_NO_INDEX, &x);

    // Get alarm[0], set it to 10 and save
    YYRValue alarm0;
    Variable_GetValue_Direct(objBase, g_VAR_alarm.val, 0, &alarm0);
    alarm0 = 10;
    Variable_SetValue_Direct(objBase, g_VAR_alarm.val, 0, &alarm0);
}
Macro FREE_RValue()
C++:
#define FREE_RValue(rvp)
Frees memory used by RValue and set its value to undefined.

YYGML_ functions
The header contains some declarations of external functions starting with prefix YYGML_ and followed by a function name used in GML, eg. YYGML_show_debug_message(), which is the equivalent of GML's show_debug_message(). These functions also have the same arguments as in GML. If you want to use some function, I would recommend you searching for it there first.

YYRValue &YYGML_CallLegacyFunction()
C++:
YYRValue &YYGML_CallLegacyFunction(CInstance *_pSelf, CInstance *_pOther, YYRValue &_result, int _argc, int _id, YYRValue **_args)
As the comment above the declaration states, "this function routes any unknown functions to the correct destination". So in case you can't find a declaration of some function withing those starting with YYGML_, you will have to call it through this one.

Arguments:
  • _pSelf - Pointer to the instance calling the function.
  • _pOther - Pointer to the 'other' instance, as eg. in the collision events.
  • _result - Reference to the YYRValue structure which will contain the return value of the function.
  • _argc - Total number of arguments passed to the functions.
  • _id - The id of the function you want to execute. Can be found in .vars.cpp.
  • _args - An array of function arguments in form of pointers to YYRValues.

Example:
C++:
#include <YYGML.h>
#include "gmlids.h"

// Defined in .vars.cpp
extern YYVAR g_FUNC_window_mouse_get_x;
extern YYVAR g_FUNC_window_mouse_get_y;

void gml_Object_MyObject_Create_0(CInstance* pSelf, CInstance* pOther)
{
    (CInstanceBase*) base = (CInstanceBase*) pSelf;

    // Create a structure for holding results of function calls
    YYRValue retval(0);

    // Call the window_mouse_get_x() and store its result into object's variable mouseX
    YYRValue* mouseX = &base->GetYYVarRef(kVARID_self_mouseX);
    FREE_RValue(retval); // Free memory first!
    (*mouseX) = YYGML_CallLegacyFunction(pSelf, pOther, retval, 0,g_FUNC_window_mouse_get_x.val, NULL);

    // Call the window_mouse_get_y() and store its result into object's variable mouseY
    YYRValue* mouseY = &base->GetYYVarRef(kVARID_self_mouseY);
    FREE_RValue(retval); // Free memory first!
    (*mouseY) = YYGML_CallLegacyFunction(pSelf, pOther, retval, 0,g_FUNC_window_mouse_get_y.val, NULL);
}
YYRValue &YYGML_CallScriptFunction()
C++:
YYRValue &YYGML_CallScriptFunction(CInstance *_pSelf, CInstance *_pOther, YYRValue &_result, int _argc, int _id, YYRValue **_args)
This function is the equivalent of script_execute. The arguments are exactly the same as for YYGML_CallLegacyFunction(), except the _id argument is the script's position in the resource tree in GM, starting from 0.

YYRValue &YYGML_CallExtensionFunction()
C++:
YYRValue &YYGML_CallExtensionFunction(CInstance *_pSelf, CInstance *_pOther, YYRValue &_result, int _argc, int _id, YYRValue **_args)
As the name suggest, this function is going to be used for executing extension functions. I haven't tried it out yet though, so more info on this one later.

gmlids.h
This file can be found in the YYC cache folder and it is included in every source file. It contains macro definitions of translations from global and object variable names to a unique index like so:

C++:
#define kVARID_global_varA 0
#define kVARID_self_varA 0
#define kVARID_global_varB 1
#define kVARID_self_varB 1
The indices are then used as argument in the CBaseInstance's GetYYVarRef() method to get the YYRValue structure containing the variable value and type. There always are both kVARID_global_ and kVARID_self_ definitions, no matter the variable scope.

The indices are also sorted by variable names, and since variables are held in an array (as is suggested by the available CInstanceBase definition and parameter names of getter/setter functions), it seems like every instance has allocated space for as many variables as there are in total. Because if you had one object with variables a, b, c and another object with variable z, the z would get index 3. So this object's instances would still need an array of size 4 for its single variable. Unless there is some other translation table of course. But this is just a speculation.

ProjectName.vars.cpp
This file can be found in the YYC cache folder. It is overwritten every time you run your project, so there is no point in doing any modifications to it, but it does contain some handy data.

YYVAR g_VAR_varname
Structures with variable names and ids (accessed with .val). The id is used when getting/setting a global and object variables. If you want to use one of the variables in your source, declare it there as extern, eg:

C++:
#include <YYGML.h>
#include "gmlids.h"

// Required to be able to access variable myVariable
extern YYVAR g_VAR_myVariable;

void gml_Object_MyObject_Create_0(CInstance* pSelf, CInstance* pOther)
{
    // You can now use g_VAR_myVariable here...
}
YYVAR g_FUNC_funcname
Structures with function names and ids (accessed with .val). The id is used for calling the function with YYGML_CallLegacyFunction(). To be able to use them in a source file, they also have to be declared there as extern.
 
Last edited:

YellowAfterlife

ᴏɴʟɪɴᴇ ᴍᴜʟᴛɪᴘʟᴀʏᴇʀ
Forum Staff
Moderator
I've been working on a tool that allows to automate C++ injection and code changes - most times you don't want to rewirte your GML code completely (else you'd use a C++ based engine, yeah?), just add strict types or native calls where appropriate.

Doing this by hand is a little less exciting, as cache can be invalidated on file changes and then you'll have to wait out a compile, replace the files again, and wait out another one.
 

lolslayer

Member
I've been working on a tool that allows to automate C++ injection and code changes - most times you don't want to rewirte your GML code completely (else you'd use a C++ based engine, yeah?), just add strict types or native calls where appropriate.

Doing this by hand is a little less exciting, as cache can be invalidated on file changes and then you'll have to wait out a compile, replace the files again, and wait out another one.
Can't wait!
 

kraifpatrik

(edited)
GameMaker Dev.
I've been working on a tool that allows to automate C++ injection and code changes - most times you don't want to rewirte your GML code completely (else you'd use a C++ based engine, yeah?), just add strict types or native calls where appropriate.

Doing this by hand is a little less exciting, as cache can be invalidated on file changes and then you'll have to wait out a compile, replace the files again, and wait out another one.
Nice! I do have a GitHub repo with tool which in its current state takes C++ code from comment blocks in GML and replaces body of the generated function with that. All you have to do is run the tool with python yyc-overwrite.py, so it's pretty convenient. But as you said, you don't want to rewrite the entire GML, but rather inject specific parts and do some modifications to what's already there, so I will be also thinking about in which direction I will be taking the repo. Can't wait to see more stuff like this and more ideas on what could be achieved this way! :)
 
Last edited:

GMWolf

aka fel666
Nice! I do have a GitHub repo with tool which in its current state takes C++ code from comment blocks in GML and replaces body of the generated function with that. All you have to do is run the tool with python yyc-overwrite.py, so it's pretty convenient. But as you said, you don't want to rewrite the entire GML, but rather inject specific parts and do some modifications to what's already there, so I will be also thinking about in which direction I will be taking the repo. Can't wait to see more stuff like this and more ideas on what could be achieved this way! :)
Whoa. Ok this is incredible! Really cool stuff!
 

Tthecreator

Your Creator!
Looks great! I wonder how usable this will ever turn out to be since the cpp code was never intended to be editable. Some tricks will probably be needed, but I have faith.
Like for example we should probably want to cast between plain old datatypes and RValue.
 

kraifpatrik

(edited)
GameMaker Dev.
As I'm writing my tool in Python, I was searching if there are any handy Python libraries available and this is what I've found. https://github.com/CastXML/CastXML parses C++ into AST and outputs it as XML, which can be then used in Python using https://github.com/gccxml/pygccxml. As for the data type changes, I think a simple regex could be enough, but this may be handy for more complicated operations. Also I've decided that I will be putting code that I use in my tool into a library which could be used by anyone, so you don't have to bother with reconstructing original paths to the files, grabbing comment blocks with C++ alongside line number their occur on etc.
 
Last edited:

kraifpatrik

(edited)
GameMaker Dev.
A little update on my tool YYC Overwrite. I've implemented typed local variables. You just put these macros somewhere in your project

Code:
#macro const ;
#macro static ;
#macro bool_t var
#macro char_t var
#macro int_t var
#macro longlong_t var
#macro float_t var
#macro double_t var
and you can then define typed vars like this:

types.PNG
It's rather naive implementation using regular expressions, so it may break in some cases, but it's still some progress :D Is anyone else working on something?
 
Last edited:

kraifpatrik

(edited)
GameMaker Dev.
A little bump, since I find this important. Using gml_pragma("forceinline") anywhere in your project causes automatical overwrite of all C++ files, even if you don't do any changes in GML, disabling their modification! Seems to no longer apply!
 
Last edited:

kraifpatrik

(edited)
GameMaker Dev.
Hey everyone, I have revisited my project YYC Overwrite and I wanted to let you know that I have implemented support for multithreading, including mutexes and semaphores, so now you can run GML in separate threads natively. You can check it out at <link removed>.

threads_2.gif
Infinite loop running in a separate thread
 
Last edited:

Yal

🐧 *penguin noises*
GMC Elder
This is some pretty cool stuff! I don't have any constructive feedback yet, just figured I should pitch in the praise and get myself an automated subscriptions to new posts at the same time. The multithreading extension in particular is some really promising stuff - one of GM's ever-present performance barriers broken.
 

kraifpatrik

(edited)
GameMaker Dev.
Thanks @Yal , I hope you will like it! Regarding the multithreading, I have created a task-system library to accompany it. It allows you to spawn as many threads as you have logical CPUs (or as many as you want really) and create tasks which are then processed in the threads. Tasks can also be joined into groups in case you want to do something when some specific tasks are finished. It also takes into account VM, where the threading is not available, so the tasks are executed in the step event (as long as they fit into a configurable time frame). It can be found here <link-removed> and installed simply using catalyst require kraifpatrik/yyc-tasks. Cheers!
 
Last edited:

rwkay

GameMaker Staff
GameMaker Dev.
WARNING! The GMS2 runtime was not designed for multi threading so DO NOT call GMS functions on multiple threads as you will risk a deadlock situation.

Multi threading is very difficult and just because it works on your machine does not mean that it will not deadlock on another machine. GMS was NOT designed for multi threading.

By all means experiment, just don't expect us to support it.

Russell
 

GMWolf

aka fel666
[...] DO NOT call GMS functions on multiple threads as you will risk a deadlock situation.
This is quite interesting: it implies that some gms functions may use synchronization internally.
I was under the impression that the GML runtime was only ever ran on the one thread.

My totally, extremely speculative guess is it has to do with the draw functions:
GML runtime runs all events on a frontend thread, except fro draw functions which run async on the backend GFX thread so as to avoid needing to lock backend resources. (Hence draw events being 'optimized' for graphics functions). Draw functions running in step event etc on the frontend thread will require syncronization with the backend thread.
A sync happens between the front end and backend when entering the draw events.

total guess based on what few info is peppered here and there, but how close am i?
 

kraifpatrik

(edited)
GameMaker Dev.
By all means experiment, just don't expect us to support it.
Everything about this thread and my tool is purely experimental, and I hope I do put enough warnings everywhere so people get the right idea. Hopefully you stepping in makes it perfectly clear 😅

The GMS2 runtime was not designed for multi threading so DO NOT call GMS functions on multiple threads as you will risk a deadlock situation.
That being said, it still runs surprisingly well when used for computational stuff - do some heavy lifting using only local variables and data structures (arrays etc.) and then use the result in the main thread.

My totally, extremely speculative guess is it has to do with the draw functions:
Rendering is one of potentially many things that do not work when run in a separate thread and it is mentioned in the docs here https://kraifpatrik.com/docs/yycboost/FeatureMultithreading.html. If anyone encounters some more, please let me know, so I can append it to the list.
 
Top