GMS 2 Custom C++ in YYC without DLLs

Discussion in 'Advanced Programming Discussion' started by kraifpatrik, May 12, 2019.

  1. kraifpatrik

    kraifpatrik Member

    Joined:
    Jun 23, 2016
    Posts:
    116
    Hello everyone,​

    recently I got into going through C++ source files generated by YYC, trying to get some understanding of how GameMaker works behind the hood. I'm putting this together and sharing with you in hopes to pontentially enable the community to make custom tools that could improve the generated C++ code, add extra functionality to it or spot incorrectly generated malfunctioning code and file bug reports. This document is still being developed and it's structure and content can change drastically. I don't have access to the full source code, I only go through files that are available on disk in plaintext form, so some information may be on the spot and some may be just a guess based on what I have seen so far and how I understand it.

    Cheers!

    Existing tools
    If you come to this thread not as a tools developer but a user, here is a list of finished/WIP tools which allow you to do modifications to the C++ files.
    • YYC Overwrite [WIP] - https://github.com/kraifpatrik/yyc-overwrite - MIT licensed tool that in its current state allows you to write comment blocks with C++ in GML and inject them into / completely overwrite the original files + define types for local variables (experimental feature).
    If you would like me to add a tool to the list, please let me know through a reply to this thread or a PM.

    build.bff
    Code:
    C:\Users\{username}\AppData\Local\GameMakerStudio2\GMS2TEMP\build.bff
    This file exists only when a project is opened. It is passed through command line arguments to Igor. It is a JSON file containing info about the project, like it's name, location on the disk, configuration etc. If you have multiple projects open, this will containg info about the project that was open/run last.

    Libraries folder
    Code:
    C:\ProgramData\GameMakerStudio2\Cache\runtimes\runtime-{version}\yyc\Win32\lib
    This path (or similar for a different target platform) contains static libraries. It is not possible to add any custom libraries to the compilation process, as clang is triggered by Igor with given set of command line parameters.

    Includes folder
    Code:
    C:\ProgramData\GameMakerStudio2\Cache\runtimes\runtime-{version}\yyc\include
    Contains header files. It is possible to add custom ones here and include them in your project through some of the default headers.

    YYC cache folder
    Code:
    C:\Users\{username}\AppData\Roaming\GameMakerStudio2\Cache\GMS2CACHE\{project}_{suffix}\{project}\default\Scripts
    This is the place where GM puts C++ files for compilation. The files are overwritten every time you make changes in corresponding GML files. EDIT: Using gml_pragma("forceinline") anywhere in your project causes automatical overwriting of all C++ files, disabling their modification! When cleaning cash with the brush icon, this folder gets erased completely. It is not possible to compile any custom file that's not generated by GM.

    Source files for object events are named gml_Object_<objectname>_<event>.gml.cpp and for scripts gml_Script_<scriptname>.gml.cpp.

    Through the file names and the project path contained in build.bff it is possible to reconstuct path to the original GML file. This could be used to inject C++ code from comment blocks, perform some optimizations etc.

    YYGML.h
    This header can be found in the includes folder. It contains includes of standard libraries, class & struct definitions (for some only forward declarations, so we can't see their implementation), declarations of external functions etc., so it's going to be the most important point of reference to us. I wouldn't recommend modifying anything that's already there, but since this header is included in every YYC source file, it is a good place to put your own includes of other libraries (if you want them to be included everywhere of course).

    Note: The code is C++, but all included libraries are standard C libraries. I tried to include C++ libs like iostream and thread, but that only resulted in compile time errors.

    Following are a few structs/classes and functions that I find important.

    class YYObjectBase
    This class (as the name suggests) seems to be the base class for GM objects. The header file contains only a forward declaration of the class, so we can't see it's implementation. It is going to be important for us, because instance pointers are casted to it when reading & writing built-in variables.

    classes CInstance and CInstanceBase
    As for CInstance, the header contains only it's forward declaration, so there is not much info for it, except that every script and object event takes a pointer to it as first two arguments (one for 'self', one for 'other'), but when actually dealing with instance variables, the pointers are always casted to CInstanceBase pointers. The definition of CInstanceBase is guarded by #if defined(...) checks without an else branch, so the real CInstanceBase class could be hidden somewhere else where we can't see it, but this one can still give us certain clues on how GM works on the inside.

    YYRValue &GetYYVarRef()
    Code:
    YYRValue &GetYYVarRef(int index)
    This is CInstanceBase's method that is used to retrieve a reference to a YYRValue structure containing value and type of a variable with given index. An index of a variable can be found in the gmlids.h file.

    Example
    Code:
    #include <YYGML.h>
    #include "gmlids.h"
    
    void gml_Object_MyObject_Create_0(CInstance* pSelf, CInstance* pOther)
    {
        // Get object variable myVar and set it to 10
        CInstanceBase* instBase = (CInstanceBase*) pSelf;
        YYRValue* myVar = &instBase->GetYYVarRef(kVARID_self_myVar);
        (*myVar) = 10;
    
        // Get the global instance
        CInstanceBase* global = (CInstanceBase*) g_pGlobal;
    
        // Get global variable myGlobal and set it to 50
        YYRValue& global_myGlobal = global->GetYYVarRef(kVARID_global_myGlobal);
        global_myGlobal = 50;
    }
    
    struct RValue
    Since GML allows us to do stuff like store values of different types into an array/list, change type of a variable throughout it's lifetime, perform operations between different variable types etc. and C++ does not, it is important to wrap GM variables into a structure that holds it's value and type. This is the base structure that handles just that. It also contains methods for retrieving the value as a different type.

    struct YYRValue
    This structure inherits from RValue and it is the structure that is actually used for storing variables. It contains overloaded operators for creating a new instance of the structure from different data types as well as performing operations between them. The definition of this structure is also guarded by checks, but since the else branch contains a definition as well (though with an empty body), I think we can take it as the real one.

    bool Variable_GetValue_Direct()
    Code:
    bool Variable_GetValue_Direct(YYObjectBase *inst, int var_ind, int array_ind, RValue *val)
    Used for getting a built-in variable.

    Arguments:
    • inst - The instance of which we want to read the variable.
    • var_ind - The the id of the variable. Will be defined in the .vars.cpp file as g_VAR_varname.
    • array_ind - Used when the built-in varriable is an array (eg. alarm), otherwise ARRAY_INDEX_NO_INDEX is passed.
    • val - Where to save the variable.

    Example:
    Code:
    #include <YYGML.h>
    #include "gmlids.h"
    
    // Defined in .vars.cpp
    extern YYVAR g_VAR_x;
    extern YYVAR g_VAR_alarm;
    
    void gml_Object_MyObject_Create_0(CInstance* pSelf, CInstance* pOther)
    {
        YYObjectBase* objBase = (YYObjectBase*) pSelf;
    
        // Get the built-in x variable and store it into local variable x
        YYRValue x;
        Variable_GetValue_Direct(objBase, g_VAR_x.val, (int) ARRAY_INDEX_NO_INDEX, &x);
    
        // Get the built-in alarm[0] variable and store it into local variable alarm0
        YYRValue alarm0;
        Variable_GetValue_Direct(objBase, g_VAR_alarm.val, 0, &alarm0);
    }
    
    bool Variable_SetValue_Direct()
    Code:
    bool Variable_SetValue_Direct(YYObjectBase *inst, int var_ind, int array_ind, RValue *val)
    Used for setting a built-in variable. Has the same arguments as Variable_GetValue_Direct, except val is now the value we want to save.

    Example:
    Code:
    #include <YYGML.h>
    #include "gmlids.h"
    
    // Defined in .vars.cpp
    extern YYVAR g_VAR_x;
    extern YYVAR g_VAR_alarm;
    
    void gml_Object_MyObject_Create_0(CInstance* pSelf, CInstance* pOther)
    {
        YYObjectBase* objBase = (YYObjectBase*) pSelf;
    
        // Get x, set it to 20 and save
        YYRValue x;
        Variable_GetValue_Direct(objBase, g_VAR_x.val, (int) ARRAY_INDEX_NO_INDEX, &x);
        x = 20; // Set x to 20
        Variable_SetValue_Direct(objBase, g_VAR_x.val, (int) ARRAY_INDEX_NO_INDEX, &x);
    
        // Get alarm[0], set it to 10 and save
        YYRValue alarm0;
        Variable_GetValue_Direct(objBase, g_VAR_alarm.val, 0, &alarm0);
        alarm0 = 10;
        Variable_SetValue_Direct(objBase, g_VAR_alarm.val, 0, &alarm0);
    }
    
    Macro FREE_RValue()
    Code:
    #define FREE_RValue(rvp)
    Frees memory used by RValue and set it's value to undefined.

    YYGML_ functions
    The header contains some declarations of external functions starting with prefix YYGML_ and followed by a function name used in GML, eg. YYGML_show_debug_message(), which is the equivalent of GML's show_debug_message(). These functions also have the same arguments as in GML. If you want to use some function, I would recommend you searching for it there first.

    YYRValue &YYGML_CallLegacyFunction()
    Code:
    YYRValue &YYGML_CallLegacyFunction(CInstance *_pSelf, CInstance *_pOther, YYRValue &_result, int _argc, int _id, YYRValue **_args)
    As the comment above the declaration states, "this function routes any unknown functions to the correct destination". So in case you can't find a declaration of some function withing those starting with YYGML_, you will have to call it through this one.

    Arguments:
    • _pSelf - Pointer to the instance calling the function.
    • _pOther - Pointer to the 'other' instance, as eg. in the collision events.
    • _result - Reference to the YYRValue structure which will contain the return value of the function.
    • _argc - Total number of arguments passed to the functions.
    • _id - The id of the function you want to execute. Can be found in .vars.cpp.
    • _args - An array of function arguments in form of pointers to YYRValues.

    Example:
    Code:
    #include <YYGML.h>
    #include "gmlids.h"
    
    // Defined in .vars.cpp
    extern YYVAR g_FUNC_window_mouse_get_x;
    extern YYVAR g_FUNC_window_mouse_get_y;
    
    void gml_Object_MyObject_Create_0(CInstance* pSelf, CInstance* pOther)
    {
        (CInstanceBase*) base = (CInstanceBase*) pSelf;
    
        // Create a structure for holding results of function calls
        YYRValue retval(0);
    
        // Call the window_mouse_get_x() and store it's result into object's variable mouseX
        YYRValue* mouseX = &base->GetYYVarRef(kVARID_self_mouseX);
        FREE_RValue(retval); // Free memory first!
        (*mouseX) = YYGML_CallLegacyFunction(pSelf, pOther, retval, 0,g_FUNC_window_mouse_get_x.val, NULL);
    
        // Call the window_mouse_get_y() and store it's result into object's variable mouseY
        YYRValue* mouseY = &base->GetYYVarRef(kVARID_self_mouseY);
        FREE_RValue(retval); // Free memory first!
        (*mouseY) = YYGML_CallLegacyFunction(pSelf, pOther, retval, 0,g_FUNC_window_mouse_get_y.val, NULL);
    }
    
    YYRValue &YYGML_CallScriptFunction()
    Code:
    YYRValue &YYGML_CallScriptFunction(CInstance *_pSelf, CInstance *_pOther, YYRValue &_result, int _argc, int _id, YYRValue **_args)
    This function is the equivalent of script_execute. The arguments are exactly the same as for YYGML_CallLegacyFunction(), except the _id argument is the script's position in the resource tree in GM, starting from 0.

    YYRValue &YYGML_CallExtensionFunction()
    Code:
    YYRValue &YYGML_CallExtensionFunction(CInstance *_pSelf, CInstance *_pOther, YYRValue &_result, int _argc, int _id, YYRValue **_args)
    As the name suggest, this function is going to be used for executing extension functions. I haven't tried it out yet though, so more info on this one later.

    gmlids.h
    This file can be found in the YYC cache folder and it is included in every source file. It contains macro definitions of translations from global and object variable names to a unique index like so:

    Code:
    #define kVARID_global_varA 0
    #define kVARID_self_varA 0
    #define kVARID_global_varB 1
    #define kVARID_self_varB 1
    
    The indices are then used as argument in the CBaseInstance's GetYYVarRef() method to get the YYRValue structure containing the variable value and type. There always are both kVARID_global_ and kVARID_self_ definitions, no matter the variable scope.

    The indices are also sorted by variable names, and since variables are held in an array (as is suggested by the available CInstanceBase definition and parameter names of getter/setter functions), it seems like every instance has allocated space for as many variables as there are in total. Because if you had one object with variables a, b, c and another object with variable z, the z would get index 3. So this object's instances would still need an array of size 4 for it's single variable. Unless there is some other translation table of course. But this is just a speculation.

    ProjectName.vars.cpp
    This file can be found in the YYC cache folder. It is overwritten every time you run your project, so there is no point in doing any modifications to it, but it does contain some handy data.

    YYVAR g_VAR_varname
    Structures with variable names and ids (accessed with .val). The id is used when getting/setting a global and object variables. If you want to use one of the variables in your source, declare it there as extern, eg:

    Code:
    #include <YYGML.h>
    #include "gmlids.h"
    
    // Required to be able to access variable myVariable
    extern YYVAR g_VAR_myVariable;
    
    void gml_Object_MyObject_Create_0(CInstance* pSelf, CInstance* pOther)
    {
        // You can now use g_VAR_myVariable here...
    }
    
    YYVAR g_FUNC_funcname
    Structures with function names and ids (accessed with .val). The id is used for calling the function with YYGML_CallLegacyFunction(). To be able to use them in a source file, they also have to be declared there as extern.
     
    Last edited: May 27, 2019
    Tthecreator, GMWolf, Dacker and 15 others like this.
  2. Samuel Venable

    Samuel Venable Time Killer

    Joined:
    Sep 13, 2016
    Posts:
    1,217
    why don't they tell us these things lol
     
  3. Fanatrick

    Fanatrick Member

    Joined:
    Jul 17, 2016
    Posts:
    58
    Inb4 locked for "confuzing new uzerz".
     
  4. lolslayer

    lolslayer Member

    Joined:
    Jun 23, 2016
    Posts:
    679
    Did that happen before?
     
    Tari likes this.
  5. YellowAfterlife

    YellowAfterlife ᴏɴʟɪɴᴇ ᴍᴜʟᴛɪᴘʟᴀʏᴇʀ Forum Staff Moderator

    Joined:
    Apr 21, 2016
    Posts:
    2,351
    I've been working on a tool that allows to automate C++ injection and code changes - most times you don't want to rewirte your GML code completely (else you'd use a C++ based engine, yeah?), just add strict types or native calls where appropriate.

    Doing this by hand is a little less exciting, as cache can be invalidated on file changes and then you'll have to wait out a compile, replace the files again, and wait out another one.
     
    Tari, zargy, Samuel Venable and 2 others like this.
  6. lolslayer

    lolslayer Member

    Joined:
    Jun 23, 2016
    Posts:
    679
    Can't wait!
     
    Tari and Samuel Venable like this.
  7. kraifpatrik

    kraifpatrik Member

    Joined:
    Jun 23, 2016
    Posts:
    116
    Nice! I do have a GitHub repo with tool which in its current state takes C++ code from comment blocks in GML and replaces body of the generated function with that. All you have to do is run the tool with python yyc-overwrite.py, so it's pretty convenient. But as you said, you don't want to rewrite the entire GML, but rather inject specific parts and do some modifications to what's already there, so I will be also thinking about in which direction I will be taking the repo. Can't wait to see more stuff like this and more ideas on what could be achieved this way! :)
     
  8. GMWolf

    GMWolf aka fel666

    Joined:
    Jun 21, 2016
    Posts:
    3,360
    Whoa. Ok this is incredible! Really cool stuff!
     
    Samuel Venable and kraifpatrik like this.
  9. Tthecreator

    Tthecreator Your Creator!

    Joined:
    Jun 20, 2016
    Posts:
    740
    Looks great! I wonder how usable this will ever turn out to be since the cpp code was never intended to be editable. Some tricks will probably be needed, but I have faith.
    Like for example we should probably want to cast between plain old datatypes and RValue.
     
    Samuel Venable likes this.
  10. kraifpatrik

    kraifpatrik Member

    Joined:
    Jun 23, 2016
    Posts:
    116
    As I'm writing my tool in Python, I was searching if there are any handy Python libraries available and this is what I've found. https://github.com/CastXML/CastXML parses C++ into AST and outputs it as XML, which can be then used in Python using https://github.com/gccxml/pygccxml. As for the data type changes, I think a simple regex could be enough, but this may be handy for more complicated operations. Also I've decided that I will be putting code that I use in my tool into a library which could be used by anyone, so you don't have to bother with reconstructing original paths to the files, grabbing comment blocks with C++ alongside line number their occur on etc.

    EDIT: YYC Overwrite now supports both C++ code overwriting and injection (using YYC_STACKTRACE_LINE calls to determine position)!
     
    Last edited: May 15, 2019
    00.Archer and GMWolf like this.
  11. kraifpatrik

    kraifpatrik Member

    Joined:
    Jun 23, 2016
    Posts:
    116
    A little update on my tool YYC Overwrite. I've implemented typed local variables. You just put these macros somewhere in your project

    Code:
    #macro const ;
    #macro static ;
    #macro bool_t var
    #macro char_t var
    #macro int_t var
    #macro longlong_t var
    #macro float_t var
    #macro double_t var
    
    and you can then define typed vars like this:

    types.PNG
    It's rather naive implementation using regular expressions, so it may break in some cases, but it's still some progress :D Is anyone else working on something?
     
    NeZvers, Lonewolff, 00.Archer and 3 others like this.
  12. kraifpatrik

    kraifpatrik Member

    Joined:
    Jun 23, 2016
    Posts:
    116
    A little bump, since I find this important. Using gml_pragma("forceinline") anywhere in your project causes automatical overwrite of all C++ files, even if you don't do any changes in GML, disabling their modification!
     

Share This Page

  1. This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
    By continuing to use this site, you are consenting to our use of cookies.
    Dismiss Notice