GMS2 DLL with Inline Assembler

[Bořek Tuleja]

I've been learning assembly language for a few past weeks so I got an idea to try to use inline assembly within a C++ DLL for Game Maker Studio 2. I wrote a simple function that negates a value, multiplies it by 2 (I'm using bit left shift for this) and the function returns this value. As you may know it makes no sense to perform these operations on a double value so everything is converted to an int. After many hours I finaly got the right results. Here is the code:

double testFunc2() {
    double a = 5;

    /*
        1. move A value to register xmm0
        2. convert double to int, store in eax
        3. neg eax
        4. eax * 2
        5. convert int to double, store in xmm0
        6. move xmm0 value to A
    */

    asm(
        "movsd xmm0, %0;"
        "cvttsd2si eax, xmm0;"
        "neg eax;"
        "shl eax, 1;"
        "cvtsi2sd xmm0, eax;"
        "movsd %0, xmm0"
        : "=m"(a)
        :
        : "eax", "xmm0"
    );

    return a;
}

The tested value is 5, Game Maker correctly outputs -10.

My problem is that I would like to return the value directly in the ASM block. I thought the only thing I need to do is leaving the value I want to return in the XMM0 register followed by LEAVE and RET instructions but it doesn't work. The DLL starts normally, function gets called without an error but Game Maker outputs "NaN" instead. I tryed to leave the result in other XMM1 - XMM7 registers but it doesn't work as well. This is why I have to return the value as "return a;".

Does anyone know which register should I use or what to do to return the correct value?

 

[Nidoking]

This is x86 assembly, right? Is it not eax?

 

[Bořek Tuleja]

This is x86 assembly, right? Is it not eax?

Yes this is x86 assembly, Intel syntax, EAX is used when function returns INT (or that's at least what I know) but the DLL can only return DOUBLE or char pointer so this is why I chose the XMM0 register.

 

[Nidoking]

Ah, gotcha. That Google search was just too quick. This page suggests that it's the st(0) register for floating point.

 

[Bořek Tuleja]

Ah, gotcha. That Google search was just too quick. This page suggests that it's the st(0) register for floating point.

Thanks I have never heard about this register before. I discovered instructions FLD and FADD which can be used to manipulate this register and now everything works fine and I'm able to return the value to GMS.

 

[Bořek Tuleja]

If someone is interested in this problematics I can offer this 2 functions to show you how to use inline assembler to work in Game Maker.

main.h

#ifndef _MAIN_H_
#define _MAIN_H_

#include <stdint.h>

#define VOIDX extern "C" __declspec(dllexport)

VOIDX double VD_Bitwise();
VOIDX double VD_Argu(double x, double y);

#endif // _MAIN_H_

main.cpp

#include "main.h"

double VD_Bitwise() {
    int32_t value = 5;

    asm(
        "mov eax, %1 \n"
        "neg eax     \n"
        "shl eax, 2  \n"
        "mov %0, eax \n"
        "fild %1     \n"
        "leave       \n"
        "ret         \n"
        : "=m"(value)
        : "m"(value)
        : "eax"
    );
}

double VD_Argu(double x, double y) {
    asm(
        "fld qword ptr [ebp + 8]   \n"
        "fadd qword ptr [ebp + 16] \n"
        "leave                     \n"
        "ret                       \n"
        :
        :
        :
    );
}

The first function shows how to convert integer to double and return the value to GMS2. The second function even works with arguments. You have to use the FPU instruction set. I'm compiling the code with Code::Blocks, GCC and "-masm=intel" flag to use the intel syntax.