Glad you got this figured out.
Gonna comment on the buffer situation as
@chamaeleon hit the nail on the head. Now, I have not done this in Java as it, as far as I understood, does not have proper 'pointer' support. I have done this in C++ so I will explain the concepts as to how I did it here and maybe you can translate it to Java as it has been a
long time since I needed to use Java.
You can create a buffer in GameMaker which is effectively allocating one solid strip of memory to storing data. You can get the location this strip starts, in memory, by calling
buffer_get_address(). This is a pointer, and also where I am not sure how you would handle this type of data in Java. You can pass this value to your extension by specifying it as a 'string' on GameMaker's side. This works because a string is technically just a pointer to a strip of data where the data is a series of 'char' values. GameMaker can't tell the difference, it just knows you are passing a location in memory.
On the extension side, in C++ I would 'receive' the data as a double pointer (aka., an array of doubles). I could then modify the data inside this array however I wished and it would automatically 'update' on GameMaker's side because it was modifying the exact same location in memory. I couldn't resize the array, as that would create a new strip of data in a new location, but I could modify to my heart's content.
That's how it works, pretty simple. If there is a way to receive this pointer and manually tell Java 'this is an array of doubles' then you should be good. If there is not then I am not sure how you would go about this. As said, it has been probably close to a decade since I touched Java.