Asset - Extension SAE - Smiechu Audio Engine

[Smiechu]

Hi!

As I mentioned couple of times already, I'm working on a custom audio engine for GMS2 utilizing only pure GML functionality. This means it should be 100% cross-platform. But on the other hand quite resource-demanding (already proved) and slow to develop.

I would like to slowly share with you my progress, concept and first thoughts. Additionally I would like to get some feedback, what users would actually need, and how they would like to use it. On the other hand this is not a "white board", I have already made some design decisions, and these are not a thing to change.

Below please find attached the first "rough" version of a document describing the systems functionality and structure:
http://bit.ly/SAE_doc_01

I'm simply recreating a well known structure of a classic analogue mixer setup - widely known from DAW software. It's a workflow structure which evolved for last 60 years of music production, there is really no sense reinventing the wheel. This means we have a master channel with basic controls and fx bin/container. Source for master channel are outputs of mixer channels, with basic controls, fx bin, sends, and an input source.

User can choose from couple of audio sources - ie. generator/synth, but the most important source is the standard audio file playback.

GMS allows only to load standard wave files to buffers, it's not possible to use build in "sound resource". Here comes the first bump. Wave files are quite large, and they need time to load into memory - a 3 minute music track, needs couple seconds to load approx. 10-15 or even more. The process is faster with YYC. Solved thanks to @hippyman !!!

Second bump, the playback of multiple sounds is quite resource demanding - playback of 8 sounds at a time is causing drops using VM by 60FPS game speed on my i7 2600 CPU, and there is nothing more - only the engine. Buffer read/write functions are taking most of the CPU time. Fortunately using YYC things are looking much, much better. Using higher game speeds also helps.

So main list of functionality will be:
- playing audio loaded from included wave files,
- incorporating a classic analogue mixer workflow with sends/fx/etc,
- basic fx - eq, filters, simple reverb, delay, echo, chorus, compression, limiting etc,
- basic audio analysis in order to trigger events in game, or control objects,
- functions for control and automation of mixer and fx parameters,
- debug GUI with mixer view, where user could manually set the parameters in order of creating desired sound scenes.

Optional functionality:
- generators, basic synths,
- record input source,
- midi playback, mod playback,
- user programmed fx,
- advanced beat detection,

I'm currently cleaning up my project, the core base is up and running, with first basic fx and most of the controls. Next week I'll be programming a nice GUI, in order to present you a demo video, and maybe some practical use examples.

Please stay tuned, and give ma a feedback if you feel something is missing here, or you have some suggestions.

 

[Misty]

Needs 3D and multiple listener location support, for 2 player games. Also, will this be compatible on all platforms? Also, a sound muffle effect would be nice, to use for sounds behind walls or sounds in the distance.

 

[Smiechu]

Needs 3D and multiple listener location support, for 2 player games. Also, will this be compatible on all platforms? Also, a sound muffle effect would be nice, to use for sounds behind walls or sounds in the distance.

3D is not on my roadmap... I assume simple stereo is all man need for type of games developed on GMS...

As already stated, till now it's pure GML so it should work on all platforms...
Unfortunately I cannot tell what the performance will be...

What is sound muffle?? You mean High Pass Filter - yes it's already implemented.

 

[hippyman]

You can load wav files WAY faster than what you're saying. Check out my external audio extension. It covers the wave importing thing and it's pure GML as well.

I just now tested it to be 100% certain and it was as fast as I remember. I just downloaded this video as an MP3 and converted it to a WAV file in Audacity.

It went from being an 86.5 MB mp3 file to a 636 MB wav file. This 1 hour wav file imports in 0.12 of a second on my i7 6700K (may as well be a normal 6700 since I don't overclock though). I would share the project but it's a lot of bandwidth.

If you really want proof of this I would recommend doing what I did and downloading the external audio project from my github repo here. Then find a long video on Youtube and download it with any of the several Youtube-to-mp3 sites available. Convert that with Audacity to a WAV file and drag it into the included files of the external audio demo project. Then open the "HowToExternAudio" object in the project and put the wav's filename in the ext_audio_load call.

You can even check how long the load time was by adding these lines of code before and after the load script is called.

var startTime = current_time;
externalAudio = ext_audio_load("filenamehere.wav");
var totalTime = current_time - startTime;
show_message("1 hour audio clip took " + string(totalTime / 1000) + " seconds to import");

Then when you start the game it will load the file super fast and tell you how long it took to load. Then it will begin playing when you close the message.



I'm definitely interested in this. It sounds like you're trying to make something along the lines of FMOD/Wwise but with pure GML. It's a seriously ambitious project. Are you planning on keeping this closed source and selling it or are you going open source? I'd be down to contribute.

 

[Smiechu]

You can load wav files WAY faster than what you're saying. Check out my external audio extension. It covers the wave importing thing and it's pure GML as well.

I just now tested it to be 100% certain and it was as fast as I remember. I just downloaded this video as an MP3 and converted it to a WAV file in Audacity.

It went from being an 86.5 MB mp3 file to a 636 MB wav file. This 1 hour wav file imports in 0.12 of a second on my i7 6700K (may as well be a normal 6700 since I don't overclock though). I would share the project but it's a lot of bandwidth.

If you really want proof of this I would recommend doing what I did and downloading the external audio project from my github repo here. Then find a long video on Youtube and download it with any of the several Youtube-to-mp3 sites available. Convert that with Audacity to a WAV file and drag it into the included files of the external audio demo project. Then open the "HowToExternAudio" object in the project and put the wav's filename in the ext_audio_load call.

You can even check how long the load time was by adding these lines of code before and after the load script is called.

var startTime = current_time;
externalAudio = ext_audio_load("filenamehere.wav");
var totalTime = current_time - startTime;
show_message("1 hour audio clip took " + string(totalTime / 1000) + " seconds to import");

Then when you start the game it will load the file super fast and tell you how long it took to load. Then it will begin playing when you close the message.



I'm definitely interested in this. It sounds like you're trying to make something along the lines of FMOD/Wwise but with pure GML. It's a seriously ambitious project. Are you planning on keeping this closed source and selling it or are you going open source? I'd be down to contribute.

Thanks! I'll definitely check this out!
I was on a business trip for last two days, so don't it have possibility to do it right away.

 

[hippyman]

No worries, just thought I'd share what I've learned on the subject. Hope it helps some!

 

[Smiechu]

No worries, just thought I'd share what I've learned on the subject. Hope it helps some!

I've checked your wave loading code and I understand now where you've gained the "speed", or rather where I've made my code terribly slow!

1. You simply load the buffer from wave file using buffer_load() where I've used file_bin_open(),
2. Than you just "copy" the raw data from this buffer, where I have manually decoded the 2's-complement signed 8-bit integer samples into 16-bit signed samples, as that seemed "logical" from the info in GMS manual.

That's really something "new" to know for me, and now the sounds are loading incredibly fast! GREAT THANKS!

 

[hippyman]

I had a feeling you were using the file_bin_* functions I just didn't want to make any assumptions. Yep reading the binary of files is way faster when using the buffer functions instead. I really have no idea why they're so much quicker.

EDIT: Oh and thanks for the credit! Wasn't necessary but I appreciate it

 

[Smiechu]

I really have no idea why they're so much quicker.

file_bin_* functions are forcing the user to iterate throug the whole file, this is where it gets slow.

 

[hippyman]

Aaah that makes sense. Since you're able to just load the entire file in one go with buffer_load.

 

[Mick]

Second bump, the playback of multiple sounds is quite resource demanding - playback of 8 sounds at a time is causing drops using VM by 60FPS game speed on my i7 2600 CPU, and there is nothing more - only the engine. Buffer read/write functions are taking most of the CPU time. Fortunately using YYC things are looking much, much better. Using higher game speeds also helps.

If the 100% GML method fails (results in being too resource demanding), the next best option would be to do most of the work in a shared library but still use the audio engine in GMS. My chiptune extensions work like that, pointers for buffers (audio file and audio queue buffers) are sent to the shared library (C++) where the buffers are directly manipulated. This should work for most exports but it requires more maintenance of course. The same code base can be used for all platforms though. I have successfully made this work for Windows, Linux, OS X and Android. It should also be possible for IOS but I don't have the tools needed so I haven't been able to try it out.

But hopefully all that is not needed, just wanted to let you know there are other options.

 

[Smiechu]

If the 100% GML method fails (results in being too resource demanding), the next best option would be to do most of the work in a shared library but still use the audio engine in GMS. My chiptune extensions work like that, pointers for buffers (audio file and audio queue buffers) are sent to the shared library (C++) where the buffers are directly manipulated. This should work for most exports but it requires more maintenance of course. The same code base can be used for all platforms though. I have successfully made this work for Windows, Linux, OS X and Android. It should also be possible for IOS but I don't have the tools needed so I haven't been able to try it out.

But hopefully all that is not needed, just wanted to let you know there are other options.

To be honest... I'm afraid that this is the only way to handle a more advanced functionality of the audio processing. Most of my time is now wasted just to find the optimal solutions for the basic stuff.
I'm not giving up, but very probably the scope of pure GML functionality will have to be cut. We'll see, I still have some ideas to test...

 

[Andrey]

To be honest... I'm afraid that this is the only way to handle a more advanced functionality of the audio processing. Most of my time is now wasted just to find the optimal solutions for the basic stuff.
I'm not giving up, but very probably the scope of pure GML functionality will have to be cut. We'll see, I still have some ideas to test...

Yes, and in the end it will be possible to come to a simple idea — in GMS2 need the integration of ready-made professional products, and not to suffer with the invention of bicycles.

 

[Smiechu]

Yes, and in the end it will be possible to come to a simple idea — in GMS2 need the integration of ready-made professional products, and not to suffer with the invention of bicycles.

Hey! Just let me have some fun??

Yeah it would be nice to have a native support for FMOD in GMS. But you can still use it, nobody holds you back... GMS allows you to use external libraries and FMOD comes with C++ and C user API. I'm quite astonished that nobody made an extension yet...
But, you must be aware that despite the native support of other Game Engines, you as a developer are still obligated to follow the FMOD licensing rules - for Indie users, it's 1 free published project every year. Everything above is 2000$ for every next project. I didn't dig so much deep into the rules, you can check them for yourself.

 

[Andrey]

I'm quite astonished that nobody made an extension yet...

Of course, because the knowledge in this area is not enough. If such there is, apparently, are played with UE4 and other.
To integrate such engines will need to be configured for all platforms (mobile, console) and test.

 

[Smiechu]

is it like audacity?

No...