1. Hey! Guest! The 34th GMC Jam will take place between August 22nd, 12:00 UTC (Thursday noon) and August 26th, 12:00 UTC (Monday noon). Why not join in! Click here to find out more!
    Dismiss Notice

GML Generating sounds in real time

Discussion in 'Programming' started by GMWolf, Feb 18, 2017.

  1. GMWolf

    GMWolf aka fel666

    Jun 21, 2016
    Hello everyone.

    I have been trying to generate sound waves, and play them back in near-real time using audio buffers.

    I came up with two techniques:
    The first was to have a number of small buffers (10ms), and queue them up onto a audio queue.
    whenever an audio async event was triggered, i would get the buffer ID, generate more data, and push it back onto the queue.

    The problem with this technique was that i needed either many buffers, or larger buffers in order for the queue to keep playing. With smaller buffers the queue would simply stop after each buffered had played once.
    The problem with larger buffers was that it ended up having too much lag. im looking for sub 20ms, optimally.

    Technique 2 was to have two longer buffers, one "active" and one "passive". The two buffers would be queud up, and every async event, i would queue them back onto the queue.
    In the step event, I would generate a bit more audio data onto the "active" buffer, in order to be just ahead of the current track position (adjusted to take into account how many bufferes already played).
    When generating samples, if i ever exceeded the length of the active buffer, i would switch the two buffers around, and start generating on the other buffer.

    This technique seemed to work, but was quite glitchy, It seemed like rather than streaming from the buffers, the queue was taking chunks from the buffer. Those chunks seemed larger than the ammount I would buffer ahead by. This resulted in some of the buffers to either not play or play some leftover garbage data.

    From these two tests, It seems to me like the qudio queues would copy a chunk of data from the queue, and start playing it. The size of chunk taken is too long for my usage.

    Do any of you know of a way to get near-real time audio generation to work?
    If not i guess Ill be building a DLL or something. (but i'd rather not)

    [edit] Not sure if important, but i was using s16 formated buffers. Would that make a difference when it comes to minimum buffer sizes?
    Last edited: Feb 19, 2017
  2. TheStolenBattenberg

    TheStolenBattenberg Member

    Jul 25, 2016
    From what you'd said, I'm taking it you are talking about 'Square', 'Triangle' and 'Sine' waves.

    What I'd do is keep a list and a pre-generated buffer with a 10ms sample of each type of wave you need, then keep a ds_qeaue, two of them. One will have your audio buffer for which wave type you want, one will have the pitch. It will then reference both while it plays them, to play the wave at the correct pitch.

    But that's how I'd do it.

    Your second method should work well though, were you using 'buffer_copy' or manually transferring the buffer with read/write? buffer_copy is a memcpy call in c++, so it's VERY fast, and what I use for everything I can. When it comes to your problem with leftover data or not playing buffers, these are the same problem, just different sides of it. Your transferal method probably has the wrong offsets/length, so you are writing to the wrong region of the buffer you're streaming from... Try clearing out the buffer for testing reasons, before you write new data.

    A good way to test this is to write the buffer out to a file, 50 or so times for each time new data is placed into the buffer. It will tell you exactly what is going on with your engine, its how I've found problems.

    Keep in mind all this comes from theory, not practice. My skill is more with making custom audio-formats, converting other formats etc. I even made an ADPCM decoder for PS1/PS2 games.

    A note to you, try and generate your sample waves with a C note, this is 440hz if memory serves me correctly. Most internet documentation is based around this, so you can get a lot of help with it.

    s16 is merely better quality audio, it's two bytes of data rather than 1, and can be non positive. Did you generate your data this way?
  3. GMWolf

    GMWolf aka fel666

    Jun 21, 2016
    Any kind of wave, actually.

    The idea is to have real time audio generation. Pre-generated buffers won't do.

    I'm writing directly to the buffer that is on the queue. Not much sense in writing to one buffer and then copy...

    I'm fairly certain I'm writing to the correct part of the buffer: I'm always using buffer write, and when I reach the end, I seek to the start of the other buffer.
    Since my buffers are continually placed back onto the queue, this should work correctly.
    But unlike the first technique, even when buffering a whole second in advance, it doesn't seem to work seamlessly. You will sometimes only get the first section of the buffer that was generated before the buffer was pushed to the queue (that is with 5 second long buffers).

    Regardless, my first method should work fine with any size buffers, but it seems like I need around 100ms of buffers ahead of the current play position. When I try around 20 or 50ms of buffers, it simply stops playing, as if I didn't keep queuing data up.
  4. TheStolenBattenberg

    TheStolenBattenberg Member

    Jul 25, 2016
    Well then the only thing I can suggest is to write out your buffer as it's playing to debug. Have a look if you have random data that shouldn't be there.

    With your first technique, having loads of buffers shouldn't be a problem. They're just locations in memory (plus a small amount of overhead to store the seek, size etc), so if they're not that big it won't matter. If keeping track of them is a problem, use a ds_list or something. Heck I use buffer's as a way to have structs in programs I make, so there ends up being more than 200 of them when I'm working with a lot of files.
  5. GMWolf

    GMWolf aka fel666

    Jun 21, 2016
    When I generate the data to a single buffer and then push it to the queue the data is fine. Its when I generate the data with not enough ahead time on a buffer that is on the queue that I get intermittent behavior.
    It is a problem: I need it to be near-real time. That means 20ms or so ahead of the current play position.
    If I use more, or longer buffers, then It is no longer near real time.
    It doesn't matter if I use many small buffers, or fewer larger buffers, they always seem to need to total up to ~100ms. Unacceptable for my use.

    Btw, I simply let the queue keep track of them.
    Whenever I get an async event, I simply overwrote the returned buffer with new data, and push it back onto the queue. Works like a charm with 100+ms buffers zone, simply stops playing with anything smaller.
  6. TheStolenBattenberg

    TheStolenBattenberg Member

    Jul 25, 2016
    This may just be a problem with audio latency then, as this is starting to sound exactly like that, rather than your code.
  7. renex

    renex Member

    Jun 23, 2016
  8. GMWolf

    GMWolf aka fel666

    Jun 21, 2016
    I'm not so sure, since just playing the audio buffers can be done in real time if I don't use a queue. (But then I get horrible gaps between the sounds).
    Thanks, Ill check it out!

    Ok, this seems to work very well!
    Im suprised playing the sound every step like that isnt giving you breaks in the sound. when i tried something simmilar, i got breaks between each play of the sound.
    ill try to implement something simmilar, hopefully itl work :) thanks again, to both of you, this was very helpful :)
    Last edited: Feb 19, 2017
    renex likes this.
  9. renex

    renex Member

    Jun 23, 2016
    I remember implementing a millisecond counter that would adjust the length of the sound based on delta time. It might be what's missing for your implementation. It's one of the numbers on the debug text.
  10. ThunkGames

    ThunkGames Guest

    Something I had commissioned a while back (I'm free to distribute it too):

    var params = argument0;
    var rate = params[?"sampling"];
    //General Information
    wave_type = params[?"wave_type"];
    var punch = params[?"punch"];
    //Time/Duration Variables
    var repeats = params[?"repeats"];
    var attacktime = params[?"attack_time"];; //No sound to full
    var sustaintime = params[?"sustain_time"];; //Full sound all the way
    var decaytime = params[?"decay_time"];; //Fading of the sound
    var duration = attacktime+sustaintime+decaytime;
    //Frequency Variables
    var freq = params[?"start_frequency"];; //The base sound frequency
    var fslide = params[?"frequency_slide"];; //Altering the frequency overtime
    var dslide = params[?"delta_slide"];; //Altering the altertion of frequency
    var freqcut = params[?"frequency_cutoff"];
    //Vibrato Variables
    var vibrato = params[?"vibrato_depth"];;
    var vibratospeed = params[?"vibrato_speed"];;
    //Harmonics variables
    var harmonics = params[?"harmonics"];;
    //Frequency Filters
    var low_limiter = params[?"low_limiter"];;
    var high_limiter = params[?"high_limiter"];;
    var ll_strength = params[?"low_limiter_strength"];
    var ll_sweep = params[?"low_limiter_sweep"];
    var hl_strength = params[?"high_limiter_strength"];
    var hl_sweep = params[?"low_limiter_sweep"];
    //Pitch control system
    var pitch = 1;
    var p1 = params[?"pitch1"];
    var p2 = params[?"pitch2"];
    var p1_time = params[?"pitch1_time"];
    var p2_time = params[?"pitch2_time"];
    var pitch_repeats = params[?"pitch_repeat"];
    var pitch_time = duration/(pitch_repeats);
    //Bit crushing algorithms (quatization)
    var bit_crush = params[?"bit_crushing"];
    var bc_sweep = params[?"bit_crushing_sweep"];;
    var gain = params[?"gain"]; //The volume of the sound
    //Create the sound buffer
    var sbuffer = buffer_create(rate*duration*4*repeats,buffer_fixed,4);
    //Fill the buffer
        for(t = 0; t<duration*rate;t++){
            var truet = t/rate; //The true time used for calculations
            var value = 0; //The value that will be written
            //Calculate the correct gain of the sound
            var finalgain = gain; //The final volume of the sound
            //Tamper with the frequency
            freq = clamp(freq,50,20000); //Clamp the frequency to audible amounts
            //Apply the filters
            low_limiter += ll_sweep/rate;
            high_limiter += hl_sweep/rate;
            var flt = -1;
            if(freq < low_limiter){
                flt = low_limiter;
                var distance = abs(flt-freq);
                finalgain = max(0,finalgain-distance*ll_strength);
            if(freq > high_limiter){
                flt = high_limiter;
                var distance = abs(flt-freq);
                finalgain = max(0,finalgain-distance*hl_strength);
            //Vibrato additions
            finalgain *= (1+vibrato*SoundGetFrequency(0,vibratospeed/2,t,rate));
            //Cuttof when time ends/starts
            if(truet < attacktime){
                finalgain *= sqr(truet/attacktime);
            }else if(truet > attacktime+sustaintime){
                finalgain *= sqr(1-(truet-attacktime-sustaintime)/decaytime);
            //Cuttof when frequency cutter does not allow
            if(freq < freqcut)
                finalgain = 0;
            //Pitch changer
            pitch = 1;
            var tempt = truet mod pitch_time;
            tempt /= pitch_time;
            if(truet mod pitch_time > duration*p1_time/pitch_repeats)
                pitch*= p1;
            if(truet mod pitch_time > duration*p2_time/pitch_repeats)
                pitch*= p2;
            var finalfreq = freq*pitch;
            var value;
            if(harmonics == 0 || harmonics == 1)
                value += finalgain*(SoundGetFrequency(wave_type,finalfreq,t,rate)*(1+random(punch/100)));
                var ff = finalfreq;
                for(i = 1; i < harmonics; i++){
                    value += finalgain*(SoundGetFrequency(wave_type,ff,t,rate)*(1+random(punch/100)));
                    //Check and stop if frequencies are geeting too high
                    var off = clamp(ff,0,20000);
                    if(ff != off){
            //Bit crushing the value in the end
            bit_crush += bc_sweep/rate;
            var s = sign(value);
            value /= bit_crush;
            value = round(abs(value));
            value *= s*bit_crush;
            /*var valper = abs(value)/finalgain;
            valper *= finalgain;
            valper /= bit_crush;
            valper = floor(valper);
            valper *= bit_crush;
            valper /= 1000;
            value = sign(value)*valper*finalgain*/
    return audio_create_buffer_sound(sbuffer,buffer_s16,rate,0,rate*duration*4*repeats,1);

    Helper Script:
    var type = argument0;
    var freq = argument1;
    var t = argument2;
    var r = argument3;
    var period = 1/freq;
    var truet = t/r;
    var tt = truet/period;
        case 0:
            //Sine wave
            return cos(2*pi*freq*t/r);
        case 1:
            //Rect Wave
            return sign(cos(2*pi*freq*t/r));
        case 2:
            if(tt < .25){
                return (tt/.25);
            }else if(tt < .75){
                return 1-2*(tt-.25)/.5;
                return -1+(tt-.75)/.25;
        case 3:
            return -1+2*tt;
        case 4:
            return ((2*get_frequency(0,freq,t,r))+sign(get_frequency(1,freq,t,r)))/3;
        case 5:
            //Tan (may rethink this)
            return tan(2*pi*freq*t/r);
        case 6:
            return cos(2*pi*freq*t/r) + .1*cos(20*2*pi*freq*t/r);
        case 7:
            //White Noise
            return random(abs(cos(2*pi*freq*t/r)));
        case 8:
            //Pink Noise
            return random(abs(cos(pi*freq*t/r)))*sign(cos(pi*freq*t/r))*.9;

    params = json_decode('{

    Introduction (copy and pasted):

    The 2 scripts: create_sound and get_frequency,
    are needed both to run correctly.

    They simulate most of the bxfr synthesizer functinality,
    However, due to GMS limitations, the effects might be quite altered,
    Its important to note that this is not a direct port,
    but a sohisticated simulation of the system, which means,
    the implementation of the features may, and will most likely,
    difer from the original.

    Parameters (copied and pasted):

    wave_type: 0 to 8
    Determines what wave type will be used for the effect,
    Different wave types alter the sound a lot.

    0 sine wave
    1 rect wave
    2 trig wave
    3 saw wave
    4 breaker wave
    5 tan wave
    6 whistle
    7 white noise
    8 pink noise

    gain: positive

    It affects the volume of the sound

    attack_time: positive
    The time it takes for the track to reach full volume from the start

    sustain_time: positive
    The time where the sound plays without volume change

    decay_time: positive
    The time it takes for the sound to reach mute from full volume

    repeats: positive
    How many times will the sound repeat

    sampling: positive
    Determines the compression of the sound, more is better. A resonable value is 40000.

    start_frequency: positive
    Determines the frequency of the sound, which affects the pitch.E.G the note A has a frequency of 440

    frequency_cutoff: positive
    Limits the possible frequency, If it falls below the frequency_cutoff it will be muted.

    frequency_slide: positive or negative
    Alters the frequency per second (it adds to it or subtracts)

    delta_slide: positive or negative
    Alters the frequency_slide parameter per second, to give more flexibility

    low_limiter: positive
    high_limiter: positive
    Those set a limit on the possible frequencies, e.g low_limiter:1000 high_limiter:9000 cuts frequencies outside (1000,9000)

    low_limiter_strength: positive
    high_limiter_strength: positive
    Those determine how harshly will the frequency limiters work respectively,

    low_limiter_sweep: positive or negative
    high_limiter_sweep: positive or negative
    Those 2 alter low_limiter and high_limiter values respectively, every second.
    This gives more flexibility to the frequency limiting

    pitch1: positive
    pitch2: positive
    pitch1_time: zero to one
    pitch2_time: zero to one
    pitch_repeat: positive
    These variables allow for calculated pitch swifts within the sound
    the pitch variables determine the itensity of the effect e.g pitch1:2 doubles the pitch
    The pitch_time variables determine when the pitch change will happen
    e.g pitch1_time:0.5 the pitch alteration will happen at the half time of the sound effect
    Finally pitch_repeat determines how many times the above user created patern will repeat,
    throughout the sound effect.

    Now for the effects

    Adds artifacts to the sound to resemble an old recording, more means greater effect

    Interlaces the sound with multiples of its frequency, thus enriching it

    Makes the sound have a vibrating effect by affecting the gain, and those 2 parameters,
    affect the speed and the itensity of the effect.
    e.g vibrato_depth:1 vibrato_speed:1 makes a vibrating volume that doubles and mutes,
    like a wave once a second

    Lowers the "resolution" or sampling rate to match a lower capability system,
    which creates interesting artifacts, Higher value increases the effect

    bit_crushing_sweep:positive or negative
    Alters the bit_crushing parameter (adds or subtracts to it every second),
    giving the bit_crushing effect greater flexibility.

    Here is the list of the parameters range (also copied and pasted from an email):
    wave_type: 0-8 (pretty obvious, each number corresponds to a wave,
    I have sent you details before)

    It's the volume of the sound.
    This requires some testing to figure it's range,
    however I found that 8000 is good enough for the max volume.
    So I recommend 0 to 8000 as the range.

    attack_time, sustain_time,decay_time:
    Each of those is measured in seconds, If you add them you get the
    full effect duration. Limiting each to five would be a good option.

    repeats: Will repeat the signal as many times as you set it,
    If the sound lasts 10 seconds, a repeat of 3 will make it 30 seconds.
    So if you allow a high value it may put a big load on the cpu.
    I recommend not giving an option to change that, or keep it
    below 3 if you don't want large sound effects.

    sampling: In exact, it measures how compressed the sound will be.
    A typical value used by many is 48.000 and will give an ok sound
    setting it too hihg will give a cleaner sound, but may affect preformance,
    especially in long sounds 5000-100000 or less will be a good range.

    start_frequency, frequency_cutoff,low_limiter,high_limiter:
    All those parameters have to do with frequency.
    The human ear can distinguish 20-20000 Hertz of frequency,
    So limiting it between these values will do.

    low_limiter_strength, high_limiter_strength:
    Those 2 sound be set with a minimum of 0 and a maximum
    same as the gain.

    pitch1, pitch2:
    They alter the pitch of the sound.
    A range of 0-3 for each of them Is fine.
    Other pitch arguments are fully described, and giver range,
    on the .txt file that I sent.

    This effect is hardcoded to accept a range of 0 to 10.
    Anything outside that will cause strange effects.
    Putting it near 10 will almost destroy the sound with noise,
    so it may not be a bad idea to set the range as 0 to 5.

    Accepts integer positive values. 0 and one will do nothing.
    2 or more will add frequency multiples to your sound.
    I recommend limiting it to 10 or less.

    vibrato_depth: Range from 0 to 1, it's hardcoded to work well
    on those values, so make this the range

    vibrato_speed: Determines the vibrato speed effect.
    20 means 20 vibrations per second.
    In fact 0 to 20 is a good range for it, but you can make it longer
    if you see fit.

    The range should be the same as the gain, more than it, it does nothing.

    All other arguments accept previous arguments per second,
    so I cannot recommend a range for these,
    however they take positive aswell as negative values,
    so 0 should be the default and should be in the middle.
    for example -10 to 10, and not -20 to 40.
  11. GMWolf

    GMWolf aka fel666

    Jun 21, 2016
    Yes, THe problem i was having when trying something simmilar was to simply play the sound when the async event triggered. But Async isnt actually async. Its very syncronous indeed! It will only fire during a step. So the sound would stop playing, but only fire the event a little after.
    The way you implemented it ensure the sound will be as long as the previous step, so if the frame rate is steady, it works quite well. Unfortunately if there is a lag spike, there is an audible artifact. This is still the best solution yet though, so i think ill be using it.
    Wow thanks! This will be super helpful!
  12. Mike

    Mike nobody important GMC Elder

    Apr 12, 2016
    I've done this quite a bit in GMS2, my C64 emulator, my Spectrum emulator and my MOD player.

    The emulators had to try and "sync" with the video rendering so was a little odd. The framerate was basically 9999 with the audio event setting a flag to say when to fill int the buffer and render the next frame. This worked but is yucky

    For "pure" music rendering, you simply sit on the audio event callback and fill buffers as they come in. You have to try and stike a balance between big buffers that will last longer but give you loads of lag, and small ones that mean you have to keep filling them in but are faster to respond to changes.

    The MOD player does ALL it's work in the audio event, sample generation, channel mixing - everything. Occasionally you may get a couple of events one game frame, and sometimes none. Because the audio stuff is on a timer it will be out of sync with video (/game...sprites etc) rendering, so the more you can put in the audio event the better. Also make sure to check for having to requeue the buffer, as when you start dragging windows around it can stop. This isn't because window dragging stops it, just simply because the buffers "run out" as nothing has refilled them.

    The manual on audio buffers is pretty perfect for streaming stuff, copy and paste and it should be streaming an empty buffer - then you just fill it in.

    You can buy the C64 and MOD play stuff on the Marketplace ;)
    Blackened and renex like this.
  13. GMWolf

    GMWolf aka fel666

    Jun 21, 2016
    THis was actually the first way i tried doing it. I queue up a couple (5) very small (10ms) buffers, and as the async event triggered, i would take the buffer, dump more data onto them and queue them back up, all in the async event.
    It didnt work. I had to increase the buffer size, or use more buffers, before it would work correctly.
    This meant too much lag. This isnt just background music, As i want the user to interact with the sound, a bit like if it was a keyboard. So it needs to be very responsive.

    I actually had a question about that: Are async events truely async, as in, they will trigger the moment the audio ends? or are they queued up and tirggered the next step?
    From one of my tests it seemed like it was doing the later.

    I still cant seem to get below a certain "buffer ahead" time when using queues. Is there a limit to how little data ahead of the play position i can put?

    I may go check out the MOD tracker :)
  14. HammerOn

    HammerOn Member

    Jun 22, 2016
    If there is a limited number of possible sounds, you should use a cache. If there isn't, you should design your project to be like this.
    Then copy from the cache to the big buffer or audio queue to avoid lag.
    But I'm almost sure your main problem is the audio async event.

    The engine may spawn a thread to do the processing (ex. load a file) but the async events are queued callbacks. They aren't called when the thread finishes the work but in the main loop of the game like any other event.
    You generate 10ms but a frame takes 16ms (if the fps is 60) plus fluctuations. There will be synchronization issues for sure.
    You can try a high framerate to workaround this but... you know... GM is a frame based engine so this is solving a problem with another problem. More workarounds!
  15. Lycanphoenix

    Lycanphoenix Guest

    Somebody said MOD player extension? I have to check that out!
    *Is a MOD-Tracker enthusiast*
  16. GMWolf

    GMWolf aka fel666

    Jun 21, 2016
    That would defeat the entire point of this project.
    Yes, I'm starting to think GM may not be entirely suitable.
    I May have to switch to using a DLL for this.
  17. Mike

    Mike nobody important GMC Elder

    Apr 12, 2016
    10ms is less than a frame so you'll need to make sure you have at least a frame - probably 2 - worth of buffer. This shouldn't give too much lag. Events are not threaded, they don't come in "any old time", they are queued for the start of the next game frame. This means if they are <1frame you won't be able to fill it until the next frame - and it'll probably stop playing.

    Also... Audio drivers a little funny with creating buffers of small "odd" sizes, so make sure the buffers actually get queued. As I said... 1 or 2 frames should do it, but make sure you queue up 3 or 4 of them so there is some overlap to game frames and giving you time to build and queue the next one.

    GML based MOD tracker: https://marketplace.yoyogames.com/assets/4951/mod-tracker
  18. GMWolf

    GMWolf aka fel666

    Jun 21, 2016
    Thanks. I'll play around with timing a little more. Try to get 4 frames at 60fps or something.

    I will check out the MOD tracker :)

    Thank you all for your help!

Share This Page

  1. This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
    By continuing to use this site, you are consenting to our use of cookies.
    Dismiss Notice