How can I fix floating point number error

Discussion in 'Programming' started by Blan, Jun 7, 2019.

  1. Blan

    Blan Member

    Joined:
    Jun 26, 2016
    Posts:
    25
    Hi ,
    Basically what I am looking to achieve is to subtract one variable from another and have a result which is accurate to 2 decimal places. Most of the time it will be the same value to be deducted
    I have tried floor and multiplying by 100 then divide by 100 and also ceiling too, but I can still get + or - 0.01 errors.
    I do not know in advance which calculations will need a floor and which require a ceiling as they are unknown variable values that are being calculated.
    I do NOT have a problem with 0.5 or any multiplications off.
    I have removed multiplying by 100 then divide by 100 for this controlled example below
    Controlled example (fixed values not variable)
    First 99.99 - 0.9 = 99.08999999999999
    Second - 0.9 = 98.18999999999998
    Third -0.9 = 97.28999999999998
    Fourth -0.9 = 96.38999999999997

    If I use the method of "multiplying by 100 then divide by 100 ", I have values which can be + or - 0.01 after 3rd or more calculations.

    any ideas much appreciated.
    Thanks
     
  2. FrostyCat

    FrostyCat Member

    Joined:
    Jun 26, 2016
    Posts:
    4,033
    Blan and Rob like this.
  3. Blan

    Blan Member

    Joined:
    Jun 26, 2016
    Posts:
    25
    Thanks FrostyCat for your links, I now have a better understanding of floating point numbers.
    Also after having watched some youtube videos I can now convert to binary using only a pen and paper.LOL
    I did manage to fix my problem, what I did was to draw a string instead of a variable.
    After doing some maths on my variable I needed to draw the resulting value; I was using something like this:-
    draw_text(400,500,Low)
    and to fix the problem
    draw_text(400,500,string(Low))
    No more display errors.
     

Share This Page

  1. This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
    By continuing to use this site, you are consenting to our use of cookies.
    Dismiss Notice