updated 64bit Jessie, using Galculator.
I type .30 x 100 =
I get 30
I type -30 =
I get -1.11022302463e-15
What?????
Scheduled Maintenance: We are aware of an issue with Google, AOL, and Yahoo services as email providers which are blocking new registrations. We are trying to fix the issue and we have several internal and external support tickets in process to resolve the issue. Please see: viewtopic.php?t=158230
Galculator math errors?
Re: Galculator math errors?
The decimal point causes the calculation to be performed in floating-point math. The result is displayed as a fixed-point approximation of the floating-point result (which is correct).No_windows wrote:updated 64bit Jessie, using Galculator.
I type .30 x 100 =
I get 30
What I'm guessing happened here is the calculation is still being done in floating-point, hence the "error". What might have been a better (I'm guessing here, no testing of the idea has been done) coding is to take the display value (which has only an implied decimal point) and proceed using fixed point math.No_windows wrote: I type -30 =
I get -1.11022302463e-15
Re: Galculator math errors?
This caught my attention because I use galculator fairly regularly. Maybe I need to find a more accurate calculator. Something I actually need to worry about, as opposed to all the recent nonsense FUD in the news and forums about CPU "flaws". Oh but wait, "E-15" ....
Re: Galculator math errors?
Galculator works perfectly in RPN mode. At least, that is the mode I always use. (It is also easier for me to use in RPN mode.)
.3 [enter]
100 [*]
Display shows: 30
30 [-]
Display shows: 0
.3 [enter]
100 [*]
Display shows: 30
30 [-]
Display shows: 0