Page 1 of 1

Galculator math errors?

PostPosted: 2018-01-12 03:50
by No_windows
updated 64bit Jessie, using Galculator.

I type .30 x 100 =
I get 30
I type -30 =
I get -1.11022302463e-15

What?????

Re: Galculator math errors?

PostPosted: 2018-01-12 11:45
by srq2625
No_windows wrote:updated 64bit Jessie, using Galculator.

I type .30 x 100 =
I get 30
The decimal point causes the calculation to be performed in floating-point math. The result is displayed as a fixed-point approximation of the floating-point result (which is correct).
No_windows wrote:I type -30 =
I get -1.11022302463e-15

What I'm guessing happened here is the calculation is still being done in floating-point, hence the "error". What might have been a better (I'm guessing here, no testing of the idea has been done) coding is to take the display value (which has only an implied decimal point) and proceed using fixed point math.

Re: Galculator math errors?

PostPosted: 2018-01-12 14:18
by tynman
This caught my attention because I use galculator fairly regularly. Maybe I need to find a more accurate calculator. Something I actually need to worry about, as opposed to all the recent nonsense FUD in the news and forums about CPU "flaws". Oh but wait, "E-15" .... :)

Re: Galculator math errors?

PostPosted: 2018-01-12 16:59
by RU55EL
Galculator works perfectly in RPN mode. At least, that is the mode I always use. (It is also easier for me to use in RPN mode.)

.3 [enter]
100 [*]
Display shows: 30
30 [-]
Display shows: 0