Galculator math errors?

Here you can discuss every aspect of Debian. Note: not for support requests!

Galculator math errors?

Postby No_windows » 2018-01-12 03:50

updated 64bit Jessie, using Galculator.

I type .30 x 100 =
I get 30
I type -30 =
I get -1.11022302463e-15

Posts: 489
Joined: 2015-08-05 03:03

Re: Galculator math errors?

Postby srq2625 » 2018-01-12 11:45

No_windows wrote:updated 64bit Jessie, using Galculator.

I type .30 x 100 =
I get 30
The decimal point causes the calculation to be performed in floating-point math. The result is displayed as a fixed-point approximation of the floating-point result (which is correct).
No_windows wrote:I type -30 =
I get -1.11022302463e-15

What I'm guessing happened here is the calculation is still being done in floating-point, hence the "error". What might have been a better (I'm guessing here, no testing of the idea has been done) coding is to take the display value (which has only an implied decimal point) and proceed using fixed point math.
Posts: 44
Joined: 2016-02-26 11:01

Re: Galculator math errors?

Postby tynman » 2018-01-12 14:18

This caught my attention because I use galculator fairly regularly. Maybe I need to find a more accurate calculator. Something I actually need to worry about, as opposed to all the recent nonsense FUD in the news and forums about CPU "flaws". Oh but wait, "E-15" .... :)
Posts: 123
Joined: 2016-05-03 19:48
Location: British Columbia, Canada

Re: Galculator math errors?

Postby RU55EL » 2018-01-12 16:59

Galculator works perfectly in RPN mode. At least, that is the mode I always use. (It is also easier for me to use in RPN mode.)

.3 [enter]
100 [*]
Display shows: 30
30 [-]
Display shows: 0
User avatar
Posts: 402
Joined: 2014-04-07 03:42
Location: /home/russel

Return to General Discussion

Who is online

Users browsing this forum: No registered users and 3 guests