Strange issue with precision on N9

[conxt]

I'm testing a calculator application and apparently got to a strange precision issue with simple operations on occasional operands. For example, operation
55*0.45
returns
24.750000000000004
The operations are executed through JS, not C++.
Issue present on both physical N9 (PR 1.1) and Qemu.
Does anyone know how to avoid this besides artificially limiting the precision?
Thanks!

[gnuton]

Hi,
You can round flat numbers by using toFixed() JS core function.
Usually 2 decimals are enough.

http://www.w3schools.com/jsref/jsref_tofixed.asp

[conxt]

That would really be the solution if the calculator were financial, but it's engineering with sines, logarithms etc., where it's not fully appropriate to limit the precision.
Anyway, I wonder how this issue is possible at all.

[gnuton]

Well, even scientific calcultors round their results.
No idea if there is a function to identify significant figures in JS.. but I don't think so.
You maybe need to write them, if you are not able to find it googling.

http://en.wikipedia.org/wiki/Signifi...ificant_digits

[conxt]

What really bothers me, is that the result in the given example is basically incorrect.
Say, I do sqr(1.0000001). The correct answer here is 1.00000020000001, it's computed correctly, and the tailing 1 does matter.
However, 0.555*25 produces 13.875000000000002 which is false, and there is no way to determine this computational mistake, the whole concept of which seems very weird.

[conxt]

By the way, I just downloaded one of the calculators from the Store and it produces exactly the same result.
So, I guess, this might be at the OS core level...

[gnuton]

Hi,
These are the result of your operations, calculated by Wolframe Alpha. http://www.wolframalpha.com
* 0.555*25 = 13.875
* sqr(1.0000001)= 1.0000000499999987500000624999960937502734374794921891113279...

If you get 13.875000000000002, I can see the error there, but to be honest I see a bigger mistake in the second one.

Do you have the same results if you run this calculations in your PC?
In any case you can file this as bug in Qt bug reporting tool https://bugreports.qt.nokia.com/secure/Dashboard.jspa

[conxt]

Wolframalpha interprets "sqr" as square root, not as ^2, so that result is correct :)
Thank you for the link! - I wanted to post this issue but I couldn't find the correct place to do it
I guess I will have to live with this problem (luckily, the set of erroneous operands is very limited) until the fix.
Thanks once again!

[conxt]

Before posting the bug I will try to compile a small sample app for different platforms (Qt Desktop and emulated Symbian) to be able to be more specific there.
Thanks!

[gnuton]

You are welcome,
BTW please if you file the bug there, paste here the URL to the bug report for reference.
Other people may run in the same issue.

Anyway it's interesting to understand if this is a platform problem or not.
If the bug is related to Harmattan itself the bug could be solved quickly.
In case this error happens on other platforms too (PC for instance), then
the problem is in the JS engine and it could take more time!

[conxt]

After some tests it came down to this:
https://bugreports.qt.nokia.com/browse/QTBUG-23074