Fuzzy optimizations in QTransform

Submitted by mimec on 2012-12-06

Before I get to the point, just a quick note: I recently released a new version of both WebIssues and Fraqtive and I'm planning to release a new version of Saladin by the end of this year, so I'm quite busy, as usual at this time of the year. Also check out the fractal animations created using the latest version of Fraqtive. There are much more impressive deep zoom animations available, but they were created using specialized tools which use high precision numbers. Fraqtive uses SSE2 to maximize the real-time experience, so it's limited to double precision, which allows to zoom the fractal about 10^13 times. But this is enough to produce some cool effects.

Now back to the main topic. For a long time there was a strange bug in Fraqtive that I thought was impossible to fix. As those who use it know, in Fraqtive it is possible to move and zoom around the fractal using the mouse, and when you release the mouse button, the new region of the fractal is recalculated. This works by converting the start and end position to a QTransform (which is basically a 3x3 matrix defining the offset, zoom and rotation) and calculating the relative transformation. It worked fine until a certain zoom level was reached; then it started producing weird results. I always blamed the limited precision of double numbers, though the effect could be observed a few orders of magnitude before reaching the maximum valid zoom.

Recently I debugged the entire code, including the calculations performed inside QTransform class and discovered that it performs some optimizations which cause the wrong results. The documentation mentions that QTransform performs some optimizations based on the type of the matrix. For example, if the transform includes translation only, the scale and rotation components are ignored. What the documentation doesn't mention, though, is that Qt uses fuzzy compare functions (such as qFuzzyCompare) to determine the type of the matrix.

The problem with qFuzzyCompare and it's undocumented qFuzzyIsNull counterpart is that they assume that only about 12 digits are significant in a double precision floating point number. This often makes sense, because limited precision can result in some subtle differences which cause the strict comparison operator to fail. We all know that 10 / 3 is 3.33333... and that value multiplied by 3 gives 9.99999.... In mathematics, this value is equal to 10, but for a computer, these numbers are not necessarily equal.

The effect of using fuzzy comparisons is that at a zoom level higher than 12, the scale factors are considered equal to 0, so the transformation is considered non-invertible, when in fact it is. Also rotation tends to be ignored at this zoom level. The solution is not to use QTransform when high precision is required or to write custom functions which do not have these side effects. See also [#QTBUG-8557] where a similar problem is discussed.

Note that fuzzy comparisons are used not only in QTransform, but also in other classes like QMatrix4x4 and QVector3D. Some time ago I came across the a similar problem with QVector3D::normalized which caused Descend to incorrectly calculate normal vectors for the surfaces. The problem is that Descend calculates three samples that are very close to one another in order to precisely determine the normal vector. In some areas of the surface it would hit the 12 significant digit limit, so I ended up writing my own version of this function which didn't use fuzzy comparisons.