Lower precision for rotation matrix check

The invalid rotation matrix warning was triggered many times because of
very small differences in the 7th or 8th decimal number. We already use
lower precision for exactly this case in our geometry predicates.