Since RDR2 only has 20 sensitivity steps you are most likely getting a calculation with a huge discrepancy. So it's limited by the lack of sensitivity precision in the game, and when you convert back the value will be wrong since it's based on a sensitivity with discrepancy.
All calculations are accurate, but in addition to the lack of precision the game has issues with smoothing and treating mouse input as an analog stick.