I don't know how to convert hex to decimal in C code, but I'm sure a programmer reading this could . . . (hint hint). Or google. I have no idea where your friend got that equation from.
But otherwise, yea your output voltage is correct. If you want to get a bit more accurate, change the 5V to the exact voltage that goes into your IC (probably the output of your 5V regulator, perhaps 5.01V for example). To avoid the range, put a small cap like a .1uF ceramic across power and ground of your IC.