I had to look up how an optical mouse works ... (
http://computer.howstuffworks.com/question631.htm) but it seems to me like you could build a scaled up version with a light and camera. The challenge seems to be in the software to process the images ... and i'm not much help there. However if my understanding is correct I don't see why it couldn't track orientation as well. My thought is the software would compare two images and find the transformation that provides the best fit. The best fit would look for a transformation that would include translation in x and y as well as rotation about the z-axis. Of course adding another degree of freedom (rotation about the Z-axis) to the transformation increases the processing required.
I am looking at a sensor for a project at work that detects translational motion using Laser Doppler Velocimetry. There is a version that can detect both x and y translation but not rotation. I would think you could use two sensors a fixed distance apart and calculate the rotation from the difference in the two sensor movements. However these sensors run about 10k for a single axis detection capability so they are probably outside of your price range.
... just some thoughts on the problem, probably not very helpful.
If you do find any sensors capable of measuring the translational movement of an irregular surface without contact (even if it is only in one axis) I would be interested.