Do you seriously need continuous positional information at 100k RPM (which is what encoders are for) - or do you just need to count the number of turns / synchronize something with a single, specific angular position?
If the latter, then any slot photointerruptor will do - Toshiba TLP841 or whatever. The signal would be barely 1.6 kHz, while a typical photointerruptor has a response time of 3-30 µs.
If the former, then yeah, depending on the resolution needed, this may be hitting the limits of the speed your ATmega chip may be able to run at.