No, you can't substitute one for the other. A solenoid has a limited path. If it's a standard one, the path is linear. If it's rotary, the path describes an arc.
On the other hand, a motor spins continuously.
Rotary solenoids are a good choice for replacing unhacked servos, for example (although you need positioning logic). Much more precision can be gained with a rotary solenoid compared to a servo, though.