what is the different between them in term of position sensor?
The encoder is typically a digital output, non contact sensor and the potentiometer relies on physical contacts and has an analogue output. Position sensors will cope with continuous rotation just fine, where most potentiometers have limited rotation - you can get continuous rotation pots but expect to pay a premium.
can the potentiometer replace the encoder?
Yes, but it is not recommended. Since potentiometers rely on physical contacts they have a limited lifespan so are likely to prove unreliable long term. You can buy better quality pots with longer lifespans but you save little, if any, over the encoder option and still ultimately have a less reliable solution.