Is there a any downsides to using larger capacitors than ~.1uf
like 10uf or 100uf?
If I asked you if there would be any downsides to using a larger knife for the next cut I have to make, then what would you answer (not knowing if I'm a surgeon, a butcher or perhaps was about to "skin" a whale).
That said, I have to say yes, as any choice of component values you make will be a compromise.
I have no idea of what you're trying to filter, but from the values mentioned, it could be a power supply of some sorts and then the 100nF (0.1µF) might be calculated for optimum performance and changing it for a 10 or 100 times larger value will mean a shift in what noise frequencies it will filter, as larger electrolytes are less efficient towards higher frequencies (and a higher ESR).
It is common use in supplies, to parallel connect 2 to 3 capacitors, eg. 4700µF//6.8µF//10nF, to get a wideband noise dampening.