Well you are not obliged to remove them. But we would to keep the consistency and to prevent any possible playback issues.
Also, if you re-uploaded these files at 192kbps rather than at 320kbps you are gaining back "almost" half of your storage capacity to add even more files.