For psa itself, even when installed using the autoinstaller, rpm should still list psa itself as installed.
rpm -q psa
should show you something like:
psa-7.5.3-rh9.build75050506.13
so to uninstall it: rpm -e psa-7.5.3-rh9.build750506.13
For the other packages which are psa specific, to find a list of them:
rpm -qa |grep 'psa-'
then just 'rpm -e <package name> to get rid of those. Any other packages which were not Plesk specific you can generally leave those (apache, perl, php, mysql, etc, etc) since you may need those for later.