I had a similar issue with sendmail not working after upgrading from 11.5.0 to 12.0.18 whilst using Postfix as mail client.
Nothing I tried fixed the problem (something about HELO = localhost being wrong) - command rejected error...
Since it was getting critical, I installed Qmail instead, then ran
Code:
/usr/local/psa/admin/sbin/mchk
to rebuild the rcpthosts file (which was now empty).
This resolved the problem for me - not ideal but needed a quick fix.