No. That tick box is for webmail (Roundcube or Horde on Plesk itself). It has no effect on the problem in question.
I can't tell for sure, but it looks like you are running Plesk Onyx, yes?
You need to go to Tools & Settings > SSL/TLS Certificates. On that page, there is a section that talks about SSL certificates currently in use by the server (see ssl-1.png)
The certificate you choose for mail must be valid (i.e. not expired) and must work for whatever server address you are going to specify for people enter enter as the SMTP server address.
For example, you already have one for Plesk itself, don't you? You can use the same one. And tell people to enter the same domain for email.
e.g. if you login to Plesk with
https://plesk.example.tld:8443 and you have an SSL certificate (e.g. Lets Encrypt) for plesk.example.tld already set up, then you can use the same one for mail. You would then ask your customers to use plesk.example.tld as the SMTP address in GMail.
There are many other options open to you in terms of what address to choose (like a wildcard certificate - see bottom of email). All that matters is that the certificate you choose for mail is valid (i.e. correctly secures and is not expired) for whatever address you tell people to enter for the SMTP server address.
What is happening is simple:
Imagine you have a domain
www.example.tld and you DO have a certificate for it, but the certificate is for
www.some-other-domain.tld. If you connect using your web browser to this domain using https:// you will get an error. And the error will be that the certificate does not match the domain.
This same thing happens with Gmail. And you'll get the same error with Outlook or any other email program is you tell it to use TLS.
Note that the certificate issue also applies to IMAP (and pop3) if you tell the email client to use TLS. You will get an error if the certificate the mail server uses does not match the domain you enter in the email program. You can experiment in Outlook or Thunderbird or whatever. Put a tick in the TLS box and try connecting to the Plesk email server using various domains that point to it. You will always get an error unless you choose a domain that matches the SSL/TLS certificate you have selected for the mailserver to use.
For the moment, GMail is only enforcing TLS, and requiring a valid/matching certificate, for SMTP. I do not know why it is not enforcing it for POP3 collections. No doubt it will soon.
Wildcard certificates: Honestly, the easiest option is to buy a wildcard certificate, or to have Lets Encrypt generate a wildcard certificate, and use that. The wildcard certificate covers *.example.tld so you can tell people to use mail. smtp. pop3. or whatever you want, and as long as you have told Plesk to use the wildcard certificate for email, it will work fine.