Resolved SUPER WEIRD port issue

[DenizGelion]

Server operating system version

Ubuntu 22.04

Plesk version and microupdate number

Plesk Obsidian Web Pro Edition Version 18.0.56 Update #4

Hi there,

I've migrated our old server to a new one, fresh install and all, but I've encountered an issue you guys hopefully will be able to figure out:

My websites cannot send mail over SMTP on the same server- I always get a timeout. If I use an external SMTP server it works- what is weird, is that I can use said Plesk SMTP mail server from ANOTHER server or send mail via my mail program, no issue. If I run a php test script to check for open ports, I get the result that all ports are closed. When I run the script (with the same login information) on another server- what the cheese, it works?

Does anyone have any clue why the ports are blocked on the server itself, but not if I try to reach it externally? This makes no sense, maybe someone has a clue.

Thank you very much in advance.


(using Postfix / Dovecot, standard configuration on pretty much everything)

 

[DenizGelion]

Running

netstat -tulpn | grep LISTEN | grep 465

yields:



So the ports are obviously open- but not for the applications on that very same server- somehow?

 

[Peter Debik]

I am having difficulties understanding the question. What do you mean by "cannot send mail over SMTP on the same server" and "I can use said Plesk SMTP mail server from ANOTHER server". Could you please provide steps to reproduce the issue?

 

[AYamshanov]

If something blocks connections, from my point of view, it is usually firewall. What about local firewall? Maybe the output from the `iptables -S` command could help to find issue/clue?

 

[DenizGelion]

Thanks for the quick reply and sorry for the late reply from my side- we had to switch to scenario 3 (from the image below) temporarily because of the current workload and I had no time to get back to you guys here.

Take a look at this:



I hope this is clear enough. The IP-addresses are of course just example addresses, but I hope you get the gist. I could eliminate software issues by now and the server somehow can't send emails from itself and it's driving me nuts. It can clearly send (B) from ANOTHER server (C) and that OTHER server (C) can send from said server (B), but the Server (B) can't send from itself (B).
The exact same settings have been used in all scenarios, nothing was altered (testwise it was altered, but that had no effect either).

@AYamshanov
The output of iptables -s is as follows:

-P INPUT DROP
-P FORWARD DROP
-P OUTPUT DROP
-N DOCKER
-N DOCKER-ISOLATION-STAGE-1
-N DOCKER-ISOLATION-STAGE-2
-N DOCKER-USER
-A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
-A INPUT -p tcp -m tcp ! --tcp-flags FIN,SYN,RST,ACK SYN -m state --state NEW -j                                                                                                                      REJECT --reject-with tcp-reset
-A INPUT -m state --state INVALID -j DROP
-A INPUT -i lo -j ACCEPT
-A INPUT -p tcp -m tcp --dport 25565 -j ACCEPT
-A INPUT -p udp -m udp --dport 68 -j ACCEPT
-A INPUT -p tcp -m tcp --dport 49152:65535 -j ACCEPT
-A INPUT -p tcp -m tcp --dport 8447 -j ACCEPT
-A INPUT -p tcp -m tcp --dport 8443 -j ACCEPT
-A INPUT -p tcp -m tcp --dport 8880 -j ACCEPT
-A INPUT -p tcp -m tcp --dport 80 -j ACCEPT
-A INPUT -p tcp -m tcp --dport 443 -j ACCEPT
-A INPUT -p tcp -m tcp --dport 21 -j ACCEPT
-A INPUT -p tcp -m tcp --dport 22 -j ACCEPT
-A INPUT -p tcp -m tcp --dport 587 -j ACCEPT
-A INPUT -p tcp -m tcp --dport 25 -j ACCEPT
-A INPUT -p tcp -m tcp --dport 465 -j ACCEPT
-A INPUT -p tcp -m tcp --dport 110 -j ACCEPT
-A INPUT -p tcp -m tcp --dport 995 -j ACCEPT
-A INPUT -p tcp -m tcp --dport 143 -j ACCEPT
-A INPUT -p tcp -m tcp --dport 993 -j ACCEPT
-A INPUT -p tcp -m tcp --dport 106 -j ACCEPT
-A INPUT -p tcp -m tcp --dport 3306 -j ACCEPT
-A INPUT -p tcp -m tcp --dport 5432 -j ACCEPT
-A INPUT -p udp -m udp --dport 137 -j ACCEPT
-A INPUT -p udp -m udp --dport 138 -j ACCEPT
-A INPUT -p tcp -m tcp --dport 139 -j ACCEPT
-A INPUT -p tcp -m tcp --dport 445 -j ACCEPT
-A INPUT -p udp -m udp --dport 53 -j ACCEPT
-A INPUT -p tcp -m tcp --dport 53 -j ACCEPT
-A INPUT -p icmp -m icmp --icmp-type 8/0 -j ACCEPT
-A INPUT -j DROP
-A FORWARD -j DOCKER-USER
-A FORWARD -j DOCKER-ISOLATION-STAGE-1
-A FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o docker0 -j DOCKER
-A FORWARD -i docker0 ! -o docker0 -j ACCEPT
-A FORWARD -i docker0 -o docker0 -j ACCEPT
-A FORWARD -m state --state RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -p tcp -m tcp ! --tcp-flags FIN,SYN,RST,ACK SYN -m state --state NEW                                                                                                                      -j REJECT --reject-with tcp-reset
-A FORWARD -m state --state INVALID -j DROP
-A FORWARD -i lo -o lo -j ACCEPT
-A FORWARD -j DROP
-A OUTPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
-A OUTPUT -p tcp -m tcp ! --tcp-flags FIN,SYN,RST,ACK SYN -m state --state NEW -                                                                                                                     j REJECT --reject-with tcp-reset
-A OUTPUT -m state --state INVALID -j DROP
-A OUTPUT -o lo -j ACCEPT
-A OUTPUT -j ACCEPT
-A DOCKER-ISOLATION-STAGE-1 -i docker0 ! -o docker0 -j DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-1 -j RETURN
-A DOCKER-ISOLATION-STAGE-2 -o docker0 -j DROP
-A DOCKER-ISOLATION-STAGE-2 -j RETURN
-A DOCKER-USER -j RETURN

The Web Application Firewall is only on "Detection only" with ModSecurity 2.9 on Apache, the free Comodo reule set and set to Fast. Changing any of these settings or completely turning it off has no impact. Turning the firewall itself on or off also has no impact on this. I hope someone can lend us a hand here...

 

[DenizGelion]

I cannot find any errors on any logs except for the shopsystem log:

{
  "exception": "[object] (Zend_Mail_Protocol_Exception(code: 0): Connection refused at /engine/Library/Zend/Mail/Protocol/Abstract.php:273)"
}

This obviously works if I choose any other mail from any other server, so the software itself shouldn't be at fault.

EDIT: If it's any help, this is the line it's referring to:

    protected function _connect($remote)
    {
        $errorNum = 0;
        $errorStr = '';

        // open connection
        $this->_socket = @stream_socket_client($remote, $errorNum, $errorStr, self::TIMEOUT_CONNECTION);

        if ($this->_socket === false) {
            if ($errorNum == 0) {
                $errorStr = 'Could not open socket';
            }
            /**
             * @see Zend_Mail_Protocol_Exception
             */
            throw new Zend_Mail_Protocol_Exception($errorStr);
        }

So it seems that it cannot open the socket somehow? How could I even check this?

 

[DenizGelion]

And my latest insight:

I ran a PHP testscript from both server B and server C about server B which look sas follows:

<?php
ini_set('max_execution_time', 0);
ini_set('memory_limit', -1);

$host = 'SERVER.B.DOMAIN';
$ports = array(21, 25, 80, 81, 110, 143, 443, 465, 587, 2525, 3306);

foreach ($ports as $port)
{
    $connection = @fsockopen($host, $port, $errno, $errstr, 2);

    if (is_resource($connection))
    {
        echo '<h2>' . $host . ':' . $port . ' ' . '(' . getservbyport($port, 'tcp') . ') is open.</h2>' . "\n";

        fclose($connection);
    }
    else
    {
        echo '<h2>' . $host . ':' . $port . ' is not responding.</h2>' . "\n";
    }
}

And the result is really something I cannot even remotely understand:


I don't understand how this is possible? I can reach all those ports externally, but not internally?

 

[DenizGelion]

After more tinkering:



Port 465 is clearly open and I can send mails externally. I've come to the conclusion that perhaps for some reason PHP scripts are not allowed to send mails?

I've used PHPMailer to try to send mails from Server B itself and the used same script from Server C (over Server B again) with the following content:

<?php

use PHPMailer\PHPMailer\PHPMailer;
use PHPMailer\PHPMailer\Exception;

require 'PHPMailer/src/Exception.php';
require 'PHPMailer/src/PHPMailer.php';
require 'PHPMailer/src/SMTP.php';

$mail = new PHPMailer();

$mail->isSMTP();
$mail->Host = 'SERVER.B.DOMAIN';
$mail->SMTPAuth = true;
$mail->Username = '[email protected]';
$mail->Password = 'PASSWORD';
$mail->SMTPSecure = 'ssl';
$mail->Port = 465;

$mail->setFrom('[email protected]', 'SERVER.B.DOMAIN');
                  
$mail->addAddress('[email protected]');
$mail->Subject = 'Test Email via Mailtrap SMTP using PHPMailer';
$mail->isHTML(true);

$mailContent = "<h1>Send HTML Email using SMTP in PHP</h1>
    <p>This is a test email I’m sending using SMTP mail server with PHPMailer.</p>";
$mail->Body = $mailContent;

if(!$mail->send()){
    echo 'Message could not be sent.';
    echo 'Mailer Error: ' . $mail->ErrorInfo;
}else{
    echo 'Message has been sent';
}
                  
?>

Results from Server B:
"Message could not be sent.Mailer Error: SMTP connect() failed. TroubleshootingSMTP server error: Failed to connect to server SMTP code: 111 Additional SMTP info: Connection refused"

Results from Server C via Server B:
"Message has been sent"

So, it seems that there is some sort of security rule SOMEWHERE that I can't figure out, that is related to PHP. The solution to this problem feels ultimately very easy, almost there...

 

[DenizGelion]

I might be wrong about the PHP part- trying out telnet from the server itself on every common port besides 8080 also yields a connection refused. I've compared all kinds of settings from Server B with our other currently running Server C and couldn't find anything. I'm officially out of ideas now

 

[DenizGelion]

As a last resorted I have contacted our Hoster (Hetzner Germany) for this, just in case there's something fishy going on in the background, but unfortunately that's not the case either. Feedback / Help very much appreciated.

 

[WebHostingAce]

Is this SERVER.B.DOMAIN behind any proxy? such as CloudFlare?

Usually the CloudFlare keeps this port 8080 open while the rest such as 465 closed.

 

[DenizGelion]

(Un)fortunately no use of a CDN here. It was just an in-place replacement of the old server and keeping the old IP address. We've used the plesk migrator to migrate most of the files after a fresh install of the latest Ubuntu LTS version and installing Plesk over CLI with the one click installer.
I've checked if there's somehow a PHP SSL issue, but the settings I could find and checking the php.ini files of the version being used on Server B with the version of Server C I couldn't find a difference.

 

[Dave W]

Is firewalld running on ServerB?

 

[DenizGelion]

@WebHostingAce

(Un)fortunately no use of a CDN here. It was just an in-place replacement of the old server and keeping the old IP address. Maybe it'll help if I say what exactly we did:
- We rented temporarily a server (which is Server B) in this case, to replace Server A
- We've installed on Server B Ubuntu LTS from scratch
- Install Plesk over CLI with the one click installer
- Install trial Plesk key
- Use Plesk Migrator to migrate files and settings
- Manually check all settings again (domains, subscriptions, users, etc. ...)
- Set up IP addresses in the corresponding files (/etc/netplan/01-netcfg.yaml) on Server B
- Shut down Server A and B
- Server A a gets an in-place replacement with Server B
- boot Server B - reinstall old Plesk key from Server A
- "everything just works" - except during testing we found the current issue

I've checked if there's somehow a PHP SSL issue, but the settings I could find and checking the php.ini files of the version being used on Server B with the version of Server C I couldn't find a difference. I am just not sure why the ports are not reachable when I use either the ip-address of the Server (B) or the domain name (which also resolves correctly) on any PHP based script on the server itself. I've installed telnet and tried "telnet localhost 465" - that works! I've also tried using the ip address- "telnet 69.69.69.69 465" - this does not work, it just throws "telnet: Unable to connect to remote host: Connection refused".
It also doesn't work if I use the domain name. My first thought was "ahah! I just use localhost instead of the ip address or the domain name in my script", but that doesn't work either.

@Dave W
no firewalld

To sum it up currently:
Mails CAN be sent from this mailserver in general, but only if I connect from an external server, but no app / script can send emails from the server itself because it refuses the connection on any port that is apparently open (confirmed by CLI and external tools).

 

[WebHostingAce]

Please check if you have SERVER.B.DOMAIN in Tool & Settings > Server Settings.

How about if you ping SERVER.B.DOMAIN from the SERVER.B.DOMAIN server itself. Do you receive a response?

Mails CAN be sent from this mailserver in general, but only if I connect from an external server, but no app / script can send emails from the server itself because it refuses the connection on any port that is apparently open (confirmed by CLI and external tools).

To me this sounds like SERVER.B.DOMAIN not resolving to a IP within the server itself. Please check.

 

[DenizGelion]

Please check if you have SERVER.B.DOMAIN in Tool & Settings > Server Settings.

How about if you ping SERVER.B.DOMAIN from the SERVER.B.DOMAIN server itself. Do you receive a response?



To me this sounds like SERVER.B.DOMAIN not resolving to a IP within the server itself. Please check.

My god FINALLY a hint, thank you so much!



Within Plesk in "Tools & Settings > IP Addresses" there's an ipv4 address and an ipv6 address- which are both correct. I've also clicked on "Reread IP" and it seems fine.
I've double checked the 01-netcfg.yaml and the other 10-plesk.yaml file and the ip addresses in those files are correct aswell. I've restarted the server several times to make sure nothing is stuck in some sort of cache.

I followed up with

sudo resolvectl flush-caches

but

sudo resolveip SERVER.B.DOMAIN

still shows the incorrect ip address. How / where can I fix that?

 

[DenizGelion]

Oh my god it was in the /etc/hosts file:

### Hetzner Online GmbH installimage
127.0.0.1       localhost.localdomain localhost
TEMPORARY.IP.ADDRESS    SERVER.B.DOMAIN DOMAINNAME
::1     localhost.localdomain localhost ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
ff02::3 ip6-allhosts

Changing the ip address to the correct one in line 2 fixed this issue immediately. I can't thank you enough @WebHostingAce . Let me know if I can repay you in any shape or form What an adventure just for one simple entry to be wrong- and here I thought I had thought about everything

 

[WebHostingAce]

No problem at all. Glad to hear your issue is solved.

 

[WebHostingAce]

TEMPORARY.IP.ADDRESS SERVER.B.DOMAIN DOMAINNAME

Usually this is 127.0.0.1 (localhost)

 

[DenizGelion]

Usually this is 127.0.0.1 (localhost)

That's the first line- looks like the installer from my hoster also adds that second line.