Issue Network drops every 12 hours [IONOS]

[learning_curve]

Hi,
The fix I found only applies to IONOS Dedicated Servers, unfortunately.

We don't need a fix, because as already posted, we haven't got any of the Network problems that you have anyway, but it's far more important and excellent news, that you've got a fix for all those with IONOS Dedicated Servers that are effected. We'll read it out of interest, once it's posted. Great job!

 

[bradz]

For a dedicated ionos server, I see

# See resolved.conf(5) for details

[Resolve]
#DNS=
#FallbackDNS=
#Domains=
#LLMNR=no
#MulticastDNS=no
#DNSSEC=no
#DNSOverTLS=no
#Cache=no-negative
DNSStubListener=yes
#ReadEtcHosts=yes

for interfaces

# /etc/network/interfaces -- configuration file for ifup(8), ifdown(8)
# Generated by debian-installer.

# The loopback interface
auto lo eth0
iface lo inet loopback

allow-hotplug eth0
iface eth0 inet dhcp
iface eth0 inet6 static
address 2607:f1c0:86d:a200::5c:19c9
netmask 64
auto eth0

For sudo networkctl status (using IP4 shared)
● State: routable
Address: 74.208.67.9 on eth0
2607:f1c0:86d:a200::5c:19c9 on eth0
fe80::225:90ff:fe74:cd8e on eth0

This beyond my skill set, but may be helpful

 

[carlosarmando85]

For a dedicated ionos server, I see

# See resolved.conf(5) for details

[Resolve]
#DNS=
#FallbackDNS=
#Domains=
#LLMNR=no
#MulticastDNS=no
#DNSSEC=no
#DNSOverTLS=no
#Cache=no-negative
DNSStubListener=yes
#ReadEtcHosts=yes

for interfaces

# /etc/network/interfaces -- configuration file for ifup(8), ifdown(8)
# Generated by debian-installer.

# The loopback interface
auto lo eth0
iface lo inet loopback

allow-hotplug eth0
iface eth0 inet dhcp
iface eth0 inet6 static
address 2607:f1c0:86d:a200::5c:19c9
netmask 64
auto eth0

For sudo networkctl status (using IP4 shared)
● State: routable
Address: 74.208.67.9 on eth0
2607:f1c0:86d:a200::5c:19c9 on eth0
fe80::225:90ff:fe74:cd8e on eth0

This beyond my skill set, but may be helpful

Hi bradz,
Is that the only output you get from
networkctl status ?

 

[bradz]

See below.
# sudo networkctl status
● State: routable
Address: 74.208.67.9 on eth0
2607:f1c0:86d:a200::5c:19c9 on eth0
fe80::225:90ff:fe74:cd8e on eth0
Gateway: 10.255.255.1 (Cisco Systems, Inc) on eth0
10.255.255.1 (Cisco Systems, Inc) on eth0
DNS: 10.255.255.4
10.255.255.3
Search Domains: onlinehome-server.com

Feb 22 08:53:21 zehr.ws systemd[1]: Starting Network Service...
Feb 22 08:53:21 zehr.ws systemd-networkd[1010]: Enumeration completed
Feb 22 08:53:21 zehr.ws systemd[1]: Started Network Service.
Feb 22 08:53:21 zehr.ws systemd-networkd[1010]: eth0: IPv6 successfully enabled
Feb 22 08:53:21 zehr.ws systemd[1]: Starting Wait for Network to be Configured.>
Feb 22 08:53:23 zehr.ws systemd-networkd[1010]: eth0: Gained carrier
Feb 22 08:53:25 zehr.ws systemd-networkd[1010]: eth0: Gained IPv6LL
Feb 22 08:53:27 zehr.ws systemd-networkd[1010]: eth0: DHCPv4 address 74.208.67.>
Feb 22 08:53:27 zehr.ws systemd[1]: Finished Wait for Network to be Configured.

 

[carlosarmando85]

Hi Bradz,
The output you get differs from the issue I had but maybe this will solve the losing connection every 12 hours:

Inside your
/etc/netplan/ directory

you will find a file, in my case is: 50-cloud-init.yaml (it may be different on yours)

Open the file it may look like this:

network:
    ethernets:
        eth0:
            dhcp4: true
    version: 2

Add this line: critical:true, this will designate the connection as “critical to the system”, meaning that special care will be taken to not release the assigned IP when the daemon is restarted.

Example:

network:
    ethernets:
        eth0:
            dhcp4: true
            critical: true
    version: 2

Save the file and run:

sudo netplan generate
sudo netplan apply

P.S.

Check if your file located here
/etc/cloud/cloud.cfg.d/99-disable-network-config.cfg

Has this option:
network: {config: disabled}

If the file doesn't exist or doesn't have this option the changes you made to the netplan file will not be saved after a reboot.

 

[learning_curve]

Great work @carlosarmando85 and @bradz Lots of checking, posting & solving been done there. Also very useful for all, as it again, underlines the config / setup differences between Dedicated and Cloud servers that are provided by IONOS.

@bradz You've posted that you don't use IPv6 yet, but in both of those configs that you've posted, you can see, that despite that, your current config is setup and ready to use IPv6 anyway. Only mentioned that, because it was an IPv6 related issue that brought the Network failures in our case (see post #13) and perhaps very unlikley, but, IF the fix by @carlosarmando85 doesn't work for you, then - just in case - your IPv6 config / setup might be worth another look.

@carlosarmando85 Yep on IONOS Cloud servers (unless they've finally changed it or will change it when Ubuntu 22.04 is released...) Netplan is still not used and ifupdown is being configured by IONOS by default, so; /etc/netplan/ is empty and /etc/cloud/cloud.cfg.d/99-disable-network-config.cfg doesn't exist.

 

[bradz]

Yes carlosarmando85
50-cloud-init.yaml is the what you had seen. I get....
# This file is generated from information provided by the datasource. Changes
# to it will not persist across an instance reboot. To disable cloud-init's
# network configuration capabilities, write a file
# /etc/cloud/cloud.cfg.d/99-disable-network-config.cfg with the following:
# network: {config: disabled}
network:
ethernets:
eth0:
dhcp4: true
version: 2

It sounds like this maybe the same fix that worked for you! In my case, it seems to be about 1 week between network drops and now we have an eye out for it, so we catch it fairly fast. I do want to solve this, however I am thinking I should explore it a little more.

learning_curve​

I do feel this could be related to IP6 configuration since I had IP issues on 2 server (I think the same 2).
May be it is time for me to learn a lot more about IP6.
I initially thought that since the server had an IP4 and IP6 allocated to it, I should set both of them to shared.
But then I had IP issues on 2 of the servers, so I stopped trying to use IP6 as shared
These are the notes I made regarding what I did (maybe not related to present issue?)
----------------
IP SYNC ISSUE
Reread IPs in Plesk: # plesk bin ipmanage --reread
Check IP addresses state: # plesk bin ipmanage -l
***Note: State 0 means the IP address is ok, state 1 means the IP address is broken
Repair broken IP addresses with state 1: # plesk bin ipmanage --repair xx.x.xxx.xx
*** Maybe able to login Plesk - go to IP Addresses then Repair
----------------
Does this make it sound like it is related?
I am perplexed that 2 of the 4 servers, same age, same setup, have not shown the network drop issue, making me think I did something during setup.
I will look at the servers regarding the above posts to see if I see a difference in settings.

Again, thanks so much!
I need to sleep on this
Brad

 

[learning_curve]

@bradz Definitely makes sense to work with the fix that @carlosarmando85 has posted first of all.
Afterwards, If / When you do investigate IPv6 further, providing that everything is configured correctly within your IONOS setup, IPv6 worls perfectly within Plesk Obsidian, including your IPv6 address being a shared addrress. We only use one IPv4 and one IPv6 address per cloud server, but even so, the attached quick screen grab confirms what I've mentioned. This, is obviously then extended out to all hosted domains / mail services / databases / utilitiy services etc.

So the fact that you can't share your IPv6 address, (as per your post above) does initially... look like there might be an IONOS IPv6 config glitch somewhere.
I'll add some IPv6 checks via CLI shortly, just in case you wnat to use these If / When you do investigate IPv6 further.

 

[learning_curve]

@bradz

# ping6 fe80::1 -I ens192

^^ Change "ens192" to the interface that you're using (visible in IONOS & Plesk - maybe eth0 in your case)

# ip -6 neigh

# ip -6 r s

# ss -plnt | grep ':53'

# netstat -tunlp

No need/reason to post all the data on here, but all of those ^^ plus the generic posts below, should give you a bit more insight of where you are now with IPv6

# systemd-resolve --status

# journalctl -u systemd-resolved -f

# ping6 google.com > Or any other known IPv6 common address <

 

[bradz]

UPDATE: After looking at many potential causes with no resolution, the IONOS support/techs did a hardware replacement on the server. It has been over 2 weeks and the server is running great with no down time. It looks like the issue is solved!!! Many thanks to IONOS support & server techs and the Plesk community (especially learning_curve and carlosarmando85). I learned many valuable server trouble shooting skills! Thanks! Sincerely, Brad

 

[thinkjarvis]

Hi,
The fix I found only applies to IONOS Dedicated Servers, unfortunately.

@carlosarmando85 Sorry to post late to the party - What fix? Can you reshare?

We are having major problems with Ubuntu 22 and Plesk on IONOS dedicated servers right now. It seems to forget its network settings and reboot randomly.

 

[carlosarmando85]

@carlosarmando85 Sorry to post late to the party - What fix? Can you reshare?

We are having major problems with Ubuntu 22 and Plesk on IONOS dedicated servers right now. It seems to forget its network settings and reboot randomly.

Hi @thinkjarvis,

Here is the fix I've found for my IONOS Dedicated Server:

Issue - Problem with /etc/network/interfaces and 2 additional IPv4 on Ionos server

Hi @ Plesk friends, 4 weeks ago I migrated Plesk Obsidian 18.0.33 with Ubuntu 20.04.2 on Ionos dedicated server and Cloud Panel. Important is the very last point. In real it is a Cloud server (BUT) with dedicated fixed environment, it's a dedicated server in Cloud administration. To realize own...

talk.plesk.com

Good luck

 

[thinkjarvis]

Hi @thinkjarvis,

Here is the fix I've found for my IONOS Dedicated Server:

Issue - Problem with /etc/network/interfaces and 2 additional IPv4 on Ionos server

Hi @ Plesk friends, 4 weeks ago I migrated Plesk Obsidian 18.0.33 with Ubuntu 20.04.2 on Ionos dedicated server and Cloud Panel. Important is the very last point. In real it is a Cloud server (BUT) with dedicated fixed environment, it's a dedicated server in Cloud administration. To realize own...

talk.plesk.com

Good luck

Thanks @carlosarmando85 I have implimented this we had a different yaml file name but added critical and ran the commands.

Still seeing a random crash though and as others have said IONOS support dont really offer software support in ANY of their contracts.

Second line support restored the network settings but I may have to direct them here to see if they can provide anything else.

Last crash was randomly early morning. No notable errors to report in the console but there was a DCHP update about 6 minutes before our crash.

If you have any spare time I would love for you to look over our network settings to work out if this is the same issue.

Essentially when the problem occurs the server is still up but it can no longer be reached via public IP.

 

[thinkjarvis]

Thanks @carlosarmando85 I have implimented this we had a different yaml file name but added critical and ran the commands.

Still seeing a random crash though and as others have said IONOS support dont really offer software support in ANY of their contracts.

Second line support restored the network settings but I may have to direct them here to see if they can provide anything else.

Last crash was randomly early morning. No notable errors to report in the console but there was a DCHP update about 6 minutes before our crash.

If you have any spare time I would love for you to look over our network settings to work out if this is the same issue.

Essentially when the problem occurs the server is still up but it can no longer be reached via public IP.

Hi @thinkjarvis,

Here is the fix I've found for my IONOS Dedicated Server:

Issue - Problem with /etc/network/interfaces and 2 additional IPv4 on Ionos server

Hi @ Plesk friends, 4 weeks ago I migrated Plesk Obsidian 18.0.33 with Ubuntu 20.04.2 on Ionos dedicated server and Cloud Panel. Important is the very last point. In real it is a Cloud server (BUT) with dedicated fixed environment, it's a dedicated server in Cloud administration. To realize own...

talk.plesk.com

Good luck

The problem is largely solved but have experienced a random crash after almost 10 days of no issues.

The issue I have with all of this (and I sure you all probably share the same sentiment) is that the software is provisioned by IONOS and the network settings and config are set up by IONOS to match their datacentre/dedicated server and monitoring/update policies. It just stinks that I've had to pay for an investigation into part of the system that is essentially provisioned by them. The server wouldnt work without those settings so in my oppinion this is a real grey area in the support offering.

I'm currently quoting for Enterprise grade hosting either cloud with perminent failover/load balancing or dedicated with other cloud companies that cover software in the SLA. Will let you all know if the cost seems fair.

Other forums have suggested scheduling a reboot every 12 or 24 hours?
Anyone think this would be of use to avoid the randomness of the reboots?

 

[Peter Debik]

IMHO a server should only be rebooted if it is an unavoidable situation, e.g. after hardware maintenance or - if Kernel Care is not used - after a Kernel update. A scheduled reboot every 24 hours is not best practice.

 

[thinkjarvis]

IMHO a server should only be rebooted if it is an unavoidable situation, e.g. after hardware maintenance or - if Kernel Care is not used - after a Kernel update. A scheduled reboot every 24 hours is not best practice.

Thanks Peter,

I do agree with you but I am in an extremely difficult position here. The issue appears to be with IONOS dedicated servers specifically and the way the DHCP and DNS works and they have effectively refused further investigation - As other customers have suggested - Clearly ongoing over a period of the last few years.

So Rock and a hard place. I was reading another thread that I cannot find right now where other IONOS customers had issues with random reboots so to help control the situation they set up a Cron to reboot the server at regular intervals to refresh whatever it was that was refreshing with IP resolution when they wanted it to rather than randomly.

I will say it has left me with an extremely sour taste in my mouth. Because IONOS VPS servers do not suffer from these problems. I have 3 of those and they have been extremely stable. The move to dedicated was the next logical step for a customer who had outgrown the core counts and ram available in typical VPS setups.

I'll respond it I get any feedback but I am going to check over the network files again to see if I can identify the cause of the issue.

 

[learning_curve]

~~

I will say it has left me with an extremely sour taste in my mouth. Because IONOS VPS servers do not suffer from these problems. I have 3 of those and they have been extremely stable. The move to dedicated was the next logical step for a customer who had outgrown the core counts and ram available in typical VPS setups.

TBH @thinkjarvis You're definitely not the first person who has had some form of issue with IONOS Dedicated Servers...

Just one thought: We have always used IONOS Cloud Servers, without any issues at all (other than some faffing around with their initial configs (at root access level) which is due to IONOS and their policy of only providing choices between consistent, initial set-up packages. Some of those those items, are already mentioned in this thread, but ignoring those for a moment, we've never, in all the time we've ever used IONOS Cloud Servers, had any random re-boots.

I've only posted this, because, not to the level of Dedicated Servers, but, you can choose from lots of Cloud Server Config setups, before you proceed (and these do vary slightly, dependent on which country you choose, to use as the primary IONOS Data Centre). So, what I was suggesting, was that IF you're happy to stay with IONOS, then dependent on what spec(s) you want, IONOS Cloud Servers (NOT IONOS VPS which you already have) might be a viable alternative option?

NB Although IONOS Cloud Servers are also in the no tech support class like your Dedicated Servers, they DO ensure that their Cloud Server uptime stats are as close to perfect as possible (which is what you'd expect with all IONOS Cloud Servers being Virtual Servers) and, if you have some genuine questions, relating to their own Networking e.g. DNS / NameServers / DANE / DNSSEC / DNS API - which you can use via CRON to update all of your SSL Certificates IF all of your DNS is managed within IONOS not Plesk, plus things like IONOS Domian Guard etc then they can be quite helpful and pretty nformative too.

 

[thinkjarvis]

TBH @thinkjarvis You're definitely not the first person who has had some form of issue with IONOS Dedicated Servers...

Just one thought: We have always used IONOS Cloud Servers, without any issues at all (other than some faffing around with their initial configs (at root access level) which is due to IONOS and their policy of only providing choices between consistent, initial set-up packages. Some of those those items, are already mentioned in this thread, but ignoring those for a moment, we've never, in all the time we've ever used IONOS Cloud Servers, had any random re-boots.

I've only posted this, because, not to the level of Dedicated Servers, but, you can choose from lots of Cloud Server Config setups, before you proceed (and these do vary slightly, dependent on which country you choose, to use as the primary IONOS Data Centre). So, what I was suggesting, was that IF you're happy to stay with IONOS, then dependent on what spec(s) you want, IONOS Cloud Servers (NOT IONOS VPS which you already have) might be a viable alternative option?

NB Although IONOS Cloud Servers are also in the no tech support class like your Dedicated Servers, they DO ensure that their Cloud Server uptime stats are as close to perfect as possible (which is what you'd expect with all IONOS Cloud Servers being Virtual Servers) and, if you have some genuine questions, relating to their own Networking e.g. DNS / NameServers / DANE / DNSSEC / DNS API - which you can use via CRON to update all of your SSL Certificates IF all of your DNS is managed within IONOS not Plesk, plus things like IONOS Domian Guard etc then they can be quite helpful and pretty nformative too.

Im just going through the enterprise cloud service for quotations. But we would have a third party doing our server support in that situation.

When I say random reboots thats actually wrong what I meant was:

Randomly having DNS/DHCP issues as described in this post. But the timing of the issues has been random with no consistency to the times that the servers crash.

So not random crashes random forgetting of network settings is what I meant.

 

[learning_curve]

~~

When I say random reboots thats actually wrong what I meant was:

Randomly having DNS/DHCP issues as described in this post. But the timing of the issues has been random with no consistency to the times that the servers crash.

So not random crashes random forgetting of network settings is what I meant.

Gotcha. Same points apply again with our IONOS Cloud Servers though i.e. Zero events of that nature with any of them, ever.
However, all of our DNS, we do manage through IONOS not via our Plesk installations, so that might be a factor in the superb continuity to date.

 

[MartinT]

Hi

I may remember to have the same issue, not on a IONOS server last year. So I don't think thas an Inonos issue.

It is something with IPv6. Im not a specialist, but I solved it till today by dropping a line out of /etc/network/interfaces:
Get rid of:

iface lo inet6 loopback

But for future you have to remember to activate all IPv6 adresses every restart of your server to be save that they are listening.

I remember that this was my soloution, I think.