Question Synchronizing applications on 2 servers running their own Plesk

[Jean-Luc GARNIER]

Hi Community!

In order to setup load balancing, I have subscribed for 2 servers with my preferred hosting partner (and a load balancer obviously). I'd like to have my servers managed with Plesk and am looking for a tool / extension to keep my servers mirrored (master --> slave, roles may be switched by the load balancer) in near-real-time. What I need is to sync tools' databases (Wordpress, wiki, CRM and others) and not Plesk database itself! This makes a difference...

Unfortunately, it seems that syncing those servers is running a huge mess onto the slave, which receives the whole database content from the master, which overwrites names, passwords, configurations, permissions and others.

Therefore, we could safely say that:

• Either we remove Plesk from both servers, come back to the bare OS (Ubuntu) and try to sync them using CRON tasks
• Or we find a method / tool / extension to have only applications' databases sync'ed, protecting both Plesk instances on both the master and the slave servers.

Should anyone have a solution for this, let him/her receive my warmest acknowledgements!

Thanks in advance for any help!

 

[trialotto]

@Jean-Luc GARNIER

In some sense, I must agree with you that perfect mirroring is not something that comes "out of the box" with Plesk.

However, to be honest, your approach to load-balancing/high-availability/mirroring is flawed in the design structure to start with.

In essence, one can create perfect mirroring by

- one MySQL cluster: MySQL servers and/or MariaDB servers (note that I mention MariaDB, because that is somewhat simpler to setup)
- one fileshare, on which all relevant files and directories are placed: files/directories can be shared amongst master/slave

and that is a rough indication of how to set up such a mirroring network.

Note that the "fileshare" can be replaced with a solution based upon DRBD, Heartbeat and Pacemaker, but that is rather complicated to setup.

Nevertheless, even though the above more or less suits your requirements, all of the above does not fit efficiency requirements.

In some way, the replication of Plesk instances on two instances is more or less a waste of resources, in both the cases of load-balancing and HA.

The "slave" server can be made more efficient by going back to the bare minimum:

- one-time rsync from master to slave: relevant web files and directories
- recurring rsync from master to slave: pull from slave via crontasks, to ascertain identical files/directories structure
- making use of Redis caching on the master: Redis server instance on the master, allowing slave to pull real-time data
- making use of the Nginx mirror directive on the master: supported in Plesk 17.8.11 (note: in older versions of Plesk, a custom Nginx binary is required)
- using a custom mail server on the slave, with a proper DNS setup (read: two MX records with equal weight, to start with)

and so on and so on, the above is not an exhaustive summary at all!

Sure, the above seems to be more work than a "fileshare" or a "DRBD/Heartbeat/Pacemaker based solution", but it actually is not.

The most easy way to create load-balancing/high-availibility is still: work in the cloud!

For instance, when using Azure cloud, everything can be achieved already and/or is already automated, at a relatively low cost.


In conclusion, it would be recommended to have a different look at what you are trying to achieve and, preferably, consider to change to the (Azure) cloud.

Hope the above helps a bit.

Kind regards.........

PS I am recommending Azure cloud, due to its nice portal with an excellent GUI.

PS2 If you want to, you can always send me a private message!

 

[Jean-Luc GARNIER]

Thanks a lot trialotto for such a detailed answer! This really helps the dummy I am in system architecture!

By the way, I need to reword to ensure I've clearly got your point:

• I only have 2 physical servers which are supposed to run Wordpress, wiki, nextCloud + future web applications. Let's call them Master (S1) and Slave (S2).
• One SQL cluster: do you mean I should subscribe for some additional servers to run a "shared" SQL server (S3) which will provide database services to the apps above?
• One fileshare: do you mean I should subscribe for some dedicated storage (HDD4) where I will point all applications? So all applications running on S1/S2 would call S3 for database services and store their files on HDD4?

As I'm still the dummy I pretend to be, would I be able to use Plesk to administrate both Master and Slave? This was a real help when I only had one server and I'm still afraid of flying on my own... ;-)

By the way, we cannot use cloud services, whether they're from Microsoft, Google or Amazon as we cannot export any data out of France...

Thanks again for your advices!

 

[trialotto]

@Jean-Luc GARNIER

This statement

By the way, we cannot use cloud services, whether they're from Microsoft, Google or Amazon as we cannot export any data out of France...

is not correct: you can use cloud services, it is not about exporting data from France.

I am assuming that you are referring to GDPR? Well, if you are, do not worry.......it is only essential to have "customer agreement", a good "privacy policy" (read: customers are entitled to know what kind of data is used, stored and how and for how long) and servers and/or storage in the European Union.

Even if you are referring to local law, note that the European GDPR will supersede any form of local law dealing with the same or similar.

Finally, note that Azure has dedicated datacenters in France: Azure France.

With respect to

By the way, I need to reword to ensure I've clearly got your point:

• I only have 2 physical servers which are supposed to run Wordpress, wiki, nextCloud + future web applications. Let's call them Master (S1) and Slave (S2).
• One SQL cluster: do you mean I should subscribe for some additional servers to run a "shared" SQL server (S3) which will provide database services to the apps above?
• One fileshare: do you mean I should subscribe for some dedicated storage (HDD4) where I will point all applications? So all applications running on S1/S2 would call S3 for database services and store their files on HDD4?

As I'm still the dummy I pretend to be, would I be able to use Plesk to administrate both Master and Slave? This was a real help when I only had one server and I'm still afraid of flying on my own... ;-)

I would kindly ask you to send me a private message, can you do so?

Regards............

 

[virtubox]

Thanks a lot trialotto for such a detailed answer! This really helps the dummy I am in system architecture!

By the way, I need to reword to ensure I've clearly got your point:

• I only have 2 physical servers which are supposed to run Wordpress, wiki, nextCloud + future web applications. Let's call them Master (S1) and Slave (S2).
• One SQL cluster: do you mean I should subscribe for some additional servers to run a "shared" SQL server (S3) which will provide database services to the apps above?
• One fileshare: do you mean I should subscribe for some dedicated storage (HDD4) where I will point all applications? So all applications running on S1/S2 would call S3 for database services and store their files on HDD4?

As I'm still the dummy I pretend to be, would I be able to use Plesk to administrate both Master and Slave? This was a real help when I only had one server and I'm still afraid of flying on my own... ;-)

By the way, we cannot use cloud services, whether they're from Microsoft, Google or Amazon as we cannot export any data out of France...

Thanks again for your advices!

Hello,
if you aleady have SQL cluster, you can easily use it for your applications by adding it as an external MySQL server in Plesk. I have already setup a MariaDB Galera Cluster with 5 nodes with haproxy as a load-balancer before adding it as an external MySQL server in Plesk, this way I was able to manage all databases directly from the Plesk interface.

And for files, you can use a solution like GlusterFS to sync your site directory on your servers, and you can still mount an additional storage using NFS.

 

[trialotto]

@Jean-Luc GARNIER and/or @virtubox

if you aleady have SQL cluster, you can easily use it for your applications by adding it as an external MySQL server in Plesk. I have already setup a MariaDB Galera Cluster with 5 nodes with haproxy as a load-balancer before adding it as an external MySQL server in Plesk, this way I was able to manage all databases directly from the Plesk interface.

OR simply use

- (relatively) low cost,
- managed by default,
- high-availability by default,
- fully scalable,

Azure database for MySQL instances?

Yes, I would recommend taking the easy way, as opposed to setting up and maintaining your own cluster of MySQL servers, with considerable limitations.

To be honest, it is just a thought for those wanting to spend time on service for customers, instead of getting hands dirty with relatively complex cluster setups.

And for files, you can use a solution like GlusterFS to sync your site directory on your servers, and you can still mount an additional storage using NFS.

The same applies for the above, the easy ways of doing replication are:

- getting a fileshare (as present in Azure) or some (virtual) disks that can be attached to virtual machines: share the space and only pay for what you use (!), OR

- use a simple tool like lsyncd: a native package to many OSes, making life of replication very easy,

and again, to be honest, it is just a thought.

Note that fileshares and/or storage space comes in various "flavours": one does not have to replicate, the cloud datacenters already replicate (read: within the datacenter, across datacenters, globally and so on) at very low cost.......... one simply has to choose a flavour.


Actually, since I am implicitly talking about Azure, a quick outline of what you have to do:

1) initiate an out-of-the-box Load Balancer: low priced or even free,

2) create two Linux VMs: they can be very, very small, so prices are again relatively low and Linux VMs are even cheaper than Windows VMs,

3) install Plesk on one Azure VM, configure it to your liking and make a snapshot from it, in order to launch a second VM based upon the template of the first Azure VM,

4) initiate a fileshare (one fileshare is max 5TB, but one can easily add multiple fileshares), choose the flavour: LRS, ZRS, GRS (all types of replication)

5) add the fileshare to both Azure VMs, so they are essentially sharing the same filesystem, '

6) add additional VMs to your liking (easy: you already have a template) OR scale the existing VMs (recommended, it is an automatic process),

7) configure the load-balancer to point to both VMs, on which an identical Plesk instance is hosted,

8) when desirable, create additional VMs which will function as separate (external) MySQL server OR use Azure database for MySQL (a bit more expensive, but you never have to do anything, it is well managed and well behaved)

9) add the VMs and the Azure database for MySQL to the same VNet (virtual network), for the sake of performance and security

10) add a NSG (Network Security Group)

and investigate all other possibilities that Azure offers and that might be valuable for you.


In short, why puzzling for days, if the whole setup (see steps 1 - 10) can be done in less than 30 mins (that is, if you are familiar with Azure).

Again, it is just a thought and just a tip.

Hope the above helps.

Kind regards............

 

[Jean-Luc GARNIER]

Hi triolotto,

We hopefully don't look at you as an Azure sales rep'! By the way, your description really seduces me but we don't have any Azure skills so far.

A bit of context now: as a SME (more on the S side), we just want to focus on our core business and secure our infrastructure. We've used a single Virtual Private Server so far to host our website + nextCloud + wiki. We've suffered from a major service issue last January and understood our infra was really too weak. We then decided to hand over the infrastructure overhaul to a subcontractor who advised to build a (2 servers + Load balancer) infra to secure our daily work.

Being already an OVH customer, we chose to stick with this hosting provider and subscribed 2 dedicated servers + LB. I asked our contractor to keep Plesk running on both machines so I can manage them easily once their mission is over but he didn't manage to get both Plesk and mirroring between servers at the same time, saying phpmyadmin + rsync would break Plesk's configuration on both servers.

Back to your description: I understand that we should, whether it's on our own private servers or on any cloud (keeping Azure as a good example)

• install our applications (Wordpress, nextCloud, wiki) + Plesk on 1 VM/private server and duplicate the whole stuff to the 2nd VM/server
• create a single SQL server which will serve all applications from either VM1 or VM2
• create a fileshare (additional HDD) and have all applications pointing to it and use automated backup (or mirror) tools to secure it

With respect to this architecture, here are my new questions:

• does duplicated (Plesk + tools) to the 2nd server keep the same Plesk licence, settings, config database and so on? cf. your point 3
• if I add any new app to the 1st VM (let's says i.e. a CRM application), would I have to install and configure it onto the 2nd VM manually? cf. your point 7
• if I use Plesk to create a sub-domain, would I have to create the same sub-domain onto the 2nd VM manually?
• does a single SQL server create a SPOF?

As you can see, I'm a bit puzzled by this kind of architecture, which I've never used. Would you be kind enough to point me to some literature on "pure cloud infra"? I really need to find some drawings now, as a good picture is worth a thousand words...

Thanks to all of you for the help!

 

[trialotto]

@Jean-Luc GARNIER

I am going to send you a private message, to be better able to answer your questions and give you a rough outline on what to do next.

Nevertheless, I can take this opportunity to comment on your last post.

With respect to

A bit of context now: as a SME (more on the S side), we just want to focus on our core business and secure our infrastructure. We've used a single Virtual Private Server so far to host our website + nextCloud + wiki. We've suffered from a major service issue last January and understood our infra was really too weak. We then decided to hand over the infrastructure overhaul to a subcontractor who advised to build a (2 servers + Load balancer) infra to secure our daily work.

I have to state that you have to ask to right questions in order to get the right answers (and that certainly applies to your subcontractor!).

Simply stated, some question leading to the answer "2 server + Load Balancer" is probably the wrong question in your case.

In essence, the above mentioned answer is mixing up two solutions: high-availability (via load-balancing) and data security (via data replication across servers).

As a general rule of thumb, any high-availability solution with only 2 nodes (read: servers) is a bad solution, the same applies to load-balancing solutions.

Furthermore, data security is not necessarily related to mirroring (read: real-time replication), it is often sufficient to create (regular) static backups.

In short, you have to ask yourself

- why you should want any of the common solutions (data replication, high-availability, scaling, etc.)
- what you can achieve with specific solutions and/or server infrastructure
- what the common pitfalls are when using a specific solution and/or server infrastructure

and so on.

For instance, your current preference for "2 servers + load balancer" will fail all reasonable conditions, since it is not scalable: any burst load and/or a considerable growth in business and corresponding data will make your preferred solution fail AND a change in the chosen infrastructure is fairly difficult (for instance, when choosing to replicate data across nodes, each node has to be increased in size when the amount of data increases).

Also, never forget the costs of a solution AND never forget opportunity costs (for instance, costs of resources that are not being used).

The above remark about costs is a perfect bridge towards the statement

Being already an OVH customer, we chose to stick with this hosting provider and subscribed 2 dedicated servers + LB. I asked our contractor to keep Plesk running on both machines so I can manage them easily once their mission is over but he didn't manage to get both Plesk and mirroring between servers at the same time, saying phpmyadmin + rsync would break Plesk's configuration on both servers.

since you are essentially paying someone for a task he cannot complete (!) AND you are having double costs (in many aspects) at a not-so-good hosting provider.

In essence, you are running a very expensive setup that does not work. Bump! That could have gone better.


Having said all of the above, I can imagine that you know by now that it is not good to answer your questions, as written down in your last post.

For that reason, I am sending you a private message.

And then we can go to the drawing board, in order to define some solution that actually works for you and is available within your budget.


Is that a good idea?

Kind regards..........

 

[Jean-Luc GARNIER]

Yes, I understand I haven't learnt enough on this subject to have a good understanding on basic requirements... :-( By the way, hiring a sub-contractor is supposed to avoid having to become an expert in any field, provided the sub-contractor IS the expert you suppose he is... ;-)

OK for a private exchange from now and thanks again for sharing your expertise!