Better hardware and pigz, yet worse backup performance

[rbstern]

I just finished a server migration, migrating subscriptions on the old server from Plesk 11, to a clean Plesk 12 install on the new server. One of the things I was most excited about was the potential for pigz to reduce the high server loads for my overnight backups. I'm getting the opposite result and don't understand why.

Original setup:

Plesk 11, CentOS 6.4, serving about 160 domains, running on an E3-1240, 8GB RAM, dual 1TB in software RAID 1 config.

New setup:

Plesk 12, Centos 6.7, same domains, running on dual cpu E5-2620's, 64GB RAM, dual 2TB in software RAID 1 config.

I have a monitoring script that checks 5 minute CPU loads every minute, and sends me emails when that load goes above 5. I am now getting routine high cpu usage emails during the backup process.

The finished backup file size is about 46GB. Old server did the backup in about 2.5 hours. New server is finishing in 2 hours, so slightly better (not particularly impressive, considering the additional resources available), but the new server is running at high server loads for much of this process.

 

[rbstern]

Bump. Hoping for some helpful direction.

 

[Oliver Meyer]

Hi,

please keep in mind that a load of 1 means that a single cpu core is working
to capacity. According to Intel, cpus with hyperthreading enabled can take
~25% more load than cpus without hyperthreading. So...

The E3-1240 has 4 cores (with HT) = max load of ~5.
The E5-2620 has 6 cores (with HT) = max load of ~7,5.

Please note, that the E3-1240 has a much better single-core performance than
the E5-2620. So it's normal that the load is higher on your new system, because
the E5-2620 needs more cores (more load) than the E3-1240 to do the same job.
Maybe you should go with another cpu like the E5-1650v3 or E5-1680v3 which
would be a real improvement over the E3-1240.

https://software.intel.com/en-us/articles/performance-insights-to-intel-hyper-threading-technology
http://cpuboss.com/cpus/Intel-Xeon-E5-2620-vs-Intel-Xeon-E3-1240
http://cpuboss.com/cpus/Intel-Xeon-E5-1650-v3-vs-Intel-Xeon-E3-1240
http://cpuboss.com/cpus/Intel-Xeon-E5-1680-v3-vs-Intel-Xeon-E3-1240

Best regards,

Oliver Meyer

 

[rbstern]

Oliver, thanks for the comments. I made a mistake in the above post. The new server is a dual cpu E5-2620.

On the new server, Pigz has 12 (with HT) cores to work with, vs. the 4 (with HT) cores of the old server.

 

[Oliver Meyer]

Hi,

it doesn't matter if you have one or two cpus - the load will be the same.
Your dual cpu setup is theoretical able to process the backup faster. Maybe
your RAID1 is the bottleneck, because you are reading and writing on a single
array of slow HDDs at the same time. You should monitor the disk usage (IOPS,
read and write speed etc.) during a backup.

 

[rbstern]

Oliver, thanks for your continued input.

When you say the "load will be the same," are you saying that Linux's reported CPU load is not an accurate estimator of how hard the server is working?

The RAID1 setup could be a bottleneck, but it seems unlikely: Writing 46GB, even to SATA drives, spread out over two hours doesn't seem like a likely bottleneck.

 

[Oliver Meyer]

I think you got me wrong. The maximum load your system can take is, because
you have two cpus, double (~15) of that, what I mentioned in my first post (~7,5).
The cpu load during your backup will stay at ~5 - no matter if you have two
or four cpus. So your backup process takes about 33% of the available cpu-power.
The load is higher on your new system because the single-core performance of the
E5-2620 is lower.

During your backup, there will not only 46GB be written to the array - much more
data will be read from the array - tens of thousands tiny files. And all that happens at
the same time. This mixed read/write task with tiny files will take the performance of
your SATA drives down. Under best conditions your array should be able deliver around
100 IOPS - not much, but common for SATA drives. Your array will deliver even less
IOPS during the backup.

 

[rbstern]

OK, I think I get it: I shouldn't use CPU load as a performance comparison indicator between two different servers. Performing the same tasks, my new, dual E5 might be working at a load of 5.0, yet still have capacity to handle more additional work than the single E3 at a 3.0 load.

Is that a fair statement?

And thanks for pointing out that the backup writes are only one part of what the drives are doing. Makes perfect sense. The new server has a lot of RAM. Should I be throwing more of that at disk caching?

 

[Oliver Meyer]

Exactly - you got it

The linux kernel is automatically using unused RAM for file caching. Your
old system had 8GB - now you have 64GB. That is a great improvement.
If you not already did it, you should spend time and RAM on tuning your
mysql-server.

 

[rbstern]

Anecdotally, I had a previous experience, with similar servers, in another migration situation involving a high volume vBulletin site on a dedicated server. Serving the same site, an E3 would bog down at high CPU loads, while a dual E5 was still very responsive at even higher CPU loads. Fits what you've described.

mySQL tuning is a bit more in my wheelhouse, and have already done quite a bit of tuning to take advantage of the additional RAM.

Thanks again! Very helpful discussion.