MAJOR FLAW (HDD FULL)

[Jay Clearvoyant]

A few days ago I got this issue when i try to go the plesk admin panel on my server:

'ERROR: Zend_Db_Adapter_Exception: SQLSTATE[HY000] [2002] Connection refused

Additionally, an exception has occurred while trying to report this error: Zend_Exception
No entry is registered for key 'translate' (Abstract.php:144)'​

After doing some digging nothing seems to work.
I have tried every solution on the internet, nothing helped me.

When I run the command 'df -h' I get this:

Filesystem Size Used Avail Use% Mounted on
/dev/root 20G 20G 0 100% /
devtmpfs 16G 0 16G 0% /dev
tmpfs 16G 0 16G 0% /dev/shm
tmpfs 16G 9.9M 16G 1% /run
tmpfs 16G 0 16G 0% /sys/fs/cgroup
/dev/md3 1.8T 68M 1.7T 1% /home
tmpfs 3.2G 0 3.2G 0% /run/user/988
tmpfs 3.2G 0 3.2G 0% /run/user/0​

My server is has 2x 2TB harddrives how the hell does this happen?
It appears /dev/root is full, but how?

I can't even backup plesk so I can reinstall my server.
Any help would be really appreciated.

 

[Bitpalast]

Your /dev/root partition is very small. It is only 20 GB. This is far too small if this is mounted on /, because / needs to carry all the interesting paths that are derived from it like /usr and /var. Instead, your /home path is very large with lots of free disk space. It looks like you should reinstall your server and partition it differently, for example give 1.4 TB to /, 400 GB to /temp (a partition that is missing at the moment), and the rest to /home. /home does not need much space for a Plesk server. The exact partition sizes are of course your choice, but / should have the most space available.

 

[Jay Clearvoyant]

Your /dev/root partition is very small. It is only 20 GB. This is far too small if this is mounted on /, because / needs to carry all the interesting paths that are derived from it like /usr and /var. Instead, your /home path is very large with lots of free disk space. It looks like you should reinstall your server and partition it differently, for example give 1.4 TB to /, 400 GB to /temp (a partition that is missing at the moment), and the rest to /home. /home does not need much space for a Plesk server. The exact partition sizes are of course your choice, but / should have the most space available.

Thank you for the info, but it seems I cannot run the plesk backup utility, is there a way to backup my server?

 

[Bitpalast]

Your / partition is 100% full. You won't be able to do much on your system unless you clear a lot of disk space from it. It's not a specific Plesk problem, it is more a general system problem that will stop many other things from working, too.

 

[trialotto]

@Jay Clearvoyant

The answer to your question

Thank you for the info, but it seems I cannot run the plesk backup utility, is there a way to backup my server?

should be: yes, there is a work-around.

Open a SSH terminal session and do the following (steps are in chronological order)

1 - create the directories: /home/temp and /home/psa/dumps

2 - check for existing backup processes and kill those processes

- run the command: ps aux | grep backup
- run the command: kill -9 [PID] [PID2] ...

and this step really is required (!), to prevent that existing backup processes consume resources and block any of the steps mentioned below,

3 - use a command line editor, open /etc/psa/psa.conf and change lines 74 to 76 from

# Backups directory
DUMP_D /var/lib/psa/dumps
DUMP_TMP_D /tmp

to

# Backups directory - original values
# DUMP_D /var/lib/psa/dumps
# DUMP_TMP_D /tmp

# Backups directory - new values
DUMP_D /home/psa/dumps
DUMP_TMP_D /home/temp

4 - run the command: service psa restart

5 - run all backups, even if that requires some manual interaction: just make a "safety" backup

and that is essentially a dirty work-around.

It is not recommended to keep the new values in the /etc/psa/psa.conf, amongst others due to the fact that is not really update/upgrade compatible.

It is highly recommended that you change your disk mounts to fit the directory structure required by Plesk.

Actually, it seems to be the case that there is nothing wrong with your disk partitions, they are a bit awkward and clumsy, so to speak.

Just use a command line editor and mount the proper directories (like /opt, /var etc.) to the device /dev/md3 by editing the file /etc/fstab accordingly.

A small tip: create all necessary backup files in the /home directory with the work-around, before mounting directories. Afterwards, after a succesful mount of all the relevant directories, change the /etc/psa/psa.conf to it's default status by removing the "/home" based backup directory and uncommenting the default values.

Hope the above helps a bit.

Regards........

 

[Jay Clearvoyant]

@Jay Clearvoyant

The answer to your question



should be: yes, there is a work-around.

Open a SSH terminal session and do the following (steps are in chronological order)

1 - create the directories: /home/temp and /home/psa/dumps

2 - check for existing backup processes and kill those processes

- run the command: ps aux | grep backup
- run the command: kill -9 [PID] [PID2] ...

and this step really is required (!), to prevent that existing backup processes consume resources and block any of the steps mentioned below,

3 - use a command line editor, open /etc/psa/psa.conf and change lines 74 to 76 from

# Backups directory
DUMP_D /var/lib/psa/dumps
DUMP_TMP_D /tmp

to

# Backups directory - original values
# DUMP_D /var/lib/psa/dumps
# DUMP_TMP_D /tmp

# Backups directory - new values
DUMP_D /home/psa/dumps
DUMP_TMP_D /home/temp

4 - run the command: service psa restart

5 - run all backups, even if that requires some manual interaction: just make a "safety" backup

and that is essentially a dirty work-around.

It is not recommended to keep the new values in the /etc/psa/psa.conf, amongst others due to the fact that is not really update/upgrade compatible.

It is highly recommended that you change your disk mounts to fit the directory structure required by Plesk.

Actually, it seems to be the case that there is nothing wrong with your disk partitions, they are a bit awkward and clumsy, so to speak.

Just use a command line editor and mount the proper directories (like /opt, /var etc.) to the device /dev/md3 by editing the file /etc/fstab accordingly.

A small tip: create all necessary backup files in the /home directory with the work-around, before mounting directories. Afterwards, after a succesful mount of all the relevant directories, change the /etc/psa/psa.conf to it's default status by removing the "/home" based backup directory and uncommenting the default values.

Hope the above helps a bit.

Regards........

Alright I have tried what you said but it hasn't resolved my issues.

1. I created the directory /home/temp however I had to create /home/psa as well for /home/psa/dumps because /psa/ didn't exist.
2. I couldn't terminate the back process as the PID keeps changing.
3. Running the command service psa restart didn't work because I get 'cannot write because no space left'

I see no way to backup my files.... Are there other methods to backup my files?
Or is there a way to move the entire plesk installation to the other mounted directory?

 

[trialotto]

Alright I have tried what you said but it hasn't resolved my issues.

1. I created the directory /home/temp however I had to create /home/psa as well for /home/psa/dumps because /psa/ didn't exist.
2. I couldn't terminate the back process as the PID keeps changing.
3. Running the command service psa restart didn't work because I get 'cannot write because no space left'

I see no way to backup my files.... Are there other methods to backup my files?
Or is there a way to move the entire plesk installation to the other mounted directory?

@Jay Clearvoyant

It is very likely that you still have some running or scheduled backup tasks, so you will have to remove those.

Please run the command plesk db "select * from longtasks" and check wether you see unfinished tasks related to backups: if yes, delete them.

You can also run the delete command directly: plesk db "delete * from longtasks"

In addition, remove all files in the /usr/local/psa/PMM/tasks/ or /opt/psa/PMM/tasks/ directory.

All scheduled backup tasks should then be deleted, after which you can kill all existing backup tasks (if any still exist) and proceed with the steps provided.

Hope the above helps......

Regards

 

[Bitpalast]

3. Running the command service psa restart didn't work because I get 'cannot write because no space left'

The problem is, that there is no space left on the / partition. It is not possible to run the backup if the / space is exhausted, because none of the programs that are involved can be started.

 

[Jay Clearvoyant]

@Jay Clearvoyant

It is very likely that you still have some running or scheduled backup tasks, so you will have to remove those.

Please run the command plesk db "select * from longtasks" and check wether you see unfinished tasks related to backups: if yes, delete them.

You can also run the delete command directly: plesk db "delete * from longtasks"

In addition, remove all files in the /usr/local/psa/PMM/tasks/ or /opt/psa/PMM/tasks/ directory.

All scheduled backup tasks should then be deleted, after which you can kill all existing backup tasks (if any still exist) and proceed with the steps provided.

Hope the above helps......

Regards

I cannot execute that command because I get:
ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (111 "Connection refused")
exit status 1
I believe this is because mysql is not running and I CANNOT run it because there is no space in the server.

The problem is, that there is no space left on the / partition. It is not possible to run the backup if the / space is exhausted, because none of the programs that are involved can be started.

This is the main problem.
How can I move plesk to another mounted partition with allot of space WITHOUT losing any data since I CANNOT access mysql to backup my database.
If I had access to mysql I would've already dumped my database and simply reformatted the server into correct partitioning but I cannot dump my database at all.
Any idea?

 

[trialotto]

@Jay Clearvoyant

In response to the question

Any idea?

I can state: yes, ideas enough.

However, it seems to be the case that you are struggling with a whole lot of issues at the same time, which are unknown to us until you mention them.

Before making matters worse, please provide some relevant information: issues, issue description, output from relevant logs and so on.

Also, it is sometimes very valuable if you tell us more about the general setup: which OS, what kind of setup for MySQL (config files) and so on.

That way, you will allow us to actually help and find a concrete solution, if there is any.

Regards......

 

[Jay Clearvoyant]

@Jay Clearvoyant

In response to the question



I can state: yes, ideas enough.

However, it seems to be the case that you are struggling with a whole lot of issues at the same time, which are unknown to us until you mention them.

Before making matters worse, please provide some relevant information: issues, issue description, output from relevant logs and so on.

Also, it is sometimes very valuable if you tell us more about the general setup: which OS, what kind of setup for MySQL (config files) and so on.

That way, you will allow us to actually help and find a concrete solution, if there is any.

Regards......

CentOS 7 and I am using MariaDB as MySQL setup.

All I need to do to fix this is to move all contents of root partition to another mounted directory and then make that directory the root partition, will this work?
Anybody can gicw me any steps to accomplish this?

 

[Bitpalast]

You probably have a /tmp, /temp and /var/log directory decending from /, because these are not partitions according to your first post. I'd probably first go into the /var/log directory, download all archived log files (.tar.gz or .gz) and then delete all these archives. This will normally free a good chunk of space. Then I'd probably look into /var/www/vhosts/<subscription>/log directories and remove the archived logs from there, too. This will free more space. In your case, your subscriptions are descending from /, too, so, this will free space on the problematic partition. This might lead to a point where at least enough space is available to restart the database and Plesk service. If you can successfully do that, I'd next look into the backups of Plesk and remove all local backups that are no longer needed, because they are stored under a path that descends from /, too.

 

[virtubox]

CentOS 7 and I am using MariaDB as MySQL setup.

All I need to do to fix this is to move all contents of root partition to another mounted directory and then make that directory the root partition, will this work?
Anybody can gicw me any steps to accomplish this?

Hello , I guess you are using a Soyoustart Server with this partition schema ?
If yes, you can reboot your server in rescue mode to be able to copy your files from a partition to another one :

mkdir -p /mnt/root
mkdir -p /mnt/home

Then use fdisk -l to identify each partition, and mount them in the folder created previously .
it will probably be like :

mount /dev/md2   /mnt/home
mount /dev/root  /mnt/root

mkdir /mnt/home/dump
mv -f /mnt/root/var/lib/psa/dumps/*  /mnt/home/dump/

Then you should be able to reboot your server and your backup will be available in /home/dump/

But you will have to reinstall your server with a proper partitions schema, by replacing /home by /var , and with a "/" partition of at least 50GB.

Otherway, try to find what folders are using too much storage with the command :

du -sh *

 

[dj ramc]

What about in this scenario ? centos 7 ovh


i want to enlarge /dev/md2 .

regards

 

[dj ramc]

hello guys any suggestions ?
i want replace dev/md2 with de/md3 without any data lost ? is that possible ?

thanks by advance

 

[virtubox]

hello guys any suggestions ?
i want replace dev/md2 with de/md3 without any data lost ? is that possible ?

thanks by advance

Hello @dj ramc, with Plesk running on your server, I do not recommand you to make changes on a software RAID.
The easiest is probably to copy all your data on the backup storage available with your server and to reinstall it with a proper partition schema.

Regards,

 

[trialotto]

@dj ramc

There is a simple reason why an answer cannot be give easily: you cannot switch between dev/md2 and dev/md3 without any (complex) procedures.

Let me explain a bit.

In essence, it is madness to have to bulk of diskspace mounted in /home and have a very small amount of space in / - something went horrible wrong in disk partitioning: you now have 1.8T mounted in /home, which is a subdirectory of /.

There (probably) are not actual devices present, the output suggests that there are two virtual devices: this implies that "switching" amongst virtual devices (read: assigning more space to one virtual device) would require that the other virtual device is assigned less disk space - but you cannot mess with the /home directory, as it is crucial for the proper operating of your OS.

The only thing you can do is simply this: create a directory in / that is linked to a directory in /home - but that is not recommended, for various reasons.

I strongly agree with @virtubox and highly recommend that you reinstall the server with proper partitions.

Regards.......

 

[virtubox]

@dj ramc

In essence, it is madness to have to bulk of diskspace mounted in /home and have a very small amount of space in / - something went horrible wrong in disk partitioning: you now have 1.8T mounted in /home, which is a subdirectory of /.

This is the default disk partitionning on all OVH servers ...

 

[dj ramc]

thank you both for the answers.
when i try to make a backup by plesk it says no space left ?
is this a good idea to change backup dir in the configuration before making the backup?
How to change the location for Plesk backup files on a Linux server?

 

[trialotto]

This is the default disk partitionning on all OVH servers ...

@virtubox

I know that a lot of things are not as they should be at OVH............ but this is (on the one hand) not surprising and (on the other hand) completely madness.

Just like I said before.

Great, OVH delivers servers which require a lot of work for a well-seasoned sysadmin to get the partitions right......... is that the factual reality?

If that is indeed the case, then I really wonder............who is sane of mind AND still can justify to create two partitions (!) with the majority of the disk in a subdirectory???

Really, wouldn't it be easier for OVH and it's customers to have one big partition in / and - optionally - a simple mount of /home (if desired).

Actually, I am not so surprised by OVH..........but more by the mere fact that they are actually selling servers.

Just a thought for @dj ramc .........why not get a server elsewhere? I think that would be more easy and convenient for you!

Regards........