Out of Disk Space without any reason

[aless]

Hello,

I am facing a very strange problem on my server. It's about a Linux Ubuntu 14.04 running Plesk 12.

I have 30GB Disk Space. It should have more or less 50% of free space, but few hours ago it went suddenly out of space so harshly that Mysql and the panel cannot even start. In fact, df -h indicates that 100% of disk space is used.

So far, to avoid out of disk space issues I checked the /var/log folder for large files, or if some backup was running from the database. Simply removing large log files or broken/failed backups usually solved the out of space issue without further complication. Unfortunately now the scenario looks like totally different. There isn't any large file or broken backups to be removed and all seems to be like usual. ncdu indicates that the folders on the whole server use about 9GB. Where is the free space gone? What could use so much disk space, and how to find it?

Does anyone had a similar issue? Any help is really appreciated!
THANK YOU
a.

 

[SibProgrammer]

Can you provide the output of the following commands?
* df -h
* df -i
* du -sh /* (under root user)

 

[serverpoint_tom]

Hi,

Can you please confirm your Plesk backup location by running the following command
====
cat /etc/psa/psa.conf | grep -w DUMP_D
====

The expected out should be something like this:
===
DUMP_D /var/lib/psa/dumps
===

This directory contains server backups, backups of resellers, clients, domains, and daily MySQL dumps.
Object backups are placed according to their ownership.

You can check the size of this directory by the following command:
===============
cd /var/lib/psa/dumps

This will list the Files accourding to their size
du -k *| sort -nr | cut -f2 | xargs -d '\n' du -sh
===============

If you want to change the backup location, please follow the below kb article

http://kb.odin.com/en/5816

Regards,

 

[aless]

Thanks a lot for your help. I actually solved the issue by powering the server off and on. Disk space is now correctly reported (c. 9GB used and 20GB free), while before the free space was eaten to zero in a few minutes. That was also the main reason of my anxiety. By the way, /var/lib/psa/dumps contains mainly small mysql dumps. I attach a screenshot below, along with #df -h, #df -i and #du -sh outputs.

However, have you got any clue about the cause of such a behavior of the server? I can't explain this kind of redundancy, and am afraid not to be able to prevent this to happen again.

Best regards,
a.

 

[SibProgrammer]

We can only guess that something wrong had happened either filesystem or some process really ate all the space. Try to run "du -sh /*" next time before reboot. Best practices are saying that it's better to have separate partitions for /tmp, /var, /usr mount points.

 

[serverpoint_tom]

Hi Aless,

It would be great if you can run the following command,
==========
du -k *| sort -nr | cut -f2 | xargs -d '\n' du -sh
==========
Before rebooting the server, when similar issues are triggered in future. This is to find which files consumes the memory and thus we can trace the process which eats up the disk space.

Thank you,