Major Disaster - Advice Needed

[thedust2010]

Last night, the hard drive on our web server went KA-PUT. Nothing can be recovered. Luckily, I had been doing some nightly backups via cron jobs into .tar.gz files. What I have backed up is this:

- every webside folder inside of /home/httpd/vhosts in a separate archive
- /etc in its own archive
- /usr in its own archive
- /var in its own archive

I am not certain if a psadump was performed. I know for certain a mysqldump was performed every night with the following command:

/usr/local/psa/bin/mysqldump.sh >/dev/null 2>&1

In the /var/lib/psa/dumps directory I now have many files named mysql.daily.dump.*.gz, so I believe those are backed up. However, I see no files indicating a psadump has been performed. What should I be looking for?

I have taken all of the data backed up for /etc, /usr and /var and extracted them to the appropriate places on the server. Now, however, I can't log in to Plesk because I don't have access (everything has root permissions).

How good of shape am I in here? What steps should I take next to restore Plesk's default permissions without overwriting any data? Any help would be very appreciated, and if in fact you think you could assist with some hands on help, we can talk about possible compensation.

 

[thedust2010]

UPDATE: It looks like after restoring the /etc, /usr and /var folders that email is working correctly. However, I still can't access the Plesk folder because the correct permissions/groups are not assigned to the Plesk files. This is the error I get:

Forbidden
You don't have permission to access /login.php3 on this server.

Any advice?

 

[ShadowMan@]

Originally posted by thedust2010
I have taken all of the data backed up for /etc, /usr and /var and extracted them to the appropriate places on the server.

Did you specify the parameters to restore the ownership/permissions to the files in /usr ? Otherwise it would just make everything probably root : root

The /usr/local/psa should be:

drwxr-xr-x   12 root     psaadm       4096 Jul  8 06:12 psa

Within that:

drwxr-x---   18 psaadm   psaadm       4096 Jul  8 06:14 admin
drwxr-xr-x    2 psaadm   psaadm       4096 Jul 11 17:43 bin
drwxr-xr-x    4 psaadm   psaadm       4096 Jul  8 06:12 etc
drwxr-xr-x    3 psaadm   psaadm       4096 Nov 24  2004 install
drwxr-xr-x    3 psaadm   psaadm       4096 Jan  6  2005 installer
drwxr-xr-x    3 psaadm   psaadm       4096 Nov 24  2004 lib
drwxr-xr-x    4 psaadm   psaadm       4096 May  6  2005 logrotate
drwxr-xr-x    4 psaadm   psaadm       4096 Jul  8 06:14 PMM
drwxrwxrwt    6 psaadm   psaadm       4096 Nov 11 15:38 tmp
drwxr-xr-x    7 psaadm   psaadm       4096 Nov 19 20:30 var
-rw-r--r--    1 psaadm   psaadm         27 Jul  6 00:32 version

And it looks like just about everything within these directories have basically the same ownership/perms. Hope this helps! The 'psaadm' group should already exist from when you installed Plesk onto the new server.

 

[thedust2010]

Yes, it appears that everything is now owned by root : root. I will give this information to our system admins. Very helpful! Thanks!

 

[ShadowMan@]

You did not go into any details as to how you are doing your backups, but you really need to work up a procedure on how to extract the backup files with the original ownership/permissions settings.

If your current method of backup does not save that information, then you really need to change your backup strategy....

This is not like Windows zipping up files... Linux is very oriented towards directory and file 'ownership' (user and group ID), and 'permissions' (User Read/Write, Group Read/Write, Other Read/Write)

When doing a backup, it is usually important to make sure that all this information is backedup as well as the file content. And of course, upon restoral as well...

Just my 2 cents...

 

[thedust2010]

The error when accessing Plesk is different now:

ERROR: Unable to connect to database: get_admin_password() failed: file_get_contents() failed: 0: /usr/local/psa/admin/auto_prepend/auth.php3:81 psaerror(string "Unable to connect to database: get_admin_password() failed: file_get_contents() failed:

How do we verify that the Plesk databases are present and have correct permissions? What steps do you think we take next?

 

[ShadowMan@]

When you setup the new installation, you did use the same password(s) as there were on the original installation, right?

To verify databases, I personally would install phpMyAdmin and use that.

Since you have already restored the contents of /etc/ you have already overwritten the /etc/psa/.psa.shadow file with the one from the previous backup. If you setup the admin password on the new drive as different, then that is the problem.

If this is true, then replace the current contents of .psa.shadow with the admin password which you used upon the new installation. Hope this makes sense.

 

[ShadowMan@]

Oh, also related to ownership of files, make sure the /etc/psa/.psa.shadow file is owned by psaadm : psaadm

#ls -al /etc/psa
-rw-------    1 psaadm   psaadm          9 Nov 11 15:38 .psa.shadow
-r--------    1 psaadm   root        46030 Nov  9 23:25 psa.key
-rw-r--r--    1 root     root         1879 Aug  9 00:07 psa.conf
drwxr-xr-x    2 root     root         4096 May  6  2005 key.d
-r--------    1 psaadm   psaadm      28359 May  6  2005 psa.key.default
drwxr-xr-x    3 root     root         4096 Aug 12 03:55 vekey.d
-r--------    1 apache   apache         13 Nov 23  2004 .webmail.shadow

You are going to keep finding more and more directories and files with improper ownership/perms (like in /var/lib/mysql and other places), you really really need to see if you can re-extract the /etc , /var, /usr again WITH ownership and perms intact...

/var/lib/mysql directory and all files within should be ownership mysql : mysql

 

[thedust2010]

Thank you once again for your excellent help! When this episode is over, I want to know your name so I can name my first child after you...

 

[thedust2010]

FYI - The information you provided was enough to shed light on the subject for our server admins and everything appears restored! An absolute miracle! I can't thank you enough for all of your help. Thank you very much!

 

[thedust2010]

Still have some problems. I can't change the FTP password for any websites:

Unable to assign variables for PHostingForm: Unable to get hardquota state: usermng: Unable change password to user: chicproc
0: /usr/local/psa/admin/htdocs/domains/hosting/phosting_setup.php:290 psaerror(string "Unable to assign variables for PHostingForm: Unable to get hardquota state: usermng: Unable change password to user: chicproc")

Also, I'm unable to add any email addresses. You can see the errors I get here:

http://dustwurks.com/permission_error.gif

Any ideas what additional permissions I would need to grant for these problems?

 

[ShadowMan@]

Originally posted by me: You are going to keep finding more and more directories and files with improper ownership/perms (like in /var/lib/mysql and other places), you really really need to see if you can re-extract the /etc , /var, /usr again WITH ownership and perms intact...

My prediction is coming true.... From the screen shot, it is apparent that your /var/qmail structure ownership and perms are now the next thing to fix.

drwxr-xr-x   11 root    qmail        4096 Sep 28 01:46 qmail

Within /var/qmail -
-rw-r--r--    1 root     root           48 Jul  3 09:01 badsubj.txt
drwxr-xr-x    2 root     qmail        4096 Nov 16 03:52 bin
drwxr-xr-x    2 root     qmail        4096 Jul  8 05:27 boot
drwxr-xr-x    3 root     qmail        4096 Nov 17 22:47 control
drwxr-xr-x    8 root     qmail        4096 Nov 17 00:47 mailnames
drwxr-xr-x    2 root     qmail        4096 Jul  8 05:27 plugins
drwxr-x---   11 qmailq   qmail        4096 Jul  8 06:13 queue
drwxr-xr-x    2 qmailq   qmail        4096 Sep 28 01:46 .spamassassin
drwxr-xr-x    2 root     qmail        4096 Nov 17 00:47 users

Within /var/qmail/users -
-rw-------    1 root     root          691 Nov 17 00:47 assign
-rw-r--r--    1 root     root         3002 Nov 17 00:47 cdb
-rw-------    1 root     root          174 Nov 17 00:47 poppasswd
-rw-------    1 root     root          170 Jul 12 23:36 poppasswd.backup

Within /var/qmail/queue -
drwx------    2 qmails   qmail        4096 Nov 17 11:59 bounce
drwx------   25 qmails   qmail        4096 Jul  8 06:13 info
drwx------   25 qmailq   qmail        4096 Jul  8 06:13 intd
drwx------   25 qmails   qmail        4096 Jul  8 06:13 local
drwxr-x---    2 qmailq   qmail        4096 Jul  8 06:13 lock
drwxr-x---   25 qmailq   qmail        4096 Jul  8 06:13 mess
drwx------    2 qmailq   qmail        4096 Nov 21 18:08 pid
drwx------   25 qmails   qmail        4096 Jul  8 06:13 remote
drwxr-x---   25 qmailq   qmail        4096 Jul  8 06:13 todo

Within /var/qmail/mailnames -
drwx------    2 popuser  popuser      4096 Oct 17 02:39 domainname1.com
drwx------    2 popuser  popuser      4096 Oct 17 02:39 domainname2.com
lrwxrwxrwx    1 root     root           31 Jul  8 06:06 user_prefs -> /etc/mail/spamassassin/local.cf

Within /var/qmail/mailnames/domainname1.com -
drwx------    4 popuser  popuser      4096 Nov 21 18:20 mailusername1
drwx------    4 popuser  popuser      4096 Nov 21 18:20 mailusername2

Within /var/qmail/control - 
all *.pem files are -rw-------    1 qmaild   root 
all other files are -rw-r--r--    1 root     root 

Within /var/qmail/bin -
-r-sr-xr-x    1 root     qmail     1260052 May  6  2005 autoresponder
-r-xr-xr-x    1 root     qmail        9496 May  6  2005 bouncesaying
-r-sr-xr-x    1 root     root       984460 May  6  2005 cmd5checkpw
-r-xr-xr-x    1 root     qmail       15616 May  6  2005 condredirect
-r-xr-xr-x    1 root     qmail         126 May  6  2005 datemail
-r-xr-xr-x    1 root     qmail         114 May  6  2005 elq
-r-xr-xr-x    1 root     qmail        9464 May  6  2005 except
-r-xr-xr-x    1 root     qmail       15388 May  6  2005 forward
-r-xr-xr-x    1 root     qmail       18948 May  6  2005 maildir2mbox
-r-xr-xr-x    1 root     qmail        8976 May  6  2005 maildirmake
-r-xr-xr-x    1 root     qmail       17428 May  6  2005 maildirwatch
-r-xr-xr-x    1 root     qmail         179 May  6  2005 mailsubj
-r-xr-xr-x    1 root     qmail       33248 May  6  2005 matchup
-r-xr-sr-x    1 root     mail         4676 May  6  2005 mm_wrapper
-r-xr-xr-x    1 root     qmail         115 May  6  2005 pinq
-r-xr-xr-x    1 root     qmail       12848 May  6  2005 predate
-r-xr-xr-x    1 root     qmail       13384 May  6  2005 preline
-r-xr-xr-x    1 root     qmail         115 May  6  2005 qail
-r-xr-xr-x    1 root     qmail       11720 May  6  2005 qbiff
-r-xr-xr-x    1 root     qmail        9812 May  6  2005 qmail-clean
-r-xr-xr-x    1 root     qmail        6488 May  6  2005 qmail-getpw
-r-xr-xr-x    1 root     qmail       47552 May  6  2005 qmail-inject
-r-xr-xr-x    1 root     qmail       59363 May  6  2005 qmail_install_post.sh
-r-xr-xr-x    1 root     qmail       45944 May  6  2005 qmail-local
-r-xr-xr-x    1 root     qmail       20344 May  6  2005 qmail-lspawn
-r-xr-xr-x    1 root     qmail       15660 May  6  2005 qmail-newmrh
-r-xr-xr-x    1 root     qmail       11964 May  6  2005 qmail-newu
-r-xr-xr-x    1 root     qmail       20544 May  6  2005 qmail-pop3d
-r-xr-xr-x    1 root     qmail       13324 May  6  2005 qmail-popup
-r-xr-xr-x    1 root     qmail       16244 May  6  2005 qmail-pw2u
-r-xr-xr-x    1 root     qmail       13508 May  6  2005 qmail-qmqpc
-r-xr-xr-x    1 root     qmail       16140 May  6  2005 qmail-qmqpd
-r-xr-xr-x    1 root     qmail       25188 May  6  2005 qmail-qmtpd
-r-xr-xr-x    1 root     qmail       16712 May  6  2005 qmail-qread
-r-xr-xr-x    1 root     qmail         375 May  6  2005 qmail-qstat
-rwsr-sr-x    1 qmailq   qmail        4464 Sep 28 01:46 qmail-queue
-r-s--x--x    1 drweb    qmail      158652 May 31 18:53 qmail-queue.drweb
-r-s--x--x    1 qmailq   qmail       15960 May  6  2005 qmail-queue.orig
-r-xr-xr-x    1 root     qmail       70856 May  6  2005 qmail-remote
-r-xr-xr-x    1 root     qmail       15880 May  6  2005 qmail-rspawn
-rwxr-xr-x    1 qmailq   qmail      144219 Nov 16 03:41 qmail-scanner-queue.pl
-r-xr-xr-x    1 root     qmail       44280 May  6  2005 qmail-send
-r-xr-xr-x    1 root     qmail       17880 May  6  2005 qmail-showctl
-r-xr-xr-x    1 root     qmail       49016 May  6  2005 qmail-smtpd
-r-xr-xr-x    1 root     qmail        6780 May  6  2005 qmail-start
-r-xr-xr-x    1 root     qmail       10684 May  6  2005 qmail-tcpok
-r-xr-xr-x    1 root     qmail       10968 May  6  2005 qmail-tcpto
-r-xr-xr-x    1 root     qmail       28976 May  6  2005 qreceipt
-r-xr-xr-x    1 root     qmail       11916 May  6  2005 qsmhook
-r-xr-xr-x    1 root     root       979872 May  6  2005 relaylock
-r-xr-xr-x    1 root     qmail       10904 May  6  2005 sendmail
-r-sr-xr-x    1 root     root        20724 May  6  2005 smtp_auth
-r-xr-xr-x    1 root     qmail        7024 May  6  2005 splogger
-r-xr-xr-x    1 root     qmail       20228 May  6  2005 tcp-env
-r-xr-xr-x    1 root     qmail        2752 May  6  2005 true

There are simply too many other possible directory/files which may also be waiting in line to be next on the hit list...

 

[thedust2010]

Is there some sort of resource that would show all of the correct file permissions for all of Plesk? I would hate to have to keep coming back here for help...

We are definately looking into psadump for the best approach to our backups in the future... thank you once again.

 

[jamesyeeoc]

In the years I have been doing this, I have never found a comprehensive resource like you are saying. I always try to keep spare test server drives with different versions of Plesk handy so I can do comparisons.

That is an intriguing idea though, although with all the linux distros which are now 'supported' by Plesk, I would lack the resources to keep up on maintaining such a list for all the possible distro versions of Plesk....

Ah, I long for the old days when it was just RH OS's, much simpler to help people here on the forums too. Now there are too many posts where they don't say up front that they are running Debian, Suse, FreeBSD, or whatever, until about 3 rounds of posting later.....

 

[thedust2010]

Do you think there is any way, given my current circumstances, of reinstalling Plesk without losing email or database data? Can I do a Plesk dump now, reinstall Plesk entirely and restore that backed up information? It would be worth it to save myself from editing each and every file's permissions as we need to...

Thoughts? Am I making any sense?

 

[jamesyeeoc]

You could always try re-installing Plesk 'on-top' of what is already there. It is always good to do backups before trying anything major, but since there is obviously still problems existing, I would not just psarestore the current backup once the re-install is done (assuming it fixes your other problems).

If the re-install on-top fixes the ownership/permissions, then you could do a selective restore (search forum and/or Plesk docs) of just the client data, do not do the entire /var, /etc/, /usr structures like the last time. (I know you know, but this is for future readers who may be noobies.)

My other advice about backups:

Spend a little bit of money and try 4PSA TotalBackup and/or ClientBackup (or go for a bundle), or for a bit more money, check into the Acronis products (lots more money, but great online backup solution).

Forum member Poke often has discounts going on 4PSA products:

- poke :: www.jbwebhosting.com
- Products for Plesk :: 10% off all products & Bundles! www.jbwebhosting.com/4psa_jb.html

 

[thedust2010]

But if I do a reinstall of Plesk over the top, would I lose my email and database information or is this unclear?