Resolved Seafile running- but "Failed to login: Internal server error" when trying to log in

[DenizGelion]

Server operating system version

Ubuntu 20.04.4 LTS (GNU/Linux 5.4.0 x86_64)

Plesk version and microupdate number

Version 18.0.53 Update #2

Hi there,

all of the sudden yesterday our seafile server just stopped working- it got installed alongside Plesk Premium mail, however all of the sudden our clients can't log in anymore. I've restarted the server multiple times, each service for each domain, no errors, the service is running (confirmed with systemctl status [email protected]", however the seahub-page is showing a "Internal server error", aswell as the client app.

How can I find out why it's not working?

Thanks in advance.

 

[Peter Debik]

Hi @DenizGelion, thank you for posting. Could you please check the log entries in /var/www/vhosts/<your domain>/logs/seahub.log for more details on the error?

 

[DenizGelion]

Hi Peter,

thanks for the quick reply- here are the results of both seahub.log and seafile.log where it seems a bit fishy:

seafile.log:
[02/18/22 17:14:47] http-server.c(181): fileserver: worker_threads = 10 [02/18/22 17:14:47] http-server.c(196): fileserver: fixed_block_size = 8388608 [02/18/22 17:14:47] http-server.c(211): fileserver: web_token_expire_time = 3600 [02/18/22 17:14:47] http-server.c(226): fileserver: max_indexing_threads = 1 [02/18/22 17:14:47] http-server.c(241): fileserver: max_index_processing_threads= 3 [02/18/22 17:14:47] http-server.c(263): fileserver: cluster_shared_temp_file_mode = 600 [02/18/2022 05:14:56 PM] start to serve on pipe client [02/18/22 17:54:25] http-server.c(181): fileserver: worker_threads = 10 [02/18/22 17:54:25] http-server.c(196): fileserver: fixed_block_size = 8388608 [02/18/22 17:54:25] http-server.c(211): fileserver: web_token_expire_time = 3600 [02/18/22 17:54:25] http-server.c(226): fileserver: max_indexing_threads = 1 [02/18/22 17:54:25] http-server.c(241): fileserver: max_index_processing_threads= 3 [02/18/22 17:54:25] http-server.c(263): fileserver: cluster_shared_temp_file_mode = 600 [02/18/22 18:09:23] http-server.c(181): fileserver: worker_threads = 10 [02/18/22 18:09:23] http-server.c(196): fileserver: fixed_block_size = 8388608 [02/18/22 18:09:23] http-server.c(211): fileserver: web_token_expire_time = 3600 [02/18/22 18:09:23] http-server.c(226): fileserver: max_indexing_threads = 1 [02/18/22 18:09:23] http-server.c(241): fileserver: max_index_processing_threads= 3 [02/18/22 18:09:23] http-server.c(263): fileserver: cluster_shared_temp_file_mode = 600 [02/18/2022 06:09:33 PM] start to serve on pipe client [02/18/2022 06:09:33 PM] start to serve on pipe client [02/22/22 13:49:20] http-server.c(181): fileserver: worker_threads = 10 [02/22/22 13:49:20] http-server.c(196): fileserver: fixed_block_size = 8388608 [02/22/22 13:49:20] http-server.c(211): fileserver: web_token_expire_time = 3600 [02/22/22 13:49:20] http-server.c(226): fileserver: max_indexing_threads = 1 [02/22/22 13:49:20] http-server.c(241): fileserver: max_index_processing_threads= 3 [02/22/22 13:49:20] http-server.c(263): fileserver: cluster_shared_temp_file_mode = 600 [02/22/2022 01:49:30 PM] start to serve on pipe client [02/22/2022 01:49:46 PM] start to serve on pipe client [02/22/2022 01:49:47 PM] start to serve on pipe client [02/22/2022 01:50:12 PM] start to serve on pipe client [02/22/2022 01:56:08 PM] start to serve on pipe client [03/07/22 21:55:45] socket file exists, delete it anyway [03/07/22 21:55:45] http-server.c(181): fileserver: worker_threads = 10 [03/07/22 21:55:45] http-server.c(196): fileserver: fixed_block_size = 8388608 [03/07/22 21:55:45] http-server.c(211): fileserver: web_token_expire_time = 3600 [03/07/22 21:55:45] http-server.c(226): fileserver: max_indexing_threads = 1 [03/07/22 21:55:45] http-server.c(241): fileserver: max_index_processing_threads= 3 [03/07/22 21:55:45] http-server.c(263): fileserver: cluster_shared_temp_file_mode = 600 [04/08/22 01:55:48] socket file exists, delete it anyway [04/08/22 01:55:48] http-server.c(181): fileserver: worker_threads = 10 [04/08/22 01:55:48] http-server.c(196): fileserver: fixed_block_size = 8388608 [04/08/22 01:55:48] http-server.c(211): fileserver: web_token_expire_time = 3600 [04/08/22 01:55:48] http-server.c(226): fileserver: max_indexing_threads = 1 [04/08/22 01:55:48] http-server.c(241): fileserver: max_index_processing_threads= 3 [04/08/22 01:55:48] http-server.c(263): fileserver: cluster_shared_temp_file_mode = 600 [04/08/2022 01:55:48 AM] ../common/seaf-db.c(707): Failed to connect to MySQL: Can't connect to MySQL server on '127.0.0.1' (115) [04/08/2022 01:55:48 AM] ../common/seaf-db.c(707): Failed to connect to MySQL: Can't connect to MySQL server on '127.0.0.1' (115) [04/08/2022 01:55:48 AM] repo-mgr.c(3646): Failed to add branch. [04/08/2022 01:55:48 AM] seafile-session.c(349): Failed to create system default repo. [04/11/2022 05:21:25 PM] start to serve on pipe client [04/11/2022 05:21:32 PM] start to serve on pipe client [04/11/2022 05:26:23 PM] start to serve on pipe client [04/11/2022 05:31:24 PM] start to serve on pipe client [04/11/2022 05:36:23 PM] start to serve on pipe client [04/27/22 16:02:04] socket file exists, delete it anyway [04/27/22 16:02:05] http-server.c(181): fileserver: worker_threads = 10 [04/27/22 16:02:05] http-server.c(196): fileserver: fixed_block_size = 8388608 [04/27/22 16:02:05] http-server.c(211): fileserver: web_token_expire_time = 3600 [04/27/22 16:02:05] http-server.c(226): fileserver: max_indexing_threads = 1 [04/27/22 16:02:05] http-server.c(241): fileserver: max_index_processing_threads= 3 [04/27/22 16:02:05] http-server.c(263): fileserver: cluster_shared_temp_file_mode = 600

seahub.log:
This one was too big for the forum to handle, so I uploaded it here: 2023-08-16 11:44:45,994 [ERROR] django.request:132 handle_uncaught_exception Int - Pastebin.com

So it looks like it's not communicating with the database correctly- but I'm not quite sure how and why not.

 

[DenizGelion]

Reinstalling the extension would maybe help, but I see no option for that within Plesk- is there a workaround for this?

 

[DenizGelion]

Actually scratch everything, I've deleted both log files, restarted the services, and got fresh files with current data:

seafile.log
[08/16/23 16:16:01] socket file exists, delete it anyway [08/16/23 16:16:02] http-server.c(181): fileserver: worker_threads = 10 [08/16/23 16:16:02] http-server.c(196): fileserver: fixed_block_size = 8388608 [08/16/23 16:16:02] http-server.c(211): fileserver: web_token_expire_time = 3600 [08/16/23 16:16:02] http-server.c(226): fileserver: max_indexing_threads = 1 [08/16/23 16:16:02] http-server.c(241): fileserver: max_index_processing_threads= 3 [08/16/23 16:16:02] http-server.c(263): fileserver: cluster_shared_temp_file_mode = 600

seahub.log is actually empty. I have no idea why it suddenly stopped working.

I've rechecked the services, and altough Plesk states it's running, I couldn't find the proper running process. Instead via htop I found a bunch of seafile server processes associated to domains, that do not even have Plesk Premium Email active, let alone seafile. Something seems to be going horribly wrong somewhere.

I could find a running seafile and seahub service for our own domain with the latest output being:

Aug 16 16:12:00 SERVER.DOMAIN.TLD seahub.sh[262742]: Stopping seahub ... Aug 16 16:12:00 SERVER.DOMAIN.TLD seahub.sh[262746]: pkill: killing pid 36084 failed: Operation not permitted Aug 16 16:12:00 SERVER.DOMAIN.TLD seahub.sh[262746]: pkill: killing pid 36090 failed: Operation not permitted Aug 16 16:12:00 SERVER.DOMAIN.TLD seahub.sh[262746]: pkill: killing pid 36091 failed: Operation not permitted Aug 16 16:12:00 SERVER.DOMAIN.TLD seahub.sh[262746]: pkill: killing pid 36092 failed: Operation not permitted Aug 16 16:12:00 SERVER.DOMAIN.TLD seahub.sh[262746]: pkill: killing pid 36093 failed: Operation not permitted Aug 16 16:12:00 SERVER.DOMAIN.TLD seahub.sh[262746]: pkill: killing pid 36094 failed: Operation not permitted Aug 16 16:12:00 SERVER.DOMAIN.TLD seahub.sh[262746]: pkill: killing pid 261954 failed: Operation not permitted Aug 16 16:12:00 SERVER.DOMAIN.TLD seahub.sh[262746]: pkill: killing pid 262101 failed: Operation not permitted Aug 16 16:12:00 SERVER.DOMAIN.TLD seahub.sh[262746]: pkill: killing pid 262106 failed: Operation not permitted Aug 16 16:12:00 SERVER.DOMAIN.TLD seahub.sh[262746]: pkill: killing pid 262110 failed: Operation not permitted Aug 16 16:12:00 SERVER.DOMAIN.TLD seahub.sh[262746]: pkill: killing pid 262115 failed: Operation not permitted Aug 16 16:12:00 SERVER.DOMAIN.TLD seahub.sh[262746]: pkill: killing pid 262116 failed: Operation not permitted Aug 16 16:12:01 SERVER.DOMAIN.TLD seahub.sh[262742]: Failed to stop seahub. Aug 16 16:12:05 SERVER.DOMAIN.TLD seahub.sh[262817]: LC_ALL is not set in ENV, set to en_US.UTF-8 Aug 16 16:12:05 SERVER.DOMAIN.TLD seahub.sh[262817]: Starting seahub at port 8001 ... Aug 16 16:12:10 SERVER.DOMAIN.TLD seahub.sh[262817]: Seahub is started Aug 16 16:12:10 SERVER.DOMAIN.TLD seahub.sh[262817]: Done. Aug 16 16:14:49 SERVER.DOMAIN.TLD seahub.sh[264992]: Stopping seahub ... Aug 16 16:14:49 SERVER.DOMAIN.TLD seahub.sh[265023]: pkill: killing pid 36084 failed: Operation not permitted Aug 16 16:14:49 SERVER.DOMAIN.TLD seahub.sh[265023]: pkill: killing pid 36090 failed: Operation not permitted Aug 16 16:14:49 SERVER.DOMAIN.TLD seahub.sh[265023]: pkill: killing pid 36091 failed: Operation not permitted Aug 16 16:14:49 SERVER.DOMAIN.TLD seahub.sh[265023]: pkill: killing pid 36092 failed: Operation not permitted Aug 16 16:14:49 SERVER.DOMAIN.TLD seahub.sh[265023]: pkill: killing pid 36093 failed: Operation not permitted Aug 16 16:14:49 SERVER.DOMAIN.TLD seahub.sh[265023]: pkill: killing pid 36094 failed: Operation not permitted Aug 16 16:14:49 SERVER.DOMAIN.TLD seahub.sh[265023]: pkill: killing pid 263855 failed: Operation not permitted Aug 16 16:14:49 SERVER.DOMAIN.TLD seahub.sh[265023]: pkill: killing pid 263860 failed: Operation not permitted Aug 16 16:14:49 SERVER.DOMAIN.TLD seahub.sh[265023]: pkill: killing pid 263861 failed: Operation not permitted Aug 16 16:14:49 SERVER.DOMAIN.TLD seahub.sh[265023]: pkill: killing pid 263862 failed: Operation not permitted Aug 16 16:14:49 SERVER.DOMAIN.TLD seahub.sh[265023]: pkill: killing pid 263863 failed: Operation not permitted Aug 16 16:14:49 SERVER.DOMAIN.TLD seahub.sh[265023]: pkill: killing pid 263864 failed: Operation not permitted Aug 16 16:14:50 SERVER.DOMAIN.TLD seahub.sh[264992]: Done. Aug 16 16:21:11 SERVER.DOMAIN.TLD seahub.sh[15966]: LC_ALL is not set in ENV, set to en_US.UTF-8 Aug 16 16:21:11 SERVER.DOMAIN.TLD seahub.sh[15966]: Starting seahub at port 8001 ... Aug 16 16:21:16 SERVER.DOMAIN.TLD seahub.sh[15966]: Seahub is started Aug 16 16:21:16 SERVER.DOMAIN.TLD seahub.sh[15966]: Done.

At the same time I managed to find the service, that isn't running, from my customer that is complaining:

root@server:/var/www/vhosts/DOMAIN.TLD/logs# systemctl status [email protected] ● [email protected] - Seafile hub Loaded: loaded (/etc/systemd/system/[email protected]; enabled; vendor preset: enabled) Active: failed (Result: signal) since Wed 2023-08-16 16:26:22 CEST; 5min ago Process: 16774 ExecStart=/var/www/vhosts/DOMAIN.TLD/seafile/seafile-server-latest/seahub.sh start 8106 (code=exited, status=0/SUCCESS) Process: 18468 ExecStop=/var/www/vhosts/DOMAIN.TLD/seafile/seafile-server-latest/seahub.sh stop (code=exited, status=1/FAILURE) Main PID: 16805 (code=killed, signal=KILL) Aug 16 16:26:20 SERVER.DOMAIN.TLD seahub.sh[18488]: pkill: killing pid 15983 failed: Operation not permitted Aug 16 16:26:20 SERVER.DOMAIN.TLD seahub.sh[18488]: pkill: killing pid 15984 failed: Operation not permitted Aug 16 16:26:20 SERVER.DOMAIN.TLD seahub.sh[18488]: pkill: killing pid 15985 failed: Operation not permitted Aug 16 16:26:20 SERVER.DOMAIN.TLD seahub.sh[18488]: pkill: killing pid 15986 failed: Operation not permitted Aug 16 16:26:20 SERVER.DOMAIN.TLD seahub.sh[18488]: pkill: killing pid 15987 failed: Operation not permitted Aug 16 16:26:22 SERVER.DOMAIN.TLD seahub.sh[18468]: Failed to stop seahub. Aug 14 09:04:41 SERVER.DOMAIN.TLD systemd[1]: Stopping Seafile hub... Aug 14 09:04:41 SERVER.DOMAIN.TLD systemd[1]: [email protected]: Main process exited, code=killed, status=9/KILL Aug 14 09:04:47 SERVER.DOMAIN.TLD systemd[1]: [email protected]: Failed with result 'signal'. Aug 14 09:04:47 SERVER.DOMAIN.TLD systemd[1]: Stopped Seafile hub. Aug 16 13:52:27 SERVER.DOMAIN.TLD systemd[1]: Starting Seafile hub... Aug 16 13:52:32 SERVER.DOMAIN.TLD systemd[1]: Started Seafile hub. Aug 16 16:09:36 SERVER.DOMAIN.TLD systemd[1]: Stopping Seafile hub... Aug 16 16:09:36 SERVER.DOMAIN.TLD systemd[1]: [email protected]: Main process exited, code=killed, status=9/KILL Aug 16 16:09:37 SERVER.DOMAIN.TLD systemd[1]: [email protected]: Control process exited, code=exited, status=1/FAILURE Aug 16 16:09:37 SERVER.DOMAIN.TLD systemd[1]: [email protected]: Failed with result 'signal'. Aug 16 16:09:37 SERVER.DOMAIN.TLD systemd[1]: Stopped Seafile hub. Aug 16 16:09:41 SERVER.DOMAIN.TLD systemd[1]: Starting Seafile hub... Aug 16 16:09:46 SERVER.DOMAIN.TLD systemd[1]: Started Seafile hub. Aug 16 16:10:25 SERVER.DOMAIN.TLD systemd[1]: Stopping Seafile hub... Aug 16 16:10:25 SERVER.DOMAIN.TLD systemd[1]: [email protected]: Main process exited, code=killed, status=9/KILL Aug 16 16:10:26 SERVER.DOMAIN.TLD systemd[1]: [email protected]: Control process exited, code=exited, status=1/FAILURE Aug 16 16:10:26 SERVER.DOMAIN.TLD systemd[1]: [email protected]: Failed with result 'signal'. Aug 16 16:10:26 SERVER.DOMAIN.TLD systemd[1]: Stopped Seafile hub. Aug 16 16:10:30 SERVER.DOMAIN.TLD systemd[1]: Starting Seafile hub... Aug 16 16:10:36 SERVER.DOMAIN.TLD systemd[1]: Started Seafile hub. Aug 16 16:12:32 SERVER.DOMAIN.TLD systemd[1]: Stopping Seafile hub... Aug 16 16:12:32 SERVER.DOMAIN.TLD systemd[1]: [email protected]: Main process exited, code=killed, status=9/KILL Aug 16 16:12:33 SERVER.DOMAIN.TLD systemd[1]: [email protected]: Control process exited, code=exited, status=1/FAILURE Aug 16 16:12:33 SERVER.DOMAIN.TLD systemd[1]: [email protected]: Failed with result 'signal'. Aug 16 16:12:33 SERVER.DOMAIN.TLD systemd[1]: Stopped Seafile hub. Aug 16 16:12:33 SERVER.DOMAIN.TLD systemd[1]: Starting Seafile hub... Aug 16 16:12:38 SERVER.DOMAIN.TLD systemd[1]: Started Seafile hub. Aug 16 16:14:48 SERVER.DOMAIN.TLD systemd[1]: Stopping Seafile hub... Aug 16 16:14:56 SERVER.DOMAIN.TLD systemd[1]: [email protected]: Main process exited, code=killed, status=9/KILL Aug 16 16:14:57 SERVER.DOMAIN.TLD systemd[1]: [email protected]: Failed with result 'signal'. Aug 16 16:14:57 SERVER.DOMAIN.TLD systemd[1]: Stopped Seafile hub. Aug 16 16:22:27 SERVER.DOMAIN.TLD systemd[1]: Starting Seafile hub... Aug 16 16:22:33 SERVER.DOMAIN.TLD systemd[1]: Started Seafile hub. Aug 16 16:26:20 SERVER.DOMAIN.TLD systemd[1]: Stopping Seafile hub... Aug 16 16:26:20 SERVER.DOMAIN.TLD systemd[1]: [email protected]: Main process exited, code=killed, status=9/KILL Aug 16 16:26:22 SERVER.DOMAIN.TLD systemd[1]: [email protected]: Control process exited, code=exited, status=1/FAILURE Aug 16 16:26:22 SERVER.DOMAIN.TLD systemd[1]: [email protected]: Failed with result 'signal'. Aug 16 16:26:22 SERVER.DOMAIN.TLD systemd[1]: Stopped Seafile hub.

I have absolutely no clue what is going on all of the sudden...

 

[DenizGelion]

Uninstalling Plesk Premium Email has immediately fixed the issue. Attempting to install Plesk Premium Email again now.

I have managed to reinstall everything successfully, thank god nothing broke in the process. No idea what caused this, but I could safely uninstall and reinstall Plesk Premium Mail.

 

[Peter Debik]

I think the root cause is that the service cannot connect to the database, so it is probably not the service, but a change brought about to the database server, e.g. a configuration change or upgrade?

But with all the follow-up actions this is now impossible to proof. I understand that the issue has been solved by reinstalling Plesk Premium EMail, is that correct?

 

[DenizGelion]

Hi Peter,

yes, sadly I had to do something quickly because people (obviously) wanted their access back asap. All I can say is, that besides regular Plesk updates, there have been lately no manual changes in any configuration files on the server. But yes, removing and re-adding Plesk Premium Email has resolved this issue. All files are still there, so this has not affected anyone.