Resolved dovecot add zlib in a custom config file

[Werner99]

Hi all,

i use dovecot with zlib plugin to gz our mails, but i have a issue to set the setting and save it from overwriting when updates are getting installed.
Over the Weekend i updated plesk and it overwrite the dovecot.conf file which i was expecting, so i added the zlib to mail_plugins section again into the dovecot.conf file. I know that i can add a custom conf file in /etc/dovecot/conf.d, so i created a file called my_custom_file.conf and added into it

mail_plugins = zlib

but it doesn't get used, i get this when i list the config "doveconf -n"

# 2.3.7.2 (3c910f64b): /etc/dovecot/dovecot.conf
# Pigeonhole version 0.5.7.2 (7372921a)
doveconf: Warning: /etc/dovecot/conf.d/my_custom_file.conf line 1: Global setting mail_plugins won't change the setting inside an earlier filter at /etc/dovecot/dovecot.conf line 64 (if this is intentional, avoid this warning by moving the global setting before /etc/dovecot/dovecot.conf line 64)
doveconf: Warning: /etc/dovecot/conf.d/my_custom_file.conf line 1: Global setting mail_plugins won't change the setting inside an earlier filter at /etc/dovecot/dovecot.conf line 64 (if this is intentional, avoid this warning by moving the global setting before /etc/dovecot/dovecot.conf line 64)

any idea how i could fix this?

 

[Werner99]

no Help here? again a update and zlib missing again, then no mails are accessible!!! even with the my_custom_file.conf file, so how can i fix that or do i need to make a job to always overwrite the dovecot.conf file with a backup?

 

[mr-wolf]

I have no direct answer for you, but the key of your problem may well be the name of your conf.d/file
It needs to start with 2 digits and a dash
In this way you can influence to order.

This mechanism is often used in Linux like in /etc/rc5.d/ and /etc/nginx/conf.d/

Do NOT use 10- to 20- and 90- through 95- as they are reserved for Plesk.

Here's a snippet of /etc/dovecot.conf where /etc/dovecot/conf.d gets included.

# Customizable part of configuration gets included below. The filenames are
# first sorted by their ASCII value and parsed in that order. The 00-prefixes
# in filenames are intended to make it easier to understand the ordering.
#
# Prefixes 10 through 20 and 90 through 95 (inclusive) are reserved for Plesk.
# Please do not occupy them or edit files with these prefixes!
!include conf.d/*.conf

You can see the expanded configuration with dovecot -n
Happy hunting!!

 

[Werner99]

yes i know about 2 digits and i also don't use them in this case, it seems that the custom config file gets loaded but since the custom configs gets loaded at the end it seems the

mail_plugins = zlib

gets not used, i think easiest way would be that the default config just has already zlib support inside, because it is no issue when it is supported because it will get only used when you do a extra config to do the compression, else it will only support to read also mails which are already compressed.

Question - Enable lz4 compression on Dovecot

95-plesk-plugin.conf

plugin {
  zlib_save_level = 6
  zlib_save = gz
}

so currently i have really no idea how i could fix it without adding zlib after every update again to the mail_plugins, which is not good

 

[mr-wolf]

I tested this and I could not get rid of the warning by renaming the file.

You could use this script

cat /etc/cron.hourly/patch_dovecot

#!/bin/bash
grep -q 'mail_plugins = zlib' /etc/dovecot/dovecot.conf && exit 0
sed -i 's/^protocols = imap pop3/mail_plugins = zlib\n\n&/g' /etc/dovecot/dovecot.conf
/etc/init.d/dovecot restart

 

[Werner99]

hmm that was the last thing i wanted todo because it means every hour dovecot needs to get restarted, anyway thanks for suggestion

 

[mr-wolf]

No it doesn't
It will only get restarted if it has changed.
That's what the 2nd line is for.

 

[Werner99]

oh nice you are right but then it is maybe better to make a custom cron which checks the file every minute, i can't tell my colleagues that for maybe 1 hour the mail has no function after every update. So thank you for this great idea, thats better then nothing

 

[mr-wolf]

Those updates don't happen that often, but you can of course make it 5 minutes or so.
In my experience it's always best to be conservative about automatic procedures....
In other words... Don't make it a minute.

 

[Werner99]

since last week i think i had to modify the config 3 times , so not that often is currently not the case, but i agree every minute is a little bit to much

 

[mizar]

Hello,

I recommend to not modify dovecot.conf, but override required service in conf.d, i.e:

#/etc/dovecot/conf.d# cat 35-zlib.conf
plugin {
  zlib_save_level = 6
  zlib_save = gz
}

protocol imap {
  mail_plugins = quota imap_quota zlib
}

 

[Werner99]

the first part i have already in an extra conf file in conf.d, if this part

protocol imap {
  mail_plugins = quota imap_quota zlib
}

works then this would be the best solution, so to understand it correct this will override the "protocol imap" inside the dovecot.conf file?

But don't i need to have also zlib in general "mail_plugins"

# Enable Maildir++ quota support.
mail_plugins = $mail_plugins quota zlib

or is it enough if i have it just in "protocol imap"

Thanks for the help

 

[mizar]

Hello,

No, I means that just create the file /etc/dovecot/conf.d/35-zlib.conf with plugin configuration and with overridden protocol imap settings. No changes in dovecot.conf required

 

[Werner99]

yes i understand what you mean, if it works that way the question was don't i need the "zlib" also in the general part or is it really only needed for "protocol imap"
This is how my current dovecot.conf looks like, as you can see i have two times zlib inside

# Enable Maildir++ quota support.
mail_plugins = $mail_plugins quota zlib

and

protocol imap {
  mail_plugins = $mail_plugins imap_quota zlib
}

Complete dovecot conf

## Dovecot configuration file

# PLEASE DON'T EDIT ANYTHING IN THIS FILE! ANY CHANGES WILL BE LOST ON UPGRADE.
# Instead add your custom configuration to /etc/dovecot/conf.d/
# See the end of this file for details on customization.

# See example configuration files in /usr/share/doc/dovecot/example-config/
# See stock configuration files in /usr/share/doc/plesk-dovecot/dist-config/


# Space separated list of wanted authentication mechanisms.
# NOTE: See also disable_plaintext_auth setting.
auth_mechanisms = plain login digest-md5 cram-md5 apop

# Valid UID range should include popuser.
first_valid_uid = 30

# List of allowed characters in username. "'&" is added here to the default list.
auth_username_chars = abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ01234567890&.-_@'

# Plesk Maildirs layout.
mail_home = /var/qmail/mailnames/%Ld/%Ln
mail_location = maildir:/var/qmail/mailnames/%Ld/%Ln/Maildir

# Enable Maildir++ quota support.
mail_plugins = $mail_plugins quota zlib

plugin {
  # Take quota limits from maildirsize file. If it doesn't exist, don't enforce.
  quota = maildir:User quota
  # LDA/LMTP allows saving the last mail to bring user from under quota to
  # over quota, if the quota doesn't grow too high. Default is to allow as
  # long as quota will stay under 10% above the limit. Also allowed e.g. 10M.
  # 0 doesn't allow going over quota (compatibility with Courier-IMAP behavior).
  quota_grace = 0
}


service auth {
  # Allow userdb lookups for popuser.
  unix_listener auth-userdb {
    mode = 0600
    user = popuser
    group = popuser
  }
}

service stats {
  unix_listener stats-writer {
    user = popuser
  }
}

# PEM encoded X.509 SSL/TLS certificate and private key.
ssl_cert = </etc/dovecot/private/ssl-cert-and-key.pem
ssl_key =  </etc/dovecot/private/ssl-cert-and-key.pem


# Protocols support.

protocols = imap pop3

protocol imap {
  mail_plugins = $mail_plugins imap_quota
}

protocol pop3 {
  # POP3 UIDL format similar to the one used by Courier-IMAP.
  pop3_uidl_format = UID%u-%v
}

# Applicable workarounds for various client bugs.
imap_client_workarounds = delay-newmail
pop3_client_workarounds = outlook-no-nuls oe-ns-eoh


# Default private namespace setup compatible with Courier-IMAP.

namespace inbox {
  separator = .
  prefix = INBOX.
  inbox = yes

  mailbox Sent {
    auto = subscribe # autocreate and autosubscribe the Sent mailbox
    special_use = \Sent
  }
  mailbox Spam {
    auto = create # autocreate Spam, but don't autosubscribe
    special_use = \Junk
  }
  mailbox Drafts {
    auto = create
    special_use = \Drafts
  }
  mailbox Trash {
    auto = create
    special_use = \Trash
  }
}


# Logging verbosity and debugging.

# Log unsuccessful authentication attempts and the reasons why they failed.
#auth_verbose = yes

# In case of password mismatches, log the attempted password. Valid values are
# no, plain and sha1. sha1 can be useful for detecting brute force password
# attempts vs. user simply trying the same password over and over again.
#auth_verbose_passwords = no

# Even more verbose logging for debugging purposes.
#auth_debug = yes

# In case of password mismatches, log the passwords and used scheme so the
# problem can be debugged. Enabling this also enables auth_debug.
#auth_debug_passwords = yes

# Enable mail process debugging. This can help you figure out why Dovecot
# isn't finding your mails.
#mail_debug = yes

# Show protocol level SSL errors.
#verbose_ssl = yes


# Log format compatible with Plesk statistics collector.
mail_log_prefix = "service=%s, user=%u, ip=[%r]. "
pop3_logout_format = "rcvd=%i, sent=%o, top=%t/%p, retr=%r/%b, del=%d/%m, size=%s"
imap_logout_format = "rcvd=%i, sent=%o"

# Customizable part of configuration gets included below. The filenames are
# first sorted by their ASCII value and parsed in that order. The 00-prefixes
# in filenames are intended to make it easier to understand the ordering.
#
# Prefixes 10 through 20 and 90 through 95 (inclusive) are reserved for Plesk.
# Please do not occupy them or edit files with these prefixes!
!include conf.d/*.conf

# vim:ts=2 sts=2 sw=2 et:

 

[mizar]

Configration override should be enough (it allow to access gzipped messages for me). But in case you want to modify /etc/dovecot/dovecot.conf it's better to create a copy of config (i.e /etc/dovecot/dovecot2.conf) and use systemd unit override to pass this config to dovecot

 

[Werner99]

Yes if the override works then it will be perfect, I will try tonight first your solution.

 

[Werner99]

@mizar the solution works as it should i tested it yesterday night and all seems ok, thanks for the help

 

[rmeden]

I know this is an old thread, but I just ran into this issue myself on a new Plesk server, space mostly taken up by email.

The 35-zlib.conf solution does allow IMAP to view compressed emails, but new emails are not compressed without putting zlib in the dovecot.conf mail_plugins.

I removed it, kept the zlib_save_level stuff and a large received file was not compressed. (checked with the "file" command). Adding it to mail_plugins in dovecot.conf does cause the mail file to be compressed.

I don't think there's a solution outside of the cron job to keep putting it back. (no big rush as worst case emails received will just be uncompressed. imap will deal with both)

plesk-dovecot-2.3.21.1-2

 

[rmeden]

plesk-dovecot-2.3.21.1-2

I know this is an old thread, but I just ran into this issue myself on a new Plesk server, space mostly taken up by email.

@mizar 's 35-zlib.conf solution does allow IMAP to view compressed emails, but new emails were not compressed.

I was able to get them compressed, but updating mail_plugins in the dovecat.conf file, but there's a better way.

Here's my /etc/dovecot.conf/95-compress.conf file (it has to be after 90-plesk-sieve.conf)

#
# this causes emails to be stored compressed and uncompressed by imap
#
plugin {
  zlib_save_level = 6 # not needed, default is 9 with bz2
  zlib_save = gz
}
# override dovecat.conf
protocol imap {
  mail_plugins = $mail_plugins zlib
}

protocol imap {
  mail_plugins = $mail_plugins zlib
}

protocol lda {
  mail_plugins = $mail_plugins zlib
}

I tested it by confirming the file in the Mailbox directory is compressed with the "file" command.