Issue Umlauts broken in mail forwards but not in mailbox

[ufreier]

Server operating system version

Ubuntu 22.04.1 LTS

Plesk version and microupdate number

Plesk Obsidian Version 18.0.48

Dear Community,

I don't even know if this is a Plesk, Postfix or Dovecot problem but I have to start debugging anywhere and because the server is managed by Plesk, I start here.

Setup:

I set up a domain's mailbox and therein a mail forward to another mail address that is hosted local on the same server. I didn't choose not to keep forwarded mails (might sound a bit strange but I only have German language and translated it back) so I receive each mail to this mail address twice, once in the mailbox itself and also in the mailbox of the forward. Spam and virus protection are disabled on both mailboxes.

Issue:

In mails that go to directly to the mailbox German umlauts are correctly displayed (and if the mail file is opened on the server in vi it shows right coding e.g. "Stra<DF>e" for "Straße") but in the forwarded mails the umlauts are broken (e.g. "Stra�e"). So it seems that on forwarding the mail body has been altered and that shouldn't be IMO. All locales on the server are en_US.UTF-8.

I'll appreciate any hint where I can search for further advices!

Many thanks, Uwe

 

[mow]

Is that a literal � encoded there or is it just a display placeholder for Schei� Encoding?

 

[ufreier]

Its literal as it is copied via clipboard from less/vi into herein (same in notepad e.g.). In a terminal in less/vi it's depicted as:

All umlauts are displayed identically so a substitution does not make sense, while php utf8_encode / decode does not produce useful output.

 

[GwenDragon]

@ufreier Check source of email you received and the one you received after redirect.
Would be interesting if that is which encoding (Quoted Printable, 8bit, 7bit or something else) and which charset (windows-1252, latin-1, utf8 or what).

 

[ufreier]

There is neither an encoding nor a MIME / Content-type or similar given in the mail headers of the affected mails. Those mails result from a web application or a middleware and look very rudimentary in general.

 

[GwenDragon]

In bash i tested my server, send a message by program mail with body äöüßÄÖÜẞ to a account which is redirected on my server.
In this case of rudimentary MTA, a charset and encoding header is missing. The sent mail and the redirected have both the same mail body and are raw UTF-8 bytes.

 

[GwenDragon]

Tested a redirect to T-Online account and the redirected mail had the same body as the sent, no changes.

I suggest, if the mail you send is from your own web app or script on server, add charset and content-encoding to mail header.
If the mail is not send by you, contact mail sender that the mil is wrong encoded.

Good luck.

PS: Some mail clients on PCs guess the missing charset and can fail to display german umlauts correct, in this case try to switch in mail cleint charset for display to Latin-1 or Windwos 1252.

 

[ufreier]

Exactly this behaviour I know from our old server (Plesk 12.5, Ubuntu 14.04, Postfix 2.11 - okay, it's very old) but the affected mails are processed without any problems there (no changes in mail body of redirects). Migrating to Obsidian 18.048, Ubuntu 22.04, Postfix 3.6 I face the problems described above the first time. Unfortunately I don't have any control over the mails, I only have to parse their contents into a database and preserving the umlauts is important therefore.

 

[GwenDragon]

If the byte sequence for ß is 0xDF in mail seems to be encoded Latin-1 (ISO 8859-1).
In UTF-8 it is 0xC3 0x9F
In UTF-16 it is 0x00DF

Unfortunately I don't have any control over the mails,

Sad. You could let some programs "guess" the encoding of mail body.
Some programs are listed here:

uchardet

HowTo: Check and Change File Encoding In Linux - ShellHacks

Сheck and change file's encoding from the command-line in Linux. Convert text files between different charsets. CP1251, UTF-8, ISO-8859-1, ASCII.

www.shellhacks.com

encguess - guess character encodings of files

docs.mojolicious.org

And the xou could recode the mail body and parse.
I am out of ideas now.

 

[ufreier]

Many thanks for these information! Maybe I can code a converter this way.