TYPO3 Core

  /typo3/sysext/core/Classes/Mail/Mailer.php:108

$replyTo = MailUtility::getSystemReplyTo();
...
$replyTo = new Address(reset($replyTo), $address);

Line 108 is setting "new Address" as replyTo using the value of the array as address and $address as name.

Issue:
The array returned from MailUtility::getSystemReplyTo() is different depending on defaultMailReplyToName is being set in $GLOBALS['TYPO3_CONF_VARS']['MAIL'].
When set, address is key and name is value. When not set address is value.

        if (!empty($mailConfiguration['defaultMailReplyToName'])) {
            $replyTo = [$replyToAddress => $mailConfiguration['defaultMailReplyToName']];
        } else {
            $replyTo = [$replyToAddress];
        }

reset($replyTo) therefore returns the name when defaultMailReplyToName is filled leading to invalid RFC validation and exception when sending mail:

RFC compliance problem
Please verify $GLOBALS['TYPO3_CONF_VARS']['MAIL']['defaultMailFromAddress'] is a valid mail address. Error message: Email "ACME" does not comply with addr-spec of RFC 2822.