Ok, this is a very highly specific post - but I hope it is useful for that sysadmin who's tearing their hair out trying to figure out wtf is going on with smtp failing with a vague error message.
Recently, I was configuring a Postfix SMTP relay on a FIPS140-2 enabled system, and had a weird error that I hadn't ever seen before:
warning: private/smtp socket: malformed response
warning: transport smtp failure -- see a previous warning/fatal/panic logfile record for the problem description
warning: process /usr/libexec/postfix/smtp pid killed by signal 11
warning: /usr/libexec/postfix/smtp: bad command startup -- throttling
warning: private/smtp socket: malformed response line is specifically what the error was.
Googling this issue turns up mostly chrooted postfix issues, or incorrect permissions on the
/etc/services file. Not a lot of useful information for my specific issue!
In this Red Hat Knowledgebase article I finally found the correct answer! Now, it's obviously paywalled behind a Red Hat subscription, however knowing the magic string to search for turns up this stackoverflow article and we see that converting the hashing function from
md5 which is disabled on a FIPS 140-2 enabled system to
sha256 by running the following commands:
# postconf -e smtp_tls_fingerprint_digest=sha256
# postconf -e smtpd_tls_fingerprint_digest=sha256
# systemctl restart postfix
You can also just add the two lines to your
/etc/postfix/main.cf file, whatever floats your boat. It would be super if they were in that file commented out, but they're not (at least not on RHEL 8.x)
That's it, that's the blog post. Go forth and send emails.