smtp socket: malformed response on a FIPS 140-2 system

Ok, this is a very highly specific post - but I hope it is useful for that sysadmin who's tearing their hair out trying to figure out wtf is going on with smtp failing with a vague error message.

Recently, I was configuring a Postfix SMTP relay on a FIPS140-2 enabled system, and had a weird error that I hadn't ever seen before:

warning: private/smtp socket: malformed response
warning: transport smtp failure -- see a previous warning/fatal/panic logfile record for the problem description
warning: process /usr/libexec/postfix/smtp pid  killed by signal 11
warning: /usr/libexec/postfix/smtp: bad command startup -- throttling

The warning: private/smtp socket: malformed response line is specifically what the error was.

Googling this issue turns up mostly chrooted postfix issues, or incorrect permissions on the /etc/services file. Not a lot of useful information for my specific issue!

In this Red Hat Knowledgebase article I finally found the correct answer! Now, it's obviously paywalled behind a Red Hat subscription, however knowing the magic string to search for turns up this stackoverflow article and we see that converting the hashing function from md5 which is disabled on a FIPS 140-2 enabled system to sha256 by running the following commands:

# postconf -e smtp_tls_fingerprint_digest=sha256
# postconf -e smtpd_tls_fingerprint_digest=sha256
# systemctl restart postfix

You can also just add the two lines to your /etc/postfix/main.cf file, whatever floats your boat. It would be super if they were in that file commented out, but they're not (at least not on RHEL 8.x)

That's it, that's the blog post. Go forth and send emails.

-BadgerOps