Background

My email needs are simple. I have an HP Color LaserJet Pro M478f-9f with scan-to-email functionality, and I have a fully-validated domain on Amazon SES that I use to send miscellaneous transactional email.

I want the printer to be able to send email to anyone through a reputable provider that will provide robust deliverability. I don’t run any of my own email infrastructure, nor do I want to.

This should be simple – just create some SMTP credentials within AWS, add them to the printer’s webinterface, and the printer should be able to send email.

Out of the box, this doesn’t work, but we can solve it by using a local Postfix relay.

Problem

Nothing is ever easy when it comes to dealing with printers, and this is no exception.

I generated credentials on AWS, and added them to the HP printer interface:

AWS SMTP Configuration on the printer

The straightforward solution yielded a cryptic error message from the HP Printer whenever I tried to send a test email or perform a scan-to-email: System Failure.

The printer didn’t generate any more detailed error messages, and nothing was printed to syslog.

Frustratingly, I was able to make the printer send email using a different email provider (Mailgun), but not with SES.

I also found that the messages, even when accepted by a different provider, were not being reliably delivered to or processed by some services, such as Evernote.

Reliable delivery is important to me. Also, I really wanted to use SES, since that’s my standard for all my other field-deployed embedded devices.

I like that I can use IAM policies to restrict authorized sending identities (i.e. so a given device can only send as office-printer@mydomain.tld and not billing@mydomain.tld). This helps limit blast radius if a set of credentials is compromised.

Printers and various other embedded devices are somewhat notorious for buggy SMTP implementations, and here we clearly have a case of the printer not playing nicely with my desired email provider. It could have been any number of things. From past experience, I’ve seen:

  • Printers that can’t accept passwords over 24 characters.
  • Printers with strange TLS bugs that won’t work with modern email providers.
  • Printers that have DNS bugs.
  • Printers that have IPv6 bugs.
  • Printers that mangle the email headers/body to the point where the provider rejects them (spoiler alert: this is the problem here, as I’ll soon find out).

Goals

I needed a solution that allows me to use this buggy printer with my preferred email provider, Amazon SES. This means I needed to figure out:

  • Why the printer was reporting “system failure”, i.e. why the SMTP submission with SES is failing.
  • A workaround that allowed me to keep using it.

Diagnosing the Issue

The first step was to figure out what was wrong. Because the printer doesn’t provide any information other than the phrase System Failure, I need another tool for debugging.

I placed a Raspberry Pi running Postfix on the same local network, and I configured it as a mail relay.

This means that instead of the printer sending directly to SES, the printer would send to the local mail relay, and the local mail relay will send it along to SES. This allowed me to get diagnostic information on why SES is rejecting the message.

There are a lot of steps involved here that are peripheral to the purpose of this post, so I’m omitting them. There are excellent guides available online about how to set up a postfix relay, for example, here.

If you decide to use this in a production environment, make sure you set up appropriate restrictions – credentials, firewalls, or postfix configuration using the mynetworks directive, to prevent your Postfix relay from being used by other systems you may not be expecting.

I configured the printer to send to my local mail relay as follows:

Local mail relay configuration on the printer

This means my Postfix mail relay will receive the messages and forward them along, logging its actions along the way.

Sure enough, after sending a test email, I saw the following rejection from Amazon SES in the Postfix logs:

status=bounced (host email-smtp.us-west-2.amazonaws.com[52.41.197.26] said: 554 Transaction failed: Expected '=', got ":" (in reply to end of DATA command))

Amazon SES was rejecting the message due to some malformed content.

Luckily, Postfix allows us to grab the full message body to diagnose:

From: "=?utf-8?Q?My=20Printer?=" <myprinter@mydomain.tld>
To: <me@mydomain.tld>
Subject:Scan-to-Email Test Message from HP Color LaserJet Pro M478f-9f
Content-Type:multipart/mixed;charset=utf-8; boundary="----=_Part_Boundary_00000001_0f18f941.5466ca24"



------=_Part_Boundary_00000001_0f18f941.5466ca24
Content-Type:text/plain;charset=utf-8; Content-Transfer-Encoding:base64

Congratulations!

The settings you have configured are correct.

Note: This email message is automatically generated. Do not reply to this message.

Device Identification

Product Serial Number: 000

There’s something subtle wrong here, and we need to figure out what.

I saved the message to a file (test.txt) and then started tweaking one thing at a time. After each tweak, I tried to resubmit the message using sendmail -t < test.txt and I checked the Postfix log to see if SES accepted it or not.

Finally, I figured it out. It’s this line:

Content-Type:text/plain;charset=utf-8; Content-Transfer-Encoding:base64

There are actually two things wrong here.

  • First, Content-Type and Content-Transfer-Encoding are two different headers, and they are incorrectly smashed together on a single line. See here for the relevant specification. This explains the SES rejection details. Expected '=', got ":" makes sense; SES is parsing the headers and this one is invalid.
  • Second, the Content-Transfer-Encoding specifies base64, but the message is not actually base64-encoded.

When these lines are adjusted as follows, the message sends successfully:

Content-Type:text/plain;charset=utf-8;

Curiously, other email providers don’t do this level of detailed validation check. They will happily accept the message and pass it through verbatim. Mailgun and Fastmail, for example, will accept the malformed message just fine, and it’s up to the system that receives the mail to deal with it. I found that Evernote will not accept any messages that are malformed like this; they will be silently dropped.

Clearly this is a bug in the HP printer firmware. I confirmed I’m on the latest firmware. I have not been able to find anywhere to report this to HP.

A Postfix Workaround

Now that I know what’s wrong, I need a hook within Postfix to fix this bug before sending the mssage along to SES.

Postfix allows you to use the header_checks directive to build regex-based fixup rules.

In my main.cf file, I added this line:

header_checks = regexp:/etc/postfix/header_check

And I created a /etc/postfix/header_check file with the following contents:

/^Content-Type:text\/plain;charset=utf-8; Content-Transfer-Encoding:base64/ REPLACE Content-Type:text/plain;charset=utf-8;

This regex will replace the offending line with a fixed version prior to Postfix relaying the message to SES.

I confirmed that this fixes the issue – the printer can send malformed emails to my local mail relay, Postfix will fix them up, and it will forward them along to SES, which accepts them and reliably delivers them to the destination, where they are reliably processed.

Another example

Shortly after, I found another similar case with a different embedded device. This time it was an APC Smart-UPS 1000 unit (SMT1000RM2U) with the UPS Network Management Card 2, running the latest firmware.

It was failing to send mail directly through SES. Debugging with the Postfix relay revealed it was sending messages with a blank Content-Transfer-Encoding: header, and SES was rejecting them.

Fixing this up with Postfix and the header_check functionality was as simple as:

/^Content-Transfer-Encoding: $/ IGNORE

This instructs Postfix that if the header is blank, it should be ignored, i.e. removed from the message before relaying.

Conclusion

It is unfortunate that embedded devices contain bugs like this, but luckily we can use tiny, low-cost hardware like the Raspberry Pi to serve as a relay, which allows us a full OS environment and robust tooling to correct these issues.

Ideally the manufacturer would fix the bug, but short of that, this experience taught me quite a bit about SMTP and Postfix and how to be resilient in the face of vendor bugs. It is satisfying that I have a toolkit to deal with this type of issue in the future with other field-deployed embedded devices that may not be quite standards-compliant.

I hope this is helpful for others who may be facing the same challenge!