My email needs are simple. I have an HP Color LaserJet Pro M478f-9f with scan-to-email functionality, and I have a fully-validated domain on Amazon SES that I use to send miscellaneous transactional email.
I want the printer to be able to send email to anyone through a reputable provider that will provide robust deliverability. I don’t run any of my own email infrastructure, nor do I want to.
This should be simple – just create some SMTP credentials within AWS, add them to the printer’s webinterface, and the printer should be able to send email.
Out of the box, this doesn’t work, but we can solve it by using a local Postfix relay.
Nothing is ever easy when it comes to dealing with printers, and this is no exception.
I generated credentials on AWS, and added them to the HP printer interface:
The straightforward solution yielded a cryptic error message from the HP Printer whenever I tried to send a test email
or perform a scan-to-email:
The printer didn’t generate any more detailed error messages, and nothing was printed to syslog.
Frustratingly, I was able to make the printer send email using a different email provider (Mailgun), but not with SES.
I also found that the messages, even when accepted by a different provider, were not being reliably delivered to or processed by some services, such as Evernote.
Reliable delivery is important to me. Also, I really wanted to use SES, since that’s my standard for all my other field-deployed embedded devices.
I like that I can use IAM policies to restrict authorized sending identities (i.e. so a given device can only send as
email@example.com and not
firstname.lastname@example.org). This helps limit blast radius if a set of credentials is
Printers and various other embedded devices are somewhat notorious for buggy SMTP implementations, and here we clearly have a case of the printer not playing nicely with my desired email provider. It could have been any number of things. From past experience, I’ve seen:
- Printers that can’t accept passwords over 24 characters.
- Printers with strange TLS bugs that won’t work with modern email providers.
- Printers that have DNS bugs.
- Printers that have IPv6 bugs.
- Printers that mangle the email headers/body to the point where the provider rejects them (spoiler alert: this is the problem here, as I’ll soon find out).
I needed a solution that allows me to use this buggy printer with my preferred email provider, Amazon SES. This means I needed to figure out:
- Why the printer was reporting “system failure”, i.e. why the SMTP submission with SES is failing.
- A workaround that allowed me to keep using it.
Diagnosing the Issue
The first step was to figure out what was wrong. Because the printer doesn’t provide any information other than the
System Failure, I need another tool for debugging.
I placed a Raspberry Pi running Postfix on the same local network, and I configured it as a mail relay.
This means that instead of the printer sending directly to SES, the printer would send to the local mail relay, and the local mail relay will send it along to SES. This allowed me to get diagnostic information on why SES is rejecting the message.
There are a lot of steps involved here that are peripheral to the purpose of this post, so I’m omitting them. There are excellent guides available online about how to set up a postfix relay, for example, here.
If you decide to use this in a production environment, make sure you set up appropriate restrictions – credentials,
firewalls, or postfix configuration using the
mynetworks directive, to prevent your Postfix relay from being used
by other systems you may not be expecting.
I configured the printer to send to my local mail relay as follows:
This means my Postfix mail relay will receive the messages and forward them along, logging its actions along the way.
Sure enough, after sending a test email, I saw the following rejection from Amazon SES in the Postfix logs:
status=bounced (host email-smtp.us-west-2.amazonaws.com[18.104.22.168] said: 554 Transaction failed: Expected '=', got ":" (in reply to end of DATA command))
Amazon SES was rejecting the message due to some malformed content.
Luckily, Postfix allows us to grab the full message body to diagnose:
From: "=?utf-8?Q?My=20Printer?=" <email@example.com> To: <firstname.lastname@example.org> Subject:Scan-to-Email Test Message from HP Color LaserJet Pro M478f-9f Content-Type:multipart/mixed;charset=utf-8; boundary="----=_Part_Boundary_00000001_0f18f941.5466ca24" ------=_Part_Boundary_00000001_0f18f941.5466ca24 Content-Type:text/plain;charset=utf-8; Content-Transfer-Encoding:base64 Congratulations! The settings you have configured are correct. Note: This email message is automatically generated. Do not reply to this message. Device Identification Product Serial Number: 000
There’s something subtle wrong here, and we need to figure out what.
I saved the message to a file (
test.txt) and then started tweaking one thing at a time. After each tweak, I tried
to resubmit the message using
sendmail -t < test.txt and I checked the Postfix log to see if SES accepted it or not.
Finally, I figured it out. It’s this line:
There are actually two things wrong here.
Content-Transfer-Encodingare two different headers, and they are incorrectly smashed together on a single line. See here for the relevant specification. This explains the SES rejection details.
Expected '=', got ":"makes sense; SES is parsing the headers and this one is invalid.
- Second, the
base64, but the message is not actually base64-encoded.
When these lines are adjusted as follows, the message sends successfully:
Curiously, other email providers don’t do this level of detailed validation check. They will happily accept the message and pass it through verbatim. Mailgun and Fastmail, for example, will accept the malformed message just fine, and it’s up to the system that receives the mail to deal with it. I found that Evernote will not accept any messages that are malformed like this; they will be silently dropped.
Clearly this is a bug in the HP printer firmware. I confirmed I’m on the latest firmware. I have not been able to find anywhere to report this to HP.
A Postfix Workaround
Now that I know what’s wrong, I need a hook within Postfix to fix this bug before sending the mssage along to SES.
Postfix allows you to use the
header_checks directive to build regex-based fixup rules.
main.cf file, I added this line:
header_checks = regexp:/etc/postfix/header_check
And I created a
/etc/postfix/header_check file with the following contents:
/^Content-Type:text\/plain;charset=utf-8; Content-Transfer-Encoding:base64/ REPLACE Content-Type:text/plain;charset=utf-8;
This regex will replace the offending line with a fixed version prior to Postfix relaying the message to SES.
I confirmed that this fixes the issue – the printer can send malformed emails to my local mail relay, Postfix will fix them up, and it will forward them along to SES, which accepts them and reliably delivers them to the destination, where they are reliably processed.
Shortly after, I found another similar case with a different embedded device. This time it was an APC Smart-UPS 1000 unit (SMT1000RM2U) with the UPS Network Management Card 2, running the latest firmware.
It was failing to send mail directly through SES. Debugging with the Postfix relay revealed it was sending messages
with a blank
Content-Transfer-Encoding: header, and SES was rejecting them.
Fixing this up with Postfix and the
header_check functionality was as simple as:
/^Content-Transfer-Encoding: $/ IGNORE
This instructs Postfix that if the header is blank, it should be ignored, i.e. removed from the message before relaying.
It is unfortunate that embedded devices contain bugs like this, but luckily we can use tiny, low-cost hardware like the Raspberry Pi to serve as a relay, which allows us a full OS environment and robust tooling to correct these issues.
Ideally the manufacturer would fix the bug, but short of that, this experience taught me quite a bit about SMTP and Postfix and how to be resilient in the face of vendor bugs. It is satisfying that I have a toolkit to deal with this type of issue in the future with other field-deployed embedded devices that may not be quite standards-compliant.
I hope this is helpful for others who may be facing the same challenge!