Background

I recently enabled SCIM provisioning for AWS SSO through Okta.

It worked for 95% of our users but failed reliably on a small subset. I had a lot of trouble figuring out why, so hopefully this saves you the trouble.

The Issue

After turning on SCIM provisioning, most provisioning calls succeeded, but some failed with this error, as reported in the Okta logs:

Bad Request. Errors reported by remote server: Request is unparsable, syntactically incorrect, or violates schema.

Okta SCIM Provisioning Error

This error was consistent even on retry. Unfortunately it didn’t provide any additional information, so it wasn’t immediately actionable.

Troubleshooting

I tried manually making a handcrafted SCIM API call to Amazon’s SCIM endpoint with the user’s info. This succeeded! So it was something about the specific format that Okta was putting into the SCIM call.

I tried paring down as many attributes as I could so that Okta was only syncing over the bare minimum. Still, no luck.

I got in touch with Okta support, but they couldn’t pull the raw API requests that they were sending to AWS so I could investigate.

I searched around within AWS, but these failed requests weren’t being logged anywhere that was visible to the user (i.e. CloudTrail).

After some phone tag, I managed to get a support call between myself, Okta, and AWS. Okta support was able to provide the x-amzn-RequestId from one of the AWS SCIM responses that showed an error, which was enough for AWS support to investigate and identify the error in more detail.

The Fix

The issue is that there were some spurious Unicode control characters in some of the attributes on the affected users’ Okta profiles.

These had gone undetected up until now because every other service that used SCIM provisioning seemed to ignore them or accept them without issue.

AWS did not accept them, and this resulted in the error shown above.

I used the Okta API to dump the full profile from one of the affected users so that I could examine the fields directly, and I was able to identify and correct the issues with spurious characters.

For example, one user had a Left-to-right mark (U+200E LEFT-TO-RIGHT MARK) at the end of their last name, like this:

"lastName": "Coldren\u200e"

When I updated it to:

"lastName": "Coldren"

The SCIM provisioning issue went away.

Conclusion

Troubleshooting this issue took quite a long time because it required working with support from multiple companies.

It would have been nice if Okta could have shown me the raw SCIM requests it was sending to AWS, or if AWS provided a more descriptive error message indicating what specifically was invalid about the requests.

We’ve gone through and pruned all spurious control characters from all personnel names, so hopefully we don’t see any more issues like this going forward.

I hope this information is useful if you’re facing a similar issue!