Production API Error Elevation
Incident Report for Alloy
Postmortem

A recent patch release of one of our third party dependencies contained a bug that was not apparent in our lower environments during testing. As a result of this bug, a release of an Alloy API component caused that component to deploy to our production environment, but fail to start. Immediate attempts were made to re-deploy the component, and to roll it back to the last known good state, all of which failed. The process manager that contained the bug was then redeployed, and the Alloy API component successfully started. A number of improvements are being implemented to prevent this type of incident in the future.

Posted Jan 27, 2020 - 10:32 EST

Resolved
This incident has been resolved. We'll post the results of a root cause analysis within the next few days.
Posted Jan 23, 2020 - 18:27 EST
Monitoring
A problem with our process manager during a deployment caused data integrations to stall from 15:01 to 15:19 EST. We are currently monitoring the situation, but have resumed full functionality.
Posted Jan 23, 2020 - 15:34 EST
Investigating
Our API integration tests have encountered an increase in errors. We are currently investigating. Stay tuned for updates.
Posted Jan 23, 2020 - 15:10 EST
This incident affected: Production API.