The Alloy API started experiencing problems with evaluations beginning at 11:49 EST today and continuing until 12:23 EST (total incident duration 34 minutes). Most evaluations run during that time were affected and resulted primarily in HTTP 500 status code errors. Our team responded immediately but took a longer than usual amount of time to isolate the bug due to the complex nature of the problem. The issue stemmed from a database transaction termination bug in our code base that was not caught in testing. We are currently working on a long term fix to make sure issues like this can't occur or have an impact in the future.
Posted Jul 13, 2020 - 13:00 EDT
Monitoring
A fix has been implemented and we are monitoring the results.
Posted Jul 13, 2020 - 12:28 EDT
Investigating
Our API integration tests have encountered an increase in errors. We are currently investigating. Stay tuned for updates.