Incident report for the unexpected 500 error returned from authentication services on Jul 18, 2024.
On Jul 18, 2024, Logto Cloud experienced a service outage with 500 Internal server error from the authentication services.
During a recent Cloud deployment, a breaking change in the database schema caused the sign-in experience API to fail during the transition between the staging and production environments.
We are currently developing a new feature called "Bring your UI", which allows users to customize the Logto sign-in experience with their own web pages. This feature requires a new column in the sign-in-exp table to store the custom UI configuration.
Due to some requirement changes during the development, the feature release was delayed, but the first part of the schema change was already deployed to the production several weeks ago, despite not being in use yet. An update of the database column was introduced in this PR.
Unfortunately, this change was not backward compatible, causing API requests from the old code to fail when communicating with the new database.
When deploying a new version of Logto Cloud, we first deploy it to the staging environment and then swap the staging and production environments. The process is as follows:
However, both environments share the same database, and the entire process takes time. So in the time window between the database update and environment swap, online users remain in the production environment with the old code but attempt to communicate with the new database.
This was the root cause of the incident and the reason why it was automatically resolved in 35 minutes.
We DO have a CI task to check the backward compatibility of the database changes. However, previously it was not required to pass the CI check before merging the PR. This is because most of the time the development phase are usually short within a few sprints, and the first and second part of the schema changes are usually included in the same release phase.
This time, the feature release was delayed, spreading the schema changes across two releases. The developer assumed the CI failure was expected and informed the reviewers that it shouldn't block the PR from merging.
A communication gap was definitely there as well, and finally the PR was merged without providing any necessary backward compatibility support.