Incident report for the Logto service outage on 2023-12-17 due to loss of production Docker image.
We have experienced a service outage on 2023-12-17 due to loss of Logto production Docker image.
| Time | Event |
|---|---|
| Early 2023-12-17 (Specific time unknown) | Logto automated GitHub image retention workflow being executed. logto and logto-cloud production images were deleted by mistake. |
| 2023-12-17 03:56:00 UTC | Logto cloud service and Logto core service became unavailable. Incident detected by our monitoring system. |
| 2023-12-17 04:03:00 UTC | Incident acknowledged by our on-call engineer. |
| 2023-12-17 04:10:00 UTC | New deployment of Logto cloud service and Logto core service triggered with the latest image. |
| 2023-12-17 04:15:00 UTC | Logto cloud service and Logto core service became available. Incident resolved automatically |
Logto service production images was deleted by our automated GitHub image retention workflow. The cloud service failed to fetch the image from GitHub Container Registry and became unavailable.
The automated GitHub image retention workflow deleted the production images by mistake. The workflow was designed to delete all untagged legacy images that are older than 3 days.
We tagged the production images with prod tag to identify them as production images. Whenever a production deployment being triggered, a new image with prod tag will be built and pushed to GitHub Container Registry. The prod tag will be removed from the old image after the new image is successfully built and pushed. The old image will become untagged and will be deleted by the automated GitHub image retention workflow.
Logto service image were build to support multiple architectures. The image was built with buildx and pushed to GitHub Container Registry with --platform flag. All the tags were applied to the root manifest list. The prod tag was applied to the root manifest list as well. All the sub-images listed under the multi-arch manifest list remain untagged.
Lack of careful review of the tag and manifest list structure of the Docker image, we simply configured the automated GitHub image retention workflow to delete all untagged images. The workflow deleted all the sub-images listed under the multi-arch manifest list.
This incident caused Logto cloud service and Logto core service to be unavailable for about 18 minutes. End users were unable to login to Logto and access their client applications. Logto cloud admin portal was also unavailable during the incident.
We have stopped the automated GitHub image retention workflow and deployed a new image with prod tag to GitHub Container Registry. The new image was successfully fetched by Logto cloud service and Logto core service. The service became available again.