Postmortem: authentication failures caused by JWKS caching and signing key rotation

Date: Jan 8, 2026 (PST)

Duration: ~60 minutes

Impact: some production tenants experienced sign-in and token validation failures; Console sign-in could be affected

Summary#

A mismatch between a rotated signing key and a cached JWKS on our *.logto.io domain caused token validation failures. We cleared the JWKS cache and reverted to the previous signing key to restore service.

Some users may still see Console sign-in issues due to browser cache. We are sorry for the disruption this caused.

Timeline (PST)#

• 04:00 PM incident started (auth/token validation errors increased)
• 04:35 PM root cause identified (JWKS cached while signing key rotated)
• 04:41 PM cache cleared
• 04:49 PM signing key reverted
• 05:00 PM service recovered; continued monitoring

Incident details#

Impact#

During the incident window, some users could not sign in and some token validations failed. This impacted multiple production tenants and could also block access to the Logto Console.

We have seen no evidence of unauthorized access related to this incident; the impact was limited to authentication failures.

Customer action#

If you still can’t sign in to the Logto Console, please try:

• opening an incognito/private window, or
• clearing/disabling your browser cache, then retrying

What happened#

1. We updated JWKS caching settings for customer tenant domains (*.logto.app). JWKS caching was mistakenly still enabled on our Cloud domain (*.logto.io).
2. We rotated the signing key for our Cloud service. Because JWKS responses for *.logto.io were cached, some clients continued using a stale JWKS and could not validate newly issued tokens.
3. We cleared the JWKS cache for *.logto.io, reverted to the previous signing key, then cleared the JWKS cache again to ensure clients picked up the reverted key set.
4. Authentication recovered. Some users may still see Console sign-in issues due to browser cache.

Lessons learned#

Key rotation is not just a key management task. It is an end-to-end compatibility change that must account for caching behavior between issuers and validators. Config drift across domains (*.logto.app vs *.logto.io) is a real risk. Changes that are safe for one domain can break another if not applied consistently.

Our existing integration tests did not cover production-like JWKS caching behavior, so this failure mode was not exercised before the rotation.

Preventive actions#

We are implementing:

• JWKS cache invalidation as a required step in the signing key rotation flow (rotate only after invalidation completes)
• keeping JWKS caching disabled until the invalidation flow is in place, then re-enabling caching for performance
• production-like integration tests for key rotation, including JWKS caching behavior and cache invalidation checks