On May 12, 2026, between 13:41 and 17:43 UTC, some services experienced delays in processing. For the Code Scanning service, 53% of check runs took over 15 minutes to complete. Additionally, notifications took an average of 22 minutes to be delivered and Slack integration webhooks took an average of 20 minutes to be delivered. The delays were caused by replication lag due to an internal database migration, resulting in insufficient worker capacity for our high rate of job enqueues.
We mitigated the impact by scaling our processing workers to handle the increased load. All services returned to normal processing times after the mitigation was applied.
We are working to create dedicated worker pools for some of our high usage shared queues to help prevent this in the future.
Posted May 12, 2026 - 17:43 UTC
Update
All services have fully recovered.
Posted May 12, 2026 - 17:43 UTC
Update
CodeQL has fully recovered. We're continuing to work on recovery for the remaining impacted services.
Posted May 12, 2026 - 16:59 UTC
Update
Webhooks have fully recovered. Continuing to work on recovery for the other services.
Posted May 12, 2026 - 16:29 UTC
Update
Webhooks is operating normally.
Posted May 12, 2026 - 16:28 UTC
Update
We've established that most delays are related to a queuing service and are working to scale out. Early signals from the scale-out are showing signs of recovery for some services. We'll provide an update when services are fully recovered.
Posted May 12, 2026 - 16:18 UTC
Update
Webhooks is experiencing degraded performance. We are continuing to investigate.
Posted May 12, 2026 - 15:44 UTC
Update
We're continuing to investigate issues with CodeQL actions workflows. We're additionally seeing delays for notifications, webhooks, and the Slack integration.
Posted May 12, 2026 - 15:42 UTC
Update
CodeQL actions are currently experiencing delays, which may result in those actions being stuck in a pending state or having failed due to a timeout.
Posted May 12, 2026 - 15:13 UTC
Investigating
We are investigating reports of degraded performance for CodeQL
