GitHub sees hours-long outage

Code hosting platform GitHub is still trying to mitigate an incident that broke multiple services overnight, at one point failing half of all archive and raw repository content downloads.

The Microsoft-owned code hosting platform opened the incident on August 17 at 11:40pm AEST, saying it was investigating reports of impacted performance across some of its services.

Within 20 minutes GitHub had added API requests, Actions, webhooks, Issues and pull requests to the list, and by 12:04am August 18 it put the error rate at roughly 20 percent for web and API traffic, with archive downloads and raw repository content failing at about 50 percent.

Package installs, container builds and Go module fetches routinely pull files from raw.githubusercontent.com rather than cloning a repository outright, which is why the high 50 percent error rate is a headache rather the web and API traffic problem.

Enterprise customers were caught at the identity layer as well, with GitHub confirming that SAML and OIDC authentication, SCIM provisioning and Team Sync had also been affected.

Overnight, GitHub said the degradation affecting API requests, Actions, Git operations, Issues, Pages, pull requests and webhooks had been mitigated, and that it was monitoring for stability.

Recovery has been sporadic, with Git operations, Issues and the API each failing again within two hours of GitHub declaring the degradation mitigated.

"We are continuing to investigate sporadic authentication failures," GitHub said in an update, adding that it had partially disabled authentication token retries, seen improvement, and was monitoring the effect before applying the mitigation in full.

Neither a root cause nor the identity of the component at fault had been disclosed at that point, five and a half hours after the incident opened, GitHub said in an update just after 5 am AEST.

The incident this week was the 15th GitHub had logged this month.