On October 6, 2020 at approximately 19:00 UTC the Microsoft Azure DevOps services began to experience “Availability Degradation.” and comes only one week after a big Azure AD Outage that essentially took down all of Microsoft Azure authentication. This Azure DevOps outage is not related to last weeks outage of Azure, and started with some service degradation in Europe that since spread to the United States and Brazil regions.

Azure DevOps is Down; Outage is MS not Your CI/CD Builds (October 6, 2020) 1
Screenshot: Azure DevOps showing 500 error “Something went wrong!”

TLDR; Azure DevOps experienced service degradation and intermittent issues across United States, Brazil and Europe regions for the Boards, Repos, Pipelines, Test Plans, and Artifacts services for about 3 hours total on October 6, 2020. The outage issue was caused by “a portion of our service-to-service traffic was impacted by issues in Azure Front Door.

Azure DevOps Outage Timeline

October 6, 2020 22:00 UTC

The Azure DevOps status page shows “Everything is looking good” as all services in all regions are now Healthy.

Azure DevOps is Down; Outage is MS not Your CI/CD Builds (October 6, 2020) 2
Screenshot of Azure DevOps status page showing “Everything is looking good.”

Also, the “Final update” shown in the Azure DevOps status event log reports the following:

The issue is now fully mitigated.A portion of our service-to-service traffic was impacted by issues in Azure Front Door. Those issues were mitigated by 21:19 UTC and our impact ended at the same time. We apologize for the impact this had on our customers.

You can view the event log for the two separate outage reports for this Azure DevOps outage at the following links:

October 6, 2020 21:15: UTC

Azure DevOps status page is showing the Artifacts service in Europe and United States as “AFD Outage Affecting Packages”. Also, the other Azure DevOps services are showing as “Availability Degradation tied to AFD issues” for Brazil, Europe and United States. Clicking into the title of the outage statuses shows a message from Microsoft that “Our engineers are working to understand and mitigate the issue.”

Azure DevOps is Down; Outage is MS not Your CI/CD Builds (October 6, 2020) 3
Screenshot of Azure DevOps status page

As far as I know, “AFD” stands for “Azure Front Door”, but I haven’t seen that confirm anywhere official from Microsoft yet.

October 6, 2020 21:00 UTC

The Azure DevOps service status page is showing “Availability Degradation” in the United States for Artifacts service, along with “Availability Degradation tied to AFD issues” in Brazil, United States, and Europe for the Boards, Repos, Pipelines, and Test Plans services.

Azure DevOps is Down; Outage is MS not Your CI/CD Builds (October 6, 2020) 4
Screenshot of Azure DevOps outage status

October 6, 2020 ~19:30 UTC

Azure DevOps started showing additional “Availability Degradation” that spread to the United States region that affected the Artifacts service. However, reports from the community online show that the service degradation has already spread beyond just the Artifacts service to Repos, Boards, and Pipelines.

October 6, 2020 ~19:00 UTC

Azure DevOps started showing “Availability Degradation” in the Europe region that affected the Boards, Repos, Pipelines, and Test Plans services.

Azure DevOps is Down; Outage is MS not Your CI/CD Builds (October 6, 2020) 5
“Investigating a service disruption”…

Chris Pietschmann is a Microsoft MVP, HashiCorp Ambassador, and Microsoft Certified Trainer (MCT) with 20+ years of experience designing and building Cloud & Enterprise systems. He has worked with companies of all sizes from startups to large enterprises. He has a passion for technology and sharing what he learns with others to help enable them to learn faster and be more productive.
Microsoft MVP HashiCorp Ambassador

Discover more from Build5Nines

Subscribe now to keep reading and get access to the full archive.

Continue reading