On Tuesday, April 20th at approximately 16:00 UTC, we began to receive tickets from users stating they were unable to log into the DNSFilter dashboard. At the same time, our API monitors indicated that there were similar issues with the authentication API for DNSFilter.
The investigation immediately uncovered that our authentication provider, Auth0, was having a major outage in their primary (US-1) data center. This outage was not only affecting DNSFilter, but also a large portion of the other 9,000 enterprises that rely on Auth0. The outage immediately triggered our incident response process, and at 16:09 UTC we posted a message to our status page indicating that we were working on the issue and that DNS resolution was not affected in any way; only the ability to log in to the dashboard.
Unfortunately, the message posted to our status page did not make clear that Auth0 was the cause of the outage and could have been more clear that there was nothing to worry about, as analytics, threat blocking, content filtering, etc. were all still fully operational.
Additionally, throughout this time, the DNSFilter support and infrastructure teams maintained the ability to pull data or make network changes on behalf of customers with an urgent need. I want to confirm that at no time were DNSFilter systems impacted. No networks were affected and there was zero downtime, loss of data, or reduction in threat protection for any DNSFilter customer.
That being said, we'd like you to know that we take this incident seriously. Auth0 was chosen as our authentication provider in an effort to provide industry-leading security and privacy for our users. Although DNSFilter has never experienced any downtime from Auth0 in more than five years of use, this incident has highlighted the necessity for us to build clearer customer communications and have a solid run book should an incident of this nature happen again, just as we do with any other critical aspect of our infrastructure.
As of this post Auth0 reports complete restoration of availability and considers the issue resolved.
Please accept our sincere apologies for any concern or frustration for miscommunication around this issue. We are fully committed to learning from this incident and continuing to provide innovative technology that challenges the way the industry thinks about DNS security.
Sincerely,
Ken Carnesi
Founder & CEO