RCA Review: Addressing Latency in the NY Region Tuesday, July 25

by Dave Raphael on Jul 25, 2023 5:48:19 PM

Today we had an incident that lasted for approximately one hour in the NYC region. I wanted to take the time to address this incident and review what happened, why it happened, what we did to resolve it, and what we’re doing to prevent future issues like this.

The incident began at 11:00 a.m. ET when some customers in the New York City region experienced high latency over our DNS1 Anycast network.

The cause of this incident stemmed from an issue with one of our hosting providers in which a circuit was inadvertently turned off. This resulted in DNS traffic to the Washington, D.C. region to be rerouted to servers in New York City. While this traffic shift is what we expect BGP to do in instances like this, the load was too great to resolve some requests in a timely manner. This resulted in the latency issue experienced by some of our customers.

Prior to receiving our first ticket, our team had already recognized the issue was with that particular upstream provider. We had visibility into the traffic moving from DC to NYC, and we were able to fully diagnose and solve the issues by 12:10 p.m. ET—24 minutes after our first support ticket was received.

But we’re committed to doing better. We’ve already had plans to increase our load capacity, including the NYC region, which will allow us to have greater resilience in the case of unbalanced loads resulting from unplanned traffic shifts. Our goal is to serve our customers and provide the best possible experience.

A while ago, I promised transparency, action, and accountability. I’m here to keep that promise.

Topics: DNSFilter Updates

7 Cybersecurity Predictions for 2026

AI and other technologies are fundamentally reshaping the security world. It’s never too early to prepare, which is why we're dropping our predictions for 2026 ahead of schedule! We went straight to the source, surveying our internal team of cybersecurity and technology specialists to find out where they see the battle lines forming. Here’s what our experts predict.

The Hire the Stakes, The Greater the Fall

Another late night, exhaustion and sore eyes setting in after sending what felt like the thousandth job application on your list, when yet another reply hits your inbox with the words, “Unfortunately we…”You don’t even need to read the rest; you’re familiar enough with how this goes. Woefully, you sit thinking about the work you’ve put into the job search: The degree, the blood, the sweat, the tears—ping. An email notification graces your frustra...

Risky Apps for Business: Top Malware and Phishing Threats

Not all apps are created equal. Some introduce serious risks for phishing, malware, or data breaches, especially when they’re adopted by employees without IT approval. And while blocking a website at the domain can prevent unauthorized access on your network, many applications have hundreds or thousands of domains—simply blocking the main domain might not be enough to prevent access.