RCA Review: Addressing Latency in the NY Region Tuesday, July 25

 

Today we had an incident that lasted for approximately one hour in the NYC region. I wanted to take the time to address this incident and review what happened, why it happened, what we did to resolve it, and what we’re doing to prevent future issues like this. 

 

Impact and cause of incident

The incident began at 11:00 a.m. ET when some customers in the New York City region experienced high latency over our DNS1 Anycast network. 

The cause of this incident stemmed from an issue with one of our hosting providers in which a circuit was inadvertently turned off. This resulted in DNS traffic to the Washington, D.C. region to be rerouted to servers in New York City. While this traffic shift is what we expect BGP to do in instances like this, the load was too great to resolve some requests in a timely manner. This resulted in the latency issue experienced by some of our customers.

Prior to receiving our first ticket, our team had already recognized the issue was with that particular upstream provider. We had visibility into the traffic moving from DC to NYC, and we were able to fully diagnose and solve the issues by 12:10 p.m. ET—24 minutes after our first support ticket was received.

But we’re committed to doing better. We’ve already had plans to increase our load capacity, including the NYC region, which will allow us to have greater resilience in the case of unbalanced loads resulting from unplanned traffic shifts. Our goal is to serve our customers and provide the best possible experience.

A while ago, I promised transparency, action, and accountability. I’m here to keep that promise.

Search
  • There are no suggestions because the search field is empty.
Latest posts
Traversing the World of AI with Judy Security Traversing the World of AI with Judy Security

Raffaele Mautone, CEO of Judy Security, recently joined us for an interview session around the increasing presence of AI in cybersecurity. This insightful Q&A session sheds light on how AI is integrated into Judy Security's operations. Raffaele also touches on the broader implications of AI for the future, making a compelling case for its strategic use in both day-to-day operations and long-term security strategies.

Exploring the Security of Free Public Wi-Fi with eero Exploring the Security of Free Public Wi-Fi with eero

There is no doubt that Wi-Fi has become an essential part of our daily lives, enabling us to stay connected whether we're at home, at work, or on the go. However, the convenience of wireless networks comes with significant security risks that can compromise personal and sensitive information. 

Revving Up the Fun: DNSFilter's IndyCar Experience Recap —Laguna Seca Edition Revving Up the Fun: DNSFilter's IndyCar Experience Recap —Laguna Seca Edition

Another exciting race weekend with Juncos Hollinger Racing and Romain Grosjean has come to a close! We co-hosted the IndyCar race at Laguna Seca road course with our pals at Pax8, our logos alongside each other on Grosjean’s No. 77 car. Clearly this teamwork paid off, with Romain finishing the race in his team-best position!

Explore More Content

Ready to brush up on something new? We've got even more for you to discover.