Our team continued reviewing logs and analyzing the additional monitoring data collected during today’s incident. While we have not yet confirmed the underlying cause, that review has helped guide the next steps in our investigation.

Tomorrow morning, our vendor will join our team to capture diagnostic information during a recurrence of the routing issue. Because we have been unable to reproduce the problem during troubleshooting, collecting this information while an incident is underway is the next step in identifying the cause. Based on recent incidents, we expect another disruption between 8:45 and 9:45 a.m., although the timing has not been consistent enough to pinpoint an exact time. If the issue returns, we will collect firewall debug logs with our vendor and implement a workaround we believe will resolve the incident and restore connectivity.

Our team will be ready to act as soon as the issue arises, and we will work to keep the disruption as brief as possible while capturing the data needed to advance the investigation. We will provide updates as the morning progresses. We recognize the impact these interruptions have on your staff and students and appreciate your continued patience. As always, if you have any questions or concerns, please don’t hesitate to reach out.

The network has stabilized. We are reviewing all the logs again this morning to identify the cause and are escalating to our vendors for answers. I will provide periodic updates, as I did yesterday.

We are experiencing network issues again this morning and are active investigating.

Network Routing Issues – Investigation and Monitoring Update

Our team has spent the day extensively reviewing logs and investigating the recurring network routing issue. While we have not yet identified the underlying cause, our investigation suggests that one of the redundant virtual routers may be contributing to the problem. As a precaution, we have taken that router offline and will keep it offline while troubleshooting continues.

We have also expanded our monitoring and logging capabilities to capture additional diagnostic information should the issue recur. These measures will help us better isolate the cause and determine the appropriate corrective actions.

Despite the significant time and resources dedicated to this investigation, we cannot yet consider the issue fully resolved. Our team will continue monitoring the environment closely and provide additional updates as more information becomes available.

We recognize the significant impact these disruptions have had on our members and want to reaffirm our commitment to identifying and resolving the underlying issue. Network reliability has long been a cornerstone of NEOnet's reputation, and we take that responsibility seriously. We remain fully committed to restoring and maintaining the dependable service our members expect and deserve.

---

Technical Details

For those interested in a more detailed understanding of the issue and our ongoing investigation, we have provided the following technical summary.

The Issue

During these incidents, CPU utilization on the firewall cores responsible for routing increases significantly, coinciding with the loss of OSPF routing adjacencies. Although session counts remain within the firewall's supported capacity, the routing instability disrupts the firewall's ability to reliably forward network traffic, resulting in intermittent connectivity and service interruptions.

What We Have Found

Our investigation has identified elevated firewall CPU utilization and increased session counts during these events. However, we have not yet determined what triggers these increases or whether the elevated session counts are contributing to the routing disruption or are a consequence of it.

We are also evaluating the possibility of an internally originating traffic flood overwhelming network resources, although we have not identified definitive evidence supporting this theory.

Additionally, we have found no evidence indicating that an external DDoS attack has bypassed our DDoS protection system.

At this stage, we are continuing to investigate all potential contributing factors rather than drawing conclusions without sufficient supporting evidence.

What We Have Done

To improve visibility and assist with identifying the root cause, we have implemented several additional troubleshooting and monitoring measures:

• Enabled more detailed DNS and firewall logging to capture additional diagnostic information.
• Deployed additional session monitoring sensors to better identify unusual traffic or connection patterns.
• Implemented an sFlow sensor to collect more detailed network traffic data.
• Taken one of the redundant virtual routers offline as a precaution, based on findings suggesting its possible involvement.
• Engaged our firewall vendor through an active support case to assist with analyzing the behavior and identifying potential underlying causes.

These measures are intended to provide greater visibility into the environment and capture the information necessary to isolate the issue if it recurs.

---

We sincerely appreciate the patience and understanding our members have shown throughout these disruptions. We understand the importance of reliable network services to your daily operations and remain committed to resolving this issue.

Please continue to report any connectivity or service-related issues through a support ticket so our team can investigate. As always, if you have any questions or concerns, please do not hesitate to reach out.

We are continuing to work through the cause of our outages over the past three days. We have identified one bug in the Firewall that could be a factor, but we are not confident it’s the source. We are attempting to bring in a third party to get a fresh set of eyes on this until we have a firm root cause. As a precaution, we may apply that patch tonight during an emergency maintenance window, but we need to verify that the upgrade will not introduce any other issues. We will continue to provide updates as we move through the troubleshooting.

We have engaged our vendors and are still troubleshooting this issue. Services are up and functional, but we are still seeing issues we need to address to get to the root cause of this. We will continue to provide updates as we move through the troubleshooting.

We are investigating internet issues again this morning. We will provide more information as we try to identify the issue.