To the Emory Community:
The network problems experienced over the past two days were corrected Monday at approximately 8 p.m., and all University and Healthcare systems were back online by 10 p.m.
We deeply regret the risk, cost and inconvenience created by these outages and are working with our networking vendor to minimize the impact associated with any future changes.
As background, over the weekend, IT took another step in replacing the core networking infrastructure that serves both Healthcare and the University. This conversion is necessary routine maintenance, replacing equipment that is over six years old. It is also a desirable upgrade to satisfy demands for increased capacity and advanced features that support Emory's multiple missions.
The changes over the weekend were intended to add the last set of features and capacity to move the healthcare networks to the new core with the already-migrated university networks. Last week, the networking team worked with our networking vendor (Cisco) to validate the new, planned configuration and performed multiple successful simulations with identical equipment in our testing lab. On Sunday, the upgrade was installed at 1:30 a.m. EST, and all indications were that it was successful. At approximately 7:45 a.m. on Sunday, network stability wavered in the face of some unseen error condition.
On Sunday afternoon, the networking team backed out a portion of the overnight changes as a workaround, and that led to a period of stability for both healthcare and university applications through Monday morning. Monday's additional load proved that not all issues were resolved by the workaround.
Continuous troubleshooting in consultation with the highest possible level of technical resources from Cisco were not able to identify a root cause, and at 6 p.m. the networking team began backing out the remainder of the overnight changes from Sunday. Our internal network engineers are now working with Cisco to further analyze the outages and formulate a revised plan for performing the upgrade. Cisco is in the process of building a lab with our exact configuration in order to identify the issue in their equipment or recommended design.
Thank you for your understanding as we work through these complex infrastructure changes.
Brett Coryell, Deputy CIO, University Technology Services
Dee Cantrell, CIO Emory Healthcare
Rich Mendola, Enterprise CIO
