To the Emory Community:
The network problems experienced over the past two days were corrected Monday at approximately 8 p.m., and all University and Healthcare systems were back online by 10 p.m.
We deeply regret the risk, cost and inconvenience created by these outages and are working with our networking vendor to minimize the impact associated with any future changes.
As background, over the weekend, IT took another step in replacing the core networking infrastructure that serves both Healthcare and the University. This conversion is necessary routine maintenance, replacing equipment that is over six years old. It is also a desirable upgrade to satisfy demands for increased capacity and advanced features that support Emory’s multiple missions.
The changes over the weekend were intended to add the last set of features and capacity to move the healthcare networks to the new core with the already-migrated university networks. Last week, the networking team worked with our networking vendor (Cisco) to validate the new, planned configuration and performed multiple successful simulations with identical equipment in our testing lab. On Sunday, the upgrade was installed at 1:30 a.m. EST, and all indications were that it was successful. At approximately 7:45 a.m. on Sunday, network stability wavered in the face of some unseen error condition.
On Sunday afternoon, the networking team backed out a portion of the overnight changes as a workaround, and that led to a period of stability for both healthcare and university applications through Monday morning. Monday’s additional load proved that not all issues were resolved by the workaround.
Continuous troubleshooting in consultation with the highest possible level of technical resources from Cisco were not able to identify a root cause, and at 6 p.m. the networking team began backing out the remainder of the overnight changes from Sunday. Our internal network engineers are now working with Cisco to further analyze the outages and formulate a revised plan for performing the upgrade. Cisco is in the process of building a lab with our exact configuration in order to identify the issue in their equipment or recommended design.
Thank you for your understanding as we work through these complex infrastructure changes.
Brett Coryell, Deputy CIO, University Technology Services
Dee Cantrell, CIO Emory Healthcare
Rich Mendola, Enterprise CIO
The Emory Wheel was founded in 1919 and is currently the only independent, student-run newspaper of Emory University. The Wheel publishes weekly on Wednesdays during the academic year, except during University holidays and scheduled publication intermissions.
The Wheel is financially and editorially independent from the University. All of its content is generated by the Wheel’s more than 100 student staff members and contributing writers, and its printing costs are covered by profits from self-generated advertising sales.
Thanks, sis!