In my nearly 20 years of service with the State of Idaho, I have never witnessed a project so well managed, planned and executed as the UPS (Uninterruptable Power Supply) replacement project.
With planning starting in 2023, the initial approach to this work was to completely shut down the datacenter for an entire long weekend. ITS leadership was not comfortable with this approach and directed staff to “find a way” to replace the UPS without a shutdown.
Adhering to the direction and accepting the challenge, this project suddenly became very complex.
“How do you keep the datacenter up and operational when the power that feeds everything has to be shut off?”
A number of ITS engineers and electrical contractors went to work on a plan. This led to a very innovative and clever engineering exercise that included building temporary power distribution panels and connecting these to number of large portable generators that were staged just outside the datacenter.
Temporary power generators
Power cables were run from the generators, through a window that was removed in the building and into the data center temporary distribution panels.
Temporary power feeds to datacenter and Temporary power distribution panels
From the temporary power distribution panels, individual rack power leads were built and staged at each rack ready to assume the power load for all the datacenter equipment in order to keep everything powered on and operational.
Much effort also went into ensuring all safety and electrical codes were closely adhered to.
Temporary power leads for each rack
Another problem to solve was cooling. With all the cooling units being powered by the original UPS, these units needed to be powered down, leaving the datacenter with no cooling. To overcome this issue, four portable cooling units were installed in the datacenter, along with exhaust ducts that were run through a temporary window opening. The temporary coolers were powered by a dedicated temporary generator.
Temporary cooler (one of 4)
Once all equipment was transferred over and running on generators, the electrical contractor wired in the new UPS. During this time ITS staff monitored power and cooling to be sure everything remained operational.
A coordination and monitoring center was established in the building 2 Coral Room with camera monitoring, break area, food, and drink. Senior management was stationed there to provide oversight.
Morning safety brief
Over the course of several months leading up to the cutover, Kevin Christensen’s team worked to create a detailed manual that contained a wide range of information that was distributed to key people on the project. This included:
- Key contact information
- Event sequencing
- Details on all equipment that was in each rack that also included what systems where in those racks and who to contact if there was an issue with any individual rack or piece of equipment.
- Based on this information they also conducted multiple tabletop exercises to review as many possible scenarios to make sure the staff was as prepared to deal with any potential unplanned event.
Using the detailed procedure manual created by ITS
On Saturday Oct. 11th, ITS staff and management were onsite at 3 a.m. to start preparation for the day.
As a result of the incredibly detailed planning, rehearsal, communication and ingenuity, the goals of the project were met after a very long day that ended at approximately 7 p.m. Congratulations and thank you to all involved.
A very special thanks to the following:
- Kevin Christensen – Lead Manager
- Aaron Archuleta – Physical Datacenter Manager
- Joe Luff – Engineer IV
- Tyler Luna – Engineer III
- Mike Sheets – Engineer III
- Wall Towler – Engineer III
- Braden Harada – Engineer III
- Marshal Shultz – Engineer III
- John Brown and Josh Hardy – Network Support
- Dan Thornock and Jon Pope – Executive Support
- Chris Haener – Team Morale Entertainment and Food Supplier
- Jennifer Gonzalez and Kristin Bartz – Communications
- Alyssa Valdez – Project Management
- Tina Fuller – Project Management
- Jesse Busack – Quality Electric
- John Beck – McAlvain Construction
