Real World Examples - docuMOP would have been the answer!
With three decades of critical IT Infrastructure experience as the base of developing docuMOP, we've learned of many situations which could have been handled better and more simply if docuMOP had been in existence. Here's a few.
- Data Center had battery explosion.
- What happened? - Pressed the Emergency Power Off (EPO) button, data center was shutdown
- docuMOP Procedure - Open the UPS battery breaker, transfer the UPS to internal bypass, then to external bypass, then shut down the UPS
- Large Co location Data Center finds urgent service need in power system.
- What happened? - Call in UPS service organization. Not understanding the system, service company advises there is no way to bypass the UPS, despite the particular system having multiple methods to easily bypass the UPS. Service company recommend waiting till collocation center can be shut down!
- docuMOP fix - Simply transfer to generator, follow the UPS bypass MOP, perform the work and eliminate the risk immediately.
- Large University schedules UPS replacement for their data center with plan to run on generator for the one week project duration
- What happened? - Plan was to transfer to generator prior to the work, then transfer back a week later. Improper procedure left the Transfer Switch logic in normal mode to allow reset to normal 20 minutes after an utility outage. Before work began, utility failed at the facility load stayed up as it was on generator. However 20 minutes later the transfer switch logic initiated a transfer back to utility power with data center load drop during the transfer.
- docuMOP fix - Follow a MOP to transfer to generator which involved locking out the possibility of retransfer while the UPS was not available.
- Small data center with UPS and generator schedules UPS PM. Ideal is to run on generator while the UPS is bypassed being fully serviced. No one at site knows how to get generator to stay on for more than a 20 minute test period which is insufficient time for UPS PM.
- What happened? - UPS service is done on utility power subjecting load to unnecessary risk if utility fails.
- docuMOP fix - develop and follow a MOP to allow generator to run as long as needed.
- Hospital OR UPS routinely trips during transfer switch testing interrupting OR critical power.
- What happened? - Testing was moved to off hour to reduce risk to OR. However in a real utility outage, same issue results.
- docuMOP fix - Obtain the proper documentation on the transfer switch using the documentation to adjust settings of the transfer to eliminate the UPS tripping.
- Small data center planning software update of UPS which requires shutdown and restart of UPS after updating.
- What happened? - Data Center personnel shut down the UPS entirely prior to service appointment to be sure UPS was ready. Unfortunately UPS needs to be powered for software upload, necessitating a restart, second shutdown and second restart.
- docuMOP fix - With a properly written MOP, the UPS would have only had to be shutdown and restarted once, simplifying the overall task.
- Medium size data center called in panic as there UPS was on battery but the facility did not seem to have lost utility power.
- What happened? - Customer call was received and we advised what breakers to look at within their facility. A breaker was found off feeding the UPS and turning it on returned all to normal.
- docuMOP fix - With a docuMOP Process Diagram of the critical power path, data center personnel would have know immediately what breakers to check on. In this case reaching us by phone solved the problem but what if the right individual here had not been instantly available?
- Medium sized data center felt data center was too warm but did not know how to change temperature setting.
- What happened? - A phone call to us allowed us to explain how to change the setting. Someone from out of town that knew how to change the setting had been in the data center and raised the temperature as they were cold. Upon leaving they had not reset the temperature.
- docuMOP fix - A MOP could have been in place that explained how to adjust the setting without needing a special phone call. An additional MOP could have been in place for sign in sign out of the data center with a confirmation line to re set any parameters of the room that were changed while visiting.
- Humidity Settings at many data centers waste energy
- What happened? - Due to no checklists for HVAC settings being developed humidity set points are varied without much engineering basis.
- docuMOP fix - In developing docuMOP checklists a sound engineering based selection would be made for set point such as humidity. Following the checklist would allow for a setting that minimizes operating cost while maintaining adequate humidity levels.
- At a major data center with two locations docuMOP personnel were to meet a facility manager.
- What happened? - Receptionist could not locate the facility manager. When asked about reaching by cell phone receptionist could not find the number.
- docuMOP fix - As this was a docuMOP client we simply looked up the customer docuMOP web on a smart phone and obtained the cell number from the Key Personnel listing available on each page!
- Equipment Preventative Maintenance visit scheduled and planned per normal procedure.
- What happened? - A 45 minute delay occurred because the equipment room was locked and no one had the key and worse no one knew where the key was kept.
- docuMOP fix - Within the Assets Module important information such as where keys are located and who is responsible is kept.
- While doing service at a major call center the fire alarm was tripped by cleaning personnel when a broom stick hit a pull station. Fire dept showed up and sirens blared internally for 45 minutes. After 30 minutes and the fire department had left call center employees were told to return to work while the internal sirens were blasting still. No one on site knew how to silence the alarm which was key protected.
- What happened? Approximately one hour later personnel arrived to turn the sirens off. No one knew whom to call.
- docuMOP fix - A MOP would have been developed to perform critical tasks. In this case the MOP would have listed where the key was, how and when to use it and whom to notify. If properly done perhaps the fire dept could have been notified preventing the dispatch.
- Equipment is in alarm
- What happened? No manuals are to be found. A call to the equipment vendor is made regardless of alarm being critical or not.
- docuMOP fix - With the docuMOP library the manual is readily accessible and a review of the alarm meaning can be done in minutes without making any calls to outside parties.
All of the above are actual examples of problems that docuMOP could improve or correct. No one in the examples acted wrongly or desired a bad result. They are simply real world examples where having and following a docuMOP would have greatly improved the situation.