On October 17 2017 at approximately 1:32am MDT, we became aware of the inability for church partners to access Church Community Builder. Our engineers immediately started troubleshooting to diagnose the issue and upon investigation concluded that our data center was affected by a city power grid failure. Once we identified the problem as a power interruption our engineers were quickly dispatched to our data center to continue analysis on-site and understand the full impact. Our data center is equipped with all-UPS (uninterruptible power supply) power all the time, a dedicated generator and fuel, and A+B Power to every cabinet with the intention of providing 100% power uptime. Normally, in the event of interrupted power delivery, the UPS supports our systems while the power generator is started. Unfortunately, the batteries in the UPS system failed despite recent 3rd party inspection findings labeling the UPS as functional and in good health. As a result, our servers needed to be manually started and brought back into our network. Our application started to respond and allow access again at approximately 3:49am MDT.
While our application started responding and allowing access and requests, the power outage did cause certain parts of our infrastructure to need manual intervention to start operating at 100%. All of our engineers continued working over the next 18 hours to verify the integrity of data on our primary database servers, restore email deliverability, reactivate all internal monitoring systems, and turn on peripheral and non-critical systems. Know that during this event your data was safely replicated, stored, and secured in our offsite redundant data center and was never in danger of being lost.
We know that this outage was disruptive and unacceptable and we do everything we can to avoid any service interruptions. The process has already started to replace the batteries in the primary UPS system that supports our systems and a secondary, redundant UPS system is also being put in place to avoid this from happening in the future. We're also using new information gained from this circumstance to make our redundant, offsite systems even better prepared for a scenario where a full failover would be required. We are dedicated to partnering with you in your mission and all our Technology Operations resources over the last 2 days have been devoted to restoring Church Community Builder to the level of service you expect from us. Thank you for your patience and understanding and for allowing us to work alongside you.
To receive updates when we are having technical difficulties, follow our Twitter account @CCBTech.