On April 1, 2018 at approximately 9:03am MDT, roughly 10 percent of our churches could have experienced either intermittent connection issues or full outages, which lasted up to 46 minutes. These connection issues were a result of an inefficient API call being called thousands of times. These connections placed stress on our infrastructure in places where we did not have adequate monitoring. Unfortunately, because of a unique circumstance where our monitoring connections were able to connect, but churches weren't, our engineers were not actually alerted to the problem until we heard from churches via our staff who started receiving emails and phone calls; this resulted in an abnormally long response to the outage. As soon as we became aware of the problem, we identified the transaction causing the problem and mitigated the source of it. This allowed our connections to return to normal and service was restored.
On October 17 2017 at approximately 1:32am MDT, we became aware of the inability for church partners to access Church Community Builder. Our engineers immediately started troubleshooting to diagnose the issue and upon investigation concluded that our data center was affected by a city power grid failure. Once we identified the problem as a power interruption our engineers were quickly dispatched to our data center to continue analysis on-site and understand the full impact. Our data center is equipped with all-UPS (uninterruptible power supply) power all the time, a dedicated generator and fuel, and A+B Power to every cabinet with the intention of providing 100% power uptime. Normally, in the event of interrupted power delivery, the UPS supports our systems while the power generator is started. Unfortunately, the batteries in the UPS system failed despite recent 3rd party inspection findings labeling the UPS as functional and in good health. As a result, our servers needed to be manually started and brought back into our network. Our application started to respond and allow access again at approximately 3:49am MDT.
Update: Aug 24, 2017, 10:45am MDT: Our Help Desk is back up and functioning normally. You should be able to successfully access all articles and videos, and Support Contacts can once again submit tickets.
On the evening of August 22, 2017 at approximately 7:38pm MDT, we became aware of the inability for roughly 13% of our church partners to log in to Church Community Builder sites. We immediately started investigating the cause and after following our process of diagnosis, we identified the problem was due to a hardware failure on one of our primary database servers. As is protocol, our engineers began the process of rerouting those church partners to one of our secondary database servers and sites became accessible again at approximately 8:09pm MDT.