At Church Community Builder we are passionate about helping churches grow disciples, and a large part of our commitment is to build reliable and high performing software that empowers you to manage your church effectively. In an effort to better our reliability and performance, we are migrating all of our supporting infrastructure and hosting to Amazon Web Services (AWS). We have planned this project in a way that will protect your daily operations as much as we can and keep the integrity of your data. We are extremely excited about the benefits of moving our infrastructure to the cloud and we’ve been preparing for a long time!
On April 1, 2018 at approximately 9:03am MDT, roughly 10 percent of our churches could have experienced either intermittent connection issues or full outages, which lasted up to 46 minutes. These connection issues were a result of an inefficient API call being called thousands of times. These connections placed stress on our infrastructure in places where we did not have adequate monitoring. Unfortunately, because of a unique circumstance where our monitoring connections were able to connect, but churches weren't, our engineers were not actually alerted to the problem until we heard from churches via our staff who started receiving emails and phone calls; this resulted in an abnormally long response to the outage. As soon as we became aware of the problem, we identified the transaction causing the problem and mitigated the source of it. This allowed our connections to return to normal and service was restored.
On October 17 2017 at approximately 1:32am MDT, we became aware of the inability for church partners to access Church Community Builder. Our engineers immediately started troubleshooting to diagnose the issue and upon investigation concluded that our data center was affected by a city power grid failure. Once we identified the problem as a power interruption our engineers were quickly dispatched to our data center to continue analysis on-site and understand the full impact. Our data center is equipped with all-UPS (uninterruptible power supply) power all the time, a dedicated generator and fuel, and A+B Power to every cabinet with the intention of providing 100% power uptime. Normally, in the event of interrupted power delivery, the UPS supports our systems while the power generator is started. Unfortunately, the batteries in the UPS system failed despite recent 3rd party inspection findings labeling the UPS as functional and in good health. As a result, our servers needed to be manually started and brought back into our network. Our application started to respond and allow access again at approximately 3:49am MDT.
Update: Aug 24, 2017, 10:45am MDT: Our Help Desk is back up and functioning normally. You should be able to successfully access all articles and videos, and Support Contacts can once again submit tickets.