On April 1, 2018 at approximately 9:03am MDT, roughly 10 percent of our churches could have experienced either intermittent connection issues or full outages, which lasted up to 46 minutes. These connection issues were a result of an inefficient API call being called thousands of times. These connections placed stress on our infrastructure in places where we did not have adequate monitoring. Unfortunately, because of a unique circumstance where our monitoring connections were able to connect, but churches weren't, our engineers were not actually alerted to the problem until we heard from churches via our staff who started receiving emails and phone calls; this resulted in an abnormally long response to the outage. As soon as we became aware of the problem, we identified the transaction causing the problem and mitigated the source of it. This allowed our connections to return to normal and service was restored.
We know this was unacceptable for our affected church partners, and we sincerely apologize for the negative impact this had during your Easter Sunday services. The engineers who were responding to the event are the same people who provide System Status Updates through Twitter, so they focused their efforts first on restoring service, and then providing a status update once the solution was in place.
We are driven to have responsive and resilient services and we will be working with our development team to understand what went wrong with this particular transaction. We have also set up new custom alerts to inform us of warning signs of this sort of incident, allowing us to prevent future outages by stopping this behavior before connections are maxed out.
Our hearts are completely dedicated to helping our church partners thrive and make disciples, especially on a day as critical as Easter, and want to be a service that enables you to focus on your people. We know that for the affected churches this morning, we failed in that. Please know that we are going to do everything we can to keep this same event from happening in the future. We are truly sorry. Thank you for allowing us to partner with you in the Kingdom.
To receive updates when we are having technical difficulties, follow our Twitter account @CCBTech.