On Friday, February 3, we made an update to one of our servers that handles sending out attendance reminder emails for upcoming events. Due to an edge case that was not present during our initial testing, the server caused the emails to backup and not be sent out over the weekend. We have rolled back our changes and our engineers have restored attendance reminder delivery.
We work really hard to keep your system up and running, but sometimes we experience some hiccups. When that happens, we want you to be in the know. Outages, downtime, service interruptions, and other technical difficulties will be reported here.
On the evening of January 9, 2016 at approximately 9:30pm MST, our engineers became aware of the inability for church partners to log in to their sites, causing a service interruption. We immediately started investigating the cause and after following our process of diagnosis, we identified that the problem was not related to software and was likely hardware related. Upon discovering the problem, our engineers were immediately dispatched to our primary data center.
On the afternoons of May 26 and May 27, 2016, our engineers became aware of the inability for church partners to log in to their sites, causing a service interruption. These interruptions lasted approximately 15–20 minutes each. Our first inclination was that a job we ran on our databases was causing the latency and thus changed the frequency of the job to help balance out database resources. On Tuesday, May 31, 2016, we experienced the increased latency again and our church partners were unable to log in to the software for approximately 20 minutes. We continued to investigate and found the outage was caused by a backline process running on our database servers to archive files. As soon as the issue was identified, we killed the process and service levels returned to normal. We are implementing new processes to complete this task in a way that will not interfere with access to your Church Community Builder software. We are wholeheartedly dedicated to presenting a consistent and beautiful experience for our church partners, and we apologize that we did not do that in this case. We are truly humbled and grateful that we get to work with all of you and understand the inconvenience this caused. Thank you so much for allowing us to strive onward with you in your ministry!
On March 17th, 2016 at approximately 4:24pm MDT Church Community Builder was the target of a Distributed Denial of Service attack. Our monitoring systems alerted our engineers to the issue, and we immediately took action and tweeted to our @CCBTech Twitter account that we were aware of the problem.