On the evening of January 9, 2016 at approximately 9:30pm MST, our engineers became aware of the inability for church partners to log in to their sites, causing a service interruption. We immediately started investigating the cause and after following our process of diagnosis, we identified that the problem was not related to software and was likely hardware related. Upon discovering the problem, our engineers were immediately dispatched to our primary data center.
As we tested our infrastructure we distinguished a switch in our network topology that had partially failed. As per our protocols this switch was rebooted and started passing traffic properly again at 11:14pm MST. As part of our recovery protocol we monitored the switch and started seeing behavior that prompted us to initiate a failover to backup hardware at 12:25am MST on January 10.
While service was restored in this monitoring period, we made the decision to fail over to our backup switch to ensure that we could provide uninterrupted service going forward. This resulted in an additional 10 minute service interruption as traffic was being rerouted. Service was again restored as of 12:35am MST and we continued to monitor behavior before sending out an all clear tweet on our twitter account, @CCBTech, at 3am MST. Our engineers will be doing an in depth analysis of our redundancy and failover protocols to ensure that we are always continuing to serve you in the best way that we are able.
To receive updates when we are having technical difficulties, follow our Twitter account @CCBTech.
Church Community Builder is dedicated to being an integral partner with you in your mission and we are driven to provide an excellent experience that empowers you to be successful in making disciples.