Recent Maintenance Incident Impacting Some EC2-Based Multi-Tenant Customers

To our valued Community edition customers, please be informed that for a period of time starting on October 24th at 5:00 UTC and lasting through October 25th at 13:00 UTC, one of ClearDB’s multi-tenant clusters in the Amazon EC2 cloud experienced an incident that impacted database writes.  As a result, if your database existed on this cluster and you performed write operations during that window of time, it’s possible that you may have experienced data inconsistency issues or data loss during this window.

This incident occurred while one of our database engineers was performing maintenance and an unexpected event resulted in the masters falling out of sync on the affected cluster.  Normally we gracefully recover from similar incidents, but in this unique case a full recovery was not possible.

The source of the problem was not immediately clear and it took us some time to understand what had happened and the scope of the issue, including who was affected.  We are initiating email notifications to customers who may have been impacted, and we have added additional safeguards and alerts to our maintenance process to ensure this kind of problem will not repeat.

If you believe that your database was affected by this incident, please open a support ticket through our web portal so our support staff can assist you.  Please note that backups were not affected should you choose to restore them.

To reach the portal from Heroku, simply click on the ClearDB add-on from your application.  Once logged in, choose the “Help & Support” tab, then look for the option to log a support incident.

For the next thirty days, any customers who it’s determined have been negatively impacted by this incident can request a full credit for the month.  Please include any requests for credit in your support ticket.

We’d like to sincerely apologize to impacted customers.  We recognize that we fell short of our high standards this month, and we greatly regret any disruption this may have caused you.  Please be assured that we take this incident very seriously and will continue to take steps to ensure that it is not allowed to recur.