Cloud Cluster DUS7 is having issues with replicated data accross the network storages

The last critical vServer / Container have been restored. We are now checking additional vServer / Container which were not affected by the incident to make sure that we didn't overlooked something.

Until further notice we consider the incident being resolved by now. You might notice a short downtime of 2-5 minutes of the couple of minutes, as we one of our technicians is exchanging network switches and cables inside the data center to rule out a hardware cause.

A full "post-mortem" containing more details will be provided for clients and partner beginning of next week. We wish you a nice remaining weekend and apologize for any inconveniences causes by the incident.

The first complete vServer / Container have been successful recovered from backups and are completely operational now. Additional vServer / Container are at the moment already restored in parallel. All vServer / Container will be restored on SSD drives.

We've fixed the core issue, and are fixing system services one-by-one now.

We still see very low read/write processes. We are in close communication with the virtuozzo deveopers and they are investigation the root cause. Besides that, we evaluate different recovery scenarios at the same time on various machines.

Restore processes have been started, and we are waiting for things to recover.

Only a few services and vServers seems to be affected in the cluster and primarily databases with large write access seem to be crashed. At the moment we are perfoming file system checks of every possible vServer / container and will recover backups of the affected systems later on.

Some vServers and containers inside this HA cluster might see downtimes of certain services (like MySQL) due to read/write errors. At the time writing we are already investigation the issues together with a Virtuozzo support engineer at highest priority.