Some systems are experiencing issues

16th February 2019

No incidents reported

15th February 2019

No incidents reported

14th February 2019

Aurora Objects Performance degradation Aurora Objects and Aurora Compute

We are currently experiencing availability and performance issues on our Aurora Objects platform and Virtual Machines in our AMS02 zone. You may see increased loading times and/or poor performance.

Our technicians are investigating the problem and we're working very hard to resolve the issues as quickly as possible. Our sincere apologies for the inconvenience.

Update #1: We have identified a possible cause of the issue and are working on a possible solution.

Update #2: We have started a process that will mitigate the current problems, in this process there will be data movement and this in itself might cause a small amplification of the issue until that part is done. This means that sometimes issues can come back, but over time the overall problems will be reduced.

Update #3: At the moment we're experiencing outages and slow requests on our objects platform again, our technicians are working very hard to resolve these issues as quickly as possible.

Update #4: The issue is unfortunately also affecting virtual machines in our AMS02 zone. We hope that recently made changes will soon have a positive effect. Because of the size of the dataset, it hard for us to give a indication at this moment but keep updating when we can.

Update #5: Because of an increase in issues on our Stamina AMS02 zone we are reverting the running operation to prevent any further issues and will be looking for other solutions.

Update #6 (15:10) For the past hour, the performance of the virtual servers in ams02 seems to be normal again.
We've added more hardware to the objects cluster to be able alleviate the pressure currently impacting service.

In the past weeks, we have prepared for hardware upgrades for objects and the complete separation of objects and virtual servers and ordered additional hardware to facilitate this. Unfortunately, the hardware delivery takes longer than expected.
We now expect that the delivery will take place within a few weeks and that we will perform maintenance in the upcoming weeks to improve performance, and the first week of march for the separation.
For both maintenances, we expect that they will both improve the reliability and the performance of the objects cluster.

Update #7 (18:30): There are certain virtual machines in AMS02 that are again experiencing issues which are still caused by the earlier data relocation. Most of the data is already available again, other parts will come back over the course of the next couple of minutes.

Update #8 (10:00): During the night and this morning, we have successfully performed operations on the storage nodes, which reduces the impact caused by the data relocation on the cluster. Nearly all data has been moved off the virtual servers part. When that completes, virtual servers are no longer impacted by the objects data. Meanwhile, we've received a part of the hardware needed to improve the performance on the objects side. We expect to be able to perform the maintenance to add that hardware today.

Update #9 (12:50): We have completed the final steps to separate the objects and the "virtual server" part mentioned in the last update. We expect the objects data move to finish somewhere this weekend. Because we are still moving objects data and the fact that we need to shutdown the machines we decided to place the hardware when the data move is completed.

13th February 2019

No incidents reported

12th February 2019

No incidents reported

11th February 2019

No incidents reported

10th February 2019

No incidents reported