Subscribe

Critical maintenance on ROSE station server.

Scheduled for Sunday 25th October 2020 at 17:00 (London)

Schedule/description of work

Following a RAID rebuild failure on Friday we are required to update our backups and restore the contents of Rose onto new hardware. Due to the nature of this task, performance may become degraded at points within the maintenance window. We expect full maintenance to be complete within 36 hours.

Scheduled start time
October 25, 2020 17:00
Duration
2 days 20 hours
Status
Finished
← History

Updates

The DC team have confirmed the server is now operating fully. This critical maintenance window has now closed. If you continue to experience issues with your station please contact help@radio.co

Posted by Radio.co DevOps

The RAID rebuild has completed. The verification process has begun which ensures data integrity. This is a background task which does not affect performance. We will keep this maintenance window open until this process completes however stations should be playing normally. If you have any issues with your station please contact help@radio.co

Posted by Radio.co DevOps

Progress update: The RAID rebuild is currently at 86%. We expect this process to complete within the next 3 hours.

Posted by Radio.co DevOps

Progress update: The RAID rebuild is currently at 71%. We expect this process to complete within the next 9 hours.

Posted by Radio.co DevOps

Progress update: The RAID rebuild is currently at 52%. We expect this process to complete within the next 15 hours.

Posted by Radio.co DevOps

Progress update: The RAID rebuild is currently at 40%. We expect this process to complete within the next 20 hours.

Posted by Radio.co DevOps

Progress update: The RAID rebuild is currently at 22%. We expect this process to complete within the next 24 hours.

Posted by Radio.co DevOps

We have disabled waveform generation (used for the track editor) for new uploads to reduce stream hangs while the system rebuilds. We ask users to refrain from uploading files if possible during the rebuild period. If you do experience your stream hanging severely during this time, please contact help@radio.co.

Posted by Radio.co DevOps

The RAID array is currently rebuilding. We expect this process to take approximately 24 hours.

Posted by Radio.co DevOps

The server is now back online. Stations should be coming back online too. We will confirm the status of the drives momentarily.

Posted by Radio.co DevOps

The server is currently rebooting...

Posted by Radio.co DevOps

The drive has been replaced, however the server must be rebooted. We are doing this momentarily. We expect no more than 10 minutes of unavailability.

Posted by Radio.co DevOps

The maintenance window for hard drive replacement will begin shortly.

Posted by Radio.co DevOps

Datacentre staff will be performing a hard drive replacement between 14:00 and 16:00GMT. We do not expect any server downtime however there may be degraded performance during the RAID rebuild.

Posted by Radio.co DevOps

The backup of the server has been completed. Stations appear to be stable. If you have any issues with playout please contact help@radio.co. Our DC team will be carrying out hardware upgrades Monday AM (UK time). This maintenance notice will be updated when this begins.

Posted by Radio.co DevOps