Skipped playlist items - Incident Write Up.

We have completed our initial investigation into the instances of playlist positions being skipped. This notice relates to the now resolved incident.

What happened

We identified several contributing factors rather than a single fault.

During periods of increased activity, particularly around the top of the hour, some requests to retrieve the next track were taking longer than expected or were not completed within the required timeframe. In these cases, the system could move on to the next playlist item.

This was made more noticeable by the way our platform was scaling to handle these periods of increased demand.

We also identified two related issues during the investigation. Changes being prepared for an upcoming feature had an unintended impact on database performance, while a separate reporting issue meant some very short tracks, such as jingles, could play correctly but not appear in Playout History.

Strict track separation rules could also increase the time required to find a suitable next track and, in some cases, result in a playlist tag being skipped.

What we changed

During the incident we made a number of changes to improve playout performance and reduce the likelihood of playlist items being skipped.

These included:

• Increasing the resources available to the services responsible for retrieving upcoming tracks.
• Increasing the number of instances of those services running at all times.
• Adding extra capacity around the top of the hour, when demand is typically highest.
• Improving request handling and increasing relevant timeouts.
• Optimising how track separation rules are processed.
• Temporarily disabling the part of our upcoming Early Access feature which was affecting database performance while that work is further optimised.
• Fixing the reporting issue affecting short tracks and jingles.

These changes resulted in a significant reduction in occurrences during our monitoring period.

What happens next

• We are continuing work to improve response times and platform performance, particularly during periods of increased activity around the top of the hour.
• We are also improving monitoring and alerting across the shared platform infrastructure so that similar performance degradation can be identified earlier.
• Further changes planned as part of our Early Access functionality will also change how this information is handled by the playout system, removing some of the dependencies involved in this incident.
• We have scheduled an internal post incident review to look at the wider incident, the contributing factors and any further changes we should make as a result.

We apologise to customers affected by this issue and appreciate your patience while we investigated and implemented these changes.