Incident Details

Resolved [04/10/2020 13:45]

Update [20/08/2020 23:15]

We have received a copy of the RFO the data centre has supplied. The outage was ultimately caused by a known single point of failure in their power design. Full details can be read here:

https://cdn.interdns.co.uk/downloads/rfo/LD8-RFO-18-08-2020.pdf

We will provide details of the Virgin and Sky RFO in due course, and from this we can determine any compensation for SLA clients impacted.

Update [20/08/2020 10:53]

Services have remained stable. We await RFO from the affected carriers and will share in due course.

Update [18/08/2020 22:30]

We can see that Virgin circuits have just been restored across both of our affected NNI.

We will endeavour to share full reason for outage notifications from the carriers as they are received in the coming days.

Update [18/08/2020 21:30]

We have chased Virgin Media for an update given that the initial estimates have passed. They are not in a position to supply one yet. They are suggesting that there remains power issues on the 4th floor as Equinix move circuits to new equipment. We will continue to share updates as they are received.

Update [18/08/2020 17:12]

Virgin's latest update is as follows:

<-- snip -->

The engineers confirmed that on site investigations are ongoing. We have reached out to the escalation contact we have and have had emails stating ERT between 19:30-21:00, we have asked further questions and if we can add any assistance to getting services restored any sooner.

<-- snip -->

They have confirmed that of 104 NNI affected they have restored 7. So restoration is happening, it's just slow progress sadly. We are not entirely sure why now that power has been clearly restored this is set to take so long. We will continue to apply pressure to expedite our remaining 2 links.

Update [18/08/2020 16:33]

We apologise for the continued disruption for Virgin circuits. We are chasing them for an update, and can only assume that power remains an issue for their location. We will apply further updates once we have anything to share.

Update [18/08/2020 14:32]

Virgin now have access to their suite, but power still hasn't been restored. We remain without an ETA or official word from the data centre on progress. Updates will flow to you as they are received.

Update [18/08/2020 13:28]

The current update from Virgin is that they are in a queue of engineers on-site awaiting access, this is due to the COVID-19 social distance rules. It's unclear whether their part of the data centre has power, or if they remain on the list to move to the new infrastructure. We apologise for the continued delay.

Update [18/08/2020 12:41]

We are seeing Sky affected circuits start to restore. Please treat them at risk for the time being.

We still have 2 further Virgin NNI to restore.

Update [18/08/2020 12:24]

Sadly there has been no further communication from the data centre as to the current state of resolution. We have no further details to share or ETA. We apologise for the inconvenience will be causing. Be rest assured we will share updates as they are received.

Update [18/08/2020 10:21]

There have been no further updates from the data centre.

Be rest assured we will update this feed as soon as more information, including any ETA, is made available. Please avoid calling our support teams where possible and refer to this feed. We are not withholding any further details.

Update [18/08/2020 09:10]

The latest update reads:

<-- snip -->

Equinix IBX Site Staff reports that IBX Engineers are currently migrating customers' supplies into new infrastructure. IBX Engineers will start to investigate the cause of the failure on UPS.
The next update will be provided in approximately 30 mins.

<-- snip -->

Update [18/08/2020 08:42]

The latest update from the data reads:

<-- snip -->

Equinix IBX Site Staff reports that IBX Engineers and the specialist vendor have begun restoring services to customers by migrating to newly installed and commissioned infrastructure. IBX Engineers continue to work towards restoring services to all customers and further updates will follow when more information is available.

The next update will be provided in approximately 30 mins.

<-- snip -->

Sorry that we don't have an ETA, and are conscious that this is likely to impact working hours. We will continue to supply updates as they are received.

Update [18/08/2020 06:51]

The latest update from the data centre reads:

<-- snip -->

Equinix IBX Site Staff reports that fire alarm was triggered by the failure of output static switch from Galaxy UPS system supporting levels 1, 2, 3, 4 in building 8/9 at LD8. This has resulted in a loss of power for multiple customers and IBX Engineers are working to resolve the issue.

The next update will be provided in approximately 30 mins.

<-- snip -->

Update [18/08/2020 06:10]

Our links remain down and no further communication has been received from the data centre about the situation or any estimated resolution times.

Our BT Wholesale DSL circuits have automatically re-routed via our Telehouse North POP. It is Ethernet circuits which remain down. Apologies for the disruption this may be causing.

Investigating [18/08/2020 05:08]

We have been alerted to a potential fire at the data centre. We don't have any further details to share other than the building has been evacuated.

Sky, Virgin, TalkTalk and BT Wholesale leased line circuits terminating to this location may be currently affected.

We will share further details as they become known.

Investigating [18/08/2020 04:58]

We are currently looking into an issue affecting leased line circuits terminating at LD8 data centre.