Fastly outage: What caused it and do internet CDNs have too much power?

Thousands of websites were down for almost an hour. - Copyright Getty via Canva

Published on 09/06/2021 - 17:25•Updated 10/06/2021 - 01:13

"Even though there were specific conditions that triggered this outage, we should have anticipated it," said Fastly.

Most people wouldn't have heard of Fastly or CDNs until thousands of websites including Amazon, Paypal and Reddit were brought down for almost an hour on Tuesday.

As is the way with infrastructure, we don't tend to think about it until something goes wrong.

Here's all you need to know about why some of the world's biggest websites went down.

What happened?

On June 8 at 11:47 CEST, users of sites including Amazon, Reddit, many news websites and even government web services were met with a 503 error message.

It took 49 minutes for 95 per cent of the affected websites to get back up and running to almost full capacity.

The reason for this is was due to a bug on Fastly, a content delivery network (CDN) used by thousands of sites.

CDNs allow websites to store data on mirror servers across 26 different countries. This means users can browse them more smoothly.

Essentially, we're talking about wires across continents.

Here’s an example: if you're in the United States and you want to watch a video on a site based in Belgium, rather than waiting for the content to come from Europe, which would take a while, your traffic could be diverted through a CND server in New York, which cuts out the loading time.

To prevent us all from getting annoyed when the internet is too slow, much of the internet's traffic is routed through one of three CDNs: Fastly, Cloudflare or Amazon’s CloudFront.

On Tuesday, the Silicon-Valley based Fastly said it experienced a global glitch due to “an undiscovered software bug" that caused an outage that affected 85 per cent of its network.

The bug "was triggered by a valid customer configuration change," the company said.

"This outage was broad and severe, and we're truly sorry for the impact to our customers and everyone who relies on them," said Nick Rockwell, senior vice president of engineering and infrastructure at Fastly in a blog post.

"Even though there were specific conditions that triggered this outage, we should have anticipated it."

What are the consequences?

Time means money on the internet. As PayPal was down, those who rely on international transfers were affected. As for media websites, many that rely on advertising, including The New York Times, would have been hit as their adverts could not be seen.

And for government websites, such as those in the United Kingdom and the United States, people could not make appointments to receive the coronavirus vaccine or receive potential benefits they may be entitled to.

Although it lasted under an hour, "such a relatively minor thing can have such wide consequences for so many websites," Dr Corinne Cath-Speth, a recent PhD graduate from the Oxford Internet Institute told Euronews.

"I think that says a lot about the consolidation that you see in the CDM market where there are only a few providers and that if one thing happens with one small company, then that affects such a large swath of the entire internet."

What are the solutions?

As we are so used to the speed we get our information online, no one will want to get rid of CDNs and return to the low speeds of the past.

The problem is two-fold, said Cath-Speth.

"On the one hand, we are very reliant on CDNs and the CDN market is very small as there are only a limited number of providers."

She said that’s because the infrastructure needed to provide really good CDN services is only possible if you are a large-scale company, as it costs a lot of money to build servers all over the world and provide the cybersecurity provisions necessary.

As part of her research, Cath-Speth said she interviewed a lot of technology specialists who work for the big CDNs. One of their frustrations, she said, was that people do not realise how fragile and complex the internet is and fail to understand that mistakes do sometimes happen.

But overcoming our impatience with the internet appears an unlikely solution.

Another fix could be for websites to use multiple CDNs, so that if one fails they can switch to the other.

But the reason many companies do not have multiple CDNs is that they are very expensive, due to the lack of competition.

"CDNs are fully aware of the fact that they have this amount of power and they don't necessarily see the first instance of responsibility lying with them because they tell their clients that they should get multiple contracts and if they subsequently choose not to mistakes happen and it’s not their fault," said Cath-Speth.

Many of us had not even heard of Fastly until the blackout of thousands of websites. But their shares on Tuesday rose 11 per cent after the debacle, bringing the company value to $6.5bn (€5.3 billion).

Comments