For Video Delivery, It's Not About A "Distributed Vs. Non-Distributed" Network

For as long as I can remember, there has been this long standing debate inside the content delivery industry regarding which network architecture is better, one that is distributed or non-distributed. While many have opinions on the subject, to me, it's the content owner that decides which is best and not the vendor. Over the years I've spoken to hundreds of customers about the topic and frankly, I think it's time for the industry to re-define what it means to have a "distributed" network and the real metrics should be used when comparing one network to another.

Fifteen years after some of the first CDNs started delivering video, many of the CDNs still want to use their network architecture as a key selling point. While that's natural of any CDN to do, the problem is the way some go about doing it. Today, customers don't ask how many servers a content delivery network has and I can't remember the last time any content owner asked me how many servers any CDN has. While Akamai has spent the last ten years trying to convince everyone that a distributed network is the only way to properly delivery content, the market has shown us that there are now many different ways to delivery video with the a very good level of QoS.

In prior year's, Akamai's distributed network mantra was accurate. When CDNs first started to crop up in the market, they would start off with a few POPs and call themselves a CDN, even though they had very little coverage or capacity. But today, times have changed. The real questions customers ask are about capacity, coverage locations, quality of service and peering, not the number of servers. Those are the details content owners want to know and as long as a CDN can deliver a good user experience, it doesn't matter to the customer if the network is distributed or non-distributed. How a CDN sets up their network is up to the CDN and most content owners don't care how they do it, only that the service they buy works with their expected level of quality.

To me, the word distributed needs to be re-defined. Today, all the major CDNs are distributed, even though some don't call themselves that. The market has called Limelight and Level 3's network "non-distributed" because it wants to distinguish their method from Akamai's approach.  In essence, "distributed vs. non-distributed" is shorthand for the market recognizing that there is more than one way to solve the problem.

But putting all those marketing tactics aside, when is comes to specifically delivering video, nearly all the CDNs do it the same way. Yes, there are differences between coverage and capacity, but anyone who knows anything about CDNs will tell you it's not just about how many servers you have. You can have the most servers on the planet and the most "distributed" network around, yet a viewer's video experience can fail due to poor capacity or peering amongst other many other factors. A lot more goes into a CDN than just the numbers of servers they have deployed.

To me, a non-distributed network in today's world is an ISP or colocation company who might have a POP on the East coast and West coast and calls themselves a CDN. That's not a real CDN and it's not distributed. But when you look at all of the major CDNs today, they all have around two dozens locations they serve video out of. That's distributed. Of course the locations they are in and the capacity they have at these locations varies, but then the debate should be about capacity and scale and not about distributed or non-distributed.

Another topic that has to be discussed when talking about distributed networks is the idea that a CDN is delivering all or most of the customer's video content from thousands of locations. While that's something Akamai's marketing language has implied in the market for many years, you'll notice Akamai has never come right out and said how many locations they deliver video from or even what percentage of their servers are used for video delivery. In one of their latest press release from March 16th, talking about video traffic, they were careful to say that customers who deliver video can, "leverage Akamai's global footprint of more than 61,000 servers".

Akamai's strategy has always been to talk high-level about their video delivery services without a lot of details so that the market perception is that they deliver video from thousands of locations. But in reality, no matter what the market may think, we know that perception is simply not accurate.

Yes, Akamai and all the CDNs use thousands of servers, but even Akamai does not deliver video from thousands of locations. There is this myth by some in the industry that all these pieces of video and large objects are somehow being stored and or cached at thousands of locations at the edge. However unlike small objects where that does happen, video files are simply too big to cache in thousands of places. Imagine any large content owner who might have 50,000 videos encoded at 750Kbps. Even if each video were only twenty minutes in length, you'd be talking about almost 6TB of storage, for just one customer. It's possible that the top 1% of the content owners content is cached, but not at thousands of physical locations.

Today, nearly all of the CDNs still want to talk a lot about their CDN architecture yet remain very closed at telling customers and the industry how content is delivered, where it is delivered from, what their capacity is for specific video formats and how their peering works. There is this false notion that some CDNs get all the free peering they want or get to place servers for free inside other ISP. In reality, that's not the way it works. While many CDNs want to say they can't give out a lot of this info due to competitive reasons, considering so many of them have sued one another, it's not like they have that many "secrets". It's not very hard for one CDN to figure out what the other CDN is doing simply by putting their networking skills to use.

Having said all of this, I know some vendors are going to want to tell me that their network is better at delivering video due to the number of locations they are in. In some cases, that may be accurate when compared to a regional CDN, but you'll notice that these same vendors won't tell you how many locations they deliver video out of. Some vendors might also say that data from Keynote or Gomez shows their network as being better than a competitor for video, but that's simply not accurate data.

The CDNs all know where a Gomez or Keynote probe is and can put a server there and then ensure some test content is always bound to that server to make their performance look fantastic. As a result, that data does not tell you anything about the user experience that one can expect from a network. And this is not just speculation on my part as many CDNs have privately said they do this and that it's a common practice amongst many of the CDNs.

But the most important part of this debate is not what I think or what the vendor thinks, but what the customer thinks. If one CDN had a network architecture so much better than another vendor, specific for video delivery, then we would not have a few major players in the CDN space doing more than $100M a year in CDN revenue. And when a content owner like Apple, Netflix, Microsoft and others use two or even three CDNs for the same content, it tells you exactly what they think when it comes to the CDN vendors different network architecture. There is more than one way to deliver video content and get the same QoS and the idea that there is only one way to do something in the online video market simply isn't accurate.

When it comes to delivering small objects, it's a completely different ball game and the kind of network and software architecture that CDNs use can make a big difference in the QoS. Doing dynamic site acceleration (DSA), application acceleration and small object delivery is very different than delivering video and large objects and I'll be writing more about those services in the coming weeks.

While I know some are going to want to argue with me or try and convince me that their network is different or better at delivering video over another CDN, simply based on a distributed or non-distributed architecture, there is nothing to debate. Customers have already answered the question for us. It can be debated what the definition of "distributed" means in today's market, but when we're talking about video specifically, the fact that all the major CDNs deliver video in a similar fashion, I think we already know what the definition of "distributed" really is.

Comments

[Stef van der Ziel]

Spot on Dan! For instance, Akamai claims to have +60K servers, but that does not mean +60K locations. Most of these servers are web caches, not streaming servers. Even if there is a streaming server nearby, Akamai may choose to send a user to a remote server because of cost savings (95%) or server load, or limited asset popularity in your region. Highest performance is not guaranteed.

Another example: Global CDNs with European coverage have a 3-5 superpops in Europe on average, close to the major internet exchanges. Because of this, streaming performance in all non-western European regions (Scandinavia, Iberia, Italy, Eastern Europe) can be pretty poor.

You don't need a distributed set of servers if the underlying network infrastructure itself can guarantee throughput and performance into many regions. But dumping traffic on just 3-5 exchanges isn't enough in a physically fragmented market such as Europe, especially now ISPs are renegotiating free peering agreements.

The most common CDN architectures are:
- Edges (many edge servers, expensive to deploy and manage)
- Superpop (limited number of edge pops, needs great connectivity)
- Megapop (two, maybe three pops, needs a highly distributed network)

These architecture designs are popular with global and regional CDNs. Telcos also deploy CDNs and most of these architectures are designed to offload on-net traffic. So we don't see many megapop designs here. Superpops and Edges are the most popular.

The benefit of on-net telco CDNs is that these CDNs control and guarantee the entire infrastructure, from hosting, distribution, delivery to transportation right into the living room and to mobile devices. I predict that Micro-edge (tiny edge servers, embedded at the outer edges of telco networks) architecture designs will become popular as well.

My experience is that CDNs tend to change their architecture design over the years. Bandwidth capacity is growing, bandwidth prices are dropping. To save costs, CDN operators change from an edges powered CDN to superpops. Other CDNs are deploying small edges in addition to their megapop sites to save on transit costs.

The most important lesson we've learned is that because of this required flexibility, CDNs should not be locked into the network infrastructure or in appliances. Operators want to be able to deploy and remove edge servers at any remote location without dependency on PoP ownership, network ownership, routing equipment or specific appliances.

You're also right that content owners aren't interested in the number of servers. Other quality factors are important: 24*7 customer support, native language support, features, manageability, API's, transparency.

[Kevin]

One thing that has always irritated me and just screams ignorance is when someone references a number of servers. This doesn't just apply to CDN's but pretty much any context it is used in. There is a big difference in the capacity and power between a $3,000 single processor quad core server and a IBM ISERIES 9406-595 going for about $750,000 and every server configuration out there between those price points. I guess it's human nature for the guy with 10 $3,000 servers to brag about having 10 times the number of servers over the guy with 1 $750,000 server, but you a gullible as hell if those members mean a damn thing to you.

Not to mention that experienced administrators and I'm sure the guys at Akamai, Limelight, and L3 fit this category, would be able to get 5 to 10 times the performance and capacity out of the same exact server over Joe Schmoe who hooked the thing up out of the box and did what he could to get it running. But that's another topic...

[Robert Mendez]

How does someone planning to scale a community site assess the things you're talking about? is there a primer on how to do this? Also, how does virtualization affect the differences between a distributed or non distributed array of servers? Are there ways to ascertain which will fit my needs longer term? thanks.

[Vincent Veauclin]

Great article! Finally a speech that contrasts with what we read too often in certain media or expert blogs. The number of servers is not indeed an absolute guarantee of the highest quality of service possible. Even if we all can recognize Akamai makes good job. But many other factors, especially local, can affect the QoS.

Each market has its own specificities, with telcos/ISPs local networks more or less centralized, and peering points more or less congested. Because of those specifics, some local CDN propose a delivery quality as good as the worldwide CDN, whether on a distributed model or non-distributed. And they prove it every day by winning deals against global CDN. Otherwise, it’s been a longtime (since the early 2000’s) that only the “Big Three” would survive on the market!

Moreover, with the irremediable rise of http streaming and intelligent player interfaces, the integrated delivery strategy chosen by some CDN as ipercast makes sense. Controlling the video delivery vertical chain, from CDN to CMS, offers the customer a better monitoring of the quality of service, optimize the end users experience, and compensate in certain situations a lesser granularity of the network.

[Rickson]

The most essential thing about your post is the introduction you have provided here. We do understand that it is a serious topic & having an insight into the subject matter is really enriching. Thanks Dan.