How Is CDNs Network Performance For Streaming Measured?

No matter how big or small a content delivery network is, they all talk about the performance of their network. Small regional service providers, global CDNs, P2P providers etc… all market the value in everything they do regarding how good their network performs. The problem is, the more customers I ask, the more of them say they
don’t know how to compare one network to another for video streaming – and neither do I.

Aside from the network or servers being down completely and not being able to get the content at all, how does a customer see and more importantly measure the performance amongst any network? I’ve asked a lot of the CDNs and they all speak to Gomez, which is considered the industry standard amongst the networks, but Gomez doesn’t support streaming. It seems that most use Gomez to show network performance for delivering small cache objects and the market just assumes that any content, static or streaming, must see the same kind of performance. But as any network knows, delivering and scaling the delivery of small static images via HTTP is very different than delivering and scaling a 300Kbps stream via RTSP or RTMP.

Keynote provides a service to check your files on a particular network, but that service does not compare it from one network to another and it’s not real-world in the sense that Keynote does not have thousands of agents checking the files as if you have many viewers from all over the world testing the content. Keynote has only a handful of locations that tests the content which is evident by the fact that every delivery network in the Windows Media Certification program for hosting has been given an A+ ranking from Keynote, no matter the size of their network or the number of servers they have. I’m not knocking Keynote’s service, but it’s just not a real-world test if you don’t have thousands of clients all over the world, on many different networks, accessing the content.

I’ve asked many of the CDNs about any tools on the market or any products they have built that customers have access to, but so far, those CDNs I have asked have agreed that there is no easy way to really compare streaming media network performance. If that is the case, what options do customers really have? What is the solution to solve this problem in the market?

If you are a customer of a delivery network, how do you judge performance and what does performance mean to you? For the CDN vendors, what data do you share with customers that gives them an insight into how your network performs? Is there anything you show them that is specific to streaming media delivery?

  • Steve

    I think Keynote currently is the best neutral monitoring party.
    The real issue is that the CDNs aren’t clear on what capacity they actually offer in total and per region. Some CDNs have just one server in a country/region. Europeans may end up watching in Japan.
    CDNs tend to redistribute viewers to servers as soon as the local edge server reaches the 95 percentile average limit. CDNs (even Akamai) prefer to cut costs instead of offering the best performance.
    Users are redirected to other servers in other regions resulting in lousy connectivity (for these users). This happens a lot. I know centralized farms that actually perform better globally than distributed networks because of these redirection to bad regional interconnected servers issues.
    CDNs also don’t guarantee peak capacity: the networks are shared. The edge servers are shared among the customers and they don’t dimension to handle simultaneous peaks from multiple customers.
    I know some smaller CDNs which offer dedicated media clusters (flatfee, no traffic charge) with guaranteed performance 24*7 and the option to use additional media server farms to handle larger peaks. And they are completely transparent for offered capacity and costs I’d consider that!

  • thePlatform has a QoS reporting tool, which consists of a couple lines of Javascript that gets embedded in the player and can offer some information regarding quality of the streams from the end users perspective, i.e. how many frames were dropped, how much rebuffering during the video etc. We had one corporate client who used the technology to diagnose their eCDN, so that if users in Japan were getting lots of rebuffering and dropping the video, they could pull an underutilized server from UK for example. Basically, it offers the client a snapshot of what their end users experience might be like when viewing a piece of video.

  • random_graph

    Help me define first what “performance” means, because “speed” is hardly a meaningful term. Customers want to know the following about their CDN:
    1) Can they scale concurrent demand to meet needs? This is qualitative question for mitigating risk. No CDN reveals their scalability limits, they just say “no problem”.
    2) Will my consumers have a good experience? QOE can be measured in terms of startup latency, jitter, buffer events, %frame arrival, local resource consumption, browser crashes, compatibility, etc. Determining many of these requires a client-side presence, or otherwise capturing MMS events for example. It’s not clear yet what the best 2 or 3 metrics are for QOE, mainly because they’re not available to customers today!
    3) Will CDN be able to do anything about #2? We hear repeatedly…”Our viewers are reporting dropped streams, but our CDN doesn’t have any idea how to solve the problem.” Move’s approach to rate selection is a nice example of what a CDN can do.
    4) Can I rely on their reports to accurately reflect what’s happening? Sample based QOS-testing is clearly a weakness as expressed above. It is by definition an incomplete view, it will never capture the real QOS profile across the market where things like local congestion play a big role in viewer experience, and customers are discounting its value.
    One could also argue that security, privacy, controllability, etc should be part of “performance”. Too broad to be meaningful.

  • Jeff Geiser

    Thanks for the mention. Keynote’s Streaming Perspective service is an offering that we have had for over 4 years and is leveraged by all of the leading content delivery networks and streaming content companies.
    We believe that the quality of streaming has a direct impact on user adoption and retention. Therefore, we provide our customers insight into the quality of their delivered streams – along with a wealth of network related and streaming related metrics.
    One of the primary uses of our platform is to provide insight into how well network providers deliver streaming content relative to each other. Not only do content delivery networks use our service to prove the performance of their delivery to that of an alternative, but many companies have used our service to evaluate the performance of these networks as part of their buying process.
    Additionally, many customers of ours that happen to leverage multiple CDNs to deliver content use our reporting capabilities to monitor in real-time their performance – allowing them to flow content and the associated traffic to the most suitable provider.
    As you may know, Maven Networks is an example of a company that offers our monitoring solution to their customers as an integrated component of their platform.
    I would be more than happy to share with you more details of our industry leading solution along with our solutions for measuring the scale of streaming networks.
    Keynote Systems is certainly the market leader in the mobile and Internet test and measurement space and we feel that our Streaming Perspective solution is a key contributor in that regard.
    I look forward to speaking,
    Jeff Geiser
    Global Director, Business Development
    Streaming Media and VoIP
    Keynote Systems, Inc.