New Chart Shows The Differences Between Performance Monitoring Vendors

Performance monitoring was a very straightforward category of service just a few years ago. Companies set up programmatic test boxes in data centers around the world and pinged services designated for testing as their business model. Companies like Keynote and Gomez provided these synthetic monitoring services, and these companies were icons in the field, responsible for creating the market. However, there hasn’t been many innovations in their services over the years, and competition was primarily always based on price. Because the locations of these servers performing the synthetic monitoring could be discovered, these services were also gamed by smart CDN’s that wanted good scores. As a result, this lead to cynicism regarding the results, making their data much less effective.

The good news is when it comes to performance monitoring, things have evolved rapidly in the last few years. The main reasons for this are that application monitoring, which was traditionally data center centric, has now been brought into the cloud. New pricing models such as Cedexis with its free Radar product are creating a commoditization of the traditional service in favor of a more valuable service (in their case the global CDN load-balancing product Openmix). In addition, RUM (Real End User Measurements) has come to the forefront as the defacto standard for monitoring and is a far cry from the methodology used in the early days by systems like Keynote and Gomez.

These three factors have made a market that was feature stable and resistant to price decline into a completely new market that defies easy categorization. The chart below takes a look at some of the current performance monitoring competitors in the market and details their focus. [UPDATE 10/3: The first version of this chart incorrectly showed Keynote as not supporting RUM. Keynote does have RUM measurements taken by inserting a W3C approved tag on the site.]monitoring-vendors-1024x534

As the chart shows, the competitive landscape is broad, and the features supported vary widely depending on the focus of the company. The prevalence of RUM measurements is also something to take note of, and while the majority of these vendors claim to have some form of RUM, it can be deceptive. The reality is that WHAT is being measured by the RUM also matters. Cedexis for instance measures every CDN (and Cloud) for the purpose of their load balancing solution. These measurements are of specific objects that have been inserted into the CDNs for this purpose. This gives a very clear view of the relative performance across CDNs. Other RUM measurements by some of the other vendors may be more directed at specific customer infrastructure, which could provide more relevant reporting only for a specific customer.

Cedexis is clearly the winner with their billions of RUM measurements a day and the fact that they have built a community out of some of the most ferocious competitors in recent history, namely the CDN vertical. This cannot be underrated as the community that Cedexis has built is not easily replicated and their “Wisdom of the Crowds” message only holds water because they have been able to get every CDN on the planet, for the most part, to participate. Conviva has a very strong offering in the video space but as that space has evolved they may find that the needs for such a specific and very expensive service dwindles. I’ve written about this before saying that I don’t believe Conviva can survive on their own, as a stand-alone service. CDNs like Akamai are now offering a competing platform to Conviva, and it’s bundled within a larger ecosystem that is much easier to sell and is a better way to buy it.

The strongest followers in the space are Dyn, DeepField, Catchpoint and New Relic, and it’s interesting to note that two of the four come from the APM space (Application Performance Monitoring). Time will tell if the APM tool companies can compete, especially when they have Cedexis giving away its monitoring product for free. Cedexis does this so that they can aggregate or pool the community data to inform their Global Load Balancing solution (the higher value product), but it’s still hard for any vendor to compete with free. I’d be interested to hear from customers who use these solutions what they like/dislike most about them in the comments section.

Thursday Webinar – Best Practices for Enterprise Webcasting

Thursday at 2pm ET, I’ll be moderating another webinar, this time on the topic of, “Best Practices for Enterprise Webcasting .” As more organizations discover the benefit of streaming their communications and events live, they are faced with a wide range of technical requirements. Join this webinar to learn the best practices for webcasting inside the enterprise. Presenters will discuss how to deliver fail-proof live webcasts to reach large audiences; creating a flawless user experience; leveraging fault tolerance and automated failover procedures and how best to manage webcasts via a central platform. Attendees will also get advice from Telestream, MediaPlatform, Kaltura and Qumu on:

  • Choosing encoder-agnostic webcasting solutions
  • Solving the unique distribution challenges of video
  • Live-to-VOD automation tips
  • Cloud transcoding for reliable, adaptive delivery to any device
  • Do it yourself vs. “white glove” managed services
  • Adding network and broadcast monitoring rules
  • Tips for producing high quality live webcasts

These questions and many more will be covered in this webinar so REGISTER NOW to join us for this FREE Web event.

Dyn Launches Internet Intelligence Tool With Unique Perspective On iOS 8 Internet Impact

Earlier this week, Dyn launched a new Internet Intelligence platform that allows customers to look at the routing that makes content delivery possible and see the performance between transit providers that impacts customer experience. Using the tool to look at Apple iOS 8 release gives a global view of the event and what is happening at any moment. Performance of Apple’s own CDN, third-party CDNs and operator networks matter, but the interconnection of the CDNs to those eyeball networks is not often examined as a link in the performance chain. Understanding where the iOS 8 download is being served from is the first step and using the specific download URLs, a map can be built that shows the location of each CDN POP and who is serving the traffic from each. As I reported earlier, Apple is delivering the majority of the traffic themselves, with Akamai,  Limelight Networks and Level 3 all sharing in the portion of traffic Apple has outsourced.

This view is a map of where the different CDNs during the release were and what they were serving, specifically for the iOS 8 download. Dyn can produce this view in real time for the actual URLs for all of the files.


If you want to look at specific parts of the world to see exactly how the CDNs are connected, you can examine each of the nodes and how they route traffic through the Internet. This view of Frankfurt, Germany gave a very interesting view, showing when Apple connected to their own CDN versus when they were connected to those of Limelight or Akamai.


Performance upstream of the CDN can cause problems across a customer base. The image below depicts a slow-down inside Amsterdam on Level 3 during the peak hours of the download. The examination can take place on a hop-by-hop basis so each provider in the delivery chain can be examined for performance changes.


CDNs Akamai, Limelight Networks and Level 3 Are In The Mix For iOS8 Downloads

While Apple is delivering the majority of iOS 8 downloads from their own CDN, third-party content delivery providers Akamai, Limelight Networks and Level 3 are getting some iOS 8 traffic. Based on traceroutes and other data from Cedexis, DeepField, DYN and internal data from ISPs, it’s clear Apple is using a multi-CDN strategy especially in specific regions outside North America, particularly with smaller ISPs. Last mile providers are seeing a huge demand for iOS 8, with traffic inside some IPS being  50% higher than normal. Combined, others saw traffic jump from 200Gbps, to a peak of 3Tbps. Apple’s new CDN has been impressive, both by the volume it has handled and also its performance.

Most of the larger ISPs in the U.S. have direct interconnects with Apple and are using Apple’s CDN. Apple put in place a lot of interconnects over the last year, including ones that are paid, and adopted a fairly broad direct peering strategy, especially in North America. For Tier-3 ISP’s, Apple looks to have split the traffic between Apple transit, Akamai and Limelight’s CDN. Akamai and Limelight appear to have gotten more CDN traffic than Level 3, but Level 3 should be the biggest winner in the iOS 8 download as Apple buys a huge amount of IP and other network related infrastructure from them.

It’s impossible to split out the percentage of traffic that is going to third-party CDNs, and Apple will change it based on availability, but looking on the data from multiple services, it’s clear Apple is delivering the majority of the iOS 8 traffic via their own CDN. Again, I can’t say an exact percentage, but it’s safe to say they are delivering the largest share. You can see when traffic started to hit some of the third-party CDNs, in certain regions like Finland, when the iOS 8 download was released. The first chart below from Cedexis shows the latency in Limelight’s network increasing, at the same times iOS 8 was made available for download. The key performance metric for downloads is availability, not latency but it’s a good indication showing which CDNs are being used in specific regions.

The second chart below from DYN, shows even more details on how the CDNs are connected, allowing you to examine each of the nodes and how they route traffic through the Internet. That chart is data from Frankfurt Germany giving a very interesting view, showing when Apple connected to their own CDN versus when they were connected to those of Limelight or Akamai.

Screen Shot 2014-09-17 at 4.38.40 PMscreen_shot_2014-09-18_at_12.31.07_pm


DeepField Data Shows Apple Traffic Jumping From 200Gbps to 3Tbps Thanks To iOS8 Downloads

Following up on the data I already published from Qwilt, DeepField has also shared some data with me showing just how much traffic spiked when Apple’s iOS 8 download was released. DeepField collects data from ISPs that participate in their Internet Observatory research project and the below chart shows Apple traffic from those ISPs, which normally sits around 200Gbps combined, peaking at 3Tbps at 6:15pm ET on Wednesday night. The data represents a cross section of consumer providers, mostly in North America, covering many tens of millions of users. Apple’s new CDN is impressive, it has massive capacity and service providers are seeing solid download rates across the board. Apple did rely on Akamai, Limelight and Level 3 for a small percentage of their overall downloads and I’ll have more details on that in a follow up post.

Screen Shot 2014-09-17 at 11.03.19 PM

For more on Apple’s CDN buildout, see these previous posts:

Feb 3rd: Apple Building Out Their Own CDN To Deliver Content To Consumers

May 20th: Apple Negotiating Paid Interconnect Deals With ISPs For Their Own CDN

July 31st: Apple’s CDN Now Live: Has Paid Deals With ISPs, Massive Capacity In Place