Here’s How The Comcast & Netflix Deal Is Structured, With Data & Numbers

There’s been a lot of speculation involving the business and technical details surrounding the recent deal between Comcast and Netflix and plenty of wrong numbers and information being used. I thought it would be helpful to detail what’s really taking place behind the scenes, highlight some important publicly available data in the market, talk about the deal size, and debunk quite a few myths that people are spouting as facts. It’s time we cut through a lot of the misconceptions of the deal, from both a business and technical level, and focus on what’s really happening. This is a long post, but if you really want to know what’s happening, I’ve tried to make it really detailed. [My first post on the deal can be found here: Inside The Netflix/Comcast Deal and What The Media Is Getting Very Wrong]

From a technical level, Netflix has their own servers that are sitting inside third-party colocation facilities in multiple locations. To connect Netflix’s servers to ISPs, Netflix buys transit from multiple providers, which then connect their networks to the ISPs. Netflix pays the transit providers for those connections and with that, gets a certain level of capacity from the transit provider. While Cogent is one of the companies Netflix is buying transit from, they are not the only one. Netflix buys transit from multiple companies, including Cogent, Level 3, Tata, XO, Telia, and NTT, with Cogent and Level 3 being the primary providers. Transit providers like Cogent then connect their networks to ISPs like Comcast in what’s called peering. This is where a lot of the confusion starts as many are under the impression that ISPs like Comcast are suppose to allow any transit provider to push an unlimited amount of traffic into their network without any compensation. This isn’t a Comcast specific policy, but rather one that is standard for all ISPs.

ISPs have something called a peering policy (comcast.com/peering), which are rules that govern how networks connect with one another and exchange traffic. ISPs like Comcast will allow transit providers like Cogent to connect to their network, for free, in what’s called settlement-free peering. However, once the transit provider sends more traffic to the ISP then they are allowed to, per the ISPs peering policy, the transit provider pays the ISP for more capacity to get additional traffic into their network. Remember, Netflix is the one paying Cogent and Cogent is selling Netflix on the principle that it can get all of Netflix’s traffic into an ISP like Comcast. As a result, Cogent has to take all the necessary business steps to make sure Cogent has enough capacity to pass Netflix’s traffic on from Cogent’s network to Comcast. But Cogent isn’t doing that.

The reason for the poor quality streaming is that Cogent refuses to pay Comcast to add more capacity, even though Cogent is taking Netflix’s money for the service. Cogent is charging Netflix for a service it can’t deliver. Some are arguing that Comcast should be the one to pay to upgrade that connection with Cogent since Comcast charges consumers to get access to the Internet and is double dipping by charging both the consumer and the content owner. In reality, they aren’t. Netflix does not need to go direct to Comcast and pay them anything, they chose to because they could not get the level of service they were paying Cogent for directly. Netflix has also decided it makes more business sense for them to build their own CDN instead of relying 100% on third party CDNs (cdnlist.com) like they used to.

You will notice that when Netflix was using third party CDN providers Akamai, Level 3 and Limelight for 100% of their video delivery, there were no quality issues. Just look at their speed ratings from 2012. The reason for this is that those CDNs already have their servers connected to ISPs like Comcast and have put in place all the necessary links, both free and paid, to guarantee, via an SLA, that they can deliver Netflix’s video. So for all the people who say that Comcast forced Netflix into paying or is strong arming them, that is not true. Netflix has multiple options in the market for delivering good quality video, but Netflix chose to build their own CDN and change their delivery strategy because they want to have more control over it and save money. Netflix’s streaming quality is based on business decisions, that’s it.

In a little known, but public fact, anyone who is on Comcast and using Apple TV to stream Netflix wasn’t having quality problems. The reason for this is that Netflix is using Level 3 and Limelight to stream their content specifically to the Apple TV device. What this shows is that Netflix is the one that decides and controls how they get their content to each device and whether they do it via their own servers or a third party. Netflix decides which third party CDNs to use and when Netflix uses their own CDN, they decide whom to buy transit from, with what capacity, in what locations and how many connections they buy, from the transit provider. Netflix is the one in control of this, not Comcast or any ISP.

As a result, this also shoots down all the arguments where people are saying that this deal is bad because streaming services that aren’t as big as Netflix won’t be able to cut the same direct deal with Comcast. Why would they? They don’t need to connect directly to Comcast as they don’t have enough traffic for it to make sense and haven’t built out their own CDN. All they have to do is use a third party CDN provider to be able to get their content to the ISP with excellent quality, guaranteed with an SLA by the CDN. Nearly every single content owner today outside of Netflix, Google, Microsoft, Yahoo, AOL, Amazon and a few others all use third party CDNs for video delivery. Pick any major broadcaster, sports league, Hulu etc. they all use third party CDNs as they are cheap and back up their service with a guarantee. It works really well.

Now that Netflix has a deal direct with Comcast, here’s how it will work. Netflix’s servers that are sitting in third party data centers connect to Comcast’s network, which is also in the building, via a cross connect that Netflix buys from the co-lo facility provider. While I have heard people say that Netflix will need thousands of physical interconnects into Comcast’s network, that’s not accurate. Comcast has a total of 18 national locations (public info) and Netflix and Comcast will initially connect in about 10 of those locations to start. Out of those 10 locations, Netflix will need 300+ 10GigE ports. Netflix gets a guaranteed level of service from Comcast but as the two companies have announced, Netflix does not get any prioritization in the last mile. This is also where many are getting confused. Some are saying that Netflix is now getting “guaranteed delivery through the last mile”, but that’s not true. That would be paid prioritization, which Comcast cannot do and does not offer.

Moving on to the deal size, I’ve seen all kinds of crazy numbers put out there, with many saying Netflix will spend $25M-$50M a year with Comcast. Some even reported that Comcast was asking Netflix to pay $400M, based on a report put out by Wedbush Securities, which I will get to in a moment. While I do have a lot of sources and are privy to a lot of details others aren’t, there is plenty of public and easily accessible data that allows you to get a good estimate on the size of this deal and debunk the numbers being reported. For starters, no one has reported how much traffic Netflix is actually sending into Comcast’s network and you need to know that before you can discuss the size of the deal. Without doing that math first, any dollar signs attached to this deal are pure guesses.

In 2012, Comcast said they were carrying 4 Tbps of traffic and with the current Internet growth rates, one could easily extrapolate that Comcast’s network is now at 8 Tbps. Based on Sandvine’s data that says Netflix account for 1/3 of traffic, Netflix would need about 3 Tbps of capacity from Comcast today, with that number growing. The way these deals are priced is on a per Mbps sustained model, also known as 95/5 or 95th percentile. Wedbush Securities put out a report that ran numbers based on a per GB delivered model, not per Mbps sustained, saying “Comcast likely sought as much as $0.01/GB transmitted”.

They then estimated that each of Netflix’s 33M U.S. subscribers consume 100 GB of data per month and came up with a total of 3.3B GB of data delivery per month, saying “Netflix would be required to pay approximately $400M per year.” While Wedbush’s numbers are wrong and aren’t using the per Mbps sustained model, the number they gave out was for all of Netflix’s delivery, across all ISPs, not just Comcast’s. However, the media didn’t read it right and went with the idea that Comcast was asking Netflix for $400M, which is sloppy reporting. Once one media outlet said it, many never second guessed the number and started re-quoting it. In the end, Wedbush said they “estimate that Comcast will charge Netflix between $25M-$50M annually”, which still isn’t right and they provide no methodology of how they came up with that range. I provide methodology below to show it’s not accurate.

What no one seems to have noticed is that Comcast has previously stated that less than .1% of their total revenue came from these kind of commercial interconnect relationships in 2013. That means that for all of last year, Comcast got paid between $30M-$60M, which included all of the similar deals they have with Google, AOL and others. So the idea that Netflix would be larger than all of those deals combined, makes no sense. If you want to get a sense, not an exact number, but an idea of what Netflix is paying use transit pricing.

I laugh when I see all of these “save the internet” people saying these deals are bad as they are clouded in secrecy and no when knows what’s really taking place. When we know how much traffic Netflix has and we know the average going rate of transit, both public numbers, you can estimate what Netflix’s cost is. That said, keep in mind a few things. Transit pricing is higher than what Netflix pays Comcast as wholesale is cheaper than transit. Also, Netflix is not the “average” customer and their contract clearly has to have a lot of variables in it, with lots of sliding scale pricing, and different ways to measure it, since the volume of their traffic increases so quickly.

So with those caveats, some of the lowest transit pricing I have seen, which I have previously published here, is around $.50 per Mbps. If Netflix needs 3 Tbps of capacity from Comcast to start, that would cost Netflix $1.5M per month. But, and this is a big one, Netflix isn’t the average customer. Their price will be special and they have a multi-year contract and a lot of variables. So to get a “rough estimate” of what this deal costs Netflix, simply pick whatever per Mbps price you want and multiply it times 3 Tbps. Some will pick a higher number, some lower. If you pick a number a bit lower than $0.50, this deal would cost Netflix about $12M per year. But run the numbers yourself using whatever variable you want, now that I have outlined how much capacity Netflix needs.

I have seen people suggest that Comcast will pull a bait-and-switch on Netflix and raise rates or not deliver good quality video now that they have them locked in a contract. That’s a lame argument as Netflix isn’t a bunch of dummies, they are anything but. It’s the whole reason why Netflix signed a multi-year deal and, this is really important to note, Netflix is getting an install SLA, packet loss SLA and latency SLA from Comcast, which guarantees quality. This is very different from what Netflix was getting from Cogent because Comcast is providing fully dedicated capacity, unlike sending it through someone like Cogent where those connections are potentially over-subscribed if a transit provider over-sells their capacity, which Cogent has a history of doing.

To date, Cogent has had peering disputes with AOL, Teleglobe, France Telecom, Level 3, TeliaSonera, Sprint-Nextel and Verizon. I find it interesting no one in the press mentioned how Cogent always seems to be the one major transit provider who continues to have disputes with so many other network providers, year after year.

A few weeks ago there were all these viral Internet reports of Comcast throttling Netflix content, supposedly backed-up by experiments where somebody would stream Netflix at home on Comcast and get a lower bitrate. Then they’d run that same stream through a VPN (which connects to a different ISP) and get a different and better bit-rate and stream quality. It was the smoking gun gotcha for a lot of folks and 100% sure-fire proof of throttling to some, even though Netflix’s own CEO publicly denied ISPs were throttling. What was happening in the VPN experiment backs-up my earlier points that Netflix was making these networking and performance decisions based on ISP and other factors.

Also, let’s play out what might have happened if Comcast gave Cogent all the capacity it wanted for free. Does that mean Netflix would work well into perpetuity and everyone would be happy? No. Netflix switches providers quite frequently. What if Netflix then moved traffic to NTT and Telia, we’d be back where we started, as those providers would then need all the capacity they wanted on Comcast. What if Netflix started making other traffic shifts to extract greater concessions from ISPs and transit vendors? Fortunately we’re now past that with this Netflix and Comcast deal, but instead of seeing the benefit here to Netflix’s customers, the picture is clouded with far-fetched negatives. The winner in the whole deal is you and me, the consumer. We get better quality video and Netflix gets a cheaper cost over time to deliver the stream to us, which keeps them from having to raise the price of our subscription to give us better quality.

Now all of this aside, I get that many people don’t think the proposed Comcast and TWC cable merger is good for consumers, that cable TV providers raise their rates every year and that Net Neutrality is something that needs to be watched. Those are all valid concerns to debate. However, don’t use the deal between Netflix and Comcast as ammunition in those arguments as it’s not relevant. If you have further questions about the deal, put them in the comments section and I will try to answer them if I can. I know this is a touchy subject for many, but be professional. Lively debates are welcomed but comments that use foul language will be removed. Also for those that have asked, I do not have Comcast as my ISP, I have Verizon FiOS as I live in the NYC area.

  • http://blog.justindorfman.com jdorfman

    Another homerun Mr. Rayburn.

  • Internetthought

    Am I mistaken or did historically companies like Comcast buy transit too? The way I understood it historically networks engaged in peering because they could save on the transit costs. It appears that at first the big transit networks were not interested in peering with Comcast. Comcast got them to peer by playing their transit providers against each other. Could it be that Comcast has turned the world of peering and transit on its head by assuming it should get paid by everyone, including the transit providers? This despite the fact it has only a North American presence. The interesting question is whether Comcast pays for transit at all. If we extend that question and ask what would happen if all ISPs around the world would expect to be paid by transit providers, than who would pay the transit providers to be in business?

    In addition you point to Comcasts peering policy and the traffic balances. Could it be you are omitting a bit of history with traffic balances? The way I understood it, historically traffic balances were used by the Tier 1 transit providers to exclude smaller networks from peering with them. The rule was, that if the smaller network send less than 1:3 compared to what it received from the transit provider, then it was too small to qualify for peering and would have to buy transit from the Tier 1. The effect was that everyone peered around the Tier 1, known as donut peering to avoid paying for transit. Particularly in Europe the many national networks peered heavily at IXPs so much so that AMS-IX Amsterdam has over 600 participants. In recent years this has been turned around, with eyeball networks pointing out the foolishness of peering ratios and turning the same rule on the transit provider. They now appear to demand to get paid, where historically it was they who had to pay in case of imbalance. Could it be that they can do this, because there is massive competition in transit? As described by Renesys’s Baker’s Dozen posts through the years? Might it be that transit providers found that this is not the case for terminating networks of ISPs?

    • danrayburn

      Sorry, but I don’t know the history of peering well enough to comment on this.

      • Tim Pozar

        Comcast has in the past (eg AT&T Transit) but eyeball networks are more in demand. Typically they can demand fees from other content providers to connect.

    • georgeou

      “Am I mistaken or did historically companies like Comcast buy transit too?”

      Comcast still buys transit the last time I checked. In fact during their public dispute with Level 3, they had a dual relationship with Level 3. Comcast paid Level 3 for transit to reach networks they couldn’t otherwise directly reach. Level 3 provided value by building expensive nationwide networks to connect unconnected networks and they sold they were able to sell transit.

      But Comcast also had a separate settlement-free peering relationship with Level 3 for “on-net traffic” which used different network connections. When Level 3 was sending twice the traffic they received from Comcast because of the one-hop routing services they sold to Netflix, Comcast accepted the imbalance. But when Level 3 demanded an additional 30 10-gigabit ports at no cost, Comcast declined. Comcast offered the additional ports for a price and Level 3 took it.

      http://www.digitalsociety.org/2010/12/video-level-3-versus-comcast-peering-dispute/

      “If we extend that question and ask what would happen if all ISPs around the world would expect to be paid by transit providers, than who would pay the transit providers to be in business?”

      Not everyone will go to the extent of video streaming content companies. Not everyone will install servers distributed at every major Internet Exchange. Those companies will always need transit providers and transit traffic (and revenue) will continue to rise.

      The peering business is not mutually exclusive to the transit business. It’s a relatively new phenomenon.

  • http://www.MaxCDN.com/ Chris Ueland / MaxCDN

    Great post. Very well done.

  • Brad

    Where did you see that Comcast is only generating 0.1% of their total revenue from interconnection?

    • danrayburn

      Was published in an article in 2012, I forget the source. But rather than rely on the article, I asked Comcast who confirmed it.

  • http://tinyurl.com/CowboyBooksBlog fgoodwin

    “Comcast does not get any prioritization in the last mile.”

    Did you mean to say *Netflix* gets no prioritization in the last mile?

    Great article by the way. Very readable, for a non-techie like me.

    • danrayburn

      Thanks for the catch, I have corrected it.

  • Michael

    Thanks for this. I have two follow up questions for you. First, do you think that it is possible for ISPs to use interconnection/peering/transit agreements in problematic (potentially anticompetitive, but let’s leave it at problematic) ways? Second, if the answer to the first question is yes, what is the best way for people outside of the agreements to recognize it when it happens?

    • danrayburn

      I have no idea, and the question is too broad. Do I “think” and “is it possible”, too many variables.

      • Michael

        how about this. Let’s assume that Comcast was acting anticompetitively in this situation. How could anyone on the outside of the deal tell?

        • Tim Pozar

          What would be anticompetitive in this case? So far, there is no indication that one content provider’s bits are pref’ed over others. In fact, in order to live up to the previous agreements when the merged with NBC Universal, they can’t,

          • Michael

            I’m not saying that Comcast is acting anticompetitively in this case. I am concerned generally that large ISPs are in a position to possibly exploit these types of agreements in ways that are anticompetitive and/or restrict an open internet. That being said, I also recognize that some of these types of agreements are legitimate. Therefore, I’m trying to get a sense of how to distinguish between the two without merely having to take the ISPs word for it.

          • danrayburn

            We don’t have to take the ISPs word for it.

            If the ISP was “exploit these types of agreements”, Netflix would be the first company to come out and complain about it. The ISPs customers will tell us if there is a problem.

            But also remember that Comcast is giving Netflix multiple SLAs, so if something happened where there was a quality issues, Netflix has a legal contract to stand on.

          • Tim Pozar

            Kinda tough. There are so many places that performance can be degraded either intentionally or not. Over-committed peering and transit networks can fubar streaming. Normally Network Ops try to keep track of these things and there is a number of different pressures to have these fixed (SLAs, network alarms, etc.)

            Much of the congestion I see for Comcast is at the last mile. This is where Comcast may have not scaled the cable infrastructure bandwidth for a neighbourhood for the current demands.

  • donaugustoperez

    Comcast has a (natural) monopoly on the access to a good deal of final users. Monopoly prices are usually higher and the quantity of the good produced, capacity/connectivity, smaller. Of course it is all business decisions, but your bargaining power when facing a monopolist is rather slim. I think that this is a fundamental part of the debate and the underlying of why many perceive Comcast as abusing from its position.

  • Armen

    I just want to thank you for explaining this so clearly and correcting my understanding of the situation which was obviously wrong based on the other shoddy reports on the issue.

  • Phalougher

    “Anyone who is on Comcast and using Apple TV to stream Netflix wasn’t having quality problems.” I watch Netflix on my Apple TV (Comcast of Boston) and had a period of about a month where I never once *didn’t* have a problem (if by “problem” one means “HD quality immediately drops to sub-SD quality and never recovers”).

  • Larrysand

    Thanks for helping me now understand why when I was streaming Netflix via my Apple TV on Comcast it worked fine, but when I used my computer it didn’t.

  • sam1331

    I think this post explained your position much better than your previous posts, and I have a better appreciation of it. However, what you state here does not remedy the basic internet access problem we face in this country.

    Where I live, if you want to stream video at a decent quality and quantity, there is only TWC. I tried the DSL provider and it was basically unusable. I suspect many others face a similar situation, where the cable ISP is a de facto monopoly.

    If there was real competition, TWC would charge both it’s customers and edge providers a rate that covered its investments and a little profit. Because of the competition, if it charged its customers too much they would switch providers, and if they attempted to charge an edge provider like Netflix too much, they would provide a poor experience so customers would switch. This is an ideal free market scenario.

    Another scenario is the de facto monopoly. TWC would charge its customers the most it could to maximize profit, and it would charge edge providers the most it could to maximize profit as well. You might believe we face this scenario today, but its far worse than that.

    Instead, TWC is currently incentivized to charge its customers and edge providers based on maximizing profit ACROSS INTERNET, TV, AND PHONE. This is a huge distinction. TWC has a huge incentive to charge more for internet than it would just to maximize profit on that internet, because it wants to protect its TV and phone products. In fact, all cable providers have a massive incentive to slow the move from cable television to OTT television, since once that transition is complete and they lose control of the cable package, they stand to make less money.

    Most ISPs today are heavily incentivized to provide a poor OTT experience and charge their customers and edge providers more than they would if they were just an ISP. Since real competition seems out of the question, it seems inevitable that the FCC will need to take a more active role in peering agreements. It’s not a solution I like, but otherwise cable companies are going to continue to work to slow the OTT transition as much as possible. That’s what all the incentives point to. We’ll get there eventually, the question is whether the transition takes 5 years or 20.

    • Patrick Henry,The2nd

      The problem is- as usually- government regulation. We have cable monopolies because of local government requirements. They don’t allow more than one cable company per area. Why? Because it protects the cable companies. How is Comcast supposed to increase into an area where it is banned from entering because, say TWC is there? Its only option is to purchase it.

      We need to remove the local monopoly regulations, and also provide tax breaks for both customers and companies to allow more build out. If a company can make money, they will build out. But its too expensive now.

      • MAS

        The ability of municipalities to restrict cable franchising to a single firm was prohibited more than 20 years ago under antitrust law. Companies such as RCN and Knology attempted to overbuild in areas served by existing cable operators and ended up in bankruptcy. Incumbents have the benefit of economies of scale and are hard to displace.

        • duelistjp

          the local governments have a legitimate interest in keeping there from being too many lines on the telephone poles. There are restrictions on people coming and putting up new poles or putting lines on existing ones. Too many would be a public nuisance. even if they were run underground that is a public easement on the property of most landowners and does indeed need to be regulated carefully. I think most

  • DroppinFrames

    “In a little known, but public fact, anyone who is on Comcast and using Apple TV to stream Netflix wasn’t having quality problems.” I have comcast and watch netflix via apple tv regularly and ~10 times in the last 14 days have seen tons of dropped frames. The audio remains clear but the video will drop down to 3-4 frames per second for minutes at a time.

    • jcburns

      I experienced the same thing, so reading that sentence made me say “the wha..?” The problem from a consumer standpoint is that all of these protocols and bandwidth issues are, well, invisible and not accurately measurable by the average user. So streaming video mythology runs rampant.

    • Dave

      The statement that Apple TV users weren’t having quality issues with Netflix is patently false. Count me in with the others who have posted here that Netflix quality on Apple TV does indeed have quality problems.

      I’m on Comcast with a 60 Mbps down/12 Mbps up connection and use Apple TV for all my Netflix streaming. Netflix started getting really bad last year around the the time that Netflix began streaming “Super HD” content to all ISP regardless of whether the ISP was part of Netflix’s Open Connect network. I’ve complained repeatedly to Comcast regarding poor Netflix quality but it hasn’t noticeably improved until around the last 4-6 weeks. Lately the quality has been good, but it was very poor prior to that.

    • wt_hayes

      Is it possible that the problem you are describing is the result of over-subscription at the last mile? I see traffic congestion issues on my service based on time of day as more of the residents that I share the cable path with log on and begin using the service. I see it on OTT and on plain old internet connectivity at time.

  • info
  • Murtaugh

    Thanks for the very interesting article. Perhaps this explains why I get better Netflix quality through my ATT LTE connection on my iPad than through my Comcast Business Internet account…

  • JDLeeConsulting

    This was a really good explanation–as was your post earlier in the week.

  • http://www.swift2.blogspot.com Swift2

    The problem with Comcast isn’t that it’s an ISP. The problem with Comcast is, it has its own streaming service that it doesn’t charge for transit, it has it’s cable network and all the elaborate levels of pay, and it gives consumers nothing in return. Oh, as ISPs they were be faster than phone copper. I want them to go back to being a public utility, competing with many other sources of the Internet on the basis of price, relability and capacity. They should be forced to sell NBC. The producers of content, like HBO, which I like paying for should be responsive directly to me, not to the cable company. All of this is without any new technology. Cable, and any corporation that delivers Internet should compete as an Internet provider. Anything else is just another coaster in your mailbox from AOL in the ’90s: irrelevant. Annoying.

  • Fanfoot

    Nicely written, thanks.

    Honestly the previous piece had just so much snark that I couldn’t trust anything you said. This piece was much better.

    You make a lot of good points. I’d challenge you on a few minor points, but overall its a good contribution. Helped me understand things a lot better.

    Personally it doesn’t change how I feel at all though. Comcast is a monopoly, provides lousy service for too much money, and we’d all be better off if their were either real competition or aggressive regulation from the FCC–and by that I mean that Comcast should be broken up along the lines of AT&T. The last mile provider should simply sell services to all comers and be prevented from offering services or owning media companies.

    I suspect we would differ on this. But regardless of that, appreciate the contribution. And understand that its tricky trying to fit this event into the picture.

    • LameRandomName

      It is precisely the government meddling that CREATES monopolies to begin with.

  • lcannell

    “Remember, Netflix is the one paying Cogent and Cogent is selling Netflix on the principle that it can get all of Netflix’s traffic into an ISP like Comcast.”

    Likewise, many Comcast customers pay premium rates for upgraded bandwidth in expectations of receiving higher quality video. So, BOTH Cogent and Comcast are making promises that they are not fulfilling.

    It comes down to this: Which company do you believe is taking the “the necessary business steps” to ensure they are able to deliver what they advertise? You seem to believe that Comcast is acting in good faith and Cogent is not.

    • duelistjp

      no comcast is claiming you have access to the internet at those speeds it does not make any claims whatsoever of a given website being able to provide content to you at those speeds

  • Julian Smyth

    Excellent article, your posts on this subject have been essential reading for me that really cut through the misinformation and lack of technical knowledge most of the tech and other media has exhibited.

    You mention at the end of the article that you have Verizon Fios. From Netflix’s ISP ratings, as well as anecdotally, I’ve heard that Fios has also suffered from slow Netflix speeds in recent months. Is that something you have experienced or could comment on?

    • uzenzo

      I also have Fios in NYC and Netflix is unwatchable. (It is fine thru VPN
      to Cablevision.) I would like to hear Mr. Rayburn’s thoughts on who is
      responsible.

    • Dek

      I have a very rudimentary understanding on how an individual’s bandwidth requirement to watch Netflix flows through the system, but I’ll give this a try. Based on what’s explained in the article, it doesn’t really matter who your personal ISP is – and I think this is the point that the author tried to make at the end of the article by clarifying that he has FIOS.

      The issue here is that before this deal with Comcast, there was no direct relationship between Netflix and your isp. The relationship, and the allocation of Netflix’s bandwidth was mediated by their transit partners. Transit partners then engage in peering with ISP’s, but Netflix does not determine how much peering their transit partners negotiate. I’m sure that management at most transit providers aims to slightly undershoot the amount of bandwidth they are allowed to use through peering, but they tell clients like Netflix that they have more than enough bandwidth secured for their needs.

      Regardless of your ISP, the limiting factor on Netflix streaming speeds up to this point was the amount of peering bandwidth the transit provider(s) secured. Of course your speed depends on your personal bandwidth, but from Netflix’s point of view – systemically – in terms of “pushing out” content your ISP wasn’t being talked to or negotiated with.
      Now that the “middle man” is out of the way, it sounds like Netflix has been able to secure an appropriate amount of bandwidth for the amount of traffic they have – directly with an ISP (Comcast). Presumably, Netflix is also in the best position to understand when and where the peaks and troughs occur in their bandwidth requirements, and has negotiated a deal with Comcast that reflects this. Cogent (and other transit providers) probably wasn’t in as good of a position to do this.

  • cosmicwonderful

    Very illuminating post — thanks for writing it.

    Hypothetical Q: if Comcast’s consumer-ISP customer base was dramatically smaller — i.e., if there was more robust competition for consumer broadband access, whether via many cable companies, Google Fiber, municipal ISPs, etc — would they be in a position to get Netflix to peer with them directly AND get paid? Or might they even pay Netflix for the privilege, hoping to satisfy their Netflix-hungry customers?

    Another way to think about it: is Netflix having a streaming problem with any other consumer ISP; assuming no, why not? Why are Netflix’s CDNs sufficient for all the other ISPs? After all, those other ISPs AREN’T getting paid by Netflix — or even paid by Netflix by proxy, via its CDNs — right?

    • MAS

      Actually the commercial CDNs sometimes do pay Comcast for the right to collocate at Comcast’s premises and deliver traffic to them. That is, Netflix is paying by proxy when it uses CDNs.

      • cosmicwonderful

        Thanks for replying, but that wasn’t my Q. Okay, CDNs sometimes pay Comcast. But Comcast is a behemoth; an exception to most rules. The question is, does Comcast get more favorable deals than OTHER ISPs that are similarly situated except for being smaller? Does Comcast’s size allow it to get paid where other ISPs have to pay or make settlement-free deals?

        • anon_coward

          not sure but some small ISP’s took netflix’s open connect CDN which they don’t get paid for

          probably because its cheaper than upgrading their connections to the backbone. those routers are huge and EXPENSIVE

        • duelistjp

          Would it matter if they did? Volume pricing is always cheaper in any market. For the most part it comes down to what is cheaper to the companies. This way is cheaper for netflix than going through cogent or others and it allows comcast to allocate less bandwidth to them so they don’t need to upgrade those connections for some time. Netflix is paying for the infrastructure needed to connect directly to comcast but so that cost is irrelevent to comcast. so moving to a direct arrangement makes sense. The question then becomes can comcast charge netflix for this privilege and clearly the answer is yes as they do this with others. the difference is that the cost is largely offset by comcast paying to send data into their networks. With netflix this is not the case as the data ratio is orders of magnitude more skewed. it may not be necessary to enter into this type of arrangement with other companies though as the number of users is small enough the bandwidth exists to transmit the data they need through more traditional methods. so yes the other companies likely could not get netflix to pay them as the economies of scale don’t make it worthwhile

  • Albin

    Very interesting. The thesis comes down to Cogent failing to live up to its commitment to Netflix. My question relates to this: end users (retail customers) have direct purchase relationships with BOTH Netflix (content) and Comcast (pipe) and both those companies have a serious stake in marketing and maintaning the “experience” for these customers.

    How can it be that both content and pipe providers have put their customers’ experience in the hands of Cogent (or other intermediaries described here) and apparently its own discretion / willingness to pay more of its own money?

    I’ve become more interested in the market and terms of service contracted by the various transit intermediaries and specifically whether Cogent (if it behaved as described) is going to be sued for the various expenses and customer-service problems it has cost Neflix and Comcast.

    • The_RS_Gadfly

      I doubt they’ll be sued. The actual service deals will have very specific, measurable facts to be measured for the price quoted.

      Best guess: The Cogent Sales guys are “better” than their tech team and sold a bean counter on their program. When the lawyers sat down and hammered out the deal, the Cogent Tech Team made sure they could deliver their legal commitment, which didn’t match the Sales guy’s pitch. But since the contract superseded the verbals and the Netflix guys signed off on it, they were stuck with it. In the end, Netflix got what they paid for but nothing more.

  • Michael Dube

    Interesting timing for this deal to have become public. Almost makes everyone forget that Comcast is trying to buy Time Warner…

  • kpedraja

    “In a little known, but public fact, anyone who is on Comcast and using Apple TV to stream Netflix wasn’t having quality problems.” This “public fact” is not a fact at all. Many Comcast-using, AppleTV owners – myself included – were having *horrible* problems with Netflix streaming starting late last fall. They persisted for months. It’s well documented on the AVS forum (http://www.avsforum.com/t/1278071/apple-tv-owners-thread/2490) and elsewhere. We were getting sub-SD quality video on our AppleTVs while getting much better streams – usually HD-quality – on other devices like our iPads, Rokus, TiVos and Xboxes.

  • Jeff Shaffer

    Helpful article. Thank you. One thing I still don’t understand: with the insufficient peering capacity between Cogent & Comcast’s networks, has all Cogent->Comcast traffic been degraded, or just netflix: i.e. has Comcast been segregating Netflix traffic at peering points? If not just netflix, why haven’t we heard of other content being degraded in recent months for Comcast subscribers?

    • jd

      Cogent->Comcast is pretty bad regardless. For me personally, when I adjusted our datacenter traffic to route Comcast-bound traffic through Level3 instead of Cogent, speeds improved dramatically. I’m happy to hear that Cogent’s bandwidth to Comcast will be getting free’d up as part of this process.

      • jd

        As for why you don’t notice it much, I’d imagine that most other content providers have chosen to similarly route around Cogent where Comcast is involved.

        • George C

          Don’t they also have performance issues with Verizon, AT&T, TimeWarner, Qwest,… the list goes on.

          I think most content providers have chosen to route around Cogent where customer performance is required. With transit… you get what you pay for

  • LameRandomName

    You know, part of the problem is that nobody wants to be honest with customers about what the cost of access really is. So they try to spread the cost around, take advantage of locations without real competition; and they keep hoping that their physical plant build outs happen fast enough to keep up with increasing demand.

    Meanwhile, people who have no idea what they are talking about complain about problems that they don’t understand while politicians look for opportunities to meddle in ways that will benefit THEM, everyone else be damned.

    The more things change the more they stay the same…

  • John Crusader

    Carriers should charge its consumers real internet rates and stop the games. Anyone can peer to the premium ports for free as we are charging the consumers for the internet bandwidth properly. Your responsible for your media delivery budget and equipment. You must connect to 10 locations to have free peering with the following rules geo delivery rules. We do not care if you steal our television rights anymore, we are the eyeball networks, this is the costs and our profit. We as carriers will stop promising people internet and refuse to deliver it.

    Premium Internet Access: $179/month for 9mbps [1080p Streaming]
    Plus Internet Access: $99/month for 4.5mbps [720p Streaming]
    Open Internet Access $19/month for 100mbps [ Best effort, no quality ] [Sites like netflix and youtube may not work if they dont buy commercial services to connect to the open network] [contact your television provider to pay for the access rights to work properly on the Public Internet]

    Call it for what it is when you buy internet from a carrier. They should not pay for everyone to give you tv for free. The internet was not designed to be television. IPtv is a commercial product to do the work, not for the carriers to get stiffed on the bill. If you want to watch tv on the internet, then the carriers need to be honest with the public. Honesty is gone from the world. Sad times we live in. PS this goes for google and your youtube as well. YouTube is a television station channel #501 and should pay rights for what they are.

    • anon_coward

      if netflix paid to put their content back onto the limelight CDN there would be no reason for these rates

      you can watch TV on the internet, just not live TV. pre-recorded if the content owner buys CDN access then there shouldn’t be any bandwidth issues

      netflix has always been cheap with its bandwidth budget but has no problem licensing crap content no one wants to watch

    • Revko

      The internet developement and implimentation was publicly funded as a communications infrastructure. Not to make a few companies rich.
      Just because I open a lemonade stand on the road doesn’t mean I now OWN the road.

    • http://laborequalszero.com laborequalszero

      Remember- All telcos and comcasts are DUMB PIPES. As far as they are concerned, all bits of data are electronically the same! The rates they charge for any bits of traffic should be equal within volume tiers and competition across telcos should commoditize pricing lower and lower until they give it away like nationwide long distance!

  • http://join.groupflix.tv GroupFlix

    I’ll certainly pass this article on. It is quite similar to the research we did over at GroupFlix on the topic of net neutrality.

    As a new comer to the game, I can safely say we don’t have any worries about being “held hostage” by an ISP. In fact, we met with some people at Comcast a couple weeks ago; they can be quite positive about start-ups depending on your position.

    In the end, its an evolutionary move. I think this was a business decision that had to be made as they move to 4K streaming and other bandwidth heavy activities.

  • lazr47

    So what is the agreement between Comcast and Netflix called. Are Comcast/Netflix now peering with each other, this seems like a better solution than Netflix deploying content delivery servers inside of each ISP’s network?

  • Ken Nielsen

    Thanks for the informative post Dan..

  • Dan States

    “The reason for the poor quality streaming is that Cogent refuses to pay Comcast to add more capacity, even though Cogent is taking Netflix’s money for the service. Cogent is charging Netflix for a service it can’t deliver. Some are arguing that Comcast should be the one to pay to upgrade that connection with Cogent since Comcast charges consumers to get access to the Internet and is double dipping by charging both the consumer and the content owner. In reality, they aren’t.”

    This statement has multiple fallacies and shows a complete lack of understanding of how transit over the internet is provided and billed for. Cogent is a service provider, whos customers are both Netflix and Comcast. Comcast pays Cogent to be able to access the rest of the internet *through* cogent, and Netflix also pays Cogent to be able to reach the rest of the internet *through* Cogent. Cogent’s network (which is their only responsibility) is capable of handling Netflix’s data demands, and is the service that Netflix pays Cogent for. Cogent is *not* a customer of Comcast, their relationship is the other way around, and they are not equal peers. Comcast’s customers are internet users such as myself. *I* pay Comcast to be able to reach the rest of the internet through Comcast and the transit providers that Comcast is a customer of. Therefore, Comcast is the party that is failing to provide the service the advertise, which is a certain transfer rate, to the *entire* internet.

    The problem lies in their subscription model, where they advertise that you have a service level such as 25Mbps of transfer rate, but in reality their network is not capable of sustaining that service level when enough subscribers are using it at the same time. This over subscription model allows the to make a lot of money, but is contingent on the gamble that not everyone will try to use all their supposedly available bandwidth at the same time.

    Because using a service like netflix (or any other streaming service) uses more of the bandwidth that customers are paying for, Comcast’s oversubscribed network’s limitations are exposed. Instead of upgrading their network to actually provide the service the advertise, they are trying to bill Netflix for the additional load that Comcast’s own customer are causing (not Netflix) when they try to use the service they are paying for (but it isn’t there for them).

    • Dan States

      That said, Netflix directly peering with Comcast means that Comcast doesn’t have to pay Cogent (or Level 3) for Netflix traffic, which means Comcast is happier because they have a cheaper transit bill.

      The fact that Comcast wants Netflix to PAY for this is still ridiculous since they would both benefit hugely from a free-peering agreement, but I’m sure it’s still cheaper for Netflix to peer directly with Comcast and pay for that direct access rather than pay a Tier 1 transit provider a higher rate to get through their network to Comcast.

    • MrK

      AFAIK Comcast is one of the few Tier 1 (maybe they lost this title since some big other Tier1 broke peering relationship with them some times ago, my memory fails me here, and I’m not from US ;) not up to date with the news here) ISPs, and has indeed a peering relationship with Cogent, not a transit contract.

  • Bryan

    You say you are using Netflix with Fios. Our Verizon Netflix service is awful. So bad we have stopped using Netflix. Any thoughts on a resolution. Would buying an Apple TV help? Thanks!,

  • dave

    your numbers don’t add up. .50 per mbps and there is 1,048,576 mb in a tb and netflix needs 3 tbps… the cost would be much higher on a per month basis… am i missing something?

    • Carl

      Possibly? What do you think the # should be? Intuitively, if it’s about a million mb in a tb, and it’s a dollar per mb, then it’s about a million dollars a month per tb. However it’s half a dollar per mb, so it’s half a million per tb. And they use three tb, so that’s one and a half million, as he quoted. That’s 18 million on the year, and he cited 12 million after saying that he thinks Netflix can get a better deal than that due to the size of their contract — in this case, 33% off.

  • Tim Investor

    Dave, if the cost is 18 million a year, that would be a rounding error for Netflix. Why would they make such a big deal over such a small cost? Also, why would they not publicly provide the costs of this deal, not even during an earnings announcement? Comcast annual revenues are over $110 billion a year. $18 million would not even be less than .0.00016% of their revenue. Why would Comcast CEO or for that matter AT&T and Verizon get in public spat with the CEO of Netflix over a rounding error in revenues. I think your calculations, although methodical and well thought-out, are probably way off. I think this is something that can cost Netflix, between all the ISPs, over a billion a year. That is why their business model is broken, that is why Netflix CEO is furious, and that is why Netflix was forced to raise prices by $2 bucks.

    • danrayburn

      I don’t know how you can try and spin facts so badly.

      Netflix has already said, during their earnings call that the cost of the Comcast contract does not have a material impact on their business. But you want to imply that these interconnect deals cost Netflix “over a billion dollars a year”? If that was the cost, they would have to legally disclose it, because of the impact to their business.

      Not to mention, Comcast has already put on record that last year they earned between $30M-60M in revenue, for ALL deals combined.

      Stick to the facts.

    • anon_coward

      it’s not the money, it’s the power

  • Chuck

    Forgive me if I’m reading this wrong… speed to the end-user makes no difference? I worked in the cable industry for 6yrs, installing and troubleshootng physical cable plant, providing tech support to end users and internal support to techs in the field. I was a computer network technician for 5 yrs in the military as well.

    I get the whole deal between Netflix and Cogent but I know for a fact the speeds quoted to the end-user for cable broadband are defined by a range. Its a range that varies greatly… especially in full copper and hybrid networks like the ones I worked in. So your saying the speed to the end user has nothing at all to do with the quality of service being delivered… I’d highly doubt that.

    I know vast areas of the multiple towns and cities I serviced that had constant issues with VoIP services that transmit around 96Kbps and marcoblocking on the digital TV services delivered over the proprietary network. Those same Networks had and some still have crap internet because of issues starting from the RDC over the plant to end-users.

    Granted these could all be seen as individual cases but I know that these cable companies know that the patch work of physical plant isn’t built for the level of service they advertise in the majority of places where they are often the only viable option for broadband internet.

    I feel like this issue has been characterized as an issue of solely Netflix being too cheap to pony up for a better 3rd party carrier to Comcast and Comcast providing rock solid Internet. It is just not true… I didn’t work for Comcast but I know people who do… and I know people that have it. It’s the same Time Warner plant they bought out… same physical limitations. The connection form Comcast and the end-user is neither 100% rock solid or out putting max streaming capacity (dwnld/upld) consistently. NO NETWORK DOES.

  • http://www.ivpcapital.com/blog Michael Elling

    Ironically, Netflix does indeed make out better in the short run. I’ve seen estimates where they can drive the cost to $0.30/mbs. That said, you don’t get into the real economics of what Netflix’ nor Comcast’s revenue and cost models per bit. Netflix, because of its flat-rate, all you can eat model needs needs to drive the cost/bit down continuously. But this is happening in the WAN at a rate of 20-40% annually. The problem is that it hasn’t been happening in the MAN (mid, first and last mile) for at least the past 12 years due to consolidation and policy that has been anti-competitive . Rates have only been declining 5-10% annually.

    So there is a significant imbalance between WAN transit and MAN transit rates. In addition to taking undue profit on the high access costs it charges, Comcast is trying to use peering agreements with Netflix to keep that WAN/MAN demarc as close to the core and keep competitive market (volume) forces from driving out to the edge.

    Finally, at present, Netflix’ content aggregation strategy isn’t a threat to Comcast, but when 4/8K begin to reach critical mass in the next 2-3 years, it could and will be a disruptive threat to Comcast’s LinearTV model.

    So, by looking at one set of issues, you really aren’t focusing on the complete set of macro issues that are relevant to current financial market and policy issues. You are incorrect in saying, “However, don’t use the deal between Netflix and Comcast as ammunition in those arguments as it’s not relevant.” It is not only relevant; but central to the issue of IP transition, interconnect, net neutrality, etc… as we look to a rapidly approaching (mostly 2-way) future of 4/8K VoD, 2-way HD collaboration, seamless mobile BB and IoT. All of these require cloud intelligence and economics to move out to the edge of the networks.

  • Mark Able Jones

    For the purpose of disclosure, how much business does Frost and Sullivan do with cable companies?

    • danrayburn

      Frost & Sullivan has almost 2,000 employees across 40 offices on six continents. So I can’t speak for the whole company, but the group I am in, the Digital Media Practice, in the nearly six years I have been at Frost, I have never worked on a project where our group has been hired by any of the MSOs. Thousands of companies have subscriptions to our online research portal, including some MSOs im sure.

  • http://laborequalszero.com laborequalszero

    As an ISP, I really have to applaud this article. Apart from a few forgivable typos, it is the clearest technical description of the issue that cuts through the hyperbolic b.s. you get from news media. After I wish they would read it, I am still sad, because they are not technical, and would not understand it.