New Data Questions Netflix’s Assertion That ISPs Are At Fault For Poor Quality

In the Netflix versus ISPs peering dispute, there are a lot of opinions and debate around who’s at fault for letting some peering points degrade and who should be responsible for upgrading them. To date, many are having a hard time separating facts from opinions because Netflix and the ISPs haven’t released any concrete data to back up their claims. In most industries, if one company accused another of doing something wrong, it would be expected that the company making the claim would back up their position with detailed data that proves their point and leaves little doubt as to who’s responsible for the problem. Netflix has yet to do that.

Most seem to be giving Netflix a pass, with very few demanding real transparency into what’s taking place, or changed, that degraded Netflix performance back in September 2013. No company should try and force us to take their word for it, they should simply make the data public and let us decide on our own. Netflix says they are bringing transparency to the debate, but they are doing the opposite by using vague and high level terms with no definition. To date, Netflix has yet to set forth any details on how they want the current business models to change, how it should be regulated, what they consider “strong” net neutrality or even submitted a proposal to the FCC.

The best example of this is how Netflix’s player recently gave out messages saying that Verizon was at fault regarding quality issues, but then when challenged by Verizon to back up their claim, Netflix announced they would discontinue showing these messages on June 16th. Originally Netflix said these messages would be rolling out in a phased deployment on all networks, but in their blog post yesterday, they now say these messages were just a “test”. To me, it looks like Netflix simply created noise in the market, again with no data, and then when pressured by Verizon to prove their case, Netflix instead decided to stop sending the messages and now release any details. Why? If the problem lies within Verizon, Netflix should let us see the data that shows this and stand behind it. Why back down if they have the data to show where the problem is coming from?

This is just another example of many where all sides simply point the finger at each other and say it’s the other guys fault, but then provide zero details to back up their claims. However, that may change soon as Netflix will likely publish network and performance graphs around a peering event, taking place in DC on June 18th, to bolster their argument. At the same time, some ISPs are actively working to release some data to the market, like the internal chart below [Figure 2] that I received from a major U.S. broadband provider, which gives us some more visibility into what’s taking place inside an ISPs network.

Netflix’s accusation is that ISPs have purposely congested their peering points in order to specifically degrade the Netflix service. What Netflix has failed to be transparent about is that Netflix has always paid to deliver their traffic. CDNs like Akamai, Limelight and Level 3 successfully managed the majority of all of Netflix’s video and were responsible for Netflix customer performance. Each of these companies successfully delivered Netflix via all the same transit paths and business relationships equally available to Netflix today. When Netflix took over the routing controls for their video traffic with their own CDN Open Connect, customer performance began to suffer as highlighted in Netflix’s own data that they shared with the Washington Post. I added a red circle to the chart to show when the Netflix changes took place and the impact to customer performance by ISPs.

Figure 1.

chart1
While Netflix was able to convince smaller regional ISPs to voluntarily offer settlement free peering, most large ISPs maintain national/international infrastructures, which require peering policies and consistent business practices to ensure fair and equal treatment of traffic. For the providers Netflix did not qualify for peering, Netflix moved their traffic onto very specific Internet paths that were not capable of handling their massive load and caused the congestion that impacted customers (as highlighted in Figure 2 below). In other words, if Netflix receives free peering, ISP customers receive good performance and high rankings and blogging praise from Netflix. But if Netflix does not receive free peering, ISP customers do not receive good performance and get low rankings and shame from Netflix.

It was Netflix that specifically chose transit paths to those ISPs who refused to give it free peering that it knew (and measured) were not capable of handling an increase in load. In some cases I was told by ISPs that traffic levels increased by 500% in only a few months where normal Internet growth with these same peers was less than 20-30% across an entire year. These ISPs’ customers did not request traffic to be served from poorly performing paths. Netflix chose to create, and use, paths that they knew were congested, simply because they were cheaper than using paths that were less congested. While some may not like that decision, Netflix is running a business and like all businesses, cost is a factor in a lot of decisions. I’m fine with Netflix having to make tradeoffs between quality and cost, but it’s not true that 100% of every path going into Comcast was “congested”.

Some of the many other transit providers I have spoken with confirm this, saying that they could have handled incremental Netflix traffic into Comcast, but that it would have been more expensive than Cogent, which was Netflix’s primary transit provider at the time. Even Cogent would not deny they were the cheapest transit provider of all the ones Netflix was using, but as we’ve all learned, cheap does not guarantee quality. During this same time, Netflix was still using other third-party CDNs for some of their video delivery. These CDNs were delivering the same Netflix service at the same time, to the same locations and with good quality. That is why some customers said their Netflix video was working great, while others said it was buffering and it is also why if some customers used VPNs, their performance improved. Netflix had control over who to give good service to and who to degrade, as shown in this chart below, from a major U.S. broadband provider:

Figure 2: Major U.S. broadband provider

Red is Netflix Video Bitrate delivered via a 3rd Party CDN
Blue is Netflix Video Bitrate delivered via Netflix internal CDN

BCC6DD36-44AB-42A5-A192-0AC7D2BE1071

When Netflix delivered video through similar sized third Party CDNs, customers received a consistent HD video stream 24 hours a day and from the CDNs I have spoken to, none of them had any problems getting Netflix’s traffic to the ISPs. When Netflix sent the video stream through their own CDN using their used newly congested transit paths, prime time viewing based on these decisions had buffering and low quality video.

Amongst those in the industry, there is a widely used term “traffic manipulation” and is “one of the most clever and devious of all the [negotiation] tactics” as described in The Peering Playbook. As the playbook details, the problems this traffic manipulation strategy introduces are three-fold. It negatively impacts customers; the levels of Netflix traffic also impacts other services sharing the same path; and if the “traffic manipulation” tactic is successful by Netflix or any other large CDN provider, it will be repeated by others creating further instability on the Internet.

Netflix’s point of view is that the Internet has changed a lot since many of these paid interconnect deals were done ten years ago. Today only a few ISPs control the vast majority of the market in the U.S. and Netflix would like to see peering and interconnect business models change. While that’s a fine argument to make, Netflix has yet to deliver any proposal or suggest an alternative other than to say it should be free. Netflix only confuses the discussion by involving “net-neutrality” in the debate and opportunistically point fingers at specific ISPs like Verizon and Comcast. But, Netflix has always had all the same delivery options as every major CDN and video provider on the planet. Many of these CDNs and large content sources have large volumes of traffic as Netflix and able to deliver Netflix and other Internet services with high quality service for their customers. The transmission decisions that Netflix makes are just as suspect to what is impacting their customers’ performance. Putting all the fault on ISPs is not accurate as both sides share the blame in not being able to make this work.

I’ve also heard from some that Netflix has told them that they have self-limited their transit decisions to only ISPs without residential access customers. Most CDNs use ISPs like AT&T, Sprint, Centurylink, Verizon and Comcast as transit options to reach the entire Internet, including each other’s network. Akamai and Limelight for example are connected to every Tier 1 provider and most Tier 2’s to deliver their service. They do not impose these business limitations in order to ensure they deliver services with high quality, so why is Netflix? These are not “peering issues”, these are first mile network decisions that Netflix is 100% in control of.

By Netflix limiting themselves to only Cogent, Level 3 and smaller International ISPs, they could find themselves buying capacity that is just not available. “Just upgrade your peering links” is not always the answer due to Internet peering policies as well as concerns that after investing capital, Netflix will just move terabits of traffic again in a few months making the problem start all over. What if Comcast gave Cogent all the capacity it wanted and footed the bill for it, and then the next year, Netflix moved away from Cogent and used Tata instead? Now Comcast would be forced to have to give Tata all the capacity they wanted and this change could happen every year. In fact, over a 3-4 year period, Netflix moved some or all of their video traffic amongst multiple CDNs including Akamai, Limelight and Level 3 and in some cases, due to lower pricing, left one provider only to come back to them a year or two later.

Netflix dominates close to 70% of the long form streaming traffic which is 10x their nearest competitor.  This market dominance allows Netflix to use their massive traffic controls in ways to force or demand special free peering privileges from ISPs rather than continuing to include these market based delivery costs in their service as others do. These costs that Netflix is objecting to have always been a cost of doing business for large CDNs in the U.S. That’s why to date, no other large CDN like YouTube, Apple, Microsoft, Amazon, Akamai or Yahoo! has come out publicly and backed Netflix’s position. Some ISPs, like Google Fiber, have said they think interconnects should be free, but that’s not the CDN portion of Google’s business, which for years, has done some paid interconnect deals.

Netflix has always paid a portion of their traffic delivery cost just like all of Netflix competitors large and small. Netflix is unhappy that they are being held to the same Internet policies as every other player. With great power comes great responsibility, and if Netflix will be operating one of the largest CDNs in the world, they must start behaving like one of the largest CDNs of the world. If they think the current model is broken or feel that market dynamics have changed enough to where new models needs to be put in place, that’s a fair argument. But you can’t make that argument and expect any change when you don’t put forth any kind of proposal. Netflix says they want changes that are “fair” and rules that are “strong”, but those words don’t mean anything without details.

Netflix hasn’t been clear with their arguments. First it was really about transit, but now they are saying it’s about “terminating access monopolies,” broadband competition and such. Problem is that is a much different issue and they’re having a hard time making that case and tying it to why they must have this or that remedy to lower their own costs. If broadband competition is now their issue, then it clearly is unrelated to net neutrality or the Comcast and TWC merger since the companies don’t overlap. There are other ways to go about addressing broadband competition constructively without bad-mouthing ISPs and confusing the public and throwing out made-up regulatory remedies like Title II or strong net neutrality.

If you talk to the major transit providers, CDNs, ISPs and the companies tied up in this debate, many will detail what has been going on behind the scenes with traffic manipulation and only using the data that makes them look good. There is more to it than that, and it’s one of the main reasons why those involved won’t detail what’s really taking place, where the problem lies, what their business motivations are for some of the decisions they make and the impact these decisions have on their bottom line. Remember, Netflix locked in a fixed rate with Comcast, for more than five years, for nearly a third of their traffic, which gives them a clear advantage over any other OTT competitor. So lets not forget the positive impact these interconnect deals are having on Netflix’s bottom line. Cheaper costs, lower churn, better quality video and the longest length contract, which I have come across, in the industry.

If Netflix really believes broadband competition is an issue, why make enemies with the companies who you can partner with to address it? To me, it all seems like an irrational strategy, especially when some of the arguments Netflix makes, don’t align with what some transit providers, CDNs and data from within ISPs shows. If Netflix wants real transparency, then they need to come out and put their cards on the table and show us what they have. It’s easy for anyone to complain and blame the other guy, but just because you might not like your ISP, think your cable bill is too high or have some other reason not to like your ISP, that’s not a reason to give Netflix a free pass and not demand they give us the transparency that they themselves keep saying they want.

  • AVonGauss

    Regarding Figure 1, I don’t believe NetFlix’s CDN per se is called “Open Connect” rather I believe it just refers to the NetFlix program where NetFlix CDN equipment is installed inside an ISP’s premises creating a mini-POP for the NetFlix CDN. Which is why when the “Super HD” flood gates were unleashed back in the fall of 2013 you can see from the chart where Open Connect members did not appear to be impacted as severely as non-members still relying on transit links. I don’t have any historical data personally but what I have also seen suggested is that is also the same time-frame when NetFlix started directing more of their traffic for non-member Open Connect ISPs to NetFlix’s own CDN rather than through traditional CDN providers.

    • danrayburn

      From Netflix’s website:

      - Open Connect is a single-purpose Content Distribution Network
      - The Netflix Open Connect Content Delivery Network

      • AVonGauss

        Agreed, they actually call it “Open Connect Appliance”.

  • TheComcasto

    Saying the networks who peer with netflix are small and regional is not an honest characterization. Aside from AT&T/vz/cc you’ll note that nobody has a problem preventing their peering into netflix from running hot, or is shaking them down for alum rents.

    • danrayburn

      Look at the number of subs and size of network of Google Fiber, Cablevision and Clearwire. They are all small and regional.

      • MrNarus

        What about COX, CHARTER, and TWCABLE, who peer with content for free? The argument doesn’t hold up. Besides, doesn’t Comcast’s larger size mean they are able to take advantage of greater efficiencies and economies of scale?

        • MrHand

          Can you site your source the that statement?

          Why doesn’t Cogent, Level 3, Sprint, Centurylink, Abovenet, XO, AT&T, Verizon, etc, etc peer with Netflix? They all have eyeball customers or are transit to eyeball networks.

          Because Netflix is not a peer and doesn’t meet their policy. Netflix has always paid someone. That is what they want to change and will do what every it takes.

        • uniquename72

          This. A simple comparison with Cox makes this whole fake article a farce.

      • Jon_Irenicus

        I don’t understand the mechanics of why that is relevant. Are you suggesting that if Comcast/verizon attempted to use the open connect style solution netflix tried to promote to them it would not scale like it did for smaller ISPs?

        I am just a lay person on these matters, so I ask honestly, why? It seems more plausible that given the option of getting paid for traffic delivery into the network vs merely lowering the cost for companies like netflix by placing more local caching nodes near an ISPs customers (that includes multiple caching nodes, a large isp in different regions would obviously include multiple nodes in different regions) inside their network, some of the larger public isps would choose the former. Better to get paid.

        That was the weakest line of argument you gave above as it glossed over the impure motivations of some of these large ISPs. Some of the rest of the arguments made sense. I get why an ISP might not want to spend large chunks of cash upgrading a peering point that is congested if netflix would switch that traffic to another node into the network at the drop of a hat.

        THAT is a reasonable worry and critique, but I can’t imagine there are not ways to get around such issues by discussing it with services like netflix.

        For the rest of the peering issues, the more I see of how things work, the more I wonder why ISPs are paid by anyone to connect to their network. But I’ve yet to see the business model and rationale laid out completely, too hidden from our eyes, so I can’t make a final judgment over whether the current system is reasonable or not. If their total peering cost is pennies on the dollar compared to their other costs though, then your argument becomes incredibly weak, and an argument for a new model become stronger. Costs matter, and if there are none, then the charges become as arbitrary as when cell phone companies were charging people based on how many texts they sent.

        But I want transit costs for services like netflix to be as cheap as possible so we can get 4k streaming and higher framerate streaming and higher color depth in video streaming here as fast as possible. I think it’s perfectly reasonable for a company like netflix to build its own content delivery network to drive their delivery costs down, and it makes financial sense to have caching boxes stored inside ISP networks as that means the same data will not have to travel through the peering points multiple times to begin with, it can travel once and be stored within the network and beyond potential congestion points. These are obvious ways to make video delivery more efficient and less congested. And the argument of verizon/comcast are too large for them to accept such deals makes no sense to me.

  • FollowTheMoney

    Dan: When did you sell out to the AEI? I remember when you once presented the industry news “fair and balanced”. You owe your readers the courtesy of letting them know that AEI, a think tank sponsored by large telecom/cable companies like Comcast, is now sponsoring your blog so there is going to be this sort of bias. Thank you.

    • danrayburn

      AEI is NOT a sponsor of my blog. All of the sponsors of my blog are listed on the right hand side, AEI is not one of them.

      Making false statements like that really don’t help in this debate. Stick to the facts.

      • EelePrewitt

        http://www.techpolicydaily.com/author/dan-rayburn/

        I’m curious as to how you’d characterize the relationship then. Are you a corporate shill who doesn’t even get paid for it? From what I understand, that would miss the entire point of what you’re trying and failing to do here.

        • DNFTT

          When you resort to inappropriatly attacking the messenger vs addressing the debate with relevent facts and data, you lose credibility and are unworthy of a response.

          DNFTT

          • danrayburn

            Do your eyes work? On the right hand side of my blog it says, “blog sponsored by” and shows who sponsors my blog. None of the sponsors are AEI, ISPs or MSOs.

        • danrayburn

          What relationship? They simply re-publishd a post of mine. So does Seeking Alpha, Forbes, CNN, and many other news outlets.

  • Dom Robinson

    Thanks Dan – I absolutely agree with this post.

    I have ‘waxed lyrical’ on a lot of this stuff from gut feel for years. But you present the argument robustly and clearly and backed up with good data and insights.

    It is a shame that voices speaking sense about CDNs gets drowned out by those in need of a good ‘traffic manipulation’ conspiracy, often gift wrapped in totally confused ‘net neutrality’ policy reasonings.

  • http://stopthecap.com Phillip Dampier

    Dan, this is just another blog from you that unfortunately seems to hold Netflix to a much higher standard than the telecom companies I see you increasingly support in your opinion pieces.

    First, you demand disclosure and facts from the players you state have been less than forthcoming on both sides. But that has not stopped you from drawing sweeping conclusions mostly in favor of Comcast, AT&T, and Verizon.

    In no instance in the above piece do readers get a verifiable roster of your sources, because you have not named any of them. We have to trust an unnamed telecom company’s self-produced (and self-interested) chart as the authority on which you base many of your arguments. Most of the rest are ‘unnamed industry insider told me,’ which is hardly compelling evidence.

    Second, while blasting conspiracy theorists for suggesting ISPs are deliberately messing with Netflix’s traffic, you take time out to suggest your own conspiracy theory that Netflix is purposely clogging its links to the provider bad boys that threaten to charge them. Yet Netflix has now signed at least two traffic-calming-for-cash agreements with Comcast and Verizon. That’s a Netflix revenge sandwich that lacks any beef. Netflix is opening its wallet, not Comcast or Verizon.

    Third, and this was LOL funny: “While Netflix was able to convince smaller regional ISPs to voluntarily
    offer settlement free peering, most large ISPs maintain
    national/international infrastructures, which require peering policies
    and consistent business practices to ensure fair and equal treatment of
    traffic.”

    Let me get this straight. Poor Verizon, Comcast and AT&T can’t offer to do the right thing by their customers and quickly resolve congestion other ISPs do not experience because they have to “fairly and equally treat traffic?” No matter they oppose doing that on behalf of their customers and have sued to make sure they don’t ever face it as a rule. There is no doubt in any rational mind that if any of these companies wanted to accomplish a fix of the Netflix traffic problem, they’d find a way in short order, especially if it was happening to Verizon/Redbox’s streaming venture.

    Bell Canada/Aliant is more pervasive and important in Canada than AT&T or Verizon is here, yet they had no trouble signing an agreement with Netflix that assures customers of a trouble-free experience. In fact, Bell Aliant’s Netflix rankings top those of U.S. ISPs.

    But since these giant telecom companies are all advocates of raking in additional compensation from fast lanes or more robust connectivity with their customers, the conflict of interest is easy to spot. There would be no market for “fast lanes” if these providers delivered the speed and performance overpaying customers should already be assured of getting.

    Follow the money my friend. Netflix has a free solution of peering and appliances that have successfully resolved the issue for other ISPs at almost no cost to those ISPs (and no problems from angry customers either). That seems exactly like the responsible solution you demand of Netflix.

    But that won’t guarantee AT&T and others another pay day. With their market control, they don’t mind alienating their customers for their own self-interest.

    By tossing in cable talking points like the Comcast/TWC merger is between companies that don’t overlap (as if that is the only issue in contention), you are risking your credibility as an independent observer and sounding more and more like a industry PR hack, which leads us back to the issue of money.

    Can you assure us you have no financial connection of any kind to any of these players, either through consulting or speaking fees, compensated participation in a group or think tank funded in part by one of these companies, advertising, or any other non-monetary compensation?

    • http://hightechforum.org/ Richard Bennett

      So Netflix deserves special treatment but nobody else does? That doesn’t strike this rational mind as defensible. Netflix wants to claim Internet protections for a cable TV business that bypasses the Internet. That’s good work if you can get it.

      • Jon_Irenicus

        As a practical matter, netflix is special because it is the largest single driver of network traffic during prime time. Finding ways to make delivering that traffic cheaper and more efficient benefits consumers. For netflix that means getting local caches for video delivery inside an ISPs network, but there is no reason neutral caching boxes could not be installed for other companies as well so that everyone could benefit from these efficiencies.

        • http://hightechforum.org/ Richard Bennett

          So much for a neutral network.

          • Jon_Irenicus

            I think a rule that any video provider could link caching equipment within an ISPs network would make a lot of sense. Kind of like how in the UK third party companies can lease the back end wires from British Telecom for internet, but also install special equipment to assist the needs of their customers like say, lower latency. When you only have a single fast broadbrand provider in an area, these kinds of rules open up service options. But we still have liars claiming that we have alternatives for high speed broadband. Not in most places since dsl can’t compete with cable, and that new faster dsl is nowhere to be found where I live.

          • http://hightechforum.org/ Richard Bennett

            US broadband is way faster than UK broadband.

          • Jon_Irenicus

            What I wish for most of all is for google fiber and municipal broadband to expand into more areas, then they can fight it out for who takes better care of video streaming logistics. Verizon would be a lot more proactive about not having these congestion issues if another competitor was playing in the same space.

            As for speeds, where are you getting that US broadband is slower?

            According to this listing, they beat us out (not by much, but they are still ahead).

            http://www.netindex.com/download/allcountries/

          • http://hightechforum.org/ Richard Bennett

            Netindex is a self-selected sample, like an Internet poll, and certain ISPs game the test. The more reliable measures of actual Internet speeds are Akamai’s State of the Internet report and SamKnows, both of which show the US way ahead of all comparable nations except Japan.

            You said yourself that you don’t want to be stuck with DSL, right? Half of the UK can’t get cable and only one percent can get fiber. Do the math.

          • http://stopthecap.com Phillip Dampier

            What? Richard left out nine other countries ahead of us:

            Country / Q4’13 Avg. Mbps / QoQ Change / YoY Change
            1 South Korea 21.9 -1.1% 57%
            2 Japan 12.8 -4.4% 14%
            3 Netherlands 12.4 -0.7% 38%
            4 Hong Kong 12.2 -2.6% 22%
            5 Switzerland 12.0 3.8% 27%
            6 Czech Republic 11.4 0.7% 30%
            7 Sweden 10.5 13% 30%
            8 Latvia 10.4 -6.7% 11%
            9 Ireland 10.4 8.4% 59%
            10 United States 10.0 2.0% 25%

            http://www.akamai.com/dl/akamai/akamai-soti-q413-exec-summary.pdf

          • JP

            Hmmm, looks like 3 of the 4 fastest regions in the world are U.S. states and 10 U.S. states are among the 15 fastest regions in the world. It’s relevant to note that South Korea spans only 1% of the area of the U.S. and Hong Kong’s area is a mere 0.01% of the U.S.

            region / 4Q13 / Avg Akamai Connection Speed (Mbps)
            1 South Korea 21.9
            2 Virginia 14.4
            3 Washington DC 13.8
            4 Massachusetts 13.5
            5 Japan 12.8
            6 Netherland 12.4
            7 Maryland 12.3
            7 Delaware 12.3
            9 Hong Kong 12.2
            10 New Jersey 12.0
            10 Switzerland 12.0
            12 New Hampshire 11.8
            13 Rhode Island 11.7
            14 New York 11.5
            14 Washington 11.5

          • http://stopthecap.com Phillip Dampier

            You are mixing data from regions/states, which offer a more tight and localized snapshot of speeds, with entire countries, something Akamai has avoided doing themselves because it’s an apples and oranges thing. Regions in South Korea, Japan, and across Europe would show even faster speed results if you sampled areas with similar population densities to the U.S. states you mention. Speed rankings for a whole country will naturally be lower than those in the fastest urban regions because rural connections tend to be slower.

            Akamai found “across the whole country (U.S.), average connection speeds rose quarter-over-quarter in just 28 states, with increases ranging from 0.1% in Kansas (to 8.0 Mbps) to 11% in Virginia and Ohio (to 8.3 Mbps). In the 21 states that saw average connection speeds decline from the prior quarter, losses ranged from 0.1% in Michigan (to 10.5 Mbps) to 14% in Nevada (to 8.7 Mbps). Wisconsin and Idaho saw average connection speeds remain unchanged quarter-over-quarter.”

            The population dense northeast and mid-Atlantic turned in the best results, but states like West Virginia, and those in the central, south and Pacific Northwest drag our country’s ratings as a whole down, mostly because of the prevalence of DSL in a lot of these areas.

          • JP

            You miss the point entirely. It is the U.S. which can not be compared to any other country. The only comparable to the U.S. is the EU. You also chose to ignore the fact that the area of Hong is only 0.01% of the area of the U.S. and the area of South Korea is only 1% of the area of the U.S. Comparables to other countries ARE U.S. states, that is the point. S. Korea’s footprint is closest in size to Kentucky’s, but S. Korea has 11 1/2 times the population. 49% of the S. Korean population lives in the Seoul metropolitan area and 69% of the population live in the 3 largest metro areas. Comparing the U.S. to any other country IS the apples and oranges comparison.

          • JP

            Just to recap, no other country in the Akamai rankings is remotely comparable to the U.S.
            The following provides a clear picture of how much larger the U.S. is than the leading countries in the Average Akamai Connection Speed chart. Seven of the nine countries span 1% or less of the United States and the other two cover less than 5%. It’s more like apples and poppy seeds.
            country / area of country/area of U.S. as %
            South Korea 1.0%
            Japan 3.8%
            Netherlands 0.4%
            Hong Kong 0.01%
            Switzerland 0.4%
            Czech Republic 0.8%
            Sweden 4.6%
            Latvia 0.7%
            Ireland 0.9%
            United States 100.0%

          • http://hightechforum.org/ Richard Bennett

            Most of Sweden is uninhabited and has no broadband service at all. The population of Sweden – 8 million people – is all clustered in a few coastal cities where everyone lives in apartments. Stockholm has 800K people, 80% in apartments, and it still took them 20 years to wire 90% of them up to public networks. That’s sad.

          • http://hightechforum.org/ Richard Bennett

            How many rural areas does Hong Kong have, Phil? Do you have any idea how the urban/rural breakdown between the US and Korea or Japan works out? I understand that you hate TWC, but that’s not my issue.

          • Jon_Irenicus

            Explain the performance of Time Warner Cable in New Your City. That is not a spread out area, it is the more densely populated city in the US, and you would expect that covering a relatively small area would yield better speeds than they have.

            Your arguments are a cop out, and excuse poor ISP build outs and upgrade expenditures.

          • http://hightechforum.org/ Richard Bennett

            30% of the entire population of Japan lives in one city, Tokyo, and 70% of Koreans live in Seoul. There are no comparable scenarios in the US. TWC in NYC is an example of what happens when a merger fails and a company splits. TWC is the most poorly capitalized cable company in he US and hasn’t been able to fund major upgrades because of the debt it carries from its spin-off. NYC is also a hard area to serve because the cable conduits are literally full, making it next to impossible to pull fiber until legacy copper is removed. Verizon wants to do that, but the consumer groups are demanding that they leave the copper in place because it runs power to phones. It’s a much more complicated picture than you realize, but feel free to call names if that makes you feel better about yourself.

          • http://stopthecap.com Phillip Dampier

            Pot to kettle about the name calling Richard.

            I’m glad to see you say that mergers are not always a good thing, which is why I am certain you will be joining us in strong opposition to Comcast buying up Time Warner, right?

            TWC’s broadband philosophy had little to do with its capital. The philosophy of its management believed its current speeds were more than enough for customers. At no time did TWC look worse financially than this past winter, yet amazingly the “most poorly capitalized” cable company in the US announced TWC Maxx speed upgrades that best Comcast and cost less.

            A much bigger debt dump is Charter, and they had no trouble implementing speed upgrades.

            Verizon, by the way, is one of the reasons why those cable conduits are in rough shape. Empire City Subway is the company in New York City responsible for maintaining underground conduits in Manhattan and The Bronx, and the manholes by which those conduits are accessed. The company was formed in 1891 as part of a plan for common utility ducts to consolidate all utilities underground.

            I’ll give you three guesses who owns Empire City Subway.

            Wait for it.

            Verizon New York

            Oh, snap!

          • http://hightechforum.org/ Richard Bennett

            Yes, Verizon has been plotting their takeover of the US economy since 1891, glad you finally caught on. Good luck with the name-calling.

          • http://hightechforum.org/ Richard Bennett

            Thanks for using your real name.

          • http://hightechforum.org/ Richard Bennett

            What part of the word “comparable” do you not understand, Phillip? Comparing the US to Korea, Hong Kong, Latvia, et al. is ludicrous. We have counties in the US that are larger than most of the nations on your list.

          • http://stopthecap.com Phillip Dampier

            When caught out, I love how you guys love to change the subject in a reflexive excuse-o-matic way to explain away this country’s dreadful broadband ranking.

            It’s frankly irrelevant to talk about geographic size when the networks providing access are already in place and have been in many cases for decades.

            Cable platforms were originally built for cable television, broadband was an add-on. Telco lines were originally built for voice but can be adapted for DSL and upgraded over time for even faster service.

            Here is where the population density argument starts to fall apart. Busan, Korea has a population of 3.5 million spread out across 300 square miles. Manhattan, the most densely packed NY borough has 1.6 million squeezed into 33 square miles.

            Using the traditional explain-it-all-way “logic,” Manhattan should have fiber everywhere with far faster speeds because population density is much higher so the cost to provide service to a less populated city should be less.

            It’s not. Fiber to the home service in Busan (which is all over) from a provider like SK Broadband provides 100Mbps service for under $20 a month. Time Warner Cable and Verizon both charge far more than this.

            Population density can make a lot of difference in the costs to place the infrastructure. A multi-dwelling unit is cheaper to wire than 100 farms, obviously. But when the wiring is already in place, especially with cable broadband, the cost of boosting speed is very low in comparison. Since cable broadband increasingly dominates American broadband market share, it is absolutely fair to compare this country’s speeds against others. In fact, while countries like Latvia have to spend huge sums to decommission copper, U.S. cable broadband providers can manage huge speed upgrades with a platform shift to DOCSIS 3/3.1 that costs a fraction of what Latvia is paying.

          • http://hightechforum.org/ Richard Bennett

            “You guys?” Knowledge doesn’t come from conspiracies, Phil, it comes from research and experience. Cable franchises in the US have traditionally required universal service, companies price uniformly for the most part, and they don’t get government subsidies like they do in Japan, Korea, and Latvia, where 400,000 homes finally got something better than twisted pair for the first time last year. I’m happy for them, but I don’t want to emulate them.

            And thanks again for posting with your real name instead of pretending to be me.

          • Sheep

            One thing I don’t understand around “rankings” is how even faster broadband improves the lives of the US people. Today most in the US can get about 25Mbps and that has increased year over year. http://www.netindex.com/download/2,1/United-States/

            Almost everything you want to do on the Internet, you can do with 25Mbps and many get much faster speed that this. Perhaps there is only 1% of the US population who’s lives are directly impacted by this so-called national crisis.

            Now we have real-world problems and real-world rankings that actually mean something to the US population like:

            The U.S. ranks 24th in literacy
            The U.S. ranks 17th in educational performance

            etc, etc

            Do we really need to download porn faster or refresh our unwatched digital libraries from USNET quicker, or spend all day binge watching OITNB? Seriously? Shouldn’t the US focus be on more important issues?

            My belief is this is all spun up my major businesses like Netflix and Google to inflame the public to get special treatment for these multi-billion dollar streamers. It means nothing for your broadband at all except for shifting their costs to your bill.

          • Jon_Irenicus

            Here is another small sample. I asked on another forum that has people commenting from overseas if they have buffering issues from services like netflix/youtube. I also asked country of origin and the speeds they are supposed to get. Here are the results so far.

            http://linustechtips.com/main/topic/164797-euros-and-other-foreigners-do-you-have-buffering-issues-with-youtubenetflix/

            With the exception of Norway, every respondent, regardless of foreign country in Europe had no buffering issues with either youtube or netflix.

            To me what that suggests, is that the operating model for the ISPs in europe, and the expectations, even for british telecom in the UK with its “slower than US speeds” by your assertion, have better actual service when it comes to streaming video content.

            Remember, a slight delay in loading a web page is FAR less damaging then the same delay in video playback.

            Dan asked for netflix to offer up a better model for connectivity norms and stop blaming ISPs, I’d turn the question around and ask you and Dan why is it that ISPs in Europe have fewer video streaming issues compared to US based ISPs? Is it merely a lower volume of video traffic consumed? (harder case to make with youtube seeing as that I still have buffering issues with them in the US on time warner)

            Or is it that maybe, just maybe, the status quo peering model Dan is defending as preferable is in fact an inferior model for delivering video traffic?

          • http://hightechforum.org/ Richard Bennett

            Video content is licensed on a per-country basis, and the Netflix portfolio in Norway is much less compelling to local viewers than it is in the USA. Consequently, Netflix has a very small number of customers in Europe and apparently has purchased enough capacity to route their streams where they need to go without a lot of buffering. The issue with Netflix all along has been their refusal to cover their own operations costs from their own purse. Neither Hollywood nor Broadbandville has any obligation to give stuff to Netflix for free.

          • Jon_Irenicus

            Peering is something I am increasingly obliged to think should be free on the ISP side. The infrastructure costs needed should simply be covered by the high margin business known as wired broadband. In particular, upgrading interconnects has to be orders of magnitude cheaper than laying fiber directly to millions of homes, but of course Verizon can’t even be bothered to do that, (and gets cheers from Dan on their laziness and apathy).

            As for operation costs and refusing to cover them. Do you know how much it costs netflix to deliver X amount of traffic to different ISPs? I suppose you think it’s reasonable for them to try to lower those costs as much as possible, which is why they want free peering deals and local caching inside an ISPs network, this all goes to facilitate cheaper costs of operation. The benefits here are even greater with the rise of 4k. Netflix is the only company delivering 4k content over the web aside from youtube, and the latter is not really geared towards premium content. I’d be curious to know how much it actually costs an ISP who chooses to allow free peering with netflix combined with their video caching boxes.

            If Dan cared about a complete picture, he might want to report that figure if he can find it. If the costs are close to rounding errors for better video streaming service, then the only argument left is not about cost and giving something to netflix for “free,” it’s about wanting to get paid from netflix for the privilege of delivering content to customers on verizons network.

            Another question for you. How much do you pay for internet service at home?

            Is it the base tier of service? If not how high did you go and for what reason? Do you and others need 50mbps+ for illegal torrents? Or for something more legal like streaming media?

            As far as I can tell, the majority of legal reasons to increase the money people pay to an ISP for higher bandwidth revolves around the desire to stream media like youtube and netflix with no issues. Some people have families with multiple screens that can be drawing content concurrently.

            If an analysis was done to estimate all the increased revenue/profit ISPs were able to make because of the existence of services like netflix, and that total subtracted all the costs of upgrading interconnects to be able to handle the increased traffic. Do you think ISPs are better off or worse off financially?

            I don’t know the answer, and seeing as how in the tank Dan is for the current ISP model of paid rents for activities they should be doing for their subscribers anyway, I don’t expect him to investigate this, but I would be surprised if netflix was costing isps anything on net. I’d bet most of them make more money off the existence of netflix with higher tier subscriber fees.

            Though I suppose even if that was shown to be true, you would retort, “I don’t care” “netflix is the freeloader, they need to pay verizon to upgrade their network so that their paying customers can access the content they paid for from netflix”

            Lovely. I hope to the god I don’t believe in you lose this fight. It’s a garbage model.

          • karl marx

            I love the “you are making good money so you should subsidize my business argument” I think this works for anyone that asks and in any industry.

          • Jon_Irenicus

            Video streaming services are already increasing the profits for ISPs like verizon and comcast. THEY are the primary legal reason people upgrade their tiers of service for higher speed. Do you deny this?

            Another totally unscientific poll, but video streaming capability seems to rank pretty high.

            http://linustechtips.com/main/topic/165912-people-who-pay-for-their-internet-service-or-influence-those-who-do/

            Is netflix asking for a cut of the additional revenue/profit generated by verizon/comcast customers who pay them more PRECISELY because they want to get better streaming capability? Or course not, nor should they, this is just an example of the added benefit that “freeloader” netflix and other video streaming services are providing to ISPs.

            Every time you and others ignore this contribution to an ISPs bottom line by video streaming services, you commit a lie of omission. You present this activity is a PURE cost to the ISP, It’s an amateurish first order analysis that only serves to run cover for lazy ISP’s who don’t want to share any of the responsibility and burden of upgrading their network/interconnects/service to provide for the EXACT type of content their customers are paying them MORE MONEY to deliver with better results.

            If you and others want to champion the pro paid peering model (like Dan is want to do), a better angle is to argue why that model produces better results for video streaming than something like free peering with the open connect alternative.

            If they can both produce similar results in quality with the primary difference being that one methods gets verizon and others money for that quality delivered vs the others where netflix saves money for that quality delivered, then what you are really arguing for is a pro rent policy. It’s not that netflix traffic is costing verizon more money on net, it’s probably making them more money once you look beyond the surface, but without the rents you go into a rage. I’d be more sympathetic to the rent argument if the activity was significantly cutting into Verizon’s bottom line, but it’s probably doing the opposite, so why the rage?

          • karl marx

            I’m going to call them out on that argument. The average US broadband speed is 10x what is required for Netflix streaming. This is just the “win-win” argument that Netflix employees use. I also don’t know how that scales or is equally applied to others.

            The 10x Netflix requirements is also why I call them out on their rankings. Nothing else suffers for me, so I am confident that Netflix is moving the cheeze and blaming the rat. It’s easy to point at the rat because everyone hates their bill and calling support. But when it is someone else moving the cheeze and placing blame… that is just bad form

          • http://stopthecap.com Phillip Dampier

            But there are ISPs with no arrangements with Netflix and are neither requesting compensation or accepting their offer for peering who have no problems with Netflix traffic.

          • karl marx

            And there are content providers with no direct arrangements with broadband ISPs that are delivering traffic with no issues. OpenConnect seems to be a solution to a problem that only exists where Netflix wants it to exist.

          • karl marx

            ” a better angle is to argue why that model produces better results for
            video streaming than something like free peering with the open connect
            alternative”

            Because you get what you pay for and that payment is competitive with others. Demanding something for free and playing traffic games will impact customers. Paying for something and getting it delivered with support is far better. Reed even said the costs were not material… just lock in the $$s

            Netflix needs to put on its big-boy pants and stop breaking the Internet

          • http://stopthecap.com Phillip Dampier

            Exactly Jon. The mystical problems afflicting Verizon, AT&T and Comcast are not ruining streaming for anyone else. While our blog host seems to suggest Netflix has a conspiracy against these providers by serving their customers from only oversold connections, I think it is much more likely these three companies made a conscious decision to put their foot down and stop upgrading connections “on their dime” to push an agenda of paid peering.

            It’s a creative way around Net Neutrality because the ISP can correctly say it is not throttling traffic. It is simply refusing to build more traffic lanes to handle traffic growth. Paying customers suffer.

          • http://stopthecap.com Phillip Dampier

            Richard, the UK broadband market for “super fast broadband” is competitive, whether you subscribe to Virgin Media’s cable Internet or one of several providers buying wholesale access to BT’s fiber-copper network, comparable to AT&T’s U-verse.

            Sitting here in Time Warner Cable country, UK citizens can buy faster Internet access at a much lower price than I can. While our providers invent excuses to slap usage caps on, UK providers are increasingly getting rid of them as competition works its magic.

            We don’t have that kind of competition here.

          • http://hightechforum.org/ Richard Bennett

            Cable modem service is only available to half of the UK, and fiber to only one percent. In the US, cable reaches 96% and fiber reaches 23%. Study the issues before speaking.

          • http://stopthecap.com Phillip Dampier

            You would think a guy paid by a K Street lobbyist as “an expert” on these issues wouldn’t embarrass himself by writing a whole mess of wrong, but here we are.

            I do happen to study the issues, which is why I can present evidence instead of drive-by claims.

            Nobody said cable broadband was pervasive in the UK, but where it does exist, speeds are fast and competitive pricing is low.

            Your numbers about fiber misrepresent what is going on the UK. There are several fiber to the home projects in the UK, but BT has primarily gone the route of AT&T, widely deploying fiber to the neighborhood service across the country for what they call “super fast broadband.” Just like U-verse, it still relies on existing copper for the last leg of the journey into customer homes, but can manage speeds far in excess of traditional DSL.

            BT is currently upgrading 2,000-3,000 cabinets a quarter and connecting between 70,000-100,000 premises with FTTN/FTTH every week. More than 75% of Britain has access to one or the other right now, not one percent.

            The issue you should have been responding to was my point about competition — faster access at lower prices. BT resells access to its network at fair wholesale prices, allowing a number of competitors to exist in the UK DSL market.

            The industry here managed to lobby their way out of mandated regulated wholesale access for resellers, so they don’t face much competition.

            The differences are striking.

            A Virgin customer today can get 50Mbps cable broadband for $12.61 a month for the first six months, $26.08 each month thereafter. Want to really go nuts? 152Mbps costs $33.66 a month for the first six, then $47.12 a month thereafter.

            Time Warner Cable, in contrast, charges $55 a month for 15Mbps service (not including the modem rental fee).

          • http://hightechforum.org/ Richard Bennett

            What K St lobbyist pays me? And where’s my check?

            BT is following the same strategy that Century Link and AT&T are following, FTTN in most urban markets, and selective FTTH. There’s very little difference in reality. But yeah, Virgin Media (UK’s one and only cable company) runs rings around them on speed to the 50% of the UK it serves.

          • Gary Oswald

            Richard: Who do you work for these days? Enough said.

          • http://hightechforum.org/ Richard Bennett

            I’m self-employed, as I have been since 1986. But thanks for asking, whoever you are.

          • Gary Oswald

            Thank you, Richard. I think that answers Phil’s question.

          • http://hightechforum.org/ Richard Bennett

            Phil didn’t ask a question, he made a false claim.

          • Gary Oswald

            Exactly.

          • http://stopthecap.com Phillip Dampier

            Perhaps this Richard Bennett is an evil clone:

            http://www.itif.org/pressrelease/statement-itif-senior-research-fellow-richard-bennett-latest-fcc-actions-open-internet

            Information Technology and Innovation Foundation | 1101 K Street N.W. Suite 610, Washington, DC 20005

            ITIF Funding: http://www.reuters.com/article/2009/01/07/usa-stimulus-technology-idUSN0749664720090107

            “The group gets some funding from industry, including
            International Business Machines Corp (IBM.N) and Cisco Systems
            Inc (CSCO.O), and also is supported by telecommunications
            companies.”

          • http://hightechforum.org/ Richard Bennett

            I’ve had nothing to do with ITIF since last August, and ITIF is not a lobbying organization in any case. My office is now on M St., across from the evil National Geographic Society. I suppose you’ll rant on them now, eh?

          • Randy B.

            Richard what happened at the ITIF? Why are you no longer there?

          • http://hightechforum.org/ Richard Bennett

            I was a consultant at ITIF, not an employee. I consult with a variety of clients. What goes on between a consultant and a client is nobody else’s business, kinda like your last name.

          • Max Cache

            Richard do you think the ITIF will be providing additional whitepapers on this topic? I enjoyed the last ones.

          • http://hightechforum.org/ Richard Bennett

            You’d have to ask them.

          • http://hightechforum.org/ Richard Bennett

            We have exactly the same kind of competition in the US that the UK does, cable companies competing with DSL. But we have it in twice as many places, where it helps overcome the effects of our more dispersed population. And we also have 50 times more FTTH. So have a ball.

          • Brian Wood

            Richard, even the slowest DSL line is a better user experience than “fast” networks where the peering is all messed up, a la Comcast and Verizon. Voice calls work, Netflix won’t rebuffer, and so on.

          • Jon_Irenicus

            Like Phillip said, even if US speeds were ten times faster than UK speeds, if the peering/video throughput from the major video providers is not right, it means jack.

            I don’t want companies like verizon to just provide gbps connectivity to my email server, that speed and capacity is needed most of all for video streaming like netflix and youtube, both of which have issues on their network. People pay verizon more money precisely to have better connectivity and bandwidth to use those services, why do you and Dan think they have to be paid directly by companies like netflix in order to provide that? Why do transit providers like akamai and netflix and level 3 have to pay verizon or anyone else at all for the privilege to connect to verizons network? Doesn’t verizon get the better end of that deal? without those connections verizon would not have proper access to the worlds sites.

            At the end of the day this seems to involve netflix seeking lower cost content delivery options for themselves by providing near no cost delivery options to verizon (those caching servers are paid for and provided by netflix not verizon and other ISPs). And verizon seems to be advocating the status quo where netflix is not allowed to lower its costs of delivery, not because it is a financial burden on verizon, but because Verizon wants more money from netflix.

            Dan ignores all of these motivations, and I can’t tell if he’s being such an advocate for the large US ISP industry because he actually believes in their peering models or if he just has too many friends in the industry that are and he feels compelled to carry the water for their arguments for access.

          • http://hightechforum.org/ Richard Bennett

            Verizon has to make a number of capacity upgrades to handle the Netflix content at the places where Netflix wants to dump it. You don’t upgrade capacity by clicking a link, it takes actual gear. Be patient.

          • Brian Wood

            Why do they need to upgrade just to sell to Netflix? The question nobody’s asking and needs to be asked is, why did Verizon let their network fall into such disrepair that this was necessary? Comcast has some of the most congested peering in the industry and this wasn’t even a problem with them.

          • http://hightechforum.org/ Richard Bennett

            Is that supposed to be a serious question? When you shift a third of the Internet’s traffic from one set of connections to another, things have to change. Netflix traffic is effectively a DOS attack unless the ISPs know where it’s going to come from on any given day. Netflix has a history of arbitrarily shifting connection points every six months, which causes carriers no end of grief. They haven’t been a cooperative partner, they’re a rogue player who has to be tied down by contract to a group of predictable connection points. Baby with an AK 47.

          • http://stopthecap.com Phillip Dampier

            They are a private business making the same kinds of decisions about traffic distribution as any other content provider. Apple and Amazon don’t call up Comcast on the phone everytime they decide to use someone else for transit. Netflix recognizes it is a traffic beast and has offered, for free, a perfectly sensible solution to help ISPs manage traffic with in house appliances and peering that should be a win-win for everyone, unless you are looking for a pay day at their expense.

          • http://hightechforum.org/ Richard Bennett

            That’s false.

          • Gary Oswald

            While true, it’s not a good idea to accuse Richard of being a lobbyist here. The last people who did that, he threatened to sue, or maybe “prosecute” was the word choice used. Let this be a warning to others: do not taunt Richard Bennett.

          • http://hightechforum.org/ Richard Bennett

            There’s more to the Internet than Gilligan’s Island re-runs. It’s a platform for speech, information, and social interaction. Who needs yet another outlet for the vast wasteland of TV re-runs?

          • Brian Wood

            Richard I don’t understand. So you’re saying the Netflix content stinks, and therefore it’s OK if there’s a reachability problem to them? Comcast doesn’t need to upgrade its peering ever?

          • http://hightechforum.org/ Richard Bennett

            I’m saying Netflix is not an Internet service, it’s a cable TV service that happens to use Internet Protocol.

          • Jon_Irenicus

            That’s exactly how comcast looks at them, no wonder they are not eager to make it easier for them to make the logistics of their business easier. All the more reason to err on the side of the companies that have to use a competitors pathways to reach customers.

            Watch the space on data caps in the future, that is the biggest most vicious tool ISPs can use against netflix, especially with the rise of 4k.

          • http://stopthecap.com Phillip Dampier

            All cable television service is moving towards IP delivery, so it’s a distinction with no difference.

          • Paul Sweeting

            In that case, shouldn’t Comcast/Verizon be paying Netflix for carriage?

          • Jon_Irenicus

            If you don’t like their content, don’t use it. Just because you dislike streaming video (interesting site to frequent with that attitude btw) does not mean it’s not important to other people.

            The vast majority of my internet activity does not involve streaming video, but when it does, slow connections and buffering are much bigger headaches than waiting an extra second for each site link click to move to another web page.

      • http://stopthecap.com Phillip Dampier

        I’m sorry Richard, I thought we were discussing that Internet scourge Netflix, the traffic Godzilla. If another content provider provokes similar reactions among ISPs, I’m all for letting them provide the ISP with similar equipment and connections at no cost to the provider to provide a better customer experience.

    • Vivien Milat

      The Bell Canada analogy doesn’t make sense.

      Bell Canada, like almost every non-U.S. ISP, buys transit in the U.S. from some of the large U.S. players. So, if they don’t set up peering with Netflix, that means the Netflix traffic is going to come in through those transit connections they pay for.

      So the incentives are very different. Bell isn’t peering with Netflix for free because they love Netflix and want to deliver the best Netflix experience; they’re peering with Netflix because some cross-connects at some carrier hotels are a lot cheaper than paying Level 3 or whoever to receive multiple gigabits/sec of Netflix traffic.

      Same thing, interestingly, with Google Fiber, which operates as a separate network from Google’s main backbone. Google Fiber buys transit from a couple large players and therefore has an obvious motivation to peer directly with content sources.

      But Verizon and Comcast somehow managed to turn themselves into peering-only ‘Tier 1′ networks…

    • Y-So Sirius

      100% SPOT ON. Guess you wont get a responce from Dan, he only likes people that feed his conformation bias.

    • danrayburn

      Funny, your argument is that “We have to trust an unnamed telecom company’s self-produced (and
      self-interested) chart as the authority on which you base many of your
      arguments.”

      Yet when Netflix’s puts out a self-produced (and
      self-interested) chart, no one questions it.

      I don’t deal in fantasy land like a lot of people making comments on this post, I live in the real world. I look at real data, talk to the companies involved and review the underlying facts.

      No one seems to be arguing with me that Netflix use to use third party CDNs with no quality issues, that Netflix is 100% responsible for picking and chosing how they send traffic into the ISPs and that Netflix paid Cogent to deliver their traffic even though they knew Cogent didn’t have enough capacity into Comcast. These are all facts.

      Trying to deflect the real issues here by suggesting or implying that I am paid by, work for or compensated by any ISP, MSO or lobbying group hurts your argument. It is very clear who I work for, and who sponsors my blog. But even though anyone can easily see that, some would rather try and debate, imply or suggest things they know aren’t true, simply to try and deflect from the real issues.

      Suggestions and comments like that I won’t respond to anymore.

      • http://stopthecap.com Phillip Dampier

        I am perfectly willing to question both charts, because Netflix has its own interest in this fight. But at least I know where that chart came from. The other one I don’t.

        Much of your reply is what we call a “non denial denial.” You don’t actually answer any of my points and questions, you just claim you live the in “real world” and imply the rest of us don’t. That is hardly convincing argument which should be able to stand on facts, not an emotional response which implies you feel offended someone would ask you questions about your publicly posted conclusions.

        If you talk to the companies involved and post stories based on those conversations, why don’t readers deserve to know your sources? Your underlying facts are largely unsourced as well, making it impossible for anyone to check it out for themselves. I think disclosed sources and citations would strengthen your arguments and conclusions enormously.

        I absolutely agree with you that Netflix has total responsibility for the CDNs they choose. I don’t have any direct evidence that Netflix knew that Cogent lacked sufficient capacity into Comcast. I have seen nobody at Netflix or Cogent verify that.

        I think readers also deserve to know, in unemotional terms, whether you personally have any financial connections to the players or the industry you are writing about. It’s only fair. I don’t suggest any of the players here sponsor your blog. I want to know (and I can’t see any reason why you feel a need to take offense) whether you personally receive any consulting fees or other compensation from any of these companies.

        From what I understand, “Frost & Sullivan is a global growth consulting firm which provides market research and analysis, growth strategy consulting, and corporate training
        services across multiple industries including automotive, healthcare,
        internet and communication technology, and more. Its headquarters are
        located in Mountain View, California, with offices in over 40 countries.” — Wikipedia.

        As this seems to be your employer, it is fair to ask whether any of the players here are paid clients of Frost & Sullivan.

        On my website, I declare without any melodrama I neither work for, nor have any financial interest (or accept contributions from) any telecommunications industry player. It’s one sentence and when I’ve been asked, I never felt offended answering because I think readers deserve to know.

        • CDNguy

          “I don’t have any direct evidence that Netflix knew that Cogent lacked
          sufficient capacity into Comcast. I have seen nobody at Netflix or
          Cogent verify that.”

          CDNs and companies like Internap have had this measurement capability for decades. It is also clearly measured by Netflix through their bit rate measurements. If they broke their measurements out by CDN in a transparent way it would be very clear.

          As others have said, I am really disappointed and unimpressed with the ad-hominem from posters. Shows not only unprofessionalism, but the weakness of their arguments.

  • andxx

    Dan you are saying that the original degradation of service experienced by Comcast and Verizon customers back in September was a result of NFLX switching over their traffic routing to their internal systems and then choosing “congested” connections to those ISPs who did not participate in Open Connect in order to save on the cost of the more expensive “less congested” connections? Seems like a strange business decision given we have all heard that this is a de minimis part of their cost structure and it obviously significantly harmed their end customer experience? Hard for me to rectify it being small cost but big impact on their consumers…what other motivations would NFLX have to do that? Conspiracy theory regarding wanting an ax to wave? Would seem a foolish business decision. Thoughts?

    And Brian Roberts has said that the deal they struck w/ NFLX resulted in NFLX paying less…I guess that doesnt necessarily mean Comcast is receiving less since theoretically NFLX could pay them more but still less than they were paying third parties to deliver the traffic previously, but the dramatic improvement of service levels that resulted from the deal doesnt make sense to me if Comcast did not come out better off in the deal?

    • Vivien Milat

      I think what Dan is saying is a little more extreme.

      i.e. that Netflix chose congested paths, not to save money in the short term, but in order to pressure big ISPs into peering with them (or hosting Netflix hardware) directly on extremely favourable financial terms (which does save them money). And given the degree of media attention their ‘speed rankings’ received, I think they’ve executed that strategy quite well so far…

      • harvey_g

        The only thing Dan’s saying is that he’s become a shill for Comcast and Verizon. Nobody of sound mind and body, without financial loyalty to these companies, could invent half this stuff.

    • danrayburn

      That’s exactly my point.

      “what other motivations would NFLX have to do that?”

      Why pay for it if you can try and get it for free? I agree that I think it is foolish.

      • Jon_Irenicus

        You could ask the same question about why ISPs like verizon and comcast don’t want to take part in netflix open connect.

        Why let netflix send out content without peering costs (that compete with their own video service) when they can get paid off that traffic?

        Netflix pitch to Verizon exec: We’ll deliver the traffic to you and store these caching boxes inside your network at no cost to you (ok, maybe a rounding error of electricity costs). This will greatly improve streaming throughput and video quality during high demand times. All of your customers, including those you up sold higher tiers of service to in part to have better connectivity with streaming services like ours will get higher quality service. It’s a win win, both your customers and ours get better service, and we get to lower our total delivery costs!

        Verizon Exec: But.. that means we don’t get paid? F*ck that, pay more or suffer with increased buffering.

        Lovely attitude you are carrying water for there Dan. They don’t care if all that netflix traffic cost them almost nothing to provide better service, if they are not getting paid, they don’t care whether it’s good or bad. THAT is the model you are propping up and defending. More than that, you seem to prefer that model, you see examples of ISPs allowing free peering and for netflix caching equipment to be installed to provide better service, and you write that activity off because they are “regional” as if that magically implies open connect cannot scale to larger ISPs (If it can’t it has yet to be explained).

        It tells me that you are not pro streaming so much as you are pro paid peering at ISP end points, that is what you keep championing as the right and proper way of doing business. That would make sense if you were a large ISP with a desire to collect as many unearned rents as you could, but you are the streaming video guy, so why are YOU so invested in that above all else?

  • http://www.vzx.tv Dorian Cole

    Well, yeah, Netflix is a big – biggest – squeaky wheel. And the public demands Netflix content and their ISP to deliver it, and they like Netflix, but commonly don’t like their ISPs, so Netflix has a louder bark, and therefore a better negotiating position. There is nothing unusual about bullies in business, and using unusual tactics to force the issue. Satellite and cable services and broadcast networks do it regularly. And part of business is living with the uncertainty you alluded to in the article, and finding ways to make things less uncertain. Netflix, CDNs, and ISPs are free to negotiate long term contracts that give them certainty of price, performance, and longevity. The question is, “What is standing in the way of their doing that?” It’s a shell game until we know the real answer.

  • CDA

    You label the situation as Traffic Manipulation tactics, however the Art of Peering whitepaper states that this is done in order to force the potential peer into the relationship through incurring high transit-based costs. Cogent appears to directly peer with Comcast, so how does this align description?

    • Vivien Milat

      That strategy, as stated in that paper, obviously only works if the target has transit providers (so, for example, if it’s a non-U.S. network or a smaller U.S. network). But Netflix, at least according to Dan (and common sense), has pursued a modified version:
      1. Pick the transit provider whose peering capacity with Comcast will be saturated when you add the Netflix traffic, e.g. Cogent.
      2. Hope (correctly) that Comcast won’t be in a mood to upgrade that peering infrastructure, at least for free or quickly. I believe Comcast and Cogent have been fighting for months over this…

      3. Watch the performance of the streams go down.
      4. Shame Comcast publicly.
      5. Hope you get free (or, as Dan would put it, cheaper than your transit provider, at least) peering direct with Comcast.

      • Jon_Irenicus

        From the horses mouth: Reed towards the end of his interview here basically says he’d like for cogent and level 3 and everyone else that peers directly with ISPs like comcast to have settlement free peering, then transit costs across the board go lower.

        http://recode.net/video/

        That does not mean those transit providers would be paying nothing to get content to comcast or verizons network, they still build and upgrade their own infrastructure and make sure it has the capacity to get the content to the ISP. That’s not “free” as Dan and others keep suggesting. But the idea that Comcast and verizon should get to collect rents on peering is the issue under debate. Should they collect fees for that privilege or not?

        Dan’s answer is categorically, Yes. He is pro ISP connection fees. Someone asked Reed about whether it should cost more for increased usage of a service, and he gave the example of listening to the radio vs using more gasoline to propel your car. There is no marginal cost to increased radio listening, there is a much larger marginal gasoline cost to increased car usage. And the key point? broadband is much closer to radio listening in terms of marginal cost, a slight increase in electricity cost, some one time charges for routing equipment and a trickle of maintenance costs, and that is mostly it. That’s why wired broadband data caps are bogus, and so are claims that upgrading interconnects is prohibitively expensive to handle netflix traffic is bogus as well.

        No company has disclosed the upgrade costs that would be required, along with the revenue they generate from subscriber fees for their service. The reason they do not do that is that these ISPs have more than enough revenue and profit and capital to cover the cost of the upgrades, and to provide settlement free peering. But the big boys have enough subscribers that they can tell the transit providers and netflix, either pay us for the privilege of connection, or suffer. It’s a racket, a naked abuse of market power to get favorable terms. Dan gladly highlights netflix as a bully using its streaming clout to try and shame ISPs (poor little comcast/verizon), but never turns any attention to ISP practices. I don’t think this is because he is bought and paid for, I think he genuinely believes this is a better model. Why? I do not know, maybe he’s just siding with his contacts in the industry and is engaged in group think.

        In a way this is not even about netflix, if they were removed from the picture, I still think it is a better standard for an ISP to be proactive in making sure its customer base can actually use internet service with as few issues as possible. Especially for video streaming because of the performance sensitivities. That is what cablevision and google fiber and cox has done by and large, but not comcast and verizon.

        I’d vote with my wallet if I could, but that is the thing about not having a proper competition constraint, when you are the only game in town, lazy bare minimum service is what you can get away with. It’s too bad Dan does not want to call them out on their apathy to the rising demands of what their customers want, better streaming video. It would be nice if their own paying customers desire for better streaming connectivity was enough to spark them into action, but they want someone else to pay for those build outs as well, however tiny it is to their bottom line.

        What terribly low expectations some bloggers have of ISPs. I hope I never become that jaded.

        • Vivien Milat

          Two things:
          1) The key question in peering politics has always been ‘who is a peer’ vs ‘who is a (potential) customer’. That issue is not new.
          2) There have been huge changes to how ‘the Internet’ works in the past decade or so.

          10-15 years ago, it was clear enough. Small ISPs pay medium ISPs for transit. Medium ISPs pay bigger ISPs for transit. And once you got to the big U.S. players (the Sprints, UUnet, MCIs, etc of the world), they peered with each other for free. But peering can obviously lead to efficiencies for everyone involved, so there has been a lot more peering going on since then…

          ISPs only want to peer with other ISPs of roughly the same nature. In non-U.S. markets, for example, there was an obvious way of identifying a ‘peer’: anyone with a direct U.S. connection is a peer to anyone else with a direct U.S. connection. So, if ISP A, B, and D each have a US connection, while C buys transit from D, then A, B, and D peer with each other, and A and B tell C to take a hike and buy transit (or some other partial-routing arrangement). There’s simply NO business/technical case for A and C to peer: A is not saving money on their U.S. transit connection, and they’re losing the chance to win C’s transit business away from D. So the only people who will be willing to peer with C are other customers of A, B, and D.

          And that, roughly, is how the traditional peering hierarchy has always worked. Small fish pay bigger fish who pay even bigger fish who peer with each other in some room in the U.S. And each size of fish try and make peering agreements with similar-sized fish so they each pay the bigger fish less. But if that doesn’t happen, then the ‘default’ is that you pay a bigger fish, the destination pays a bigger fish too, and you both expect your respective big fish to play nice with each other.

          What changed about 10 years ago is that Big Content (e.g. Facebook, Google, etc) realized that, instead of paying a few transit providers for all their traffic, they could buy up some cheap fiber and start peering directly with big residential ISPs, including foreign ones. For as long as those ISPs buy transit from bigger ISPs (which all the foreign big eyeball providers do), this is a win-win proposition – why would a Canadian cable provider pay Level 3 or whoever to receive gigabits of Google traffic when they can just string a piece of fiber between their router and Google’s in a building in Chicago or Ashburn? (and if Google wants to extend their fiber network to Toronto, even better!)

          The problem with Comcast/Verizon is that somehow, they have gotten so big that THEY are peering-only networks at the top of the food chain, just like the Sprints and UUnets and MCIs of the late 1990s. And suddenly the ‘new’ model of Big Content and Big Eyeballs peering to avoid both paying Big Transit… falls apart.

          So, we’re stuck with this situation, which can be viewed two ways:
          1) Comcast customers want Netflix. Therefore Comcast should bend over backwards to make the Netflix traffic flow. This seems to be the perspective of most people criticizing Dan, or,
          2) The ‘default’ arrangement is Netflix should be paying a bigger ISP for transit. That’s how it’s always been. That’s how everyone connects to the Internet unless they are one of the big peering-only Tier 1 U.S. networks. That’s what every other content provider does, including the ones who are too small to do any serious peering. And then that bigger ISP does whatever it takes to deliver Netflix’s packets where they need to go. (Of course, given Netflix is 1/3rd of Internet traffic, that makes things much more difficult.)
          And in that mindset, if Netflix wants to deviate from that, the arrangement needs to be mutually beneficial.

          Remember, my Canadian ISP peers with Google not because I love Gmail and YouTube, but because it saves both them and Google money on their Level 3 or AboveNet transit bills.

          Where is that rationale for Netflix vs Comcast/Verizon? ‘My customers want to reach you’ has NEVER been how peering politics have worked. The normal response to someone knocking at a bigger fish’s door and saying ‘I/my customers want to reach you/your customers, let’s peer directly’ has always been ‘then go pay me, or go pay someone I consider a peer to deliver your traffic to me’.

          Frankly, I think is why so many people are hostile to Dan here. People who have no background in peering politics presume that ‘A’s customers want B’s content’ (or ‘A’s customers want to communicate with B’s’) is, by itself, sufficient justification for A to peer with B for free. It has never been.

          • Jon_Irenicus

            “People who have no background in peering politics presume that ‘A’s customers want B’s content’ (or ‘A’s customers want to communicate with B’s’) is, by itself, sufficient justification for A to peer with B for free. It has never been.”

            It should be, especially when the company in question (netflix) is directly driving increased revenue/profit for that larger ISP by inducing their customers to upgrade to higher tiers of service with vertually ZERO marginal cost increase to the ISP.

            Also, the “bigger fish” transit providers have several other large transit providers that compete directly against each other. That is an argument given as to why transit prices have fallen so dramatically over time.

            Where is the competitive constraint on comcast for their peering costs? Their word? In most markets they are the only high speed broadband game in town. In small pockets you might have verizon fios, but both of them are behaving like typical rent seekers. Some people point to the fact that current peering costs for netflix are lower than what they were paying others to do the same job getting traffic to comcast. What else would it be? Comcast has a microscope over them with the pending time warner cable merger. What fool expects them to turn the screws on peering costs right now? This is a long game, not some over night show.

            Video streaming will be the bulk of internet traffic in the years to come, and it will only gain in total share as 4k becomes more widespread. And a precedent has already been set, comcast ought to get paid Dan says, because that is how it always has worked with peering. Has = ought now? This is a basic logical fallacy, highlighted hundreds of years ago by Hume.

            I don’t like that precedent, I don’t trust large cable companies, whatever the current rates may be. We ALL know what happens with these companies with their pay tv rates, the costs climb ever higher. They blame the content creators of course, but instead of allowing margins to thin, they “innovate*” with things like 15 dollar a month dvr boxes and other fees which seem to climb ever higher over time.

            *Sidebar, anytime you her some sock puppet use the phrase “innovate” in these discussions, grab your wallet.

            At least in the pay tv market you have an escape valve with satellite. Where again do most people go to escape a monopoly wired broadband provider? Dsl? I’m over 12 thousand feet from the branch, and even if I was closer, that speed is too anemic. I have ZERO trust that these large ISPs will constrain themselves, when they have no reason to . Competition is the mother of all constraints in a capitalist system, and these companies have none in most areas.

            And like I said before, what they are charging today is irrelevant, this is a long game. The comcast/verizon execs are like the kids here who delayed eating the marshmallow right away.

            As for tomorrow? Once the mergers are complete and the precedent of paid peering is even further entrenched? Bend over, so it was with tv fees, so shall it be with broadband fees, on both ends.

            Business model approved by Dan.

          • Vivien Milat

            But Netflix still has its choice of “bigger fish” transit providers. The Cogents, Level 3s, AboveNets, etc of the world are happy to take Netflix’s money and deliver their traffic to Comcast. Netflix just doesn’t want to pay them! But basically, that puts a cap on the price Comcast can charge Netflix – it always has to be less than what Netflix would pay a big fish transit provider…

            I thought about this some more after my earlier post, and I think all peering for the past 15-20 years can be summarized with two rules:
            1. Peer with your transit provider’s customers, or your transit provider’s peers’ customers.
            2. Do not ever peer with your peers’ transit customers.
            (and 3. If you don’t buy transit from anyone, then you have to peer with everyone who also doesn’t buy transit. But reaching that stage is hard…)

            If I, for example, started some random content provider in Canada tomorrow and knocked at the doors of Big Cable and Big DSL, asking for free peering, their answer would be “please call our business services department that will give you a quote for transit (or maybe some paid partial routing table arrangement).” And it makes sense – my hypothetical content provider needs transit from someone (unless I’m going to convince the U.S. Tier 1s to peer with me), so if Big Cable peers with me, that just increases the odds that Big DSL’s business division will be my transit provider.

            This has nothing to do with the cable industry’s greed (something which is legendary across the world, I think) or new business models. It’s simply an application of how things have always been, i.e. the way you connect to ‘the Internet’ is to pay a bigger fish than you.

            And this is what I (and Dan) find incredulous: Netflix has somehow decided that they can bypass paying a transit provider like everyone else and go to a big fish peering-only Tier 1 ISP and demand free peering (on the sole basis that that big ISP’s customers really really want Netflix’s traffic.) Something that, say, a large foreign ISP could not do – if a large French or Canadian or UK ISP showed up in Ashburn, VA and demanded free peering with Sprint or Level 3, they’d be told to take a hike. And if Netflix is told no, then have their PR/lobbyists make a big stink about it.

            What I find interesting, too, is that Google, Facebook, and Microsoft have direct connectivity to Comcast… but Yahoo is reached through Level 3, who gets paid by Yahoo. I wonder whether any money changed hands for that arrangement with Google/Facebook/Microsoft. If it did, then that confirms my/Dan’s view. If it didn’t, then… I suddenly am far more sympathetic to Netflix.

            Finally, I don’t want to digress too much, but I think the idea that Netflix makes big cable ISPs money is naive. Go read up on how DOCSIS works – you have a couple hundred subscribers on a node sharing one (DOCSIS 2) or multiple (DOCSIS 3) 38 megabit/sec channels. This is a technology that assumes almost all customers are idle most of the time. If people are suddenly consuming a steady 3-5 megabits/sec in Netflix streams for hours, the providers have to spend considerable money on node splits and other upgrades. But THAT has nothing to do with the peering fight – if it did, it wouldn’t explain why smaller, transit-buying cable ISPs are happy to peer with Netflix…

          • Jon_Irenicus

            Question. What % of comcasts internet revenue comes from transit/peering deals vs subscriber fees?

            According to Om Malik in the comments here:

            http://gigaom.com/2014/02/12/comcast-and-time-warner-cable-forget-tv-it-is-all-about-broadband/

            They never make the margins/costs on their broadband business known.

            The reason I ask is because presumably the primary source of revenue and profit for a tier transit provider comes from people paying it for transit and differential peering levels. It is reasonable for them to request fees because that is the way their networks are paid for. Do you think that a large scale ISP should have the same standards of peering as a company whose primary revenue stream is peering?

            Based on your comments, I would assume that your answer is yes, they should have the same standards. It seems to me that a consumer facing ISP, that gets the vast majority of its revenue from paying customers ought shoulder more of the costs of peering and upgrade costs. Since the these ISPs are often the sole option for faster broadband access they have captured audiences that cannot be easily moved to other alternatives. If they systematically chose to raise their peering costs to netflix and other tier 1 providers, that would raise transit costs into their network across the board would it not?

            I suppose the other tier 1 networks could just respond by increasing their own transit costs of sending comcast traffic to the rest of the internet, but which entities have the bigger stick here?

            At the end of the day, I want the ISPs to shoulder more of the cost burdens. If the reports of their margins are anything close to accurate, they are not hurting for cash, and in a more competitive market for customers, I don’t think they could get away with slow rolling upgrades for connectivity. If we ever do get a more competitive high speed wired broadband marketplace, having better netflix service reliability WILL be a selling point in favor of the ISP that can deliver better video streaming. That holds whether companies like netflix is paying for that privilege or getting free peering with the isp. The fact that some of these large ISPs do NOT have to fight and be proactive to keep their customers just goes to show how noncompetitive the broadband markets we have are.

            And I think you underestimate the benefits of video streaming on the bottom line of ISPs like comcast and verizon with fios service. Customers are increasingly shifting away from dsl because the speeds on those services were not up to the task. They mostly went to cable. I hear there is a faster version of dsl that can be slightly more competitive, but if they still have similar distance limitations that is not going to make a credible dent in the cable markets.

            I have very little faith that anything is going to stop the march of the market power of the larger ISPs except more competition (and that’s not happening anytime soon in most places), but I will say one thing. As annoyed as I and others am about netflix having to pay verizon and comcast to reach customers on those networks, If they start to implement and enforce data caps on wired broadband… world war 3.

            My fury, and that of everyone else Dan is repulsed by, will be terrible. There will be orders of magnitude more rage and fire bombs thrown. It’s like the old metered texting model verizon championed and migrated to internet service. I suspect even you and Dan would be opposed to that, but if you are not, if/when it comes, god help you.

          • Vivien Milat

            No one’s ‘primary revenue stream’ is peering. ISPs, whether they are Comcast or Level 3 or Cogent or whoever, make money from selling IP transit, whether it’s residential or business. But this Comcast/Netflix thing isn’t about money from Comcast’s perspective – it’s about principle. The principle that someone who bought transit from Comcast’s peers OR who bought CDN services from people who may pay Comcast or Comcast’s peers shouldn’t be able to bully their way into free peering with a tier 1 peering-only network. (I am still surprised that Comcast apparently managed to make themselves into a tier 1, Verizon did because they bought UUnet when Worldcom imploded)

            (And let’s make it clear, if _I_ was Comcast, I would not be waging this fight. Netflix has won the hearts and souls of just about everybody, whereas cable companies are among the most despised businesses. So even though I think Comcast is right in principle, I think this is a fight they are unlikely to win…)

            It’s funny that Comcast doesn’t disclose Internet revenue. Neither does my Canadian cable company, which blends Internet with TV in their financial reports. _I_ have always thought it’s because Internet makes way less money than the TV side of the business (which scales much, much better), and they don’t want to highlight that to their shareholders. Everybody else thinks cable companies are printing money selling DOCSIS Internet access. Who knows who is right on that.

            And BTW, data caps are the reality up here in Canada. But let’s not get into a discussion about that right now…

          • DNFTT

            “(And let’s make it clear, if _I_ was Comcast, I would not be waging this
            fight. [...] So even
            though I think Comcast is right in principle, I think this is a fight
            they are unlikely to win…)”

            Because the best way to stop bullying is to give up your lunch money. Other bullies will see this and leave you alone. Right?!?

            The main question is “who is the bully?”

            Netflix is saying that ISPs are the bullies by forcing Netflix to pay them. ISPs respond saying Netflix is the bully by wanting to stop paying their network delivery costs and strategically using their massive traffic controls to create “peering wars” and in turn customer problems.

            While I agree the love/hate relationship people have with their providers is greatly factoring into this discussion, if you look at it objectively there is more going on than peering congestion.

          • Vivien Milat

            If you know the bully who wants your lunch money has sufficient influence over the school administration that you have a good chance of getting suspended if you resist, then yes, I think giving up your lunch money is the wiser course of action.

            Assuming Netflix is the bully (which is certainly my view), then they have the following strategic advantages:
            - a product everyone wants
            - opponents everyone hates
            - an outstanding PR team
            - a ‘David vs Goliath’ story
            - opponents who can’t put together a coherent story, but rather just put out vague executive business-speak that helps Netflix’s case
            - collective ignorance of how peering has always worked, so for many people, the first thing they learn about peering is what Netflix PR wants them to know

            - a willingness to run to regulators or legislators and complain
            How is it reasonable to try and fight this?

          • Jon_Irenicus

            How peering has always worked is not an argument about how it should work.

            It’s not crazy to think that an ISP whose customers pay it for internet access ought to have some responsibility to make sure that its paying customers can receive the content they are requesting on the internet. And if some of that content is taking up too much bandwidth and causing congestion along certain pathways into the ISPs network, why should the ISP have zero responsibility to expand those lanes?

            free peering to an ISP =

            -happier customers
            -happier content distributors
            -selective unhappiness from some of the larger ISP’s (gawd, these hippies want to drop our margins from 85% to 82% !!!… and then it goes back up to 85% after a year or two once the upgrades are paid for)

            /tissue

            The details on what it actually costs the ISP to perform these types of upgrades and that impacts their bottom line could either vindicate my view or shut me down.

            Given the choice to patronize an ISP that only connects to a content distributor fpr better cpnnectivity to their paying customers because they were paid vs another that allowed the connection for free because it offered better connectivity, I’d choose the patronize the latter.

            Again, it seems like to you and Dan, you see no difference in quality and motivations between those two ISPs. It’s all about the balance of power and getting as much as they can get. There is no room for any sort of idealism for providing better service for its own sake. Is it so wrong to want my ISP run by Ned Stark vs Cersei Lannister?

            Maybe Ned is a bad example due to his outcome, but you get the idea.

          • Vivien Milat

            The funny bit is, what you are recommending actually entrenches the power of Big Content vs Small Content. In your world, Big Content would peer for free with big ISPs, while Small Content pays the Level 3s of the world for transit to both big and small ISPs. So Big Content has a considerable cost advantage compared to Small Content.

            But here’s the fundamental thing: ‘the Internet’ has always operated on a very simple model: each participant is responsible for paying a bigger fish that pays an even bigger fish, and so on until you reach the big U.S. peering-only backbones, which are expected to play nice with each other and keep plenty of peering capacity between each other. (Back when I first signed up for dialup in Canada in 1996, there was my very small ISP, a bigger regional ISP that they bought a T1 from, a bigger national Canadian ISP, and then that ISP had a couple T3s to MCI, their chosen big U.S. peering-only network. So that’s four sizes of fish that were being paid upstream of me…)

            One of the things that makes ‘the Internet’ very different from other forms of telecommunications is precisely this. It’s what makes ‘the Internet’ more democratic: for as long as ‘you’ (whether you are a residential customer or a content provider or anything else) are willing to pay someone who will take your traffic closer to the big peering-only U.S. ISPs, you can exchange traffic with anyone else. You don’t have to pay that ‘anyone else’, you don’t have to pay that ‘anyone else”s provider, etc.

            Sometimes your route will be ugly (for many years, most traffic between different Canadian ISPs went south to the U.S., possibly through a second big peering-only U.S. ISP, and then back north), but the traffic will make it through.

            What you’re suggesting is that residential ISPs have some kind of responsibility to deliver the content that their customers want. I think that’s actually a dangerous slippery slope to go down that’s likely to turn ‘Internet access’ into ‘IP-based access to selected content sources’. Something that, actually, I suspect Comcast executives would love!.

            Their responsibility is to take their customers’ packets and deliver them to someone else who will deliver them closer to the big U.S. peering-only networks, or, if they are a big U.S. peering-only network themselves (and I suspect Verizon/Comcast are the only two big residential ISPs that are), their responsibility is to deliver those packets to all the other big U.S. peering-only networks.

            Netflix’s responsibility (just like any other participant on the Internet) is to hire someone who will deliver those packets closer to the big U.S. peering-only networks, which they did for many years. What Dan is claiming is that they have now deliberately chosen to hire transit providers who don’t have enough peering capacity with Comcast/Verizon to actually provide the service they were paid for.

            I’ll restate the key principle here: the big U.S. peering-only networks are the ‘route of last resort’. If both ends have good connectivity to big U.S. peering-only networks, then traffic will flow just fine between them. And in the late 1990s and early 2000s, as I said above, most traffic between Canadian ISPs went that route, as absurd as it may seem (why send traffic to Chicago 1200km away when the other end is across the street?), and things worked just fine.

            (The big European ISPs, facing much higher costs for transatlantic links until the early 2000s, organized themselves much more efficiently to prevent traffic from going to the U.S. and back.)

            (And FWIW, I am a strong supporter of ‘net neutrality’, properly defined, namely that any IP packet, once it’s reached a given provider’s network, should not be prioritized compared to IP packets from another source network.)

            For the record, in terms of choosing a residential (or business) ISP, I will tell you what I will favour:
            1) An ISP that invests in infrastructure that brings them closer to the big U.S. peering-only networks. (My current very large Canadian cable ISP operates their own routers in the U.S. and buys transit directly from a couple of big U.S. peering-only networks.),
            2) An ISP that chooses upstream providers with a reputation for reliability and network quality (not all big transit providers are necessarily alike),
            3) An ISP that maintains adequate capacity to its upstream providers and peers,
            4) An ISP that maintains adequate capacity between all points in its own network (including the ‘last mile’ – from what I hear from U.S. friends, a lot of cable cos in your country struggle with that during peak hours. My ISP may meter my usage, but I do get the 150 megabits/sec that I pay for at 9PM just as well as at 3AM),
            5) An ISP that seeks to complement their well-chosen transit providers (see #2) with reasonable peering with reasonable peers (My current ISP is not perfect, but compared to 12 years ago when they peered with no one anywhere, things have massively improved),
            6) An ISP that will not do any kind of traffic prioritization based on side deals with content providers.
            (And yes, because Comcast/Verizon are ‘big U.S. peering-only networks’, the above formula doesn’t apply without modification, I know. But it applies to just about everyone else…)

            If you have an ISP that meets the above 6 criteria, then you should have outstanding performance from ANY content provider (whether it’s some dude on BitTorrent, somebody’s $30/month VPS, the university across town, or a giant like Netflix) without any need for any direct interconnections, paid or unpaid. If not, then it’s that content provider’s fault – either their servers/routers/etc are slow, their network is congested, they picked lousy providers, etc. Your ISP did their part. And they did their part FAIRLY, without favouring some content providers over others.

  • http://stopthecap.com Phillip Dampier

    Well folks, it appears the FCC is unconvinced by arguments that the Netflix-Comcast debacle is innocent and pure and has launched an investigation.

    The agency says they’ve obtained the agreements signed between Comcast,
    Netflix and Verizon, and have been asking around for additional detail
    on deals, which are historically kept very secret.

    “Consumers must get what they pay for. As the consumer’s representative we need to know what is going on,” Wheeler said.

    But lobbyists need not panic yet.

    “To
    be clear, what we are doing right now is collecting information, not
    regulating. We are looking under the hood,” insisted Wheeler. “Consumers
    want transparency. They want answers. And so do I. The bottom line is
    that consumers need to understand what is occurring when the Internet
    service they’ve paid for does not adequately deliver the content they
    desire, especially content they’ve also paid for.”

    So on the one hand, I hope we all welcome an investigation from an agency that can compel document production. The unfortunate part is they are unlikely to make that documentation or the investigation public.

    • http://hightechforum.org/ Richard Bennett

      That’s good, the only way to tamp down the hysteria around this issue is to disclose the facts.