IBC News Recap: HEVC, Cloud Workflows, Media Management, Video Encoding & Optimization

I wasn’t at the IBC show this year, but here’s a run down of the news announcements that I saw. Lots of focus on HEVC as expected, but less talk about 4K compared to last year. If I had to pick one theme from this show it would be cloud based media management platforms. I found two announcements to be particularly interesting; Brightcove’s new stand-alone video player platform, de-coupled from their OVP services and Microsoft Azure Media Services new live streaming platform. I’ll have more details and thoughts on both of these new offerings later in the week.

Inside Apple’s Live Event Stream Failure, And Why It Happened: It Wasn’t A Capacity Issue

Apple’s live stream of the unveiling of the iPhone 6 and Watch was a disaster today right from the start, with many users like myself having problems trying to watch the event. While at first I assumed it must be a capacity issue pertaining to Akamai, a deeper look at the code on Apple’s page and some other elements from the event shows that decisions made by Apple pertaining to their website, and problems with how they setup storage on Amazon’s S3 service, contributed the biggest problems to the event.

Unlike the last live stream Apple did, this time around Apple decided to add some Javascript JSON (JavaScript Object Notation) code to the apple.com page which added an interactive element on the bottom showing tweets about the event. As a result, this was causing the page to make refresh calls every few milliseconds. By Apple making the decision to add the JSON JavaScript code, it made the apple.com website un-cachable. By contrast, Apple usually has Akamai caching the page for their live events but this time around there would have been no way for Akamai to have done that, which causes a huge impact on the performance when it comes to loading the page and the stream. And since Apple embeds their video directly in the web page, any performance problems in the page also impacts the video. Akamai didn’t return my call asking for more details, but looking at the code shows there was no way Akamai could have cached it. This is also one of the reasons why when I tried to load the Apple live event page on my iPad, it would make Safari quit. That’s a problem with the code on the page, not with the video.

Because of all the refresh calls from the JSON-related JavaScript code, it looks like it artificially forced the player to degrade the quality of the video, dropping it down to a lower bitrate, because it thought there were more requests for the stream than there was. As for the foreign language translation that we heard for the first 27 minutes of the event, that’s all on Apple as they do the encoding themselves for their events, from the location the event is at. Clearly someone on Apple’s side didn’t have the encoder setup right and their primary and backup streams were also way out of sync. So whatever Apple sent to Akamai’s CDN is what got delivered and in this case, the video was overlaid with a foreign language track. I also saw at least one instance where I believe that Apple’s encoder(s) were rebooted after the event had already started which probably also contributed to the “could not load movie” and “you don’t have permission to access” error messages.

Looking at the metadata from the event page, you could see that Apple was hosting content from the interactive element on the apple.com event page on Amazon’s S3 cloud storage service. From what I can tell, it looks like Apple setup the content in a single bucket on S3 with little to no cache hit ratio, with poor bucket configuration. Amazon didn’t reply to my request for more info, but it’s clear that Apple didn’t setup their S3 storage correctly, which caused huge performance issues when all the requests hit Amazon’s network in a single location.

As for Akamai’s involvement in the event, they were the only CDN Apple used. Traceroutes from all over the planet (thanks to all who sent them in to me) showed that Apple relied solely on Akamai for the delivery. Without Akamai being able to cache Apple’s webpage, the performance to the videos took a huge hit. If Akamai can’t cache the website at the edge, then all requests have to go back to a central location, which defeats the whole purpose of using Akamai or any other CDNs to begin with. All CDNs architecture is based on being able to cache content, which in this case, Akamai clearly was not able to do. The below chart from third-party web performance provider Cedexis shows Akamai’s availability dropping to 98.5% in Eastern Europe during the event, which isn’t surprising if no caching is being used.

akamaiThe bottom line with this event is that the encoding, translation, JavaScript code, the video player, the call to S3 single storage location and the millisecond refreshes all didn’t work properly together and was the root cause of Apple’s failed attempt to make the live stream work without any problems. So while it would be easy to say it was a CDN capacity issue, which was my initial thought considering how many events are taking place today and this week, it does not appear that a lack of capacity played any part in the event not working properly. Apple simply didn’t provision and plan for the event properly.

Updated Thursday Sept. 9th: From talking to transit providers & looking at DeepField data, Apple’s live video stream did 6-8Tbps at peak. World Cup peak on Akamai was 6.8Tbps. So the idea that this was a capacity issue isn’t accurate and the event didn’t generate some of the numbers I see people saying, like “hundreds of millions” watching the stream.

Updated Thursday Sept. 9th: While some in the comments section want to argue with me that problems with the Apple.com webpage didn’t impact the video, here is another post from someone who explains, in much better detail than me, many of the problems Apple had with their website, that contributed to the live stream issues. See: Learning from Apple’s livestream perf fiasco

Internet Traffic Records Could Be Broken This Week Thanks To Apple, NFL, Sony, Xbox, EA and Others

Thanks to so many large scale live events and large file downloads taking place this week, it’s going to be a huge week of traffic on the Internet with content delivery networks and last mile providers preparing for what’s to come. Tomorrow in particular will be a big day on the net with so many things taking place all on the same day. Apple product announcements always make for a busy day on the web and while iOS 8 won’t be available for download tomorrow, here is a list of all the other events taking place tomorrow, or this week.

  • Monday Night Football (WatchESPN)
  • Apple’s Product Announcement (Tuesday)
  • Microsoft Security Patches (Tuesday)
  • NFL Now/Game Rewind Highlights (Tuesday is busiest day for NFL videos)
  • Yahoo! Aerosmith Concert (Tuesday)
  • Bungie’s Release Of Destiny Game (Tuesday)
  • EA Sports Fifa 15 Game Beta (Tuesday)
  • EA Sports NHL 15 Game (Tuesday)
  • League Of Legends NA/Europe Seed Matches (Tuesday)
  • Xbox Free Game Releases (including Halo: Reach)
  • Sony PS4 White Destiny Bundle (Tuesday)
  • New York Fashion Week Live (Monday-Thursday)
  • Apple’s iOS 8 Download (Probably Thursday)
  • President’s ISIS Speech (Wednesday)

Delivering video over the Internet at the same scale and quality that you do over a cable network isn’t possible. The Internet is not a cable network and if you think otherwise, you will be proven wrong this week. We’re going to see long download times, more buffering of streams, more QoS issues and ISPs that will take steps to deal with the traffic, knowing it will have a negative impact on the user experience. When iOS 8 comes out, some last mile providers are going to struggle and some will rate limit their network connections, as we saw the last time Apple’s iOS download was available. For some ISPs, iOS 7 downloads took up 40% of their traffic. Also, all other content providers are going to have to compete with this traffic and many I spoke to about it are keeping an eye on their quality guarantees and SLAs with their CDNs this week.

As for which CDNs are delivering all this content, I’ll be doing a lot of traceroutes this week, but Akamai, Limelight and Level 3 are all in the mix. I know Akamai will see the most web traffic from news sites covering Apple’s product announcements. Last time I looked, Microsoft’s security patches were being delivered by Akamai, Limelight and Level 3. Yahoo’s concert is being done by Akamai. Level 3 and others do the Xbox releases, Limelight was doing a lot of Sony downloads last I checked and when iOS 8 is available, I expect a lot of it to be delivered by Apple themselves, with Akamai and maybe also Level 3. Many of these events, both live and downloads, push more than 1Tbps via a single CDN, let alone those that use dual vendors, so it’s going to be a very busy week on the web for CDNs.

For all the talk of paid interconnects being such a bad idea, or causing great harm to the Internet, none of what is going to take place this week would be possible if these paid connections between CDNs and ISPs were not in place. So complain all you want, but it’s why the Internet works the way it does and why hopefully, all will go smoothly this week.

I will be collecting as many traceroutes as I can from multiple regions during all of these events, so if you can do a traceroute from your location, please send it to me at mail@danrayburn.com

Telstra To Acquire OVP Provider Ooyala: 5x Revenue Valuation, Will Remain Independent

Online video platform provider Ooyala announced that Australia based telco provider Telstra will acquire the company, which will become a subsidiary of Telstra, and will operate as an independent business under the leadership of its existing management team. Telstra, which already owned 23% of Ooyala via a previous $61M investment, is paying $270M to have a 98% ownership in the company. In the past seven years, Ooyala had raised $122M in funding from Telstra and other investors and the company is projected to do $65M in revenue this year, which puts Ooyala’s valuation at just above 5x projected 2014 revenue. That’s a nice multiple considering that Ooyala’s largest competitor in the space, Brightcove, has a market cap of just under $200M, but is expected to do around $122M this year, or double Ooyala’s revenue.

This is a smart deal for Ooyala and the right time to do it as the company has been experiencing growing pains with their business that has negatively impacted some of their customers. Any small company can only manage growth so well with the resources they have and Ooyala simply needs more resources from a larger company to continue to grow. Now with Telstra’s backing, Ooyala should be able to scale their business much faster and avoid some of the headaches they have experienced, hopefully enabling them to provide a much more consistent quality service to every customer. As Ooyala’s CEO mentioned in a letter to employees, scale is the key to their business, and that’s what Telstra gives them. Ooyala couldn’t get there by simply taking more money; they needed a partner and backing of a much larger company that can take their cloud offering and integrate it into a much larger platform.

Vendors that want to compete in the MSO space have to have deep pockets and spend a lot of money on people, infrastructure, R&D and all the accompanying costs that go along with a cloud based service. To put it in perspective, OVP providers Ooyala, Brightcove and Kaltura have raised more than $400M in funding combined, but are on track to just over half that in 2014 revenue. Their business models require a lot of cash and more importantly, scale. To win in this market, they have to build a much larger platform, like what Cisco did when it acquired Inlet Technologies, Extend Media and BNI Video to create their Videoscape TV everywhere solution. When the major players in any market are companies like Cisco, which to date has invested half a billion dollars into their TV everywhere platform, the writing is on the wall for smaller companies to become part of a larger ecosystem.

While I believe this deal is good for Ooyala and their customers, as an industry, we have to set the proper expectations on the size and growth of the different markets that vendors sell into. Companies are always so quick to throw out huge numbers on how big a market will grow to in a few years, but they are always wrong. Brightcove made this mistake when in their S-1 filing they said the “total potential market opportunity was approximately $2.3 billion in 2011, growing to approximately $5.8 billion in 2015.” This set wrong expectations with not just those on Wall Street, but also with those who track the industry. In an open letter to Ooyala’s employees, their CEO said, “The market for the technologies and services we provide is will be worth tens of billions in the next few years.” Those are not realistic market sizing numbers.

If you combine the revenue of all the companies in this space that are selling cloud based services similar to Ooyala, it is somewhere in the ballpark of $500M. And that number includes a large percentage of revenue from reselling bandwidth, not platform license fees. [See: OVPs Still Getting Too Much Of Their Revenue From The Re-Sale Of Bandwidth] So even if the industry as a whole grows 30% a year, for the next three years, that’s combined revenue of about $1.3B globally. That’s a long way away from Brightcove’s “$5.8B” number or Ooyala’s “tens of billions” estimate.

While Ooyala still has to execute properly in the market, Telstra’s backing gives Ooyala everything they need to scale their business to a level that most other competitors won’t be able to get to. They can hire more employees, spend more on R&D, have a true global presence, create very large and meaningful partnerships and drive faster adoption for their services. I would argue they have more work to do now than before the acquisition, but with the new backing and new resources, Ooyala should be able to grow and scale their business at a level they simply would not have accomplished on their own.

Here’s The Program For Streaming Media West Show: Now Placing Speakers

Screen Shot 2014-08-06 at 4.49.26 PMThe advance program for the 2014 Streaming Media West show (#smwest) taking place Nov. 18-19, at the Hyatt Regency Huntington Beach Resort & Spa in Huntington Beach, CA is now complete.[view the entire agenda here] Speaker placement has officially started and our first batch of speakers from Netflix, FOX Networks, DIRECTV, NY Times and YouTube have already been confirmed. Speaking spots will go fast, so if you are interested in being considered for one of the speaking spots, or placing someone from a company on a session, please contact me. As always, vendors who bring customers will always get picked before vendors that don’t bring customers.

If you have any questions at any time about the program, what a session will cover, how the selection process works, or need help choosing the right session, you can call me (917-523-4562) or send email. However I find it is much easier to discuss it on the phone rather than email and many times, I can give you an answer on the spot. So please feel free to call at any time.

You can view the entire agenda here, and below is a list of all the session titles:

  • How To: Selecting The Right Video Management Technology
  • On-Premises Or In The Cloud: Choosing The Best Live Video Workflow
  • Best Practices for Webcasting Sporting Events
  • Netflix’s Video Workflow: Transcoding, Codecs, and 4K Streaming
  • How YouTube Lost at Becoming The Next TV Network
  • Designing The Right OTT Workflow For Premium Content Owners
  • How To: Building A Chromecast Application
  • Streaming Live Events When There Is No Room For Failure
  • The Business Of Building An Enterprise Online Video Network
  • How To: Multi-Cam Shooting, Editing and Delivery On An iPad
  • TV Everywhere Trends And What It Means For The Future Of Television
  • Using YouTube For Original Content Distribution
  • How To: H.265 vs. H.264: Choosing The Best Options
  • Content Management Strategies for Enterprise Content Platforms
  • Measuring The ROI On An HEVC Deployment
  • Best Practices for Search & Discovery in a Connected World
  • How To: Producing and Distributing HEVC
  • Replacing Flash: Adaptive Streaming and DRM in HTML5
  • Streaming Deployment Architectures in Higher Education
  • How To: Building Enhanced Video With JW Player
  • The Risks and Rewards of Employee-Generated Video
  • The Business of Delivering Multi-Platform Content
  • How To: Building a Streaming HTML5 Video Player
  • Using Media Optimization To Improve Streaming Performance
  • Building Streaming Apps: Choosing The Right Devices & Platforms
  • How To: Changing Viewer Engagement with Second Screen Experiences
  • Building an Immersive Online Television Experience