Inside Apple’s Live Event Stream Failure, And Why It Happened: It Wasn’t A Capacity Issue

Apple’s live stream of the unveiling of the iPhone 6 and Watch was a disaster today right from the start, with many users like myself having problems trying to watch the event. While at first I assumed it must be a capacity issue pertaining to Akamai, a deeper look at the code on Apple’s page and some other elements from the event shows that decisions made by Apple pertaining to their website, and problems with how they set up storage on Amazon’s S3 service, contributed the biggest problems to the event.

Unlike the last live stream Apple did, this time around Apple decided to add some Javascript JSON (JavaScript Object Notation) code to the page which added an interactive element on the bottom showing tweets about the event. As a result, this was causing the page to make refresh calls every few milliseconds. By Apple making the decision to add the JSON JavaScript code, it made the website un-cachable. By contrast, Apple usually has Akamai caching the page for their live events but this time around there would have been no way for Akamai to have done that, which causes a huge impact on the performance when it comes to loading the page and the stream. And since Apple embeds their video directly in the web page, any performance problems in the page also impacts the video. Akamai didn’t return my call asking for more details, but looking at the code shows there was no way Akamai could have cached it. This is also one of the reasons why when I tried to load the Apple live event page on my iPad, it would make Safari quit. That’s a problem with the code on the page, not with the video.

Because of all the refresh calls from the JSON-related JavaScript code, it looks like it artificially forced the player to degrade the quality of the video, dropping it down to a lower bitrate, because it thought there were more requests for the stream than there was. As for the foreign language translation that we heard for the first 27 minutes of the event, that’s all on Apple as they do the encoding themselves for their events, from the location the event is at. Clearly someone on Apple’s side didn’t have the encoder set up right and their primary and backup streams were also way out of sync. So whatever Apple sent to Akamai’s CDN is what got delivered and in this case, the video was overlaid with a foreign language track. I also saw at least one instance where I believe that Apple’s encoder(s) were rebooted after the event had already started which probably also contributed to the “could not load movie” and “you don’t have permission to access” error messages.

Looking at the metadata from the event page, you could see that Apple was hosting content from the interactive element on the event page on Amazon’s S3 cloud storage service. From what I can tell, it looks like Apple set up the content in a single bucket on S3 with little to no cache hit ratio, with poor bucket configuration. Amazon didn’t reply to my request for more info, but it’s clear that Apple didn’t set up their S3 storage correctly, which caused huge performance issues when all the requests hit Amazon’s network in a single location.

As for Akamai’s involvement in the event, they were the only CDN Apple used. Traceroutes from all over the planet (thanks to all who sent them in to me) showed that Apple relied solely on Akamai for the delivery. Without Akamai being able to cache Apple’s webpage, the performance to the videos took a huge hit. If Akamai can’t cache the website at the edge, then all requests have to go back to a central location, which defeats the whole purpose of using Akamai or any other CDNs to begin with. All CDNs architecture is based on being able to cache content, which in this case, Akamai clearly was not able to do. The below chart from third-party web performance provider Cedexis shows Akamai’s availability dropping to 98.5% in Eastern Europe during the event, which isn’t surprising if no caching is being used.

akamaiThe bottom line with this event is that the encoding, translation, JavaScript code, the video player, the call to S3 single storage location and the millisecond refreshes all didn’t work properly together and was the root cause of Apple’s failed attempt to make the live stream work without any problems. So while it would be easy to say it was a CDN capacity issue, which was my initial thought considering how many events are taking place today and this week, it does not appear that a lack of capacity played any part in the event not working properly. Apple simply didn’t provision and plan for the event properly.

Updated Thursday Sept. 9th: From talking to transit providers & looking at DeepField data, Apple’s live video stream did 6-8Tbps at peak. World Cup peak on Akamai was 6.8Tbps. So the idea that this was a capacity issue isn’t accurate and the event didn’t generate some of the numbers I see people saying, like “hundreds of millions” watching the stream.

Updated Thursday Sept. 9th: While some in the comments section want to argue with me that problems with the webpage didn’t impact the video, here is another post from someone who explains, in much better detail than me, many of the problems Apple had with their website, that contributed to the live stream issues. See: Learning from Apple’s livestream perf fiasco

  • CraigM

    The encoding and audio track mixing was broken from T-15 or more when the auditorium feed went live. An obvious auditorium audio feed was overlaid by a second audio of pop tracks.

    Interestingly, when it immediately glitched at the top of the hour and lost sync when it recovered we picked up the translation track and a number of efforts seem to have been made to drop the volume as they seemed unable to drop the spurious track.

    The trouble was that each stream restart brought the audio levels back up, so mist of the fixes were rolled back each time.

    In short, quite the least professional stream I’ve seen from Apple in many years of watching … can’t believe heads won’t roll …

    • MIke

      heads will have to roll. Every year, every stream is a disaster, but this year was a train wreck.

      • JLang

        Really? The last two streams had been almost impeccable for me.

    • Apple rolls heads like it does everything else: Shrouded in secrecy.

  • FeisalK

    1. it’s JSON (not JSCON)

    2. You can cache pages with javascript code – it only runs in your browser and doesn’t change the pages on the server side.

    3. calls from the JSON code do not refresh the page; they just change the loaded document in designated areas; unless they are accessing the streaming server, they shouldn’t affect the stream (in this case, just twitter feeds?)

    • Scott Lindsey

      ^^^ This. Some parts of the analysis make sense, but “JSON made the page uncacheable” is clearly wrong.

  • itsgene

    But this doesn’t address why the feed on AppleTV was a complete disaster as well, jumping back to the beginning over and over and repeating segments – when it loaded at all.

    And I’m pretty certain the website didn’t cause the Japanese translation to play over it.

    • I agree. The AppleTV feed was even less reliable than the one on their website.

    • Net Luddite

      Yes. This was true. Apple TV with an enternet connection was worse than watching on a Macbook via wireless, on the same network.

    • No one said it did. THe Chinese was audio intended for the Chinese audience stream – operator lmixing error, that’s all.

  • Homey the Clown

    Total mess of the stream. Your analysis does make sense. When it did fail and reverted back to the beginning I had to pause the stream. Once there I was able to select Live event to get the correct video back. Somehow toward the end they were able to solve the issues.

  • Javier

    Dan’s right. This was a problem by Apple, not the CDN. And having worked with Apple on events like this in the past, at a former employer, I’m sure as usual, Apple didn’t even notify Akamai or Amazon what their real plans were. So while some might suggest that Akamai should have made this all work before hand, you can’t advise a client who doesn’t want the help and won’t even disclose what their setup is going to be for the webcast.

    Case in point, why encode on-site? Put the video on a sat and downlink and encode it at a facility with redundancy. Encoding and ingesting a stream via the Internet, for such a big event like this, leaves a lot of things that can and do go wrong.

    • EJ

      You’re really doing a lot of guessing to think they didn’t even talk to Akamai before this event. And there is a lot of supposition in this article too. E.g. Tweet stream breaking video? Why guess that?

      • Because that issue happens on and Watershed ALL THE TIME.

        • JLang

          None of this explains the issue that even AppleTV was affected.

  • Joel Webber

    I didn’t attempt to watch the stream, so I didn’t have a chance to analyze this myself, but there are several points being made here that sound odd to me:

    1. With the tweet stream, I presume we’re talking about Javascript in the page to render the tweet stream, not JSON (subtle difference to be sure, but relevant). My understanding from listening to others complain about the stream was that the video was having trouble, not the rest of the site. While it’s not entirely impossible that requesting a metric ton of twitter cards and their attendant images could interfere with client bandwidth so badly that there wasn’t enough bandwidth (or HTTP connections) available for the video, that sounds implausible at best.

    2. As pointed out in another comment, scripts embedded in a page have zero impact on caching, or the use of a CDN. Dynamic endpoints — e.g., dynamically rendered content, or API calls — obviously can’t be distributed to a CDN, but static pages and scripts that reference these endpoints can. In fact, it’s incredibly common to do so.

    3. While bad S3 bucket configuration could account for slowness of resources hosted on S3, I don’t see how this relates to video performance. Unless for some incredibly bizarre reason Apple attempted to stream live video by breaking it into chunks and uploading them to S3 during the event. This seems about as likely as serving by carrier pigeon.

    AFAICT from looking at the (not live) video being served on the site now, the video’s being served by Akamai ( I don’t know much about Akamai’s streaming products, but it’s highly likely any issues with Apple’s implementation of the site surrounding the video had much of an influence on the streaming itself. If I had to guess, I suspect something like Javier alluded to, where they either failed to notify Akamai of the impending flood of traffic, or suffered some other operational issue in the streaming setup.

  • Jack Carter

    Would someone explain to Mr. Rayburn the difference between “setup” and “set up” . He uses “setup” about 15 times when he means “set up”. .

    • John Willkie

      this comment is absurd. If there is a semantic difference between “setup” and “set up”, just what is a “set down?” And, what is the “set?”

      • Superalias

        Jack Carter’s comment is spot-on. The verb is “set up”. (Unless you’re in the habit of “setupping” computers and using them once they’ve been “setupped”. Is that how you do things?)

        A writer who uses “setup” as a verb (repeatedly; it’s not a typo) hints at an attitude that’s not suited to good reporting: “I don’t care about getting things right. I’m just here to fill a page.” Which, as many commenters have noted, it what the technical content of the page suggests as well.

        All that aside, a corrected (and edited) version of the article from Mr. Rayburn would be welcome. He picked a good topic; a lot of us are genuinely curious as to why the stream went so haywire.

        • John Willkie

          Ah but you jumped the shark. A “setup” is not a verb. “Setting up” is an adverb. If it’s clear to you that every reference to “setup” was a verbization, I would mention that I knew what he was talking about, and I suspect you did as well
          “Consistency is the hobgoblin of little minds.” Say what you will about Dan, but he isn’t a “little mind.”

          • Superalias

            “Ah but you jumped the shark.”

            ?? Do you know what that means?

            “A “setup” is not a verb.”

            Correct. “A setup” is a noun. “Set up” is a verb. It should take all of several seconds for the author to learn that.

            “”Setting up” is an adverb.”

            No. It’s a verb (either a present participle or gerund, depending on function).

            “Say what you will about Dan, but he isn’t a “little mind.””

            Glad to hear it! As a smart, professional writer, then, he won’t quote trite phrases and complain that good writing is for “little minds”. Instead, he’ll thankfully accept feedback, spend a moment to take in a correction, and welcome the improvement in his writing.

          • P. A.

            @John Willkie, it seems further clarification is in order; perhaps I can simplify with  synonyms:

            “To set up” means “to configure”, both verbs, both in their infinitives; conjugate as needed. “Setup” means “configuration”, both nouns. 

            Substitute these and the problem becomes clear: 

            “…And problems with how they [configuration] storage on Amazon” is obviously incorrect, while “…and problems with how they [configured] storage on Amazon” is the correct usage.  Since the issue involves a word and phrase that share almost identical spellings, I suppose this could seem pedantic, but had Dan used forms of “configure”, the problem would be so plain we wouldn’t be having his discussion. Your “hobgoblin” remark belittles Superalias’s objection to badly edited or just plain lazy writing, which is hardly unique to Dan, but is distracting and lessens his credibility. 

            Finally,  to “jump the shark” describes the moment when a TV show’s storyline quality begins to decline unrecoverably. Your use here is confusing at best, and seems misplaced. 

          • John Willkie

            So many trolls, so little time.
            Oxford English Dictionary:
            Setup (noun) [not news to me] usually in singular; informal [note INFORMAL] yet you guyz persist in attaching formal rules. [set up] is a verb, but I didn’t see Dan using setup as a verb, nor set up as a noun. I note you obscure the examples.]
            Going on with the OED (online)
            “1.1. An organization or arrangement;
            1.2. A set of equipment needed for a particular activity or purpose;
            1.3. (in a ball game) a pass or play intended to provide an opportunity for another player to score”
            Did you note that ALL three apply to Dan’s usage? Did you notice that the latter can be verbizized by prepending an “a?”
            Did you notice that not a single one of those definitions mentions — or even alludes to “configuration?”
            No? perhaps that is because you are in the realm of the last OED definition:
            “2. A scheme or trick intended to incriminate or deceive someone.”
            “2.1 chiefly North American a contest with a prearranged outcome.”
            Just to make sure, I checked all the synonyms. Not a word about — or allusion to — configuration.
            Yet, you lecture me about “jumping the shark.” Obviously, you and others are 1) humor-impaired and 2) continually wrong.
            Don’t bring a knife to a gun fight. Or, perhaps your ramblings are a higher authority than is the Oxford English Dictionary?
            “So many fools, so little time.”

          • Jack Carter

            Another grammar lesson for you: “Retard” can be both a noun and a verb, depending on which syllable is accented. When I read your comments the noun version is what I think of.

          • John Willkie

            see-troll! (also a noun and a verb)

      • Jack Carter

        This has nothing to do with semantics, and everything to do with literacy. My comment may be absurd to you, but to literate people, and especially people who publish on websites, “setup” is a noun, and “set up” is a verb. No one would ever understand the statements “That’s a great equipment setup” and “Set up the equipment” to mean anywhere near the same thing because “setup” and “set up” have different meanings..

        • John Willkie

          literary? Yes, but only if YOU DIDN’T UNDERSTAND what Dan was saying. I get paid from time to time (including in some time back) to write. I also write and read technical manuals ALL THE TIME. There is always “Setup” (mounting, powering up and making all the physical connections) and there is always “Configuration” which is arranging the device logically, often (don’t pronounce the ‘t’) through a graphical user interface.
          I note you don’t want to touch the Oxford English Dictionary definition of setup. Frankly, your ill-informed opinion is no match for the OED.

  • Tobias Nettie

    I hit refresh a whole bunch, then it worked.

  • Kiriki Delany

    It really makes you think about HLS and the engineering thought going into next generation streaming architectures. The stream jumped back and forth in time, it seemed like chunk synch issues. Dan brings up an interesting point about defeating the caching and overloading the origin. Apple would still have benefited from the Akamai network reach though. I suppose this is a learning experience.

  • Ryszard2

    The web stream, after about half an hour, started to work flawlessly. Unless you claim that Apple completely rewrote the web page code in that time, I don’t see how your theory could possibly be correct. And as others here have noted, the AppleTV feed also had issues, although the faulty page coding you blame was not part of it.

    Something went wrong, but it was something more than what you claim.

    • Adam Thurmond

      I saw the familiar rainbow bar screen with some words about the event and the TV truck numerous times when trying to get the video load and reload. The kind of headers you sometimes see by mistake before something is broadcast. Between that and the foreign audio track I’m inclined to think the onus is more on the video/broadcast production team at the scene.

  • arty

    But…I still don’t understand the periodic appearance of the Bars with CG when the live TV feed was interrupted….all the way up to 10:39am. Does that mean that Apple knew what was going on and inserted the Pattern?
    Also, at the same time that this was happening on a Safari based Mac, the ipad stream was showing black, but the tweets were still advancing.

  • Jav

    Quick question then. If the problem was with the code on the webpage, how come this happened on every platform? Meaning, even on the Apple TV.

    • BC2009

      Because the author does not know what he is talking about.

      • Because Apple Tv was getting the embeded video from the webpage, which is the point of the article.

        • The Apple TV isn’t even mentioned in the article?

        • Daniel Chatfield

          No it wasn’t.

        • Scott Lindsey

          Not the case at all. It is the video stream itself that was constantly dying.

  • One major reason to use client side javascript code that dynamically loads JSON encoded data is that it *IS* cacheable unlike a page generated on the server.

  • BC2009

    I would not give the author any credibility here. He speaks as if he understands dynamic web applications, but fails to understand the difference between JSON, JavaScript and AJAX.

    If the dynamic content was hosted on S3 and the video was not then these HTTP requests would have gone in different directions and not interfered (unless there was a proxy server to avoid cross-site scripting errors, but even then that would sit in front of the video stream server and so that server would never think there was added traffic from AJAX calls for a JSON feed of updates)

    Also, the statements on cacheing seem way off. JavaScript can be cached as can any dynamic web page. These web pages are not changing on the server – they are changing at the client. The client makes background AJAX requests for more data periodically and then JavaScript code is called to update the page content (DOM) on the browser client – not on the server.

    I can’t see how the author can rule out heavy viewership as the cause and then pin it on an AJAX call that maybe ram every 3 minutes on the client to fetch page updates. The added traffic would be nominal since those AJAX calls would be quickly processed as compared to the fire Hose of data feeding the live video.

  • Philippe Rambourg

    Dear All,
    From my understanding, this first issue is clearly at the encoder side (before cdn) as all devices were affected.

    The second issue, what people do in that case ? They massively refresh their browser until it works….

    • Rusty Hodge

      Or it could be an issue between the HLS encoder and the main HTTP server feeding the caches. But refreshing the browser is only going to pull a m3u8 index file from the HLS CDN initially, and that’s a lightweight file.

      It really seems like a cache corruption issue to me, with parts of the CDN serving up stale m3u8 index files or m3u8 files that were freshened before the media element files were available from the CDN server being accessed.

  • SamRobinson

    Did the author of this article see the massive web attacks via IPViking on San Francisco during the webcast. Many came from Brazil and South Korea at the beginning of the live webcast? They even knocked the homepage offline.

    Brazil attacks were only 500-600 in one minute, in the two hours I monitored, and all focused on San Francisco. Brazil never launched another attack until a few hours until after the live stream.

    It looked like losing missile command on the Atari 2600.

    The dual language was Apple’s fault. The outages were web attacks.

    If I had a tinfoil hat….

    • Rusty Hodge

      IPViking is a great marketing tool. Hats off to Norse for getting so many people to get so freaked out about “cyber attacks”.

      • John Willkie

        Even better, when the DDOS was caused here by bad Apple code, and everything focusing on a single Akamai server. Let us also not forget the Elemental (software — another black eye) encoders that were rebooted once, twice or — during the event. Hardware encoders reboot with at best loss of 6-8 frames. Software encoders reboots take seconds to minutes to resume encoding.

  • Design by Adrian

    JSON isn’t code, it’s a data type – a set of keys and values. JavaScript is code, and downloads JSON data, and uses said data on the page. Usually of intervals of hundreds of milliseconds, not a few. The page is typically never refreshed. JSON rarely contains image, thus is incredibly light. Half of your theory is clearly wrong.

    • Juno

      JSON and the Javascript Object type are sufficiently interchangeable in this context for your argument to be a purely semantic one.

      • Design by Adrian

        My argument was that JSON is text, very small, doesn’t get polled every few milliseconds, and the page isn’t refreshed – meaning that JSON loading hardly affects the streaming video.

      • r

        They are interchangeable? Not really. And. it shows that the writer has no clue about what he’s talking about. The page may not have been cached because of that but that should not have any impact on the caching of the streams

  • freediverx

    How does this explain the same issues occurring on Apple TV, where there were no Twitter feed updates?

  • Jerrod

    This is classic click-bait and we all fell for it. There is nothing here but baseless speculation and a fundamental lack of understanding of how “caching” works on the web. Please stop sharing this article.

  • howlindog

    One of Apple’s objectives for the live event stream was to prevent anyone who didn’t own an Apple product from viewing it. Apple was more successful than intended in accomplishing this goal, ultimately excluding everyone. The issue here is choosing goals. Don’t fire anyone, just do better. It will be revealing to see what, if anything, Apple says to all its fans.

    • Peter J

      Since when do you need an Apple product to view an MP4 stream on What nonsense.

      • John Willkie

        this is prime fanboisim. Apparently, this uninformed position ignores the folks with VLC who had problems no worse than Apple viewers, on PCs running windoze.

  • g

    It looks like an expert view on the topic but from the details (ie. JSON code, caching , asuming wrong s3 configuration details ) this appears like amateur work – trying to blame someone without any solid evidence

  • Juan Pablo Solano

    I just cannot believe apple engineers don’t know async/ajax javascript. Really weird explanation.

  • Costa K

    The stream worked fine on my iPad Air. It’s just that languages other than English were coming through.

    Anyone else experience this?

    • Scott Lindsey

      On my iPad I was able to watch the fifteen minutes before the start, until suddenly I got the bars. Then it was a constantly dying stream, time skips, “authorization denied” errors on reload, etc. I didn’t stick around past 45 minutes in.

  • CARose

    How does your missive pertain to watching the feed via AppleTV? Many of the same issues.

  • I played the stream via Quicktime using the mov stream, it had looping issues from the stream itself and audio issues sometimes when the video was playing. So the translation, JavaScript code, the video player, the call to S3 single storage location and the millisecond refreshes I don’t believe were at all an issue here.

    • Yeah, I also used a direct link ( in VLC and had many of the same issues.

      • On the website I did notice about 6 forbidden ajax requests returning every second, the website itself was definitely messed up for this event.

        • Yeah, but I think OP missed the fact that the stream was also corrupted on other direct streaming sources. Whatever the issues, they appeared to stabilize about half-way through the event.

          • It kept going back 3 minutes when U2 were on and in various places, did you get that with VLC?

          • I think I had a couple time-travel glitches towards the end, yeah, but they quickly resolved themselves and were nothing compared to the stream-drops and timeouts that I was getting earlier in the event. The entire first half of the phone section was impossible. Apple Pay was pretty rough. We got perfect coverage for the Watch, and then just minor glitches at the end.

          • I’ll have to watch the first half again, thankfully they kept the recording.

          • Apple also maintains an iTunes Podcast with ALL their recent events… going back to the original iPhone I believe. Search for “Apple Events”.

  • hcoverlambda

    You’re wrong, it was clearly a buffer overrun in the async semaphore API node as a result of RJ-45 race conditions in the polymorphic monad stack.

    • istumbler

      Can you enhance that? It’s still blurry…

    • TheMonster

      I don’t know what I like more – your comment or your avatar. Now, let’s try a test… Blucher!

    • BadMothafucker

      yup, ∆∆∆ this guy knows what he’s talking about.

  • dougwo

    That doesn’t explain the Mandarin over-dub though. 🙂

  • Nathan Hubbard

    This is speculative click-bait. Do not share.

    • narg

      NO! It has to be true! It was on the Internet! 🙂

  • Pablo Silva

    I’m surprised people are not tearing this apart.

    I’m sure they had problems and messed things up on their end, but this ‘article’ is chock full of misconceptions.

    – JSON is a data format, and you’re getting it confused with JavaScript.
    – Yes, you can cache only portions of a page.
    – Things like a Twitter stream are usually loaded asynchronously, so they won’t affect the entire page load.

    – Why would page load times degrade the video stream? Video players make that decisions based on network performance to deliver adaptive bitrates when it’s available (and depending on the player even things like OS, processor type, plug-in version, etc.).
    – I’m not even going to get into how Akamai could do more than just caching whole pages / websites…

  • rob

    There is no such a thing as JSON “code”. Do you really know what you are talking about?

    • istumbler

      Strictly speaking all JSON *is* Javscript “code”

      • Evan

        No, strictly speaking, all JSON is a language-agnostic STRING. If you’re going to be pedantic, at least be right.

        • istumbler

          Uh, while it’s not a turning complete subset of the language, every valid JSON string is also valid Javascript code.

          Nearly all software is written in STRINGs (not sure why the caps were needed there) and the question of weather it’s code or data is really up to the interpreter.

          For the purposes of this article, OP is clearly a little clueless, but the distinction between declarative languages and turing complete ones isn’t always obvious. HTML is typically referred to as ‘code’ despite the fact that it’s just a STRING, and I’ve seen functional languages used for system configuration data (albeit, to my horror).

          • John Willkie

            odd, then, that I’m a programmer, and while some of my code is strings, all of it is converted to byte values by the encoder. Are all bytes strings to? Then how do I represent an unsigned 16 bit integer as a STRING?

  • Incredulous

    I could make a better argument for the current phase of the moon causing the streaming issues.

  • If Apple doesn’t know how to stream video, why not just do the live stream on YouTube?

    • narg

      Anywhere would have been better overall. Multiple providers would have been ever better. They just wanted to force folks to only watch it on Apple products. That was their failure.

  • Hum, I think all developers like me will agree : you obviously don’t know what you’re talking about 🙂

  • Peter

    So why was the Apple TV stream borked as well ??

    • narg

      Really? You need to learn what Apple TV is…. It’s just another stream of data, and the problem caused ALL streams of data to get borked.

      • Scott Lindsey

        No, Peter is asking a good question. Clearly the author has not thought this through.

        • see my post above, but the gist:

          Ie. the people watching via the website
          were DDOS’ing the stream provider because of bad javascript. You experienced glitchy video via apple tv as a result.

          • Scott Lindsey

            I think OP is just misguided. Pages that are loading some JSON from an s3 bucket are not going to do anything besides overloading that bucket. A stream coming from a second location would not be impacted.

            We’re looking at either capacity problems and/or hardware failures, combined with a series of screw-ups such as the mandarin audio.

          • Luca Candela

            I think YOU don’t understand how CDNs work.

          • Scott Lindsey

            Care to elaborate?

  • JZ

    How do you explain the fact that the video was equally glitchy when viewed using VLC?

    • If I understand the post correctly, the uncacheable requests were making large volumes of bogus reload requests to the cdn hosting the stream. This javascript, while degrading the experience for people watching the web stream, also cascaded and degraded the video stream itself, just by the sheer number of reloads the browsers are sending to the video stream provider.

      Ie. the people watching via the website were DDOS’ing the stream provider because of bad javascript. You experienced glitchy video via VLC as a result.

      • Scott Lindsey

        The post is wrong. Javascript polling an S3 bucket would not have any impact on a stream coming from Akamai.

    • That’s what I want to know. When I started having problems, I wanted to eliminate as many variables as possible, so I did an Inspect Element on the video, found the URL, and opened the address in Safari by itself. When that failed after a few minutes, I loaded it up in VLC instead. Another failure.

      If it’s true that Apple basically DDoS’d themselves by constructing things as they did, then that would explain the issue, but I’m not convinced that was the problem. They seemed to mostly get it stabilized (at least for me) after about 30 minutes, and by then I was back to just trying Apple’s live event page again and refreshing it every time it stalled out. I had very few issues after that point.

  • While I get what you are saying with streaming via, this shouldn’t have interfered that much with the AppleTV stream? Which was totally down too.

    Even people streaming directly via the network (with for example VLC, using the AppleTv stream) had exactly the same issues – unreliable streaming and Chinese translators.

  • hkim823

    Let’s get past the semantic about “JSON code” and just say that it’s plausible that was JS that was part of the interactive aspect of the site poorly designed that caused lots of requests per second (again because it was poorly designed) to whatever backend that wasn’t scaled properly (which probably had a different set of people working on, and had little to no interaction with developers writing the code itself) to handle the expected load, and that it’s plausible that Akamai wasn’t able to cache it and that they didn’t use a CDN in front of S3 aka CloudFront and that’s a poor choice. That’s still multiple layers of fail and a systemic failure in human processes implementing technology.

    This is all conjecture and hearsay. No one really knows what really happened, we can only glean what we information we can get. But in the end my gut feeling tells me that it’s not the lack of talent that’s the problem, it’s the lack of communication between said talent and a culture that’s not able to speak up about issues before it’s too late.

    • Matt W

      The most plausible part of the whole thing was they were using the S3 URLs instead of the CDN URLs. That is in fact a likely scenario. The page being uncacheable is irrelevant. Live video is not cached anywhere but the CDN edge. The page elements were delivered from different locations.

      If he is suggesting the JS was forcing a full page refresh and completely restarting the video once a second for all the users and thus overloading the servers, he is wrong. Why? Because that didn’t happen and no one claimed it did happen. If that was happening, no one watching on the web would have seen anything ever.

  • Salvatore Johnson

    I am sorry but all of you guys arguing on this is wrong.
    Who gives a crap if this guy didn’t know what “JSON” code really is.
    What the writer of the story did was be the first to really try and explain the problem.
    What I find very strange over all of this, Apple is such a great company and what happened yesterday in the live feed for the first half of the show was unacceptable. I also think that at least one and probably several folks responsible for this got fired either yesterday or today.

    But don’t climb all over the writer, unless you know for a fact what the true story is.
    Apple screwed up big time and they shouldn’t of, that is the whole point.

    • Scott Lindsey

      Oh, it’s ok for the writer to speculate and analyse but it’s not ok for readers to do the same?

    • But doesn’t it seem kind of odd to have someone write an article about a technical glitch when they don’t understand the technical side of it? The fact that the data was in JSON format has nothing to do with anything at all. In fact, a lot of what he talked about doesn’t have much to do with the problem at all. The Web is full of top level pros and time is short. I can’t understand why this article was written by someone who doesn’t understand the technical side of things. People should stick to what they know.

    • Daniel Chatfield

      >try and explain the problem.
      He calls himself an expert and his “analysis” is so completely wrong it is almost amusing. A junior web dev would be able to pick this article apart easily. It is nonsensical rubbish.

    • This comes after a period where Apple has had about four video events covered flawlessly — maybe two or three jitters, but that’s it — including the WWDC, which I watched on my Apple TV with no problem. I think it’s likely they went for the sky– video streaming from, from Apple TV, from iPhones and iPads — and it had to be flawless as the most recent attempts. There were larger numbers, they tried it all, live, and they likely tried more elements. I don’t think the next one will be like this. Either a contractor won’t work there again or he’ll fix it.

    • John

      No. You can point out why the author is wrong without knowing the answer. That is valuable. That doesn’t mean you should be an asshole, but pointing out flaws that could lead to a better understanding (or even, gasp, a correct one) is a completely reasonable response to a technically inaccurate theory.

    • Matt W

      Why do they have to know the true story? They (and anyone who has read the comments here) know he was wrong. His explanation does nothing to explain what actually happened. He took the first idea that popped into his head and ran with it without thinking it through. I don’t think he is stupid. I think he just rushed to get a story out and now is reluctant to admit he was wrong.

      There is absolutely no way the explanation in the article (JSON refreshes causing the page from being cached) has anything at all to do with what happened. First of all, it is not even a thing…

  • Dan

    Way before you get to the magic of streaming, the video and audio signals are going through some kind of digital control room which managed to mix the translation audio over the main feed audio. They also managed to put put up color bars several times instead of the live feed. I would not blame all the screw ups on the streaming components. The TV tech crew did mny all on their own.

    • narg

      You really don’t get it. The bars were there because the video company was testing their setup to make sure it wasn’t on their end. Bars are static, so they would feed easily around the massive DOS the java script was causing.

      • Dan

        Sure. And they were testing the main audio by mixing in a translation. Nope.

        You don’t put up bars. You look at your outfeed monitoring which would have shown them that their feed was leaving there OK.

        • The bars got put up because the video literally crashed. Multiple times. Rather than fade to black, you want to see if the channel is there and is capable of carrying a signal, even though the stream just blew up.

          • florean

            The bars were there because you got kicked to the beginning of the stream (it can happen with HTTP Live Streaming). You put up color bars at the beginning when you’re testing your feed and have no content to broadcast. You don’t put them up part way through a broadcast when your CDN is having capacity problems.

          • John Willkie

            if the video literally crashed — You are Guessing — then bars wouldn’t have shown up. Black wouldn’t have shown up. Audio wouldn’t have shown up. Those might have been inserted due to LOV (loss of video) but having a bars/cg/tone or Chinese audio isn’t what a professional would put up due to loss of video. Industry practice is to insert a “Please Stand By” slide. Bars have the opposite effect of “Please Stand By.”

      • florean

        What? You have no idea what you’re talking about. Bars are encoded into video just like everything else in the stream. They were used at the beginning of the stream and some people kept getting kicked back to the beginning. That’s why you saw bars.

      • John Willkie

        just what “video company” “tests” their circuits DURING a live telecast? Hint: none that want future business. Perhaps Apple couldn’t afford a video transport arrangement where you do pre-testing and certification using bars ahead of time, and you use “in-service analysis tools” to test and monitor the streams during the transmission?

  • Salvatore Johnson

    One last thing, I am not a person that understands coding but from watching past keynotes was wondering if Apple screwed up because it was the first time that you didn’t push a button to get the stream. This was embedded into the webpages along with all of the “tweets” or live stuff below it.
    I just figured that they were trying something new (which they were) and didn’t work it out properly and that was the problem.

    But with millions around the world watching it was a total embarrassment towards Apple and I can bet that it won’t ever happen again.

  • jortsu

    This doesn’t explain why the stream was totally sucking on iPad, Apple TV and iPhone too.

    • Salvatore Johnson

      Because nobody really knows the reason but Apple and the company that screwed up.
      Have a feeling that Apple won’t say anything about this and will make sure the problem won’t happen again.

  • Absolutely amazing that a tech company as big (and controlling) as Apple would have made such a boneheaded tech decision. Absolutely stunning – thx for the insight.

    • John Willkie

      no, this is on a par with their bad iPhone “Find My iPhone” security bug and their blaming their customers for others exploiting a bad Apple security setup.

  • Rob

    great explanation of the tech side of the problem. Obviously some one in Apple marketing decided on the coolwow factor of this implementation but didn’t do the full due diligence on the technical side. Gotta watch out for those Marketing Execs…..

    • just a guest

      no, completely bogus explanation. this text has so many flaws it hurts.
      the endpoints are not the same, JSON does not execute and has nothing to do with video, etc. the person who wrote this know the same about this than g w bush knows about anything: zero.

  • Kuji

    That is bogus. Stop writing things. Who are you anyway?

    • Daniel Chatfield

      Supposedly the “voice for the streaming & online video industry”

    • Brandon

      Seriously. This article is pure speculation based on things that don’t even have anything to do with each other, and isn’t “inside” Apple’s live event whatsoever. JSON requests “every few milliseconds” for twitter feeds causing video player stream downgrading? Guy here has no clue about how these different web components are delivered. Clickbait at its finest…

  • Rob

    Xiaomi stole Apple’s live feed for the unveiling of the new Xiaomi phones and watches in China.

  • spaceage

    Poorly written and researched story…JSON is a common data format, not code that executes. Also any JSON-serving endpoint/URL would be distinct from any video serving endpoint/URL and therefore not a cause of the video streaming issues witnessed yesterday. Kinda blows the credibility of this publication out of the water…

  • Craig Jacobs

    JSON code doesn’t execute.

    • howie__feltersnatch

      Yet another case where XML code is superior. Excuse me while I execute some XML.

      • John Willkie

        be sure to flush twice after executing.

  • akersmc

    I don’t agree with this theory. It doesn’t explain why the exact same issues were encountered when watching the stream with and AppleTV where there is no website or javascript involved, just the video stream.

  • Christopher Levy

    How did that prevent the iWatch Video from playing? It was on it’s own HTML page….

  • florean

    Some general points:
    First, don’t claim “Inside” if you have no access to the inside. And
    if you want to provide some analysis without any access to the server
    side of things, you better be looking at some packet dumps. And you
    better be looking at them from multiple locations covering multiple
    zones. Still anecdotal, but at least you have hard data across multiple
    CDN edges.

    Second, everything is cached. The “dynamic” JSON you talk about was going out to tens of millions of people with no user customization; you don’t regenerate the exact same data for millions of people. You cache it and refresh your cache. They aren’t idiots.

    Now some other random points:

    I believe that Apple’s encoder(s) were rebooted after the event had
    already started which probably also contributed to the “could not load
    movie” and “you don’t have permission to access” error messages.

    Your belief is misplaced. You don’t reboot encoders in a live stream. Those are both CDN problems.

    By Apple making the decision to add the JSON code, it made the website un-cachable… Because of all the refresh calls from the JSON code, it looks like it
    artificially forced the player to degrade the quality of the video… Without Akamai being able to cache Apple’s webpage, the performance to the videos took a huge hit.

    Wait, are you saying there was no caching involved? Because if the JSON is “un-cachable” and hitting the main servers, then why would that affect the CDN video? In fact, it would take a huge request burden off the CDN, improving video performance.

    Akamai didn’t return my call asking for more details… From what I can tell, it looks like Apple setup the content in a single
    bucket on S3 with little to no cache hit ratio, with poor bucket
    configuration. Amazon didn’t reply to my request for more info, but it’s
    clear that Apple didn’t setup their S3 storage correctly.

    Really? Akamai and Amazon wouldn’t give you any details about their customer’s configuration? Huh. But go ahead and speculate about it anyway.

    • wayaway

      Just FYI, rebooting encoders on a live stream is extremely common… not desirable, but extremely common!

  • Samuel Caro

    I find it interesting that some people are so quick to say Dan is wrong, yet all of those commenting have provided absolutely no alternative explanation for what happened to make the event fail. Their argument is that javascript objects can be cached, but that does_not_mean that they were. Why hasn’t any shown, with proof, that they were cached?

    No one besides Dan, that I have seen, even noticed that Amazon’s S3 service was involved in the event. He’s got data from the event, traceroutes from all over the world during the event, data from third party provider Cedexis and published on his Twitter feed Akamai traffic info from DeepField, which showed that the traffic wasn’t as big as some might think. If this was a capacity issue, where is the data to prove it? Other Akamai customers on their network would have been impacted at the same time if it was a capacity problem, yet we haven’t heard of any complaints.

    Dan refused to comment when I asked him if he spoke to Apple, Akamai, or Amazon about what happened after the event, but also remember that he was the one to blog about the news that Apple was building their own CDN, he blogged about it when it went live and he’s been right when he’s blogged about details on Apple’s relationships with both Akamai and Level 3. So while he won’t admit his sources, it’s clear he’s got some deep relationships inside Apple. He didn’t write this in a bubble.

    Yes, he got some of the terminology wrong when talking code/data formats but he never said he was some kind of code, development or programming expert. Video is what he focuses on.

    I have yet to see anyone offer an alternative to what went wrong. And this IS the first time Apple has ever added any interactive elements to their live stream page. So if the problem isn’t what Dan outlined, where’s the other explinations of what caused it? It does no one any good to simple say he’s wrong, if you don’t provide your own explination of what happened. What are the alternative issues that could have caused the problems we saw?

    • John

      Pointing out why this analysis is wrong does not require an accurate alternative explanation. In many cases, it’s exponentially easier to identify what did not go wrong than to figure out what did.

      It DOES do good to point out reasons why his theory may be wrong – it may spur him to learn something new and potentially find the actual issue, and it limits the spread of false information.

    • normm

      I tried watching using both the Web site and Apple TV. Both had the same problems. There was obviously no “uncacheable Web page” issue for Apple TV.

    • Matt W

      Everyone who said he was wrong did point out why he was wrong. I can no someone is wrong without knowing the right answer…

  • florean

    The problems were quite easy: their CDN got overwhelmed by a huge volume of traffic.

    • Couldn’t load the page or page resources.
    Simple CDN problem. The server couldn’t keep up with requests.

    • Didn’t get JSON updates
    Simple CDN problem. JSON is a page resource, see above.

    • Weird access denied page.
    Simple CDN problem. The server crashed or was otherwise overwhelmed.

    • Couldn’t load video.
    Simple CDN problem. The client couldn’t load the HTTP Live Streaming playlist file because the server couldn’t keep up with requests.

    • Video died while playing
    Simple CDN problem. The client tried to load the next video file in the playlist over HTTP and the server couldn’t keep up with requests.

    • Video skipped around
    Simple CDN problem. The client has to reload the playlist file during streaming and got a stale version, causing it to load earlier video files.

    • Chinese audio
    That’s on Apple. They either fed the wrong stream to the CDN (wrong input) or assigned the Chinese stream to English clients (wrong config).

    Granted, I wasn’t “Inside Apple’s Live Event Stream”, but sometimes the obvious answer is the correct answer.

  • Simon England

    I noticed that about an hour into the event, Apple, or someone else, had changed the configuration setup on S3 and was no longer trying to serve images from a single bucket. I don’t know anything about the other stuff you are all discussing, but Dan was right about S3 not being setup right.

    • John Willkie

      “set up” right.

      • Joshua


        • Grey

          I wonder if you grasp the irony of calling that fellow out as a pedant. Moreover, it’s not pedantic to correct someone on basic grammar, especially someone who should know better, and — as importantly — should also care. Our language deserves respect.

          • John Willkie

            he’s humor-impaired. Note the earlier setup/set up discussion.

    • Daniel Chatfield

      >no longer trying to serve images from a single bucket.

      you say this like a bucket has some sort of capacity and that using more buckets would help? Buckets are used for organisation and access control they have the full capacity of s3.

      • danrayburn

        You don’t have the “full capacity” of S3 unless you choose that. You have have to assign the region where S3 will store the buckets you create. If you pick only one location, that’s a problem.

        • Daniel Chatfield

          If you pick only one location then there is extra latency but the capacity is not a problem.

  • Sigivald

    Unlike the last live stream Apple did, this time around Apple decided to
    add some JSON (JavaScript Object Notation) code to the page
    which added an interactive element on the bottom showing tweets about
    the event.

    Whether or not this had any performance impact at all, who thought that was helpful or a good idea?

    The last thing anyone should want to see there is Random Or Even Popular Tweets About It.

    Not everything needs to be “social-ed” in realtime.

    • Steven

      Random & popular tweets ARE precisely the things many want to see, despite whether or not you believe it “should” be the case.

      • jimpocket3d

        Many ADD types may want to see “random and popular tweets,” but they mess up the visuals of the video page completely. The video of the stage event is mostly dark, while the background for the wanker tweets is a blinding white. Eyes are strained needlessly. Careless, messy, unnecessary, like the stupid crawls on CNN. Bad decision on Apple’s part.

  • Michael

    Mr. Rayburn has tried to pass himself off as a streaming expert for 15 years. He’s just an opportunist with a huge ego. Move along, nothing to see here.

    • John Willkie

      You have the timing right, you have the attitude right, but c’mon, Dan hasn’t learned anything in 15 years? We’re not friends, but he’s come a long way in that time frame. And, you?

  • florean

    Dan now:
    “It wasn’t a capacity issue.”
    Dan yesterday:
    “Some IP network providers Akamai uses for overflow tell me traffic has
    jumped from 100Gbps to 350Gbps within 10 minutes. Akamai not prepared”

    • danrayburn

      Yes, my initial thought, like many, when the stream first started was a capacity issue. But once I learned some of the details on how the event was configured and looked at some of the data, it’s clear it was not a capacity issue with the CDN.

      • florean

        So Akamai was prepared? And your original sources were mistaken? It’d be nice if you could correct the record regarding what they told you.

        What’s funny is that your earlier tweet was pretty much correct: the problem was with their CDN serving HTTP Live Streaming video. No need to blame how modern web pages work.

        • John Willkie

          Isn’t it safe to say now that Akamai stuff went through a single machine, and that machine would probably have worked had not Apple’s “push” architecture, multiple full image loads, short time to live limits and just bad system architecture put more strain on Akamai’s machine than all parties planned on, paid for and anticipated?
          This is what happens when a company DDOS’s itself.

  • Kenn Fong

    Did anyone else know that Bono’s mic was stone cold at the beginning of his chat with Tim? Someone potted it up midway into the convo.

    • Andy

      Also still had heavy reverb on it from their performance so sounded very weird while they were talking.

  • danrayburn

    There is more to this than some of you think. Ask around, do some research. Look at the vendors who were involved and how they were setup in the mix. No one even noticed Amazon’s S3 service was part of it. What other vendors were used that no one has yet discussed? Why was the stream fixed an hour into the broadcast? That’s wasn’t by chance, changes were made on the fly. Many are making assumptions based on past Apple events, this one was much different.

    This event had a very_complex setup and it was a perfect storm so to speak of many things that went wrong, that had an impact on the stream, even if you watched it on your Apple TV, without going to Apple’s website. I’m not giving out any more details, people can keep digging on their own if they want. From a traffic standpoint, this event wasn’t as big as some have suggested and it wasn’t a capacity issue.

  • Orva

    Two words:

    Lowest bidder.

    • John Willkie

      no “software encoder” is the two words.

  • John Fak

    Baseless speculations.
    The stream had the same problem on Apple TV. It has nothing to do with JSON, Twitter and the Apple website.
    The Chinese female translator voice overlapping in the first 35 minutes was also not a bandwidth or JSON issue.
    There was a major fuckup, but it has nothing to do with Akamai or JSON/Twitter.
    Dan is an idiot.

    • Stephen Perry

      So you didn’t actually READ the article at all then… or you would know that indeed, JSON code caused the cascade that eventually led to low availability for EVERYONE, including AppleTV users. You would ALSO know that the foreign language issue was Apple’s fault at the encoder level…nowhere in this article was this been blamed on JSON or Akamai.


      • brucerb

        I don’t think the article clearly states that the problems on the web site led to the poor stream availability for viewers not accessing through the web site – you have to make a (admittedly small) leap to get that. I had to read it a second time to catch it. It would have helped his case if Dan had added a couple of sentences addressing the AppleTV issues or clarified this in comments. (But yes, I do agree he addressed the foreign language issue clearly).

    • danrayburn

      You “assume” its speculation.

      More details on why Apple’s live stream failed. The website problems, and complex setup, impacted the stream.

      • brucerb

        Note that the article you link does NOT really address the impact on the stream for users not going through the website and in fact closes with
        “P.S. The actual media stream was delivered via a different origin (also via Akamai). Did the above site perf problems affect the streaming? Maybe, hard to say. Perf fail is often additive in bizarre and interesting ways.”
        Note “maybe”. (Not saying you’re wrong about what happened, just that the linked article doesn’t state what you are saying it states about the website problems impacting the stream).

  • Kunal

    this is a good analysis +1 for it .. but with no references to code snippet / .. this will be bit speculative for few .. this might be classic example of Performance Testing ??

  • barcadad

    This doesn’t make sense, since Apple TV had the exact same issue and didn’t have the JSON.

    • halo

      but it the server that the video was being hosted on was getting pinged over and over again for JSON updates then it would have effected the streaming quality of the video.

  • BAzerty

    Please, let the technical stuff to technical people. I mean REAL people who know their stuff.

    You could have written : JSON couldn’t be cached, therefore the dinosaurs disappeared and internet started to roll but I’m hatin’. At least it would make more sense seriously.
    Go to IT school first please…

  • Avatar1337

    The issues must be on the server because I tried the stream in VLC. The same problems…

  • Leon Green

    Really good piece and very timely as I’m about to embark on a new website for which has plans to attempt exactly what Apple tried (albeit with obviously less viewers) of live video with social updates underneath! Paying close attention to the details here…

    • John Willkie

      unless you must have multiple language “bottom line” tweets (or low-quality encoder) I’d advise you to mix in the tweets in the “uncompressed domain” (keeping a “clean feed”) which won’t involve any additional server/stream overhead.

  • I started reading this article assuming you were talking about the feed itself. I was watching from an AppleTV and there were severe problems with the video stream (including Chinese translator audio overlay for a good portion of the beginning). The problems didn’t stop with the changes to the Apple website.

  • Matt

    Does anyone have the code snippet that supposedly caused these requests that invalidated the cache? The way this blog post is written makes me think the author doesn’t understand JSON.

    JSON simply structures data for use by other code; it doesn’t do anything by itself. JSON wouldn’t “make a request” anywhere or for anything. Other Javascript might make a request and send or receive JSON, but the JSON itself isn’t doing anything. Putting “JSON code” on a page wouldn’t make anything happen. It wouldn’t render anything or do anything, maybe just show the raw JSON. There’s code somewhere else that would do something WITH the JSON.

    Blaming JSON for making too many requests just doesn’t make any technical sense.

  • Matt

    Correction: The Sept 9th update with the link to “Learning from Apple’s lifestream perf fiasco” is a good explanation and makes sense. The translation here is nonsensical from a technical standpoint

  • William D

    Even with the two appreciated updates I fail to understand why there was the chinese voiceover

    • John Willkie

      “somebody pushed the wrong button on a routing switcher”

    • FeisalK

      from what I gather, everyone this side of the earth got the Chinese translation automatically. Never mind we could be Indians, or Malaysians, Indonesians or Filipinos, Koreans or Japanese….

  • Richard

    other then the post dan linked to in his update he’s the only person to even disect why the stream failed and has had high level contacts at apple for years. his track record on apple stories speaks for itself. so for those who want to say the javascript had no impact on the video just because you watched the stream in vlc or because dan didn’t use the term right you don’t undestand all of the elements involved in the webcast, how complex the setup was or other factors that took place. i have yet to see one person who wants to argue with dan show any evidence based on source code traceroutes, etc….. that was being cached or that all of the elements in the page had no impact on the quality of the video. where’s the data to back that up?

  • not_dan


    Funny how you can claim that no one else has data points that prove Dan’s wrong, yet Dan doesn’t provide much in the way of evidence either. He seems to talk as if he knows what he is speaking to, without really understanding the elements involved. Friend’s in the right places doesn’t at all mean that you actually know/understand what you are trying to explain. Having a high level person at Apple doesn’t mean much in terms of getting technology information correct, none of us know who these “sources” are and therefore don’t consider the information reliable.


    It was Postano that did the Apple Social Hub.