Google Has A Problem: VP8 Is Not As Good As H.264, On2’s Quality Claims Unfounded

When On2 announced their VP8 codec in September of 2008, they made some pretty big claims comparing the improved quality of VP8 over H.264. Specifically, On2 said that VP8, "delivers over 50% bandwidth savings compared to leading H.264 implementations" and stated that VP8, "surpasses the compression efficiency and performance of every other video format on the market." While the industry was quick to take note of these statements, many were also quick to question On2's claims. Now, based on test results published, it's clear that the quality of VP8 is not as good as On2 said it was and makes one wonder why Google isn't setting forth proper expectations since they now own the VP8 codec.

For more than a year, myself and others asked On2 time and again to provide the technology to the industry so that third-party evaluations comparing VP8 to H.264 could take place. While On2 had been in the codec business for a long time, they also garnered a reputation for making a lot of big performance claims with their other codecs, like VP7, that turned out not to be true. When Google decided to buy On2 last year, most people assumed that whatever On2's claims were about VP8 must be true or else why would Google be interested in the codec? While that would be a reasonable assumption to make, based on new test results, it's clear that VP8 is not better than H.264 at all.

To be fair to Google, at no time that I recall did Google ever say VP8 was 50% better than H.264. That was language that came from On2 and when Google acquired the company, they didn't make any performance claims comparing VP8 to any other codec. But the problem that Google now has with VP8 is that they also didn't refute anything On2 had been saying for the past 18 months. On2 set expectations for what VP8 could do and since Google didn't say otherwise, many are now going to be disappointed when they find out those claims aren't real.

Based on test results from two different codec experts, Jan Ozer (test results) and Jason Garrett-Glaser (test results), they both came to the conclusions that the VP8 codec provides similar quality to H.264, but in most cases, H.264 is still better quality wise than VP8. Both also stated that most won't notice the difference between VP8 and H.264, but that's not what VP8 was suppose to be about. VP8 was touted as the video codec that was suppose to replace H.264 because it could offer better quality at half the bandwidth, something both reviewers said is not possible.

Google did mention yesterday at their I/O
conference that by encoding videos for YouTube with the VP8 codec it would enable Google to save money on their bandwidth costs. But what Google didn't say is how much it would save them or even give any kind of comparison numbers highlighting the differences between H.264 and VP8 when it comes to the number of bits delivered. Considering that's suppose to be one of Google's selling points for VP8 and no details are given anywhere on their website regarding this subject, one has to wonder what Google is referring to.

After reading Jason Garrett-Glaser's long and very detailed review of VP8, it's clear that Google has some major problems that go a lot deeper than the subject of bandwidth savings. Jason found so many major problems with VP8 that he summed up his review by saying that, "VP8 is not ready for prime-time", "the encoder’s interface is lacking in features and buggy" and most importantly, it appears that Google has not improved upon the code they acquired from On2.

Jason says that, "the VP8 software basically is the spec–and with the spec being “final”, any bugs are now set in stone. Such bugs have already been found and Google has rejected fixes." While the specs may be final, in fairness to Google the website does say that the encoder/decoders are just preview releases and one would expect those to be improved upon over time. But if you are Google, why rush VP8 out to the market so fast when clearly it still has a lot of flaws? This is not a short-term play for Google and it's not like they rushed VP8 out the door to make money off the licensing.

Also, keep in mind that VP8 was out in the market for almost 18 months before Google acquired it and in that time, not a single content owner that I saw ever adopted VP8 from On2, even though On2 made a big stink about how many content owners were testing their VP8 codec. Now we know why we never saw any press releases in those 18 months announcing any customers for VP8.

Frankly, the way Google has handled the On2 acquisition and the release of VP8 simply has me puzzled. Google's not a bunch of dumb folks and I don't see why they would rush an inferior product out to the market, make it open-source and then hope it would unseat H.264 all while not getting any input from developers before the initial first release, even if it is beta. If Google permits changes to the spec and allows the developer community to help fix some of the glaring issues with VP8, I could see that approach. But at no time during Google's presentation yesterday did they frame the release of VP8 that way or ask for help from the development community to fix issues that they must know exist.

There are a lot of unanswered questions here and hopefully someone from Google will address these points on their blog since it's only going to be another 12 hours or so before these issues with VP8 come out of the small developer community blogs and into the major media spotlight.

  • Info

    Is this really a surprise? Let’s put it in some perspective: Microsoft had 120 codec engineers working on VC1. It was better then H264 when 264 launched but 264 has surpassed it, maybe 1.5 years ago. Now, a little company with fewer engineers is supposed to have built a codec that is 50% better then H264 which has had an additional 18 months of work?
    You are right to be a skeptic. More importantly, it is time to call into question Google’s real motives, whatever they may be. It is obviously not bandwidth savings.

  • JJ

    Vp8 is not as Good as h.264 simply because until now no One has used and optimized it yet, in while h264 has been optimized to the limit. See this:
    Near HD quality @ 250kbits in the flash player!

  • rs

    “Both also stated that most won’t notice the difference between VP8 and H.264, but that’s not what VP8 was suppose to be about. VP8 was touted as the video codec that was suppose to replace H.264 because it could offer better quality at half the bandwidth, something both reviewers said is not possible.”
    Eh? When was that ever what VP8 was “supposed” to be about? Yes, On2 made claims that put there codec in a favourable light, but all anyone has cared about since Google moved to acquire On2 was whether or not Google would release it on a royalty-free basis.
    A common video codec for the Open Web – *that’s* what VP8 is about. WebM is a big victory for Xiph, for Matroska, for Mozilla, for Opera, for Google, and most importantly of all for everyone on the Web.
    I can understand the value of sexing up your articles to garner more readers, but this criticism is a bit silly.

  • Wilhelm Reuch

    If it is not better quality. And not even the same quality. Why bother?
    Quality may not matter with low-quality youtube type teenager clips.
    But to deliver a proper experience for movies online we need every bit we can get. We need better than h.264 – else there is no use. Just another codec to bloat/complex our systems.
    Maybe open-source/google-fanboys are happy. but they dont produce any content so who cares.

  • guardianaj

    Its not about quality. Its about money. Google doesn’t care about quality (as long as its not horrible and even these reviewers state that while vp8 doesn’t beat 264 like claims, its not far behind) they also don’t care about bandwidth. They do care about cost savings though. And hate patent issues. The cost savings come from having an open. No cost codex. The MPEG consortium. Says they wont charge royalties till 2015. So its like a crack dealer. First few are free then when everyone hooked ur gonna pay. Well most royalties are by play amount. When Google serves a billion views a day with YouTube that’s a crap ton of money they’d have to pay in royalties.

  • Mark Vim

    If they are equivalent but VP8 is royalty-free, isn’t this a big advantage?

  • Relax.
    Dont get hung up he said she said mumbo jumbo 🙂
    Google does not have a problem.
    Google has gifted this technology to the open web.
    And as an Open Project, the community of technologists will evolve it over time.
    It’s all good.

  • Tim F.

    guardianaj, I think you are poorly understanding the situation. MPEG-LA is not charging patent licensing fees currently for noncommercial distributors, but Google is a commercial distributor and is happily paying the licensing fees NOW. These patent fees have caps so the fees are essentially inconsequential to Google.
    MPEG-LA could double, triple, or even quintuple the fees and extend the ceiling ten fold (something they would never do — the patent fees are likely to only go up according to inflation or more likely to go DOWN as patents expire and competition increases (including MPEG-LA’s own future h.265) — the only major threat of a major increase would be a new patent holder coming out of the woodwork showing substantial holdings of a large percent of related patents — and Google would still have a capped, fixed licensing cost, making their licensing cheaper than any other commercial enterprise paying for broad distribution of h.264.
    This cost IS inconsequential. Certainly far less than their current plan to encode all video in 3 different formats, store it, and distribute it.
    What Google does care about is being able to use the web to reach EVERYONE. And Mozilla and Apple currently stand in the way of that.
    Of course, even if the open-source freebie folks benefit from Google’s move, Google’s motives are far from altruistic. They win in every scenario. If the situation remains that 3 formats need to be supported because Apple doesn’t support webm, Google is far ahead of all other video distributors AND reaches everyone. While other commercial entities could easily pay MPEG-LA licensing fees, they can’t (at least not as easily or as cheaply as Google) encode, store, and distribute 3 different formats. And even if everything but webm disappeared, Google can still do it more cheaply and easily than everyone else AND owns the technology AND reaches everyone. And if webm is exposed as infringing MPEG-LA patents and goes the same path as WMV/VC-1, Google gets a nice PR win, distracts the entire community but Apple for months if not years, dupes Mozilla and the hardcore open sourcers into supporting a patent-encumbered format temporarily, and ends up right back where we are while temporarily being able to reach everyone, etc…. Win, win, win.
    And Google LOVES patents. Ask them to rip up all of their search algorithm patents and see what they say.

  • Guru

    and BETAMAX was better than VHS…quality doesnt mean success!

  • M.
  • FORD

    Wow wow wow !
    How is it possible ?
    this guy really know how to use Flash and H.264, see also this:
    So now that Google TV uses Flash Player 10.1 this kind of quality will be available at low bitrate on TV too. Who needs VP8 ?

  • Glenn

    As others have said, I’d let the clearly untrue bragging from On2 about VP8 go unless they or Google repeat it now. For the moment the question really isn’t about whether its better or not, but whether its GOOD ENOUGH, and whether is actually free from royalties.
    It appears it IS good enough for generic YouTube stuff. Personally I wouldn’t want to watch something tricky like an ‘HD’ action movie or ‘HD’ sports over the web encoded in VP8 since it isn’t as good and I can already see enough stuff even with h.264 once you crank the rates down to 4-5Mbps to get them to work well on typical US home networks. But for a lot of stuff it’s likely fine. The issues with the crap standard based on a buggy implementation are another thing entirely.
    But the most obvious question is whether its really royalty free or not. Given yesterdays article:
    where MPEG LA was asked directly whether they were assembling a patent pool for VP8 and said YES, YES THEY ARE, I’d say we’re back where VC-1 was a while ago. They’ll go over the implementation and decide which patents VP8 infringes, and then decide on a royalty structure and make it available. This will take a while, and so nothing will happen for a bit. If in six months nothing has happened, maybe we’ll be able to breath a little easier about VP8. But my guess is it won’t in fact be royalty free, and all this current hoopla will die down.
    Sure they never when after Ogg Theora. Who were they going to sue that had any money? But the idea that they won’t go after Google who has masses of money and will be deploying VP8 broadly is just somebody in denial.

  • I just ran four VP8 tests. I transcoded the same home-made “youtube style” clip taken on a digital camera at 640×480 using Adobe Media Encoder CS4 into .avi, .mov, .mp4, .wmv.
    I then used the Miro transcoder to transcode each into .webm4 format, and played back with current release candidate of Videolan player.
    The webm format files, regardless of transcoding source, are definitely “good enough.”
    I’ll run some high def stuff eventually- this was just a quick test to see if it works- and it does work.
    Thanks to Google for having massive profits so they can affort to maintain a “free” codec that may finally become the “ntsc” of internet video… but I’m sure that the competition will do everything they can do prevent this.
    Video codec work is difficult and expensive… it requires massive subsidization – but maybe now many generations after the original internet friendly codecs do we have enough of a canon of material to lower codec mainentance costs to the point that its open source…

  • slight typo above- its .webm format… not .webm4 (Dan feel free to fix and remove this post)