page 1 of 4
Consumer Reports did not recommend the 2016 MacBook Pro lineup, due to a perceived battery issue. This article explains why their test was flawed.
Consumer Reports is trusted to provide unbiased product evaluations. Their product approval carries a lot of weight. When they don’t recommend a product, it’s as good as telling consumers not to buy it.
One would think, with this great responsibility, Consumer Reports would be thorough and perform comprehensive tests. After all, coming to the wrong conclusion could cost a corporation millions and even billions of dollars. Unfortunately, their tests often leave much to be desired, especially when it comes to technology. Let’s face it, Consumer Reports is a non-profit. The best and brightest technology experts aren’t clamoring to work there. I think even most lay people may see how their testing could be flawed.
Recently, Consumer Reports tested the new 2016 MacBook Pro lineup. Their results revealed highly inconsistent battery life across all models. The MacBooks performed well on the first test, with diminishing results for each subsequent test. The Internet went wild and virtually every tech writer misconstrued the actual details of the test. I have to wonder if they even read Consumer Reports’ own article about this.
The Consumer Reports MacBook Pro Battery Test
The Consumer Reports MacBook Pro battery test isn’t the gold standard for testing battery life. They automate Safari to access 10 web pages serially, in a loop, until the battery dies. Battery life is recorded for each test run. I do applaud them for setting up the web pages on an internal server. It eliminates the inconsistencies that would occur if the pages were on the Internet. That said, even an intranet-based test has its problems. Ideally, a battery life test should be run solely on the machine, and not depend on external servers.
The main problem with this test is that it doesn’t measure battery life. It measures how long the battery lasts when doing one task — browsing the web with Safari. The Consumer Reports article explains that they tested it again using Chrome, and didn’t find this same problem:
Once our official testing was done, we experimented by conducting the same battery tests using a Chrome browser, rather than Safari. For this exercise, we ran two trials on each of the laptops, and found battery life to be consistently high on all six runs. That’s not enough data for us to draw a conclusion, and in any case a test using Chrome wouldn’t affect our ratings, since we only use the default browser to calculate our scores for all laptops. But it’s something that a MacBook Pro owner might choose to try.
Instead of revising the test, they decided not to recommend MacBook Pros, a week before Christmas. This was an incredibly irresponsible move. They didn’t question their own methods. They didn’t come up with a better test. They effectively told people not to buy a MacBook Pro, because their flawed testing revealed battery life problems. This move may have cost Apple millions of dollars. The damage to their reputation could take years to repair.
If the Consumer Reports Test is Flawed, Why Do MacBook Pro Users Complain About Battery Life?
Readers may be wondering, if Consumer Reports is wrong, why are some MacBook Pro users complaining about the battery? It’s because they use Safari. Apple’s lame web browser is the culprit, not the actual hardware. Look at any discussion about this, and you will see some users experience the problem and others don’t. It’s a problem with Safari and not the MacBook Pro battery.
I stopped using Safari on my Mac years ago, because it is incompatible with many sites. It doesn’t work well with Google Docs. I was unable to transfer money from my bank using Safari, but it worked in every other browser — even Firefox. Recently, PetSmart launched a web app for booking dog grooming appointments. It didn’t work at all in Safari, but was functional in every other browser. Beyond the sites that don’t work, Safari just isn’t a smart browser. It does a poor job of automatically re-sizing sites to different browser window sizes. So many sites just look wrong in Safari. Safari is just one of many Apple-developed apps that are mediocre at best. (continue…)