December 27, 2016 at 4:14 p.m. PST
Consumer Reports did not recommend the 2016 MacBook Pro lineup, due to a perceived battery issue. This article explains why their test was flawed.
Consumer Reports is trusted to provide unbiased product evaluations. Their product approval carries a lot of weight. When they don’t recommend a product, it’s as good as telling consumers not to buy it.
One would think, with this great responsibility, Consumer Reports would be thorough and perform comprehensive tests. After all, coming to the wrong conclusion could cost a corporation millions and even billions of dollars. Unfortunately, their tests often leave much to be desired, especially when it comes to technology. Let’s face it, Consumer Reports is a non-profit. The best and brightest technology experts aren’t clamoring to work there. I think even most lay people may see how their testing could be flawed.
Recently, Consumer Reports tested the new 2016 MacBook Pro lineup. Their results revealed highly inconsistent battery life across all models. The MacBooks performed well on the first test, with diminishing results for each subsequent test. The Internet went wild and virtually every tech writer misconstrued the actual details of the test. I have to wonder if they even read Consumer Reports’ own article about this.
The Consumer Reports MacBook Pro Battery Test
The Consumer Reports MacBook Pro battery test isn’t the gold standard for testing battery life. They automate Safari to access 10 web pages serially, in a loop, until the battery dies. Battery life is recorded for each test run. I do applaud them for setting up the web pages on an internal server. It eliminates the inconsistencies that would occur if the pages were on the Internet. That said, even an intranet-based test has its problems. Ideally, a battery life test should be run solely on the machine, and not depend on external servers.
The main problem with this test is that it doesn’t measure battery life. It measures how long the battery lasts when doing one task — browsing the web with Safari. The Consumer Reports article explains that they tested it again using Chrome, and didn’t find this same problem:
Once our official testing was done, we experimented by conducting the same battery tests using a Chrome browser, rather than Safari. For this exercise, we ran two trials on each of the laptops, and found battery life to be consistently high on all six runs. That’s not enough data for us to draw a conclusion, and in any case a test using Chrome wouldn’t affect our ratings, since we only use the default browser to calculate our scores for all laptops. But it’s something that a MacBook Pro owner might choose to try.
Instead of revising the test, they decided not to recommend MacBook Pros, a week before Christmas. This was an incredibly irresponsible move. They didn’t question their own methods. They didn’t come up with a better test. They effectively told people not to buy a MacBook Pro, because their flawed testing revealed battery life problems. This move may have cost Apple millions of dollars. The damage to their reputation could take years to repair.
If the Consumer Reports Test is Flawed, Why Do MacBook Pro Users Complain About Battery Life?
Readers may be wondering, if Consumer Reports is wrong, why are some MacBook Pro users complaining about the battery? It’s because they use Safari. Apple’s lame web browser is the culprit, not the actual hardware. Look at any discussion about this, and you will see some users experience the problem and others don’t. It’s a problem with Safari and not the MacBook Pro battery.
I stopped using Safari on my Mac years ago, because it is incompatible with many sites. It doesn’t work well with Google Docs. I was unable to transfer money from my bank using Safari, but it worked in every other browser — even Firefox. Recently, PetSmart launched a web app for booking dog grooming appointments. It didn’t work at all in Safari, but was functional in every other browser. Beyond the sites that don’t work, Safari just isn’t a smart browser. It does a poor job of automatically re-sizing sites to different browser window sizes. So many sites just look wrong in Safari. Safari is just one of many Apple-developed apps that are mediocre at best.
Apple Makes Excellent Hardware
Apple creates excellent hardware and few dispute this fact. When you go beyond the basic specs, Apple gives the consumer an exceptional value. If you find a Windows PC that measures up to a Mac, it usually costs more. One has to look beyond the basic specs — RAM and processor speed. When you look at aspects like bus and memory speed, which are often ignored, MacBooks are actually reasonably priced.
Beyond the specs, every Apple device I have owned has lasted a long time. I still have my iPod I bought 12 years ago, with the original battery. It still works. I use my old iPhone 4, which is 6 years old, as a coffee table device. It still has the original battery, and I charge it once or twice a month. I’ve never had a problem with any Apple device I have owned. This is why I was so surprised when I found out about the Consumer Reports battery test.
Macs are integrated systems. Each component is manufactured to Apple’s specifications and the entire system is tested thoroughly. While other manufactures do this to some extent, they also create many different models. PC manufacturers are also known to swap out components for cheaper options, within the same model. Apple does some of this too, as they have problems getting enough components from suppliers.
The point is, Apple only makes a few Mac models and they test them thoroughly. PC manufacturers squirt out many different models, and if one of them is a dud, it’s not a critical loss. Virtually anyone can build a PC from off-the-shelf components. It’s not rocket science. It’s not even computer science. It’s a little more complicated than plugging in a Nintendo cartridge. Apple ensures that the components are high quality and work together. They don’t use cheap motherboards, faulty storage devices and slow RAM.
When I upgraded my Mac Pro, I purchased an official Apple hard drive. It was Apple branded, box and all, however it was manufactured by Western Digital. The average PC tinkerer would assume one could just go to Fry’s and get the same drive for half the price. I checked the specs on the model, however, and it was a server grade drive with manufacturing tolerances that far exceed most hard drives. It was manufactured to Apple’s specifications. Indeed, Apple works with suppliers and has stringent standards for most components. This is why none of my Apple devices have failed — ever. I can’t say that for DELL, HP or others. In fact, one of the reasons I started buying Apple products was because I was fed up with laptops that broke after 2 years. Apple makes great hardware. Period.
Apple Needs to Take App Development Seriously or Abandon Efforts
Safari is the culprit and this battery fiasco is partially Apple’s fault. As much as I can blame Consumer Reports for such a simplistic, one-dimensional battery test, Apple obviously introduced a battery draining bug in Safari. This is why Apple’s own tests didn’t find this problem. Unfortunately, it demonstrates that Apple’s horrible first-party apps have compromised the reputation of their excellent hardware.
Beyond Safari, so many Apple apps are either average or horrible. Reminders is still quirky after almost 5 years. It’s just a list manager, but they can’t even get that right. iTunes is just awful. Apple Music was so bad, I switched to Spotify and then Google Play Music. Both are vastly superior, as it seems that Apple Music was created by summer interns. Maps? News? It’s hard to think of any Apple-developed app that’s decent. GarageBand is pretty good. Their professional apps, such as Logic and Final Cut Pro are excellent.
When it comes to the free apps they include with macOS and iOS, Apple just delivers poor quality. It’s quite a dichotomy — the worst apps running on the best hardware. Fortunately, you don’t have to use them. Use Chrome instead of Safari. Apple’s claim that Safari saves battery life is laughable. Safari cost them millions of dollars in MacBook Pro sales. Will Cupertino wake up to the reality that their apps are atrocious?!? There seems to be too much cheerleading and group-think going on at Apple and not enough internal reflection and criticism.
Something is wrong with Apple’s engineering process. It doesn’t work for apps. Complex engineering processes work well for large projects, such as devices and operating systems. They completely stifle app development. I can only imagine, fixing a bug in an app takes several meetings and a lot of documentation. Having worked in the industry for some time, I am well aware that certain engineering processes can be stifling. It’s partially for a good reason. It prevents over-caffeinated product managers from constantly changing the requirements. They’re forced to think about requirements and work them out before even a single line of code is written. If they need to change something mid-project, they have to write it up, submit it and it has to be approved by upper management.
Restricting change usually results in better software quality, unless it applies to fixing defects. Believe it or not, there are managers who refuse to let developers fix critical bugs. This happened to me, and that manager went on to work at Apple for a year. This seems to demonstrate that Apple may have problems attracting top software talent. I mean, they hired one of the worst managers I have ever had the displeasure of working with, and he held a top position. They do get the very best hardware and industrial design talent, but top software people go to startups. They want equity, and Apple can only offer overpriced stock options.
Another problem is that these engineering processes don’t work well in a crisis. They also don’t work well when it comes to fixing bugs. Couple that with a highly differentiated division of labor. Software engineers all have their expertise. Some work on the front-end, some on the back-end and others work on all points in between.
I don’t work at Apple, so I can only draw these conclusions based on what I have seen in my own experience. I am also well aware that Apple is a very large corporation, and they tend to have very complicated engineering processes and lots of specialized developers.
Apple has to either figure out how to make decent apps or get out of that business entirely. Instead of Safari, bundle Chrome with macOS and work with Google so it integrates into the operating system. Ditch iWork, no one uses it. They would be better off integrating macOS with Microsoft Office. You can get MS Office for the Mac, but if they really synergize it with macOS, it would be even better. They could even bundle a light version with macOS for free.
This problem with the MacBook Pro’s battery life is really a problem with Safari and Apple’s culture. They cannot create decent apps. The problem is so bad now, they are going to lose millions of dollars in MacBook Pro sales, because of their atrocious web browser.
What do they gain by providing Safari to end users? The reality is, a serious Safari bug ended up compromising the integrity, reputation and profitability of their actual product — the MacBook Pro computer. Apple makes profits by selling hardware. They lose money by working on Safari, even if it doesn’t compromise the hardware. Now they are losing a lot of money, thanks to Safari.
Consumer Reports’ Damaged Reputation
Consumer Reports does not emerge from this fiasco unscathed. The non-profit provides reviews, ratings and recommendations on everything from lawnmowers to cars. Some products undergo remarkably thorough testing. Their battery life test for computers, however, is deeply flawed. It really just tested Safari and not the MacBook Pro.
This revelation was lost on the blogosphere, as tech writers failed to mention that Consumer Reports ran the same test using Chrome, with favorable results. At that point, Consumer Reports should have revised their test, but they didn’t. Instead, they chose to tarnish the MacBook Pro by withholding their recommendation.
This isn’t the first time this has happened. When the iPhone 4 came out, Consumer Reports fell for “Antennagate”. Although this flaw was something common to all cell phones, they withheld their recommendation for the iPhone 4. I bought one of them anyway, and I still own it. I simply never experienced a problem with reception, no matter how I held it. I also still have my old Samsung flip phone, which still had a sticker on it, informing me not to hold it near the antenna, as it could attenuate the signal. It’s a problem common to all phones, but I never experienced it with my iPhone 4.
Consumer Reports has become reckless. They’ve lost a lot of credibility. Their tests for battery life, antennas and other aspects of technology products are woefully inadequate. Unfortunately, their recommendations carry a lot of weight. People use Consumer Reports to make purchasing decisions. Without a doubt, their mishandling of the MacBook Pro battery issue will cost Apple millions of dollars. Because this issue is Apple-related, it reverberates across the Internet with the usual distortion.
Apple isn’t blameless in this matter. They obviously have a serious problem with Safari. In certain situations, their browser will rapidly diminish battery life. This should be a wake up call to Cupertino — take your apps seriously or cease developing them altogether. I think Apple would be better served by pursuing vertical integration strategies for their profitable business — devices. They should acquire suppliers and manufacturers. Developing apps like Safari is counterproductive. This time, their sloppy app development has cost them more than they bargained for.