Why reviews can only give you part of the story

It doesn't stop us trying, but there are several factors that conspire to ensure that any review of a new product is inevitably going to be incomplete.

Much as the world's technical journalists do their best, there are some things that equipment reviews just can't do. Clearly, we can't cover things we don't have or can't possibly know and there are other factors on which nobody could say anything but speculation. Unqualified speculation is something best kept out of reviews, especially technical writing about equipment that has to do some very specific things. The result, though, is that there are several factors which are extremely important to equipment purchases, but about which no journalist can reasonably write.

One of them is reliability. There is no standard review period for equipment, although practically speaking, the answer is usually somewhere from a week to a month, depending largely on the demand for review items from other publications. Yes, this means the trade press is probably more rushed, less complete and less accurate, on average, around the time a product is launched, which puts the scramble to get a review out into perspective. It's one reason why we've occasionally returned to products some months later, especially once a phase of firmware updates has been released. Even so, we'd expect a camera to last years, even if it'll only be the latest thing for a few months. The problem is that figuring out whether it will last for years would take — well — years.

The confounding factor here and the second issue that's very hard to discuss in print is customer service, at least until a sufficiently large number of people are actually in the process of taking a company to court. Even failing that, it's clear that any company sending a review item to a journalist is going to be on its best behaviour, so it's almost impossible to make any realistic assessment of the way that company might interact with a hypothetical customer on a hypothetical future occasion. Journalists don't generally interact with the same people who answer the customer service line.

There are only really two sources of information on these factors and neither of them are objective enough to satisfy even the least uptight editor. The first is comparative price, where we might reasonably expect less reliability and a long phone hold from a less-expensive but otherwise equivalent product. When we buy a product, we aren't just paying for its design and fabrication, we're paying for the corporate apparatus which supports it. The problem is that very few products have precise equivalents between companies, and it's a matter of degree: nobody is going to give a percentage score for anticipated reliability based on the comparative price of one product against another.

The second source of information (if that's the right word) about reliability and service is reputation and rumour. It's presumably fairly clear why it's best not to base technical writing on the collective mood of the intern. Selection bias ensures that the negative view floods online with much greater alacrity than the positive, and we've talked in the past about how the internet can quickly amplify almost any characteristic of a perfectly good product to a storm of dissent.

There have been good examples, details of which we'll tactfully redact, of perfectly good products which became the undeserving subject of an internet pitchfork mob. It's happened with compression on lower-end cameras, something that's a straightforward engineering compromise between quantization noise and blurring. These compromises can always be tweaked, but the compromise ultimately remains. The real solution, if you need to operate a camera at the edge of its capability, is something other than the least-expensive member of a range.

Furthermore, various cameras from various manufacturers have been accused of producing green or magenta vignetting. In truth, this is usually a factor of people using old stills lenses which are far from image-space telecentric on a sensor with reasonably deep-set photosites and a quick web search reveals that almost every camera in the indie filmmaking world has been accused of this at some point. Without thinking carefully, it can be difficult to get to the bottom of forum conversations on the subject, because people tend to see it as a camera fault and often don't even mention the glass they were using when they shot tests.

The internet is an echo chamber of dissent, often not a well-informed one, and not one on which journalists should be basing articles. There are, sadly, things we just can't get involved in.