Opinion: Why camera comparisons don't work

Replay: Perhaps reviewers should refrain from direct camera comparisons (or at least rethink their approach).

I really like reading multi-camera shootouts. They're really fascinating and sometimes produce some surprising results. But, ultimately, I'm just not convinced that we really learn much from them. Here's why.

First of all, let's look at what you'd need to do to accurately and fairly compare two cameras. The first thing you'd have to do is restrict the scope of your test. In fact, this applies to testing anything. Imagine what would happen if you didn't do this. Let's say you're testing melons.

You might say to the two testers: "which melon was better?"

One might say, "Obviously this one, because it's perfectly round."

And the other might claim "obviously it's this one, because it's a brighter yellow."

That's an example of a completely useless test. It might seem obviously flawed, yet it's surprising how quickly this type of flawed methodology can creep into a superficially sensible test.

Troubled comparisons

Recently (at the time of original publication), someone compared a Sony A7S (a camera that's famously good in low light, because it has fewer than average pixels, but, therefore, bigger than average pixels, so it is more sensitive and hence less noisy at a given illumination) with a Sony A7R II, a camera that's got a 42 megapixel sensor, with necessarily smaller pixels.

The user complained that in low light, the A7R II was noisier than the A7S. This is about as surprising as the moon having gravity. Of course, per pixel, it's noisier. And we know exactly why. The only flaw in anything here is the understanding of how things work.

And there's another issue with this same example: if you only focus on one aspect of a camera's performance, you might entirely miss another, which completely compensates for the first issue.

In this case, the sheer fact that the A7R II has so many pixels means that you can average out multiple pixels on the sensor to give you an interpolated pixel in the final image, which will, I guarantee, have less noise than a single pixel.

Just to repeat: if you're going to compare two complex products like cameras, then you have to be incredibly specific about your terms and points of comparison.

Preference rules all

The fact is that today's level of technology with cameras is so high that it's getting very difficult indeed to measure performance in absolute terms. You can look at charts and you can take account of ergonomics. However, none of this takes into account the needs and, yes, the preferences of the user.

My favourite camera of all time is still the (slightly unusual, but still great) Canon 1D C. This top-end DSLR only has an 18 megapixel sensor, but is rugged, weatherproof, takes lovely still pictures and stores huge but immensely accessible MJPEG 4K files. Completely oddball, but completely right if you're like me. I like rugged cameras. I like taking stills. And I don't like having to grade everything just to get an idea of what the pictures look like. The images from the 1DC always look crisp and saturated. That's what I want from a camera.

Others will disagree. They'll disagree about the images, the ergonomics, the codec, the file format and virtually everything else.

That's exactly why we can't say, typically, that one camera is 'better' than another.