Cameras: Tests and spec sheets don’t give you the whole picture

An AJA Cion very definitely out of the lab

I’ve seen a lot of reviews of the AJA CION camera now and opinions are pretty polarised. Most agree that the ergonomics are very good, and there is also some agreement on the character of the pictures, but there’s complete disagreement about whether the way the look is good or bad.

We recently published this article RedShark Review: AJA CION out in the real world, where the author tested the camera and liked it so much that he bought one, which suggests that all the negative stories swirling around about it can't be the whole story.

What I think we’re seeing here is too much reliance on specifications and test results.

This is not to say that tests and specifications don’t have their place: there’s nothing wrong with having objective data on hand to support a camera choice, but unless the tests positively rule out certain uses (like a camera only working in monochrome when colour is needed) it’s absolutely vital to test the camera in the real world as well as testing it in a lab.

Here’s a good example. Some of the very best new lenses you can buy today are not the best optically. They have flaws. Their colour performance isn’t perfect, and they introduce slight but noticeable distortions in comparison with other, more precise lenses. And yet, they’re among the most sought-after lenses on the planet, and they’re expensive, too.

If test data was the only basis for choosing a lens, then nobody would buy them at those high prices. But in real life — they look fantastic.

It’s the same with microphones. Almost no-one wants a clinically accurate microphone. Typically, people prefer microphones with “character”, and for “character” read “distortion”.

It sounds contradictory but it’s pretty much the same reason that you don’t get a specification sheet and test results with a new boyfriend or girlfriend.

Where this really starts to matter is when test results can completely skew opinions of a camera. Lab results for dynamic range, for example, don’t necessarily describe the performance of a camera as it is perceived in the real world, with a real person, looking at real pictures. I’ve always liked the look of the pictures from the AJA CION. As far as I can see, it’s capable of some great results, right out of the camera, without any post processing. But some early tests suggested that the dynamic range is restricted. I don’t think the tests were wrong, but I do think they influenced a lot of people to not even try the camera.

And yet, now that people are getting out and about with the CION, it seems they are liking it and buying it.

I’m not saying that we shouldn’t test cameras, and I’m not saying that we shouldn’t read specifications. Some tests and specs are absolute and if you’re looking for a four seater car, the specification sheet will prevent you from accidentally buying a two seater. But you wouldn’t go buying a car without a test drive, and you probably shouldn’t with cameras either.