In our article How should we test cameras? we asked what was better: unblinking scientific testing or judging on the percieved quality of the image, and we discussed some of the issues surrounding camera evaluations. But we thought it was only right to get the perspective from the people on the front-end doing the reviews themselves. Here, Simon Wyndham presents The Reviewer’s Conundrum.
I have been reviewing cameras and video gear for around 10 years now. When I first started it was back in the days of XDCAM. I was one of the first people in the country to own this new system when it was released as an SD disc-based affair.
Panasonic had their alternative P2 system, and there was great rivalry between them. These were the first professional tapeless systems. There was even a fair amount of opposition to both systems! So I was approached to write my opinion of the XDCAM system from a real world perspective for a well known paper magazine (remember those?) of the time.
I focussed almost entirely on the usability and the workflow. That the camera also produced a fantastic picture was an aside. My logic being that it was a fully professional 2/3” ENG camera, and that if it didn’t produce a great picture then something would be pretty badly wrong!
Not long afterwards HD started to become mainstream, due in no small part from JVC’s initial offering and Sony’s own ubiquitous Z1. Ever more prosumer style cameras were released with ever more professional features, including many of the setup options that were once the sole preserve of the upper tier of equipment.
I used to write a fair amount at round this time about setting up cameras and what the various detail adjustment and knee features did, and how to get maximum recorded contrast range. What is now known as a flat picture profile. I also focussed a lot on testing cameras with resolution charts. I now wish that I hadn’t!
My blog was fairly popular and my articles fairly widely read, and while I cannot lay claim to being solely responsible for the obsession with camera tweaking, I am afraid that I played my part!
Resolution charts can be useful but they do not tell the whole story. While I did catch out one manufacturer for making a false claim about the resolution performance for their new camera, it overshadowed the fact that said camera actually produced a very nice looking picture despite the technically low chart performance.
I read many online comments about cameras, and there is so much talk about the noise performance, dynamic range and the like when examining the picture quality that many other, important aspects get left behind. Some of the issues that may be thrown up are what I suppose I could term “first world camera problems”. These are issues that are flagged up as issues, but in the grand scheme of things really aren’t when comparing them with equipment from only a few years back.
Noise is one of these areas. We have become so spoilt with the amazing performance from cameras such as the A7S and FS700, and the FS7 and their ilk that when another piece of gear doesn’t match them they are somehow considered inferior.
The Blackmagic cameras are a case in point. They do not match the Sony cameras for low light or noise performance, but does it matter? Not really. For my money, and this is a purely personal opinion, I prefer the more organic look of the BMD cameras. I’m not even that bothered by the noise because in the grand scheme of things it isn’t that bad, and actually contributes to the look.
Low light performance is all well and good, but is it really so important? Yes, it opens up some doors that were previously closed, but when all is said and done a good cinematographer will be looking for the nicest light, not the worst, to film in. If you need low light for a documentary then there is plenty of choice out there. But that low light performer may not be the best for, say, fictional film production.
One of the features of the AJA Cion that has gone undocumented by, well, every review I have read for it is that its ISO ratings are calibrated to precisely match those of film. This is not the case for other cameras. ISO800 on a Sony F5 is different to the same equivalent on a Panasonic camera, or a Canon. The Cion on the other hand is matched, which means that your light meter will work properly with it without needing any sort of compensation of calibrating to your specific camera.
Why do I mention this? Because AJA has taken the line that it is fine to match traditional film stocks in terms of light sensitivity. Low light performance was not their priority. Great colourmitry and contrast performance on the other hand was. Remember, price, performance, quality, pick two. AJA went with what they considered important. The result is a camera that on paper might not match an Alexa, but in reality produces a wonderful picture indeed.
With the dearth of opinions abound, how should a reviewer tackle such things? Personally I do not believe that you can buy a camera that performs badly any more. A cheap consumer handycam these days performs a magnitude of times better than a similar camera from only five or six years ago. If you have an interesting subject and interesting light you can create a picture that will leave people interested in the content and the subject matter. If someone asks what camera you filmed something on, then you have done something wrong.
Instead I prefer to focus on what matters to most camera users. How accessible are the controls? Are there any funny bugs? What are the audio controls and performance like? How heavy is it? How well built is it? What are the features that assist me in nailing the picture like? You get the idea.
Technical paper spec picture quality is way down on my list simply because I would expect nothing less than at least a good picture on a modern camera. That plus “technically” good is in no way directly translatable to “pleasingly” good. A case in point is when a picture is set up to ChromaDuMonde colour charts from DSC Labs.
These charts allow you to set up your camera so that black really is black, and your skin tones line up precisely etc, etc. DSC themselves advise that setting your camera up to such a chart will give you a technically correct picture, but that this is only a starting point because it won’t necessarily look aesthetically pleasing.
Another aspect that is often looked at is bitrates. Currently most cameras now record fairly high bitrate images. But this wasn’t always the case, and some cameras such as the FS700 still record 25Mbit/s internally.
People used to compare bitrates on cameras a lot, but failed to realise that not all compressors are the same. Sony have a lot of expertise with them, and often they could get much more quality out of the same bitrate as other manufacturers.
So it boils down to this. Is the camera nice to use? Will it last? Is the picture it produces pleasing? Picture “quality” is subjective, which is another reason why I don’t place a focus on it in reviews. It could be argued that technical charts would place all cameras on an equal playing field, but I disagree.
If resolution performance or any other aspect of performance is crucially important to you, then you should test. A camera review can give you an overview, it can make you consider pitfalls you may not have thought of previously, and it can give you an idea of what to expect from the performance. Clearly if something is completely amiss it is our job to tell you. Likewise if we really love how a camera performs, we should tell you.
But what you can’t do is list numbers such as 8-bit or 10-bit etc and expect them to give an indication of quality or how pleasing an image is. These sorts of numbers are to assist decisions on purchases based upon the end use of the camera. They are practical choices. If you don’t do heavy grading, should you really be bothered about 8 or 10-bit colour depth for example?
This is why, while I will tell you important technical specs so that you are aware, I will focus in on the real world usability of the device.