Do we want more pixels? No. What we want is better pixels

Do we need better pixels

What's the way forward with video? It's not, it seems, more resolution. What people want is better, not more, pixels

We’re living through what’s possibly the most exciting (and sometimes frightening) period of our entire history. That's not just in video, but in the whole world. The rate of change is getting hard to take on board. And now, in the science of moving images, the debate has changed from “What might it be possible to do” to “what *should* we be doing”. It’s changed because we are actually able to do most of the things we’d like to do. 8K? That’s been done. High frame rates? We can do those too. High Dynamic Range? Dolby has demonstrated it and most high end cameras can already achieve it.

So, in that sense, there’s no debate: it’s all here, ready to go.

Well, no. Not really. Not when you take things like cost and practicality into account.

While we might know exactly how to achieve these improvements in video quality, all three of them are undeveloped and none are completely ready for commercialisation. But with the astounding rate of change in technology, doesn’t it mean that we’ll have all of these in just a few years? Possibly, but most likely not. And it won’t be for purely technological reasons.

It will more likely be because of what people think, and what they want. It also depends on who you ask. Most of all it will be because of what they can fit in their houses.

It's the public that will decide

So, ultimately, it will be the buying, viewing public that decides whether a new format or video innovation succeeds. As we now know, it was by largely ignoring the opinions of viewers that 3D on television was such an expensive dead end.

But, before that, the video creators and distributors have to agree about future directions, and I have a much better idea of where this is going now, as I spent a day in the company of industry thought leaders and master practitioners at the UK Screen Association conference on More Pixels: Better Pixels on Wednesday this week in London’s Soho. I was lucky enough to chair a discussion on the headline subject at the end of the day’s presentations. We also took an audience vote, and the results were as stark as they were surprising.

The panel at the UK Screen Association event in London's Soho. From the left, there's David Shapton, Editor In Chief, RedShark, Peter Sykes, Strategic Technology Development Manager at Sony Europe Limited; Andy King, Controller of Technology, BBC Television; Mark Horton, Strategic Product Manager Encoding Portfolio, Ericsson; Rick Tester, ONSIGHT

I’m not going to recount the proceedings in detail but just want to go through the key issues.

First, and most basic: do we really want to have more pixels? Even the question of 4K isn’t completely settled, because there are still cinematographers - and probably quite a few news and documentary camera operators who are perfectly happy with HD. Even though I doubt whether we’ll see many or even any more SD cameras from the manufacturers, HD does have some advantages over 4K in some situations. 4K sensors tend to be bigger, with a narrower depth of field. This means that unless all your scenes are well lit, you’re in danger of being out of focus. There’s nothing worse than framing that great shot - at just the right moment - and then finding it’s not sharp enough to use.

Quite a few people tell me they love their new 4K cameras and shoot in 4K before immediately downsampling to HD. They get much better pictures that shooting in HD alone.

But It’s not the production side of things that I’m going to talk about here. I’m going to look at what consumers are going to want in their living rooms, because that, by far, is what is going to influence what content is produced.

The reason I’m not going to talk about production is simply that you can down-convert any high resolution to any lower resolution, and do it really well, with only a few exceptions (to do with frame rates and shutter angle/motion blur). So, while it’s obviously important to consider the cost and practicality of shooting in a very high resolution, it doesn’t ultimately matter to the viewer, who will probably be watching in a lower one.

Remember as well that the relevance of very high resolution depends enormously on the content. You will probably have a different outcome to a debate about high frame rates depending on whether you talk to a TV sports director or a drama director.

Ask the biggest audience

The issues that matter most are the ones that affect the biggest audience: viewers at home, in their living rooms (and, increasingly, in their bedrooms, on buses, trains and anywhere else you can get a mobile data signal).

And it’s here that you have to ask: exactly how much more resolution to we need?

I’m grateful to Andy King, Controller of Technology, BBC Television, for these thoughts about the dynamics of living room resolution, which I hope I’ve captured accurately. (Note that the BBC is officially neutral in these matters and any opinions expressed here are mine).

You can’t put a huge screen in a typical European living room. Anything over 60” is considered to be very big and you probably can’t expect to go above 84” without architectural changes. But to see the benefits of higher resolution you need to go even bigger. With 8K, you’re looking at over 100”.

Price comes in to it as well. To many people’s surprise, mid-range 4K sets are now very cheap, and it seems likely that they’ll reach near parity with HD sets within a few years, if not a few months - at which point, no-one will buy HD sets at all.

So 4K in the home seems assured, but not 8K. Not even slightly (we’ll come back to this later).

So with 8K effectively out of the way, that leaves two candidates for radically improving picture quality: increased frame rates, or better dynamic range.

The problem with higher resolution is that motion blur becomes bigger as a proportion of the size of the objects that you are able to see on a screen. Even though I’m a great admirer of the sheer technological heft of 8K, it was quite clear to me at IBC this year that there are issues. I stood in front of a massive screen (it was taller than me) and watched a football match in 8K, and when nobody was moving (a rarity in a football game) it looked like a photograph. But when the players started moving, they turned into ghosts. That was at 30 FPS. At 60 FPS it was better. But not better enough. I suspect that you’d need a rate of 120 FPS minimum for it to look merely OK. And that’s not the response you want when you show something that has 64 times the data rate of 30 FPS HD: that it’s “merely OK”.

But there’s the problem: data rates - and, of course - storage for all those additional frames.

You’d probably be better off, certainly in budgetary terms, with 4K at 60 FPS, which is a mere 8 times HD at 30 FPS. We tend to assume, mostly rightly, that Moore’s law will take care of issues like this, but as Mark Horton of Ericsson pointed out: bandwidth is not currently on Moore’s law, more for economic reasons than technical, I suspect.

Dynamic Range

So that leaves Dynamic range.

We’ve already identified this as the “biggest advance in video for ten years” and we will be writing about it again soon.

Simply put, conventional TVs and displays don’t even get close to reproducing the range of brightnesses that we can perceive with our eyes and internal image-processing systems. The full range from direct, bright sunlight to being in a cave is about 20 stops. Conventional displays manage 10 or probably less. HDR video would give us around 14 stops, which is about what higher-end cameras can capture now, and when you consider that each stop is a doubling of brightness, that’s a lot more light. (There are all sorts of versions of these figures - your milage may vary but the bottom line is that HDR displays need to have massively more contrast than conventional ones).

To implement this would entail changes along the whole production and delivery chain, but not as much as you’d think.

Modern cameras have to actually reduce their dynamic range output by applying log cinema curves, which are then modified in grading to produce a decent-looking picture, but nevertheless one which has to play well with whatever type of display its being sent to.

There are several ways to improve the dynamic range of displays. Probably the easiest is to increase the brightness of the display screen’s backlighting. You can even do this selectively so that while one part of the screen might be showing bright sunlight, other parts, with the backlight turned off, might be showing complete darkness.

None of this is impossibly difficult. It will mean that manufacturers will have to build sets that can do this but some, according to Dolby, already are. What’s essential if this is going to take off is that HDR video is compatible with “standard DR video”, and that’s achievable, either by creating multiple grades, one for HDR and the other for SDR (expensive, time consuming) or by including metatada in the stream to allow the receiving display to dynamically adapt the DR of the incoming pictures, all the way back to SDR if necessary.

Now, towards the end of our discussion, in a completely unscientific way, I asked the audience to vote for the video enhancement that they’d most like to see, if they had to pick just one. Based purely on my rough assessment of the number of hands raised I would say that the results were (leaving out the "don't knows"):

A bit of a trend, there, then.

Many thanks to Neil Hatton, Sarah Mackey, and Angela Cook of the UK Screen Association for organising a great day. Thanks too for the participants in the RedShark discussion: Andy King, Controller of Technology, BBC Television; Peter Sykes, Strategic Technology Development Manager at Sony Europe Limited; Mark Horton, Strategic Product Manager Encoding Portfolio at Ericsson and Rick Tester, ONSIGHT