Video technology still has a long way to go

The current state-of-the-art? Panavision's DXL 8K camera prototype

More is more: Simon Wyndham looks at some of the upcoming improvements for video tech - higher resolution, higher contrast and brightness, higher frame rates - and reckons we need to be continually pushing the boudaries if we ever want to emulate the way the natural world, and our natural eyes, work.

In Phil Rhodes' new series on HDR technology, he threw up an interesting conundrum: after we have a 4K, 120fps, 3D, HDR OLED television, what will the TV manufacturers do?

The cynical amongst you will suggest an 8K, 120fps, 3D, HDR OLED television. But let's take a look at what this really means for us all. We could after all be chasing resolution and other specifications for all eternity. Why do we need 8K? Why do we need ever increasing HDR capabilities?

Resolution smoothness

The increases in resolution that we see often come under heated debate. An interesting thought to consider about resolution, however, is that the benefits that it offers have less to do with actual detail reproduction than you might think. It isn't about being able to see every tiny bit of fluff that might fall on an actor's clothing, although, of course, finer and finer textures will be visible and, hence, give a much more lifelike picture.

Instead, the increases in resolution gives us better smoothness – an odd thought, but a very accurate one. This has been covered by RedShark before, but to briefly sum up, imagine if you have a very hard and straight, high contrast edge –  for example, a flat roof against a sky. Now imagine that it is only ever so slightly off-level. Ever higher resolutions allow such features of a picture to be reproduced with minimal aliasing. This, of course, follows through into textures. Even in bog standard 1080HD, some fine details in people's clothing can cause issues, much like it did in the days of SD. So, ever-increasing resolution helps to reduce this. But the crux of it is that higher resolution allows such minor, yet troublesome aspects of picture reproduction to be solved.

This line of thought can be carried through to graphic aspects such as fonts and text. You are unlikely to be zooming in to view every pixel of a font when you are generally reading. But one of the net benefits of the new 5K computer displays that are now appearing on the market is that text is much easier to read and clearer on the eye. In video, this is of huge benefit to ultra-detailed and sharp on-screen graphics and text.

Of HDR and perception

But what of HDR? Phil Rhodes suggested that watching high contrast imagery constantly would be tiring on the eyes and that HDR may well draw the limit at the suggested maximum of 1000 nits brightness. With each new technology, content creators can be tempted to overdo the effect. I am sure that this will happen with HDR, but it will be a limited period.

I would argue that it is perfectly fine for manufacturers to keep upping the brightness capabilities, as and when technology allows. After all, we might reach a point in the future where cameras are capable of such high dynamic range that we end up needing log gammas to display their content on what we currently call an HDR display!

Watching extremes of contrast will be fatiguing. But the solution here for content creators is to look more closely at how we see the world with our own eyes. We do not often see the full extreme of contrast, in overwhelming quantities, 100% of the time. We do, however, see lots of these high contrast details in the subtly of our world.

I am looking outside of my window right now and there are a few factors at work. My window blind is half-closed because it is fairly bright outside. This has made my office darker. There are lots of "black" objects, such as speakers, amplifiers and the like. As such, they are full of very subtle shading, including very, very fine specular texture detailing on some parts – something that I doubt 1080p could replicate accurately.

Outside, things are brighter. There are white rafters on the end of houses, while there are also a lot of trees and foliage. These are not only full of very fine texturing, but some of them have very fine specular details, but still containing colour and highlighting on shiny leaves. There is also some dark, but visible shadow areas.

The sky itself is fairly cloudy, despite the apparent brightness. But the clouds themselves are full of fine shades.

In the grand scheme of things, or at first glance, this isn't a scene of extremes. But if you look very closely, you can find them in the fine detail. This wouldn't be the type of scene that would be fatiguing to watch on an HDR display, but such technology would, much like some of the fine textures I have mentioned with regard to resolution, be able to display such subtly much more easily.

The net result of this is that such details might not be immediately apparent and jump out at the viewer. But, like good sound design, they would be noticeable if you went back to watching the same scene on a standard display. In other words, when HDR is used well and its use has matured, it shouldn't jump out at us. Used well, we shouldn't actually notice. But it gives us the creative abilities and choice to show what we need and want to show, rather than be dictated to by the technology.

For big films, this means that we have a bigger scale to save up on for big pay-offs, too. Going back to sound mixing for a moment, since it is a good parallel to HDR, if you are creating a sound mix for a feature film, you are not going to use the full volume scale all of the time.

If you mixed your dialogue such that it was using the full range of the volume scale, you would not leave yourself any room for subtle sounds. Further, when you got to that scene with the big explosion, you would have no room to make the big bang louder than your dialogue, and therefore give the impact that you would like.

The same will happen for HDR. Scenes may well make subtle use of the range available, but then when the big bad alien invades the earth using its big super sun gun, such a scene can be made to stand out much more.

Higher frame rates

That is HDR, but what of other technologies? Remember what was said about higher resolutions being about smoothness rather than outright detail? The same could be said for higher frame rates. Whether these will ever gain acceptance is another discussion. Although, frankly, I would love to see sports, such as the Olympics, broadcast in ultra HFR.

Much discussion about going to ever increasing frame rates brings up arguments about how there is no point going above a certain speed because our eyes cannot distinguish it. Sound like a familiar argument?

Once again, we shouldn't limit ourselves to what we think we can technically see or not. At current frame rates and even at higher frame rates, such as 120fps, we still see motion blur created by the recording and we can still, on a NLE timeline, step through the frames. That our eyes still fill in the gaps by optical illusion, no matter how subtly, means that we could still do with having much higher frame rates.

If we can record at an incredibly high frame rate, then when we go to our NLE timeline, stepping through the frames even on some of the fastest objects showing only minor changes, then something more interesting happens.

Our eyes can only see or capture a certain amount of movement. But if we get to the point where motion is so smooth and captured at a high enough frame rate that it is many times better than the capabilities of our eyes, this will mean the ultimate in realism.

Yes, our eyes only capture up to a limit of motion detail in reality, but what needs to be taken into account is that reality itself, the actual world we live in, is not limited by such things. Ask the average house fly! Tried catching one? It can done, but it is very difficult, due to the fact that even the fastest speed at which you can move appears to be in slow motion to a fly.

Once we reach such incredibly high frame rates on recordings and displays, we will be at the point where it is our brains that are creating the natural motion blur and movement, rather than being fooled by an idea of limited frame rate and having the blur hard coded onto the image. It will be less like watching a pre-recorded video than it will be watching real life as it happens.

As a result, I feel that the march of technology will continue and that we should never limit ourselves. After all, there was a time when there were discussions about whether we needed 1080p rather than 720p. How time flies!