Prisms are key to a new type of camera that could revolutionise how cameras work

Researchers at the University of Maryland have developed a new camera that works by emulating how human vision works, and it's not what you might think.

If anyone told you that someone had developed a camera that mimics the way human vision works, you might be forgiven for thinking that we already have it. After all, isn't it all about the lens optics, iris, and light capture? Well, yes it is, but it's not the whole story, because human vision has one component that until now, cameras have not been able to emulate; microsaccades.

What is a Microsaccade?

So, what the hell is a microsaccade? A standard saccade is the rapid, jerky eye movement that the human eye does to scan the world around us. The reason that the human eye does this is because we can only see high resolution imagery that is captured within the high density of photoreceptors that are situated near the centre of the retina. In short, the saccade movements helps us 'fill in the blanks'.

However, that's not the end of the story. Even if we try and fixate our vision on, say, an object or a fixed point in space, our eyes don't stop moving. This is where microsaccades come in. When we fixate on something, our eye still perform very tiny, almost imperceptible saccade motions. These motions are sometimes otherwise called fixational saccades. 

Fixational saccades amount to around 15 minutes of arc, which is the equivalent of holding a magazine at arm's length and looking at the left hand side of an individual capital letter, and then looking at the right hand side. In other words, absolutely minute. Microsaccades generally happen at a rate of between two to three times every second.

Generally, the reason the human eye makes these movements is to keep an object of interest within the fovea area, the point of the retina that has the highest concentration of cones. If we perform a task, such as tying our shoe laces, or performing any other intricate hand held task, the number of microsaccades will be at their maximum, to keep the task at hand focussed on the point of the eye with maximum acuity.

What do microsaccades have to do with cameras?

Okay, so that's all very interesting, what has this got to do with cameras? Well, researchers at the University of Maryland (UMD) have developed a camera that mimics these saccade movements to capture clear and stable imagery over time. The researchers recently published a paper about the development in Science Robotics. The type of camera in question is called an "event camera", and specifically in this case a new type called an Artificial Microsaccade-Enhanced Event Camera (AMI-EV), and it solves one of the main drawbacks of standard cameras, particularly when it comes to machine vision applications.

Illustration showing the differences between a standard Event Camera and the new Artificial Microsaccade-Enhanced Event Camera (AMI-EV). Image: UMIACS Computer Vision Laboratory

A standard camera is great for capturing imagery that us humans might like to watch, such as for the latest Netflix series or a YouTube video. But, standard cameras are not great at capturing sharp, blur free imagery where there's a lot of motion. As you can imagine, when it comes to machine vision, such as for a robot or for a self driving car, having the sharpest vision over time is imperative for accurate tracking objects in high motion situations.

You can probably see a bit of a conflict approaching. If the problem with a standard camera is motion blur in high movements situations, and we now have a camera that emulates micro eye movements, wouldn't that introduce motion blur of its own? You'd be absolutely right, and the crux of the new research was about how to mimic the micro movements of the eye without introducing any extra blur.

The team behind the AMI-EV emulated the microsaccade movements by installing a rotating prism inside it to redirect light beams captured by the lens. Rotating the prism in this way successfully emulated the way the eye continuously moves, and allowed the AMI-EV to stabilise the textures of a recorded object in a similar way to a human. Specialist software was used to compensate for the prism movement and created stable images from the shifting light.

What is a microsaccade camera useful for?

There are profound implications for a camera like the AMI-EV in real world applications. I've mentioned self-driving cars, which will not only be able to track objects much more accurate, but will also be able to distinguish what different objects actually are, even within a high motion situation. The researchers also point out that the low latency and low power consumption of a camera like this would also be highly useful for virtual reality applications.

During the testing phase of the AMI-EV it was apparently able to successfully capture,  display and identify movement such as human pulse detection and rapidly moving shapes. Even more interesting is how the camera could revolutionise high frame rate capture.

The researchers found that the AMI-EV could accurately capture motion in the tens of thousand of frames per second, opening up the door to anything from much better augmented reality experiences to using it for astronomical applications to capture imagery in space. For example, capturing more finite movement to assist in helping to identify the possible locations of planets and other extraterrestrial bodies. Further more, a camera like this excels in low light situations as well, giving it potential use in security situations.

To conclude, what might seem a fairly innocuous and curious development at first glance could have profound implications for how cameras are used in the future.