Digital image sensors: 60 years of history and an exciting future

Sensor sensibility: Some of the biggest names in the history of digital image sensors who have made significant breakthroughs in CCD and CMOS sensors have collaborated on a history of the technology and a look at its potential futures.

The Annual Review of Vision Science might not be a publication at the top of most people’s normal reading lists, but the content is often fascinating for those that are interested in the visual sciences. These are defined by the publication as “a cross-cutting set of disciplines which intersect psychology, neuroscience, computer science, cell biology and genetics, and clinical medicine,” so there’s an impressively wide field for the publication to cover.

With thanks to DP Review, one of the articles that caught our eye recently was titled Digital Image Sensor Evolution and New Frontiers, partly because it was written by three of the developers of many recent imaging sensor developments, namely Nobu Teranishi, Albert Theuwissen and Eric Fossum, who between them made key breakthroughs in the development of both CCD and CMOS sensors.

The history is interesting. As the Abstract says, the article reviews nearly 60 years of solid-state image sensor evolution, from early work in the 1960s, through the development of charge-coupled device image sensors, to the now ubiquitous complementary metal oxide semiconductor image sensors. Along the way it discusses some of the highlights in the evolutionary chain, while taking a look at new innovations such as 3D stacked technology, photon-counting technology, and others.

It’s a technical read but an interesting one as you would expect. However, it was that latter section that really captured our attention and the authors’ discussion of where image processing might be going.

Future directions

As they write, 3D stacking has enabled increased integration of sensors, allowing high-density focal-plane image processing and increasing throughput while reducing system power (co-author Eric Fossum first wrote about this in 1989). Computation that can be performed in a massively parallel manner will probably benefit the most from 3D stacking, leading towards on-device benefits. This is opening up a move towards smart cameras utilising edge computing.

Elsewhere, “Photon-counting image sensors have advanced rapidly in the past five years and have enabled new applications,” they write. “The technology will likely be incorporated as part of mainstream CIS technology before long, enabling larger dynamic range and the ultimate performance in low-light imaging, especially when combined with computational imaging.”

They also note that “recently, thin-film transistor image sensors on silicon readout integrated circuits with quantum dot photodetectors have incorporated PPDs to improve their imaging performance. These sensors also permit better response than silicon detectors in the shortwave and near-infrared wavelength regimes.”

Lastly there are some developments that could lead to potentially significant changes in camera design and even form factors if the technology can scale. Devices still have to be usable by human hands, sure, and ergonomics will always play their part as a result. But the electronics inside might be able to become both smaller a lighter.

“On the optical side, the possibility of moving from Bayer red, green, blue, green (RGBG) color-filter kernels to receiving R, G, and B signals for each pixel site (and thus avoiding color aliasing problems, long a dream of the image sensor community) seems to be nearing reality given the development of so-called perfect color routers (Catrysse et al. 2022),” the authors say.

“The related development of metalenses and their integration on-chip may simplify camera design and reduce the size and weight of cameras.”

tl;dr

• The article "Digital Image Sensor Evolution and New Frontiers" provides an overview of nearly 60 years of solid-state image sensor development, highlighting key innovations in CCD and CMOS technologies.
• Advances in 3D stacking technology are enabling high-density focal-plane image processing, leading to smart cameras that leverage edge computing and increased integration of sensors.
• Rapid advancements in photon-counting image sensors are expected to become mainstream, offering larger dynamic ranges and improved low-light performance when combined with computational imaging.
• Developments in optics, such as moving to R, G, and B signals for each pixel and integrating metalenses on-chip, could simplify camera design, reduce size and weight, and address color aliasing issues.