PTB: Are there any promising new imaging sensor technologies on the horizon?
LaBelle: Many in fact. The flexibility of CMOS is leading to specialized detectors with multispectral imaging capabilities, sensors capable of 3D imaging, and availability of a wide range of sensor formats that go far beyond what CCD offered. SPAD sensors – arrays of avalanche photodiodes – are also very interesting for high frame rate scientific imaging due to the promise of true single photon imaging.
Longval: The rate of innovation in sensor technology is tremendous. Every year we see a number of new sensors coming to market. In recent years we have seen scientific CMOS and global shutter CMOS. There have also been major improvements to CCD technology to improve sensitivity or increase speed.
Bridges: We are always looking at new technologies that can improve our speed, light sensitivity, noise levels, etc. These might include back-illumination systems, ISIS technology, among many more.
Bode: While there are no completely new technologies on the horizon that could replace CCD or CMOS completely in the near future, there are some new technologies that could significantly increase the range of applications for digital cameras [such as] light-field imaging. Virtually all cameras today produce images with a more or less narrow depth of focus. If the image captured was not in focus, or focused on the wrong distance, the image is without value. Light-field imaging can correct this disadvantage. [And] hyperspectral imaging, [which] uses various different technologies to create multiple images in narrow spectral bands. The spectral information can then be used to infer information about various aspects, such as the health of vegetation or the composition of materials.
PTB: Rolling shutter vs. global shutter – what are the relative strengths and weaknesses of each? Which applications best suit each type of shutter?
Bridges: To my mind, a high speed camera has to utilize a global shutter. A rolling shutter is limited to whatever the framing rate is, i.e., a one thousand frame per second (1K fps) can only provide shuttering at 1/1,000th of a second, or 1ms. Plus pixels are typically exposed one row at a time, meaning if something within the image is moving fast, it will spatially shift one row from the next, producing some pretty funky results.
LaBelle: The knock on rolling shutter is related to geometric distortion from motion. As cameras with rolling shutters increase in sensitivity and frame rate, the level of rolling shutter distortion often drops to insignificant levels. In the case of fluorescence microscopy, the distortion can also be mitigated by “global shuttering” illumination. There are cases where the impact of motion can’t be mitigated and the downsides of global shutter – lower frame rates and slightly high noise – are acceptable. We think both modes will be relevant for some time.
Longval: A rolling shutter sensor can be very fast and is very affordable to manufacture thanks to its simpler transistor structure. They also offer relatively good low noise performance – not as good as CCD but better than global shutter CMOS. The problem with a rolling shutter is the image artifacts generated from any moving objects within the field of view or from camera movement. For applications where fast motion is part of the application, a global shutter sensor, be it CCD or CMOS, becomes a must have.
Bode: The global shutter of CCD sensors and the rolling shutter of CMOS sensors used to offer clear distinctions between the two types of sensors, but since many CMOS cameras now offer global shutter modes, the distinction has become less obvious. In global shutter mode, all pixels of a sensor are exposed at the same time and for the same duration, while in rolling shutter mode the lines of a sensor are exposed at different starting times (sort of like a trigger wave rolling across the sensor). The most obvious effect of this is that a fast moving object can shift position between the times this wave moves from one line to the next, which results in a distorted object in the final image. If the sensor is a color sensor, this can also lead to noticeable color fringes around the objects, and for handheld devices it leads to a “jelly effect” where objects wobble as if they were made from jelly. On the other hand, the noise performance of rolling shutter cameras can be better than that of global shutter cameras.
PTB: Looking into the future, what area do you predict will see the next big breakthrough in high-speed/scientific imaging technology?
Longval: I would say embedded vision is going to be a major growth market in the future. As technology evolves, imaging will find its way all around us: inside toys, automobiles, medical instruments, our houses, everywhere. The processing capacities found in embedded devices will be able to handle the high data associated with high speed image sensors. High end imaging will get out of the lab and find its way into our daily lives.
LaBelle: Certainly today’s scientific microscopy cameras already move between traditional high fidelity image capture, to collecting data that is a representation of underlying biochemistry, images that represent data that could never be visualized by eye. The use of increasingly sophisticated optical systems combined with computational approaches will take microscopy and scientific imaging into new realms, where rich data is extracted from the light field beyond working in low light and photometric imaging.
Bode: For high-speed cameras connected to a host computer, the bandwidth of the connection is the biggest bottleneck. To significantly increase the bandwidth, new protocols and connections need to be developed. With the current PCIe interface or Thunderbolt technology, we can reach 20+ Gbps, slightly more with special hardware (Coaxpress). But this is not the end. PCIExpress 3.0 offers a throughput of roughly 8 Gbps per data lane, and each PCIe connection can have up to 16 data lanes, which would theoretically allow 128 Gbps. To put that in perspective, this would allow a 1MPixel sensor to transmit 16,000 frames per second.
Bridges: Concerning the broader high speed market, I think the desire for ever higher resolutions at speeds around or even exceeding one million frames per second – at a price that will not break the bank – is one area that is ripe for change. [Another] is ever improving light sensitivity, ideally with a standardized way of every manufacturer presenting their real light sensitivity to one standard such as ISO 12232 Ssat, as opposed to the ad hoc system some use now.