Revolution in microscopy: Roarke Horstmeyer on multi-camera array microscopes

The Department of Electrical and Computer Engineering (ECE) hosted a seminar on Sept. 26 to showcase the research conducted by Assistant Professor of Biomedical Engineering at Duke University Roarke Horstmeyer. The talk, titled "Computational 3D Video Microscopy with Multi-camera Arrays," explained the design and algorithm behind the state-of-the-art multi-camera array microscopes (MCAMs) and several use cases. The findings were published recently in Optica.

Horstmeyer's research motivation traces back to his experiences working as a postdoctoral in a neuroscience lab, where one of the tasks involved was to observe the movements of zebrafish. To accurately capture and record these movements, there is a need for high-speed, high-resolution 3D video with a broad field of view (FOV).

Traditionally, these competing trade-offs are mitigated with large and expensive lenses and detectors. Horstmeyer's solution to this problem is the use of MCAMs.

“The first step is to take a standard optical microscope and shrink it down to a small package, which is relatively easy to do, and then we pack them in an array. In the simplest use case, each camera captures a unique point of view... We can stitch all those images together into a large composite frame,” Horstmeyer explained.

Perfecting the design required years of effort and consideration of multiple factors. After numerous design iterations, his team settled on an iteration that synchronized all cameras, providing a high imaging speed, high resolution and full coverage of a vast FOV.

“The FOV of this updated version is approximately the size of an index card, a format commonly used in biology known as the well-plate format,” he added, “In the design, we put 54 micro-cameras, each one 13 megapixels, like the ones you have in your iPhone. We normally record 10 frames per second and can go up to 240 frames per second by trading off resolution.” 

Synchronization between micro-cameras is also performed to facilitate the process of image stitching. Furthermore, machine learning algorithms are developed to track objects, reducing the volume of data that needs preservation.

Remarkably, the team didn't stop at this juncture but delved further into extracting 3D information from 2D image captures, incorporating depth information alongside RGB color data. To obtain a height map, the team employed a technique called “computational orthorectification” to reverse the intrinsic image distortions.

These ground-laying efforts have already rendered a series of novel applications both in academia and industry. One such application revolves around 3D video capture of zebrafish hunting paramecia. In a video Horstmeyer showed during the talk, machine learning algorithms precisely monitored the eye movements of the zebrafish when hunting and overlaid them on the raw video.

“The red things [in the video] are the eye direction that we are tracking,” he explained as he pointed at the video, “Generally the measurement is when they want to hunt, their eyes converge, so it's looking both eyes at the prey before it hunts.”

This technology is valuable not only for understanding animal behavior but as it also offers crucial insights to researchers, who can now directly observe the effects of drugs on the behavior of test subjects. The same technology lends itself to the examination of macrophages, antibody cells and the body's responses to infections.

Another remarkable application lies in the realm of intraoperative rapid on-site evaluation of lung fine needle aspiration procedures. Multiple instance learning played a pivotal role in quickly training models and calibrating classifiers to ensure accurate results.

Horstmeyer’s research has been of considerable interest to researchers and students at Hopkins, with attendees including undergraduates, doctoral students — many from the Department of Electrical and Computer Engineering (ECE) — and faculty. 

Associate Professor in the Department of Biomedical Engineering (BME) at Hopkins Nicholas Durr, who also hosted the seminar, explained Horstmeyer’s connections to Hopkins in an interview with The News-Letter.

“Professor Horstmeyer has met with professors from ECE, BME, Data Science, the Wilmer Eye Institute and [the School of Medicine’s] Pathology [programs], who were all interested in different aspects of his work,” he said. “The talk not only brought together a diverse group of researchers in the same room for discussion but also gave students some perspective about where scientific fields are moving. Trainees should think about contributing to the world as it will be five and 10 years from now, not a world 10 years ago they [might’ve understood] from textbooks.”