Aim

Co-designing advanced CI algorithms and computing hardware to reconstruct images with a new regime of precision, robustness, efficiency, and scalablity, enabling in particular addressing low-resource setting applications, with a specific focus on low-SWAP platforms.

Objectives

1. algorithm theory, towards interpretable solutions shipped with calibration and uncertainty quantification functionalities;
2. co-design of specialised high-performance computing hardware compliant with SWAP platforms;
3. validation for specific imaging modalities (3D, hyperspectral, depth, dynamic).

Description

Across science and technology, imaging devices are designed to observe scenes of interest and enable image-driven decision-making. An increasing number of modern modalities operate in low-resource settings, due to physical constraints or requirements to design low-footprint devices and requirements for low size, weight, and power (SWAP) platforms. This implies severe data incompleteness, calibration challenges, and measurement noise, leading to significant uncertainty about the image to be formed.

In this context, challenging mathematical inverse problems arise for image reconstruction, requiring advanced “Computational Imaging” (CI) algorithms to regularise the problem and simultaneously deliver precision, robustness, efficiency, and scalability. With current algorithms unable to deliver such capability, the project envisions a new paradigm at the interface of the deep learning and optimisation theories. The project will research our recent paradigm R2D2, standing for “Residual-to-Residual Deep neural network series for high-Dynamic range imaging”. R2D2 postulates a new framework synergistically generalising advanced plug-and-play approaches and diffusion models, which rely on learned regularisation denoisers.

Furthermore, general computing devices such as CPUs and GPUs have high power requirements for fast computational approaches, but in power constrained environments, careful design of the computing hardware hand-in-hand with the imaging algorithms, can reduce the power and energy required to produce high quality images quickly. Working with reprogrammable devices such as FPGAs, or custom processors for machine learning tasks, can enable sculpting both devices and algorithms that work together within the constraints of SWAP platforms.

R2D2low will thus span (i) algorithm theory, towards interpretable solutions shipped with calibration and uncertainty quantification functionalities, (ii) co-design of specialised high-performance computing hardware compliant with SWAP platforms, and (iii) validation for specific imaging modalities (3D, hyperspectral, depth, dynamic).

References

1. https://arxiv.org/abs/2210.16060 (R2D2 concept)
2. https://arxiv.org/abs/2403.05452 (R2D2 astro sensor array)
3. https://arxiv.org/abs/2403.17905 (R2D2 medical sensor array)
4. https://arxiv.org/abs/2403.18052 (R2D2 uncertainty quantification)