Aim

This project consists of developing spiking neural networks to process time-of-flight (ToF) data for 3D reconstruction from SPADS sensors.

Objectives

1. Identification of tractable network architectures to discriminate pseudo-periodic spatio-temporal patterns in spiking data
2. Extend architectures adapted to regular SPAD (single-photon avalanche diode) arrays to arbitrary networks of SPAD sensors (e.g., used as bucket detectors)
3. Investigate the use of such architectures for other modalities (e.g., neuromorphic cameras)

Description

Spiking neural networks (SNNs) can offer increased processing speed and reduced power consumption, especially when implemented on dedicated hardware (neuromorphic chips or FPGAs). Standard SNNs are typically fed with spiking data (e.g., streams of binary values), and the output of each layer remains spiking. Thanks to their ability to capture spatio-temporal patterns, such networks appear as promising candidates to process asynchronous streams of individual photon detection events, especially when those photons are labelled (e.g., with time-of-flights (ToFs) or colors). Current spiking architectures are not equipped with neuron models adapted to periodic patterns, which results in overly complex architectures.

In this project, we propose to leverage compact representations of spiking data via sketches, which enables label/ToF encoding for more efficient processing of single-photon Lidar data, overcoming the need for ToF histograms. We first propose to investigate such representations for 3D imaging with SPAD cameras, but plan to evolve towards imaging with a few SPAD sensors (capturing returns from the whole scene, with or without structured illumination). The methodology can be applied to other modalities, such as neuromorphic cameras, where the proposed architectures could be used to identify periodic patterns (e.g., propellers).

The findings of this project find application in a variety of imaging and detection scenarios, from long-range 3D imaging to UAVs detection. It a defence and security context, it will enable more light-weight architectures, easier to train (as needing less training data) and faster at inference time than non-spiking alternatives.