Fig. 4: Fully hardware implementation of the neuromorphic visual system based on the multimodal resistive memory arrays with ORRAM mode array for in-sensor pre-processing and the ERRAM mode array for near-sensor high-level processing.

a Architecture of the fully hardware-implemented neuromorphic visual system composed of an ORRAM mode array, an ERRAM mode array, a memory array peripheral control system and an STM32 system. b Photo of the fully hardware-implemented neuromorphic visual system. c In-sensor convolution operation executed in the ORRAM mode array. d Demonstration of in-sensor image feature extraction through in-sensor convolution operation and image erasing through NPM effects in the ORRAM mode array. e Demonstrations of in-sensor image denoising and contrast enhancement through PPM and NPM effects in the ORRAM mode array. f Convolution implementation in the ERRAM mode array. Confusion matrices of the classifications of “bear”, “fish”, and “butterfly” images (g) before and (h) after 50 training epochs, yielding an accuracy of 97.3%. i The conductance distributions (weight maps) in the ERRAM array under the initial state and different training epochs. j Comparison of the image recognition results with and without the ORRAM mode array for pre-processing. k Comparison of the image recognition results implemented through hardware and simulation. l Image recognition capability of the full hardware-implemented visual system. The images of the bear, fish, and butterfly were labelled by the digit of 0, 1, and 2, respectively. These images can be recognised by our neuromorphic visual system with high accuracies.