Introduction

Earth Observation (EO) from low Earth orbit satellites provide large volumes of data which are used in a diverse number of applications, including land cover monitoring1, food security2 and gas leaks3. Machine Learning (ML) techniques have become a critical tool for processing the enormous quantity of EO data4,5, with new ML models being designed for change detection tasks that specifically use EO data6,7. An area in which ML using EO data can have a significant impact is the management of natural disasters8,9,10. For the data to be most effective for first responders, low latency is required between data capture and its arrival to decision makers. One of the largest bottlenecks is the downlinking of data due to the limited bandwidth and coordination of available ground stations11. This problem is expected to grow in the future as more data is collected on board satellites due to advances in sensor technology (e.g. hyper-spectral cameras) and an increasing number of satellites12. One way to overcome this challenge is to process some of the data on-board and prioritise pertinent data to be downlinked, particularly for time-sensitive applications like natural disaster management. On-board processing will also be critical for future spacecraft deployed to remote sites in the solar system, or for monitoring solar weather in near-real-time.

Traditional algorithms have been developed to detect changes in EO images, for example BFAST13, which has been applied to detecting changes in forests14. Algorithms such as BFAST rely on spectral indices as input. These are a relatively simple pixel comparison method to highlight objects of interest such as burn areas15, diseased crops16 and flood mapping17. These indices are effective as they utilise domain knowledge regarding the object of interest. For example, the normalised difference vegetation index is commonly used to incorporate knowledge about the wavelengths that the Chlorophyll pigment absorbs18. More sophisticated indices, which involve more than just a normalised difference between two spectral bands, have been proposed to improve the detection15,19. The advantage of spectral indices is that they provide a simple, low computational cost method to determine objects of interest quickly, with reasonable accuracy. Additionally, they do not require training or large datasets to deploy, unlike ML models.

AI models based on deep ML, specifically Convolutional Neural Networks (CNNs), have been applied to several natural disaster tasks using EO data, including flood segmentation20,21, detecting earthquake affected buildings22 and volcanic eruptions23. To increase the performance of these AI models, often the number of layers and weights within the model needs to be increased. Chintalapati et al.24 showed how the architecture of CNN models has evolved to increase performance on an EO image classification task, which resulted in an increase in the number of weights in the models, with the two best performing models, CoCa25 and BASIC-L26, using 2.1 and 2.44 billion parameters, respectively. It was noted that most deep learning AI models have so many parameters that they would exceed the memory budget on most EO satellites. In recent years, work has been done to design smaller AI models specifically for use on-board satellites27,28. Furthermore, missions such as \(\Phi\)-Sat-129 and OPS-SAT30 have also designed satellites to accommodate AI models on-board. Although these models have been specifically designed for on-board use, all the training still occurs on the ground because of the large computational overhead. Only inference is performed on-board. Since AI models are trained on the ground, they often cannot be trained with the on-board sensor data, leading to degraded performance31,32. Work has been done to retrain models, once operational, with new data as it is acquired33,34.

In this study, we introduce the e Physics Aware Neuromorphic Network (PANN), a AI model with a novel architecture modelled after physical neural networks comprised of nano-electronic circuit elements known as memristors (nonlinear resistors with memory) that mimic synapses35,36,37. Like biological synapses, the memristive network weights continuously update in response to varying input signals, constrained by physical laws embedded within the model. The PANN, therefore, actively learns as new inputs are fed into the network, such as from a sensor on-board a satellite. As such, the PANN takes a significant shift away from the conventional artificial neural network AI paradigm, as the gradient-descent, back-propagation training of weights is completely replaced by physics-constrained dynamical equations. This is also in contrast to physics-informed neural networks, where the physics is embedded as a component of the model in the artificial neural network paradigm. Typically, this is done by modifying activation functions, gradient descent optimisation techniques, network structure, or loss functions38,39. The added physics normally takes the form of a partial differential equation and is task specific, for example, incompressible Navier-Stokes equations for fluid flow problems40, whereas with the PANN, the physics is independent of the task.

Physical memristive networks, which the PANN is modelled after, have demonstrated learning tasks such as image classification and sequence memory41, voice recognition42,43, long-/short-term memory and contrastive learning44, as well as regression tasks of varying complexity45,46,47 and chaotic time series prediction48. The PANN model has also been used to demonstrate these and other ML tasks, including transfer learning and multi-task learning49,50,51,52,53,54,55,56. These ML tasks were implemented using reservoir computing, where only weights in a single output layer need to be trained. This approach offers a substantial advantage over deep neural network models as it does not need computationally intensive training algorithms and thus does not consume as much energy. These properties make the PANN an ideal candidate for use on-board satellites.

In this study, we use a natural disaster dataset released by Růžička et al.28 to evaluate the performance of the PANN model on a change detection problem, that contains a time series of multi-spectral EO images from Sentinel-2. We benchmark the PANN against a state-of-the-art AI model and achieve comparable or better results. We also evaluate the performance of the PANN on a secondary dataset of the same natural disasters using multi-spectral EO images from LandSat-8, to compare the models performace on different sensors. Additionally, we use a distance measurement to detect changes, bypassing the need for an output layer altogether. Hence, we do not need to train any part of the model and the PANN does not need to be trained on the ground prior to deployment.

This article is organised as follows: in “Results” we show the results from the PANN model and compare them to a deep learning model for the same change detection task. We also show visualisations of the features the PANN model extracts. In “Discussion” we outline key findings from the results and their implication for use on-board satellites. Finally, “Methods” presents the details of the PANN model and workflow.

Results

The PANN model is used here to detect regions of change from natural disasters using EO multi-spectral images. The change detection task requires the model to take in a sequence of five images, where the final image is after a natural disaster event. The model outputs a change feature map that highlights the regions affected by the natural disaster, by breaking the images into smaller tiles and assigning a change likelihood score for each tile. The PANN model used in this study is an adaptive network, with weights that adjust with each new input. This allows the PANN to generate features from the inputs without requiring any training (see Methods for details). To determine if a tile has changed, the distance between inputs are compared in the feature space rather than the input space (pixel comparison). By applying a distance-based metric directly to the feature space, no training is performed at any stage in the model. The dataset includes four different categories of natural disaster events: fires, floods, hurricanes and landslides. Additionally, each natural disaster event is accompanied by a change mask which is used only to evaluate the performance of the PANN model.

Figure 1 shows the output change maps produced by the PANN model for each type of natural disaster event. Alongside the change maps are the RGB images directly before and after the natural disaster events and the associated change mask, which also includes clouds from the images directly before and after the natural disaster event. Although only one image is shown before the natural disaster event in Fig. 1, a total of four images before the natural disaster event were used to create the displayed change maps. For the four natural disaster events, the main areas of change have been correctly identified as changes in the change maps, with a limited amount of noise in the images being considered as a change, such as the vegetation colour change in the fire natural disaster event (top row), or the sediment being washed into the ocean in the flood natural disaster event (second row). Clouds are detected as changes, but are masked out of the change maps shown in Fig. 1, since they are ignored when evaluating the change maps to allow for a more direct comparison to a state-of-the-art deep learning model (see Methods for details).

Figure 1
figure 1

RGB images (columns 1 and 2) directly before and after a natural disaster event, the corresponding target change mask (column 3) and the output change map produced by the PANN model (column 4) for each type of natural disaster: Top row—an example fire natural disaster event; Second row—an example flood natural disaster event; Third row—an example hurricane natural disaster event; Bottom row—an example landslide natural disaster event. Cloud-cover has been masked from the images and corresponds to yellow in the change mask, with genuine differences coloured aqua. Satellite images are from Sentinel-2 complied by Růžička et al. and available in their git57.

Full size image

For each natural disaster event a change map was created, with the order in which the natural disasters were passed into the PANN model being randomised. We evaluated the quality of the change maps by calculating the Area Under the Precision–Recall Curve (AUPRC), which we create tile-wise. The change likelihood score for each tile is given by Eq. (3) in Methods, which depends on the distance metric used to measure the position of the tiles in the feature space. Table 1 reports the overall AUPRC as a percentage for each of the natural disaster categories using three different distance metrics, Euclidean, Cosine and Correlation distances. The Correlation distance significantly outperforms the other distance metrics and is also much more consistent across the different natural disaster classes.

Table 1 Comparison between different distance metrics used to calculate the PANN change score for different natural disaster classes.
Full size table

We benchmark our PANN model against a state-of-the-art variational autoencoder model58, called RaVAEn, which was specifically designed to be a relatively small AI model that could be deployed on-board satellites. We also compare the PANN model to a baseline method that uses the same change score equation but compares the different tiles in the pixel space rather than the feature space like the other two AI models. When comparing models, it should be noted that the PANN has an automatic feature engineering process incorporated into the model setup that is not present in the RaVAEn model or the baseline method. Table 2 reports the performance of the three different models for each of the natural disaster classes. Both AI models, RaVAEn and PANN, outperform the baseline method in all classes. The RaVAEn and PANN models achieve comparable results in the fires, hurricanes and landslides, with the PANN slightly ahead for the fires and hurricanes, while RaVAEn is slightly ahead for the landslides. With the flood natural disasters, the PANN achieves the highest results by a significant margin with a score 14% higher than that of RaVAEn.

Table 2 Comparison between change scores predicted by different models for different natural disaster classes.
Full size table

To assess how well the PANN model generalises to data from different sensors, we evaluated the model on a subset of the natural disaster events using EO images from LandSat-8 and compared the results to the Sentinel-2 data (see Methods for details). Table  3 reports the performance of the PANN and baseline models for the LandSat-8 data and the comparative results for the equivalent subset in the Sentinel-2 data. From the results it can be seen that the PANN model outperforms the baseline method for both the data from the Sentinel-2 satellites and the LandSat-8 satellite for every natural disaster category. Both models achieve higher performance when using data from Sentinel-2 compared to LandSat-8. Additionally, the difference in performance between datasets is much larger for the baseline methods, which shows a significant decrease in performance when using the LandSat-8 data.

Table 3 Comparison between change scores for different sensors by natural disaster classes for the PANN model and baseline method.
Full size table

Feature space

The AI models RaVAEn and PANN outperform the baseline method, since they can extract the most relevant features from the tiles and group similar tiles together, while keeping dissimilar tiles further apart in their feature spaces. To visualise the high dimensional feature space of the PANN, we used the Uniform Manifold Approximation and Projection (UMAP)59 method to reduce the dimensions of the feature space into 2D. The UMAP used 20 neighbours, a minimum distance of 0.4 and the correlation distance, unless otherwise stated. Figure 2 shows the UMAP 2D projection of the feature space for a flooding event. The left-hand side shows the position of the tiles in the image directly before and after the flooding event. For tiles after the flooding event, the percentage change in each tile is denoted by the colour bar. The right-hand side of the figure shows the same projection of the feature space but with the tile images instead of points. The change tiles are grouped together at the bottom of the projections and the remaining tiles are also sorted according to the features in the tiles, with greener farmland tiles on the left-hand side and tiles with the white objects on the right. Tiles that have missing values, from the north alignment of the swath, are also grouped towards the bottom right, with the tiles that have the missing values across the top being separated from the tiles with the missing values on the right-side of the tile. For comparison with the UMAP projection for the pixel space, see Supplementary Fig. S1 online.

Figure 2
figure 2

2D UMAP projections of the PANN feature space, using the correlation distance for a flood natural disaster event: Left—positions of before (blue) and after (orange) tiles, with percentage change in the after tiles indicated by the colour bar; Right—the corresponding tile images. The tiles with flooding occupy a distinct arc-like cluster at the bottom of the projection.

Full size image

Like most dimensionality reduction techniques, UMAP aims to reduce the number of dimensions while maintaining the overall structure and information of the original, higher dimensional space and thus depends on the distance metric used to measure the distances between points. Figure 3 shows the UMAP projection of the feature space for a hurricane natural disaster for the three distance metrics used in Table 1 and shows how the clustering of the tiles varies depending on the distance metric being used. This provides a visual qualitative tool to understand the large range in results reported in Table 1 due to the change in distance metric being used.

Figure 3
figure 3

Visual comparison between the 2D UMAP projection of the PANN feature space for a hurricane natural disaster using three different distance metrics: Euclidean, Cosine and Correlation distances.

Full size image

Discussion

In this study, we introduced the PANN model for a natural disaster change detection task and compared it against the state-of-the-art AI model RaVAEn and a baseline method. All three models receive the same multi-spectral EO images for the inputs and use the same change detection equation given by Eq. (3), for each tile in the image. The difference in the model performances, therefore, is attributed to their ability to extract the most relevant features from the inputs, while ignoring noise within the inputs. We note, however, that while the PANN model receives the same inputs as the other models, only the inputs corresponding to the bands selected by the automatic feature engineering process, on a scene by scene basis, are passed to the relevant individual networks. This inherently allows the PANN model to ignore irrelevant information as determined by the domain knowledge embedded in the feature engineering process. The baseline method does not extract any features from the inputs and instead does the comparison directly in the pixel space. This makes the baseline method more sensitive to noise in the images and, as expected, it exhibits the lowest performance as reported in Table 2. The AI models, on the other hand, extract meaningful features from the tiles and hence outperform the baseline method in every natural disaster category. The PANN and RaVAEn models achieve a comparable result in three of four of the natural disaster categories, demonstrating that both models extract equally useful features for these categories. For flood natural disasters, however, the PANN model outperforms the RaVAEn model and thus arguably extracts more meaningful features.

Although being able to extract the features is the most important task for the PANN model, how the feature space is navigated is also important for the performance on the PANN model. This is seen in Table 1 where the same feature space can produce significantly different results based on the distance metric used to navigate the feature space. This is visualised in the UMAP projections in Fig. 3, where different distance metrics are used to create the projections. The feature space produced by the PANN has a high dimensionality, as such the relative Euclidean distance between points tends to decrease60. This would result in the change likelihood score decreasing, particularly for change tiles and is likely why the Euclidean distance does not produce the best results. What is surprising is the large difference between the Cosine and Correlation distances, given that both metrics are angle-based measures. The Cosine distance is given by Eq. (4) and measures the angle between the vectors in the feature space. The Correlation distance is the same as the Cosine distance except each sample is mean centred first, as given by Eq. (5). The mean-correction for the Correlation distance effectively changes the location of the origin and hence the angle between the vectors also changes61. One possible reason why the mean correction has such a large impact on the performance is that it helps reduce covariate shift in the output sequence that can arise from the dynamical nature of the PANN. The covariate shift effect can be seen in Fig. 3 (and Supplementary Fig. S2), where there is a much greater overlap of points at the different timesteps (that are not changes) in the Correlation projection compared to that of the Cosine projection. Interestingly, batch normalisation layers were introduced to deep neural networks to help reduce the covariate shift between layers in the networks62 and have been widely accepted to speed up training times and increase performance. We note that the PANN does not have any batch normalisation layers, while the RaVAEn model does, and that the best performing metrics for the models are the Correlation and Cosine distance, respectively.

Figure 4
figure 4

Comparison between the change maps produce by the PANN model using the Sentinel-2 (S2) and LandSat-8 (L8) dataset, for the same landslide natural disaster event. The corresponding change masks are also included.

Full size image

From the results in Table 3, two clear trends are evident. The first is that both the PANN and baseline methods perform better when using data from Sentinel-2 than LandSat-8 across all natural disaster categories. This can largely be explained by the decrease in resolution. The Sentinel-2 data has a spatial resolution of 10 m, with a tile size \(32 \times 32\) pixels and a tile resolution of 320 m. While the LandSat-8 data has a spatial resolution of 30 m, with a tile size of \(16 \times 16\) pixels and tile resolution 480 m. As a result, the models are not able to resolve finer details when determining if a change has occurred in a LandSat-8 tile. This difference in resolution can be seen in Fig. 4, which shows the change mask and output change map for both datasets for one of the landslides. Additionally, in Fig. 4, more noise is evident in the LandSat-8 change map, compared to the Sentinel-2 change map, particularly in the areas where there is no change. This would also contributes to the decrease in performance. The second trend that can be clearly seen in Table 3, is that the decrease in performance using the LandSat-8 data relative to the Sentinel-2 data is significantly less for the PANN model compared to the baseline method. This shows that the PANN model is able to generalise to different sensor data much better than the baseline method, further showing that the PANN model is able to extract meaningful features from the input data.

Our results show that the PANN model achieves comparable-to-better results than the RaVAEn state-of-the-art AI model for the natural disaster change detection task conducted in this study. Most importantly, the PANN was able to achieve this result by creating a mapping from the input space to the feature space without any training. The PANN is training-free as it continuously adapts with each new input. The model equations are relatively light–weight and thus all the processing of the data by the PANN model was able to be carried out using only CPUs in this study. The compute resources, time and memory, were measured for the PANN to process a patch sequence of \(574 \times 509\) pixels over 5 frames, corresponding to roughly a 25 \(\hbox {km}^2\) region. Using 8 cores on an Intel Xeon Scalable processor, the PANN model takes 19 s and uses approximately 1 GB of memory (see Supplementary Fig. S3). We note that the model could be optimised to further decrease memory requirements by reducing the floating-point precision of the PANN from 64 bit to 32 or 16 bit precision. We also note that the processing time for the PANN model is not directly comparable to the inference time of RaVAEn or other AI change detection models, as the processing time to evaluate a scene also includes updating of the PANN weights with each input. Unlike AI models, the PANN model does not require any additional time and compute resources for training. The physics–based equations that constitute the PANN model are a significant deviation from conventional deep multi-layered bipartite neural networks that typically use mathematical activation functions to perform nonlinear transformations and stochastic gradient decent with backpropagation as the learning algorithm to train networks weights into an optimal static state. As such, the iterative training process is computationally expensive and usually requires cloud-based GPUs63, to produce a trained model. As the PANN simulates a physical device, based on memristive nanowire networks, we expect the computational resources to significantly decrease when tasks are implemented on the physical hardware (see e.g.41,44,48,51,64).

The PANN model is relatively sensor agnostic since it does not need to be pre-trained with data from a specific sensor and the model design of having a separate network for each spectral band (see Methods for details), allows it to receive images with varying number of spectral bands. Having a separate PANN for each band also allows a simple training–free method to add domain knowledge to the model in the form of an automatic feature engineering mechanism that determines which spectral bands and hence PANNs are used (see Methods for details). In this study, only relatively minor adjustments to the feature engineering mechanism were required when changing the sensor data from the Sentinel-2 satellite to the LandSat-8 satellite. In addition, having a separate PANN for each band combined with the feature engineering means only a subset of the PANNs are required, further reducing the computational resources. Having multiple smaller networks, however, means that interactions between bands may not be fully captured, as compared to a single larger network, where information from all the bands is integrated while being processed. Alternatively, another model design that could be considered is multiple networks with a hierarchical structure65.

The training-free nature of PANN makes it an ideal candidate for extreme edge computing applications, particularly for EO analysis on-board satellites. Specifically, there are two main challenges that are inherently addressed by the training-free nature of the PANN. The first is that ML models typically do not generalise well and need to be trained for a specific sensor. When ML models are trained using data from a different sensor, there is often a notable decrease in performance31,32. Furthermore, training a ML model with only simulation data can be difficult24. The training-free nature of the PANN therefore makes it a promising candidate to circumvent this issue. The second is that ML models are often retrained as new data from the desired sensor become available to increase performance and to account for any data-shift problems. This process requires the new data to be downlinked, to retrain the model on the ground before uplinking the updated model to the satellite33. The dynamical nature of the PANN would allow this whole process to be bypassed as the PANN inherently updates and learns from each new input from the sensor.

In this study, we have demonstrated the feasibility of the PANN for detecting changes from natural disasters within the context of their potential deployment on-board satellites. Based on these simulation results, future work would include testing a prototype of the hardware device, based on memristive nanowire networks, which the PANN is modelled after. Based on previous studies with such neuromorphic devices 41,44,48,51,64, it is expected that implementation on the low-power hardware device will substantially reduce processing times and energy consumption, and could be achieved using only the on-board compute. The methodology outlined in this study could be used when implementing in hardware with one key change: the physical network devices may need larger electrode grids than the \(16 \times 16\) grid used here, as additional electrodes would be needed to read out voltages from the larger physical networks. To date, devices with up to 128 electrodes have been fabricated 66, although interestingly, experimental results suggest that using readouts from only a subset of electrodes may be sufficient to capture the most salient output features 41. This may be attributed to how the electrodes integrate signals from many nanowires they make electrical contact with. This means that when implementing in a physical device, the option of using one large network that processes all the spectral bands together may be viable. Multiple configurations for the electrode placement would be possible, provided the contact electrodes evenly sample the underlying nanowire network.

In conclusion, we demonstrated a training-free AI model, the PANN, for a natural disaster EO change detection task that contained four types of natural disasters: fires, floods, hurricanes and landslides. We benchmarked our model against a state-of-the-art AI model and achieved better results for floods and comparable results in the remaining three categories, without any training. We also showed that the PANN model can generalise to other EO sensors. We found that the PANN was able to achieve these results due to its ability to extract meaningful features from the inputs and sort them accordingly in the feature space. Being able to navigate the feature space, therefore, plays a vital role in the model’s performance. The training-free nature of the PANN makes it an ideal candidate for use on-board satellites, processing EO data at the extreme edge.

Methods

Data

The primary dataset used in this study is a time series of multi-spectral EO of natural disasters released by Růžička et al.28. The images are level 1C pre-processed multi-spectral images from the Multi-Spectral Imager67 camera on-board the Sentinel-2 Earth observation satellite constellation. Only the 10 highest spatial resolution bands are in the dataset. All bands have a spatial resolution of 10 m. The bands that are not naturally at 10 m resolution were interpolated to the higher resolution. Table 4 shows the specific bands used in this study along with the corresponding index number used to reference the bands. The dataset covers four categories of natural disasters: fires, floods, hurricanes and landslides. Each category includes five natural disaster events, except for floods which have four, giving a total of 19 natural disaster events. The individual samples were created by tiling each event, with no overlap, into \(32 \times 32\) pixel sequence tiles, resulting in a total of 62,655 samples used to evaluate the PANN model. For each natural disaster event there are five sequential images; the first four are all before the natural disaster event and the final image is after. Along with each natural disaster event, there is an accompanying change mask that contains three labels: unaffected regions, affected regions, and clouds from the fourth and fifth images. An example of a hurricane disaster event is shown in Fig. 5, where damage to the island vegetation can be seen in the fifth image and the change mask image shows the unaffected regions in purple, the affected region in aqua and the clouds in yellow. The change mask in this study is only used to evaluate the performance of the proposed model.

In this study, we created a second dataset to evaluate the performance of the PANN from a different sensor, the LandSat-8 Operational Land Imager68. This was done by selecting the natural disaster events in the Sentinel-2 dataset that did not have any clouds obscuring the change regions in the change mask. This left 8 natural disaster events: 2 fires, 2 floods and 4 landslides. The Top of Atmosphere images from the LandSat-8 satellite with the 9 highest spatial resolution bands of 30 m were used. To create the time series of images for each natural disaster event, only images with 40% cloud cover or less were used. Like the Sentinel-2 dataset, each natural disaster event has five sequential images, with the first four before the event and the final image is after. The corresponding change mask from the Sentinel-2 dataset was used after resizing to 30 m resolution using the nearest-neighbour method and adding the clouds in the fourth and fifth images only. Clouds were detected automatically using the Google Earth Engine LandSat simple cloud score algorithm69, with a score \(\ge 55\)% considered a cloud. As with the Sentinel-2 dataset, individual samples were created by tiling each event, with no overlap, into \(16 \times 16\) pixel sequence tiles, resulting in a total of 8,199 samples.

Table 4 Sentinel-2 and LandSat-8 bands used in this study and the corresponding index used to reference each band.
Full size table

The data was preprocessed by first taking the log of the RGB pixel values and then scaling the values to the interval [-1, +1] for each band in the data. This process is described by Eq. (1):

$$\begin{aligned} x^\dagger = \log (x) \qquad , \qquad x^\ddagger = 2 \left( \frac{x^\dagger -\text{ min }(x^\dagger )}{\text{ max }(x^\dagger ) - \text{ min }(x^\dagger )} \right) -1 \end{aligned}$$
(1)

where the min and max values for each band are fixed and were chosen to be consistent with Ref. 28 for the Sentinel-2 dataset, to allow a direct comparison with their state-of-the-art variational autoencoder model. A different set of min and max values were manually selected by visual inspection for the LandSat-8 dataset. Missing values were set to a value just above zero of 0.005.

Figure 5
figure 5

An example of a hurricane natural disaster scene, with the first 4 images before the hurricane and image 5 is after. The change mask (last column) shows the affected (aqua) and unaffected (purple) regions along with the clouds (yellow) in the images directly before and after the hurricane. Satellite images are from Sentinel-2 complied by Růžička et al. and available in their git57.

Full size image

Model architecture

The PANN is based on a physically-motivated model of a physical neuromorphic network comprised of self-organised nanowires with resistive switching memory (memristive) electrical junctions 49. This is done by first modelling a nanowire network and then converting it into a graph representation which is used in the PANN. This process is shown in Fig. 6. In this study, the network for the PANN was created by distributing 803 nanowires over a \(158 \times 158 \, \mu\)m 2D plane with nanowire centres sampled from a generalised normal distribution with a beta value of 5 and with nanowire orientations sampled on \([0,\pi ]\). Nanowire lengths were sampled from a Gaussian distribution with an average of \(30 \, \mu\)m and a standard deviation of \(6 \, \mu\)m. As nanowire networks are nano-electronic systems, input signals are delivered via electrodes, which were modelled with a diameter \(5 \, \mu\)m and with evenly spaced placements in a \(16\times 16\) grid over the 2D plane with a margin of \(15 \, \mu\)m, giving a pitch of \(8 \, \mu\)m between electrodes. Where the nanowires overlap with other nanowires or electrodes, electrical junctions are formed whose internal states evolve in time, as described below. Figure 6a depicts the nanowire network model, with the nanowires displayed as grey lines and nanowire–nanowire junctions as small grey dots. The electrodes are shown as the large green circles and the nanowire-electrode junctions are shown as small dark green dots. Figure 6b shows the corresponding graphical representation of the network used in the PANN. The black nodes correspond to the nanowires and the green nodes correspond to the electrodes, used as input nodes. The edges correspond to the electrical junctions. The resulting network had a total of 1059 nodes and 12,279 edges. 256 nodes were selected as input nodes and from the remaining 803 nodes, 400 were selected at random as output nodes (note that in a real physical device, electrodes would also serve as output nodes).

Figure 6
figure 6

(a) PANN model nanowire representation. The nanowires are the grey lines and the electrodes are the large green circles. Nanowire–nanowire junctions are shown as grey dots and nanowire–electrode junctions are shown as dark green dots. (b) graph representation of the PANN. Green nodes are the input nodes which correspond to the electrodes. Black nodes correspond to the nanowires, while edges correspond to the nanowire–nanowire and nanowire–electrode junctions.

Full size image

The network in the PANN has a heterogeneous, neuromorphic topology and operates like a complex electrical circuit with nonlinear circuit components known as memristors53,70. As such, inputs are treated as input voltage signals and Kirchhoff’s conservation laws are solved at each time step to calculate the node voltage distribution across the network. The edge weights are conductances that evolve in time according to a memristor equation of state:

$$\begin{aligned} \frac{\text {d}\lambda }{\text {d}t} = {\left\{ \begin{array}{ll} \left( |V(t)| -V_{\text {set}}\right) \text {sgn}[V(t)] \; , & |V(t)| > V_{\text {set}} \\ 0 \; , & V_{\text {reset}}< |V(t)|< V_{\text {reset}} \\ \left( |V(t)| - V_{\text {reset}}\right) \text {sgn}[\lambda (t)] \; , & |V(t)| < V_{\text {reset}} \\ 0 \; , & |\lambda | \ge \lambda _{\text {max}} \\ \end{array}\right. } \end{aligned}$$
(2)

with all edges set to an initial state \(\lambda (t=0) = 0\). In this study, the following parameters were used: \(V_{\text {reset}} = 5\times 10^{-3}\,\)V, \(V_{\text {set}} = 1\times 10^{-2}\,\)V and \(\lambda _{\text {max}} = 1.5\times 10^{-2}\,\)V s. Full details of the model and its validation against experimental nanowire networks are provided in Refs. 51,53. In contrast to artificial neural networks, the dynamic nature of these physical neuromorphic networks means that the weights are not trained, which not only makes them more energy efficient, but also allows all the data to be used for evaluating the model.

Model setup

Figure 7 shows the process of feeding the natural disaster image sequence into the PANN model. First an automatic feature engineering step determines which bands to use for that specific natural disaster event. This is motivated by a physical understanding of the event: for example, water exhibits the greatest contrast compared to land at infrared wavelengths. The images are then subdivided into a series of \(32\times 32\) pixel sequence tiles, \(x^{a,b} _N(t)\), where a, b is the location of the tile and \(N \le 10\) is the number of bands as determined by the feature engineering process. The images are tiled column by column starting at the top left hand corner. The tile sequence is passed through a max pooling layer that has a pooling size (2,2), a stride of 2 and does not use any padding, hence halving the spatial resolution of the tile sequence to \(16\times 16\) pixels. The tile sequence is then split into individual bands, creating the input signals \(U^{a,b}_m (t)\), where m is the index to each PANN for each spectral band as given in Table 4. As such, there are a total of 10 identical PANN networks, each taking inputs for a specific band. The readout values of the PANNs used for the given tile sequence are concatenated to give the final features, \(F^{a,b}(t)\) for that tile location. Hence, not all 10 PANNs are used for every natural disaster event, as demonstrated in Fig. 7, where only PANN numbers 01-03 and 09 are used and the greyed out PANNs (PANN 04 – 08 and 10) are not used for this natural disaster event.

Figure 7
figure 7

Diagram of the PANN workflow showing the pipeline from the input images to the output features, \(F^{a,b}(t)\) from the networks. First, automatic feature engineering is applied to the input images to determine which bands of the images to use. The images are then broken into a sequence of small \(32\times 32\) pixel tiles, \(x^{a,b} _N(t)\), where a, b is the location of the tile and N is the number of bands. The creation of the tile sequence is shown by the coloured (red, orange, yellow, lime and green) boxes and are then fed into the max pooling layer. The bands are separated, as denoted by \(\bigotimes\), with each band being fed into a unique PANN model. In this example \(\hbox {PANN}_{01-03}\) and \(\hbox {PANN}_{09}\) are used. The outputs from all the PANNs that are used are concatenated together, as denoted by \(\bigoplus\), to produce the feature sequence \(F^{a,b}(t)\) for tile \(x^{a,b} _N(t)\). Satellite images are from Sentinel-2 complied by Růžička et al. and available in their git57.

Full size image

To determine whether a change from a natural disaster occurred at a given tile location, the distance between the images of the tile sequence is measured in the feature space. It is assumed that if there is any change it is in the last image of the sequence, so only the distances between the final image and each of the previous images are compared. This is the same change score method as introduced in Ref. 28. Formally, the change score, S, is given by

$$\begin{aligned} S(x^{a,b}(t)) = \min _{i=1,2,3,4} \left[ \text{ dist }(F^{a,b}(t-i),F^{a,b}(t)) \right] \qquad , \end{aligned}$$
(3)

where dist is an arbitrary distance metric. In this study we focus on three different distance metrics: Euclidean, Cosine and Correlation. Unless otherwise stated, the Correlation distance was used. The Cosine distance is an angle-base metric and is given by

$$\begin{aligned} \text {CosineDist}({\textbf{u}},{\textbf{v}}) = 1 - \frac{{\textbf{u}}\cdot {\textbf{v}}}{||{\textbf{u}}|| * ||{\textbf{v}}||} \qquad . \end{aligned}$$
(4)

The Correlation distance is the mean corrected version of the Cosine distance, given by

$$\begin{aligned} \text {CorrelationDist}({\textbf{u}},{\textbf{v}}) = \text {CosineDist}({\textbf{u}} - \text {mean}({\textbf{u}}),{\textbf{v}}- \text {mean}({\textbf{v}})) \qquad . \end{aligned}$$
(5)

Distances between the tile sequence images are compared directly in the features space, so that it is not necessary to learn a mapping function from the feature space to a target variable. This means it is not necessary to train any part of the model.

To evaluate the quality of the change maps produced by the PANN we use the AUPRC71. To calculate the curve, we treated each pixel in the output change maps as a positive or negative example, across all natural disaster events in that category. As such, a separate precision-recall curve was produced for each class of natural disaster and an AUPRC value. Areas labelled as clouds in the change mask were ignored when calculating the AUPRC. This was done to be consistent with Ref. 28 and to allow for a more accurate comparison between the PANN and RaVAEn models.

Automatic feature engineering

The automatic feature engineering process is performed once for each natural disaster scene and is performed first, before the normalisation and tiling stages. The feature engineering process leverages common indices such as the normalised burn ratio index72 and the normalised difference vegetation index73 to calculate natural disaster class scores. Using these scores, a binary decision tree is followed, as shown in Fig. 8, to determine what class of natural disaster is in the scene. The score for each natural disaster class is calculated by creating an index image for the images directly before and after the natural disaster event using Eq. (6), where BX and BY are predetermined bands depending on what type of natural disaster is being assessed. The images are converted to a binary image using a predefined threshold value and a difference image is created by a pixel-wise comparison. The final score is the percentage of pixels that record a change. Once a natural disaster type has been selected, the feature engineering process then selects a subset of bands to be used in the rest of the model depending on the natural disaster class. The specific values and bands used for the natural disaster class score are shown in Fig. 8. See Supplementary Fig. S4 online for the specific values and bands used for the LandSat-8 data.

$$\begin{aligned} \frac{BX - BY}{BX + BY} \end{aligned}$$
(6)
Figure 8
figure 8

Decision tree for the automatic feature engineering step. At each branch a natural disaster score is calculated for the different categories. For each score the bands and threshold values used to create the index image are given. The bands used for each natural disaster type once selected are also shown.

Full size image

We note that more sophisticated index images74,75 could be used to categorise the natural disasters and potentially further improve results. Furthermore, similar feature engineering processes could be used to tune the model to detect changes from specific natural disasters or even clouds. The focus here is to show that the simplest automatic feature engineering process, one that does not require training, can be used to easily incorporate domain knowledge and improve the performance of the model.