Background & Summary

Background

In the last decade, the convergence of big data, automation, and artificial intelligence (AI) has made the possibility of building- level mapping at global scale a reality. In step with these technological advancements, the value of large-area building mapping continues to grow across sectors. Government agencies and NGOs rely on accurate locations and descriptive information about buildings and related occupancy for emergency preparedness, response, and recovery actions associated with natural and anthropogenic hazards1,2,3. Within urban planning, real estate, and insurance industries, detailed spatial and attribute information informs mobility4, site selection5, and risk applications6. Climate and energy are intrinsically linked, and the morphological and functional aspects of the built environment play an important role in understanding the role of energy efficiency on future climate impacts and mitigation strategies7,8. Across all sectors, practitioners are moving toward creating and enriching digital twins with increased realism, to improve Building Information Models9, to simulate urban dynamics10,11, and to test what-if scenarios in a safe digital space12.

A specific catalyst for the work described here was the requirement to map any building(s) in the world to be used as foundational input to geospatial models of human settlement and associated population counts, particularly the Oak Ridge National Laboratory (ORNL) LandScan family of gridded population data. Initially, a support vector machine (SVM) method was employed to extract settlement features with an output resolution of 8˜ m from high resolution imagery13,14. However, SVMs do not generalize well to large, diverse target areas15,16. As such, training data and models were developed per image scene, which becomes impractical for global-scale mapping applications. As an example, hundreds of models were developed in order to map all the settlement areas for the country of Nigeria. Further, in many cases the output resolution of 8 m was too coarse to adequately represent between-building spacing and variability of this factor between settlements. Although this approach could still produce a high quality “footprint” of human settlements, this variability had a non-negligible impact when developing estimates of population within those settlements.

With the maturation of deep learning for remote sensing applications17,18,19, we moved to a Convolutional Neural Network (CNN) method for building extraction20. Unfortunately, a significant amount of labeled training data is required to learn an accurate model. SpaceNet21 and Replicable AI For Microplanning (ramp)22 offer high resolution labeled training data and accompanying reference imagery for 10 and 22 population centers, respectively. However, these datasets are partial to heavily populated areas. To work toward a model that could be generalizable to the variety of geographies across the globe, a robust training library needed to capture that variability in terms of structural, terrain, and imagery characteristics. To date, there are no open-source datasets that can provide the quantity, quality, and variability available in the labeled training data library that is presented here. ORBITaL-Net23 consists of 128,000 500 × 500 pixel chips and 1,490,000 hand-labelled buildings drawn from 9,800 distinct high resolution satellite images. Figure 1 shows the spatial distribution of this dataset along with SpaceNet and ramp.

Fig. 1
figure 1

Comparative distribution of source image locations for the ORNL, SpaceNet, and ramp open building training datasets.

Full size image

Toward a generalizable model

For the last decade, CNNs have been at the forefront of computer vision and now enable the creation of highly capable remote sensing feature extraction models that can operate over complex datasets of a size impossible using earlier techniques24,25,26. However, the generalization ability of a CNN model depends largely on the amount and quality of the training data used in supervised learning settings27. More complex CNN models tend to result in better overall performance, however, the number of parameters also increases as the complexity of a deep feature learning models grows. This demands more labelled training samples in the model optimization and learning process. The remote sensing community is aware of the need for an open, comprehensive labelled training dataset to leverage the advances made in artificial intelligence (AI) and enable large-scale mapping with high resolution earth observation data. As a result, a number of community collaborations and open data initiatives have lead to the development and availability of several impactful labelled training datasets, such as Functional Map of the World28, SpaceNet Challenges29, BigEarthNet30., xView Challenges31,32.

Training CNN models with such a massive amount of data is also computationally expensive, which largely limits the impact of those labelled data. The transfer learning strategy has been adapted, to take advantages of the pre-trained models without duplicating the efforts and time to build a model from scratch. The labelled data we created in this work provides a base model for building mapping for anywhere in the planet.

Supervised building extraction models could be over-fitting to the training set. For example, we observe that a given trained model can perform very well on a set of training labelled samples, however the model fails to generalize well (i.e. providing good or comparable accuracy on new/unseen labelled samples) as the models only memorize the training samples rather than learn the underlying feature patterns. With the goal of creating a generalizable model for extracting buildings across very large geographic areas from high-resolution satellite images, we recognized the need to build a labelled data set to cover variability in building appearances and image characteristics. These differences often present challenges to a pre-trained model as these models do not often “see” the rare representations of buildings and domain shifting of the pixel values under the impact of various imaging conditions. Diversifying training data has been effective in improving model generalizability. This strategy can expose those edge cases to the models during training and optimization and reduce the model bias. This strategy can also help fit models with an enormous number of parameters, as recent studies show that larger models with higher capacity need abundant data to reach their full potential33. The performance gains from increasing model size diminish without sufficient training examples.

Diversity of training samples

In order to ensure enough positive sample variability existed in the labelled training set, a variety of buildings with varying uses (e.g. residential, commercial, industrial), construction materials (e.g. mud bricks, steel, wood, straw), and shapes (e.g. square/rectangular, circular, irregular) were identified and introduced to the model. With sufficient quantity, this variability increased generalizability across large, diverse areas. As an example, a single model was used to automatically extract buildings from the countries within the Arabian Peninsula–an area of approximately 3.1 million sq km.

Core to our direction of producing a globally generalizable model is the development of a training dataset that includes variability in both image acquisition contexts and vernacular architecture and cultural settlement patterns. Key variables that can affect the appearance of include the viewing angle of the image sensor, which affects the amount of building facade visible, and the sun elevation, which affects scene contrast and presence and degree of shadows. Some challenging image acquisition conditions are illustrated in Fig. 2. The considerable variation in architecture, settlement configuration, and land cover contexts within a single country is illustrated in Fig. 3, which highlights three of the numerous regions of Ethiopia. On the left hand side of each image pair is a training sample and on the right the output from our current global model for a nearby area.

Fig. 2
figure 2

Examples of challenging variation in image acquisition conditions included in our dataset (A) High off-nadir view angle, mid sun elevation (B) Low off-nadir angle, low sun elevation (C) Mid off-nadir angle, high sun elevation.

Full size image
Fig. 3
figure 3

Illustration of just a few examples strong regional variation in building style and context that can be found across an area like North Africa. For each pair, left shows a single sample from our database, right the output of our global model for a nearby area. Illustrated: (A) Sikasso Province, Mali (B) Jungoli Province, South Sudan (C) Bench Sheko Zone, Ethiopia. ©Maxar 2023.

Full size image

In addition to showing the wide spatial distribution of ORBITaL-Net in Fig. 1, we have compiled basic geometric statistics on the building outlines as well as their environmental contexts. Shown in Table 1 are the central tendencies and standard deviation in four variables: area, perimeter, complexity (the ratio of area to perimeter), and isoparametric quotient34, a measure of how close to a circle a shape which is calculated as:

$$ISP=\frac{(4\ast \pi \ast area)}{perimete{r}^{2}}$$
(1)
Table 1 Descriptive Statistics for the structure polygons in the dataset (excluding structures >1,000 m2).
Full size table

Figure 4 then shows the percentage of buildings that fall within a given biogeophysical zone, as defined by the World Wildlife Federation and Fig. 5 the second level Koppen climate zone.

Fig. 4
figure 4

Percentage of buildings in the dataset within each biogeophysical region as defined by WWF.

Full size image
Fig. 5
figure 5

Percentage of buildings in the dataset within each level 2 Koppen climate zone.

Full size image

Impacts of the developed training dataset

As labelled data continued to be added to the library, less variation between the pre-trained model and the resulting fine-tuned model was noticed, leading to fewer and fewer samples generally being needed to adapt the model to a new project area of a given size. For instance, between the Arabian Peninsula, a project area we used early in the compilation of our training library and a far later project area of Central and northwest South America, the labelling effort in terms of work hours decreased by almost half. Since we have completed the full, global training dataset, and used it to train a Global Base Model we have eschewed the need for obtaining additional labelled data in most circumstances. Examples of areas where we have successfully used our Global Model with no fine-tuning include Taiwan, Sudan, Ukraine, Poland, and Belarus.

To illustrate the improvement in generalizabilty of our models over time, as a greater number and variety of training samples added, we performed a retrospective validation of previously trained and fine-tuned models used for building layer production against a set of 9,500, globally distributed validation samples. For this analysis, we used three standard metrics: (1) precision, which measures how reliable predictions are, (2) recall, counterpart to precision that measures how complete predictions are, and (3) F1-score, the harmonic-mean of precision and recall which can be used to balance tradeoffs between these two metrics. These are calculated as:

$$perimeter=\frac{tp}{tp+fp}$$
(2)
$$\mathrm{Re}call=\frac{tp}{tp+fn}$$
(3)
$$FI=\frac{2\ast perimeter\ast \mathrm{Re}call}{perimeter+\mathrm{Re}call}$$
(4)

The results are plotted in Fig. 6. This is presented in chronological order, from the initial project areas of North Korea and Yemen all the way to the current, Global Base Model trained on our full dataset. There is a clear trend over time (and thus expansion of the global training dataset in quantity and variety of samples) towards models that are more generalizable according to both the increase in overall F1 score and the narrowing of the ratio between precision and recall scores, a general indicator that a model has learned a more “stable” prediction function.

Fig. 6
figure 6

Performance of ORNL building segmentation models, fine tuned for project areas, measured on a set of 9,500 withheld samples drawn proportionally from imagery across our global dataset. This presentation in order of recency (left to right) shows the steady improvement of the generalizability of our fine-tuned models as a more extensive and diverse training dataset is used.

Full size image

Summary

We summarize ORBITaL-Net with a listing of what we consider it’s most salient strengths and weaknesses.

Strengths

  • A broad geographic coverage including a variety of architectural styles, settlement patterns, and biogeophysical contexts across four continents.

  • Incorporation of many medium and low population areas rather than a focus specifically on major urban areas.

  • Inclusion of a large number of pure negative (no buildings present) samples, which we have found to be very beneficial for obtaining a good model precision across large, diverse geographic areas.

  • Use of a large number of individual source satellite images that better represent the variety of image acquisition contexts seen in real-world applications.

Limitations

  • The source imagery used is exclusively from Maxar very-high resolution sensors (primarily WorldView-2 and WorldView- 3); therefore, a model trained using this dataset may have considerably worse performance when used with image data that are substantially different to these without the use of transfer learning.

  • As we labelled all visible parts of the building as such, if a user wished to have a different definition, such as extracting rooftops only, the utility of this dataset may be lower.

  • Due to the fact that these samples were designed to train a model to extract all visible parts of a building instead of the footprint, there may be discrepancies between the area of building features predicted by such a model and the true building footprint area, particularly when working with imagery at a high off-nadir angle and/or very tall buildings. This can have implications for such applications as assigning population by building area.

Methods

ORBITaL-Net was created over a span of five years (2017–2021) through a series of large-scale building extraction projects. Project areas were selected to be generally homogeneous in terms of climate and architecture/settlement patterns and ranged in geographic coverage from a cluster of provinces within a single country to multi-country regions. For each project area, we created a set of hand-labelled samples for fine tuning our pre-existing base building segmentation model, the first iteration of which was described in20. These new samples were added to the full training pool, increasing the overall generalizability of our base model.

Overhead imagery was sourced from Maxar and consists primarily of 4-band (red, green, blue, and near infrared bands) images collected from the WorldView-2 and WorldView-3 satellites. Images from the QuickBird-2 and GeoEye-1 satellites were used where suitable WorldView imagery was not available. The pan-sharpened resolution of the resulting image collections is 0.45 m with a minimum acceptable resolution of 0.60 m and maximum off-nadir view angle of 40 degrees. The median resolution for samples in the building dataset is 0.47 m with a standard deviation of 0.09 m. We received this imagery at Level-1B (sensor corrected, un-projected (raw) imagery product) and applied our in-house georegistration, orthorectification, and pan-sharpening, as described in35. The bulk of the imagery dates between 2015 and 2020, with a median acquisition year being 2017, as shown in Fig. 7.

Fig. 7
figure 7

Histogram of all individual samples by year of source image acquisition.

Full size image

Once the source imagery was pre-processed, the next step was identifying a suitable set of samples for labelling. To address the challenge of quickly identifying a highly salient set of samples across a large set of imagery with multi-modal variability, we developed the ISOSCELES program. This is a fully automated program, described in more detail in36 that performs a multi-level unsupervised clustering to select first the most representative satellite image scene from the full dataset and then the most representative chips (image subsets at the scale of the inference model the user chooses) within each scene. Features included for the selection of satellite images included sun elevation, image sensor viewing angle, and overall spectral statistics. In turn, image texture and spectral statistics are used to aid in the selection of representative image chips.

This set of chips was then divided into positive (buildings present) and negative (no buildings present) samples. In accordance with the general practice of using training data sets balanced between classes, we have found across numerous project areas that a mixture of 55–60% negative to 45–40% positive samples provides the best trade-off between false positives and false negatives.

Hand-labelled training data

Our initial proof of concept of the feature extraction at scale20 utilized existing LiDAR building footprints to generate labelled samples. While that was a successful demonstration, we identified the model generalization issue when we investigated the feasibility of adapting the learned building extraction models to other countries. The main challenges stem from the differences in images at various aspects, including the inferior model performance when the image quality is poor, or variation in structure types due to cultural or socioeconomic differences. To address this issue, we decided to use a domain adaptation approach to fine-tune our best performing model to each new target project area. However, we quickly discovered that the lack of reliable labeled data became a significant barrier to this approach. Without labeled data that is both diverse and reliable, it’s difficult to improve the model’s ability to generalize to new environments.

A team of 30 analysts leveraged GIS software to digitize features of interest. Analysts were trained how to distinguish building structure features from non-building features in imagery with emphasis placed on consistent, uniform digitization of building features.

Each building was digitized as a discrete, non-overlapping polygon (Fig. 8), with all visible portions of the building labelled, including facades (Fig. 9). Imagery properties (brightness, contrast, stretch, band combinations, etc.) were adjusted for each image chip so all buildings in the image could be positively identified. Some examples are shown in Fig. 10.

Fig. 8
figure 8

Building features were digitized as non-overlapping polygons. (a) sample image. (b) sample image with labels. ©Maxar 2023.

Full size image
Fig. 9
figure 9

Building facades were included in labeling. (a) image. (b) image with labels. ©Maxar 2023.

Full size image
Fig. 10
figure 10

Different band combinations were used to help differentiate buildings from vegetation. (a) shows the RGB combination of bands 3,2,1 which is true color, (b) shows the RGB combination of bands 4,2,1 where the vegetation appears red, and (c) shows the RGB combination of bands 3,4,1 where vegetation appears bright green. ©Maxar 2023.

Full size image

Our definition of “building” was guided by the general use-case of producing building datasets that could be used as foundational spatial inputs for population modelling. Thus, the rule-of-thumb was if a person might plausibly occupy the structure, be it as a dwelling, workplace, school, store, etc., it would be considered a building. Table 2 gives a more detailed description what features were digitized as buildings, what features commonly found on roofs were also labelled as building, and which human-built features were not labelled as buildings. In cases where portions of buildings were obscured by shadow or vegetation, those pixels were labelled as a building if there was partial occlusion and non-building if they were fully occluded.

Table 2 Details of the rubrics used to determine whether an object in an images should be labelled as building or non-building.
Full size table

Variation in how buildings were digitized stemmed from multiple facets of imagery interpretation, including architectural differences (Fig. 11) and challenging image acquisition conditions (Fig. 12). Defining the outline of a building requires an experienced analyst to resolve the artifacts of view angles (e.g. shadows), identify salient morphology features (e.g. walled boundaries between adjacent buildings, non-permanent features such as awnings), and interpret the geographic variation of building types and styles. Furthermore, the regional variation of building styles and types required that analysts had experience interpreting those differences and translating them to labeled training data. For example, walled compounds in Pakistan as opposed to huts in Nigeria. or more broadly how local culture and geography manifests in the built environment. To aide in the task of interpreting regional variations, analysts typically used additional reference imagery to learn the specifics of a region’s architecture and settlement patterns and resolve the interpretation of ambiguous features. Examples include publicly available sources, such as street view imagery from Google, Bing, and Baidu maps, aerial images showing an oblique view, and publications and image libraries of vernacular architecture.

Fig. 11
figure 11

Mixed roof types (natural material and metal. (a) image, (b) image with labels. ©Maxar 2023.

Full size image
Fig. 12
figure 12

Training samples included imagery with (a) partially snow-covered ground and buildings, (b) fully snow-covered ground and buildings, (c) grainy imagery, (d) haze, (e) clouds, and (f) extreme off-nadir. ©Maxar 2023.

Full size image

Data Records

ORBITaL-Net consists of 128,270 samples, each comprised of a reference image, a binary training label, and georeferencing data stored in json format, for a total of 384,810 individual files, deposited at Figshare37 https://doi.org/10.25452/figshare.plus.25282225.v3. The source imagery dates from between 2003–2020, with the vast majority being from 2010 onwards. The files are in GeoTiff format with the reference imagery held in unsigned 16-bit integer arrays and the labels in binary integer arrays.

The naming convention is as follows:

[source image] [individual image chip ID] [data flag] [sample component]

e.g. 11APR17162634-502717212090_P001-0017_1_img.tif

source image: 11APR17162634-502717212090_P001

individual image chip ID: 0017

data flag: 1

sample component: img

Data flag indicates whether the sample contains building pixels (1) or is purely negative (0)

Sample component indicates whether the raster is the sample image or one of several types of labels. Values are:

img: the aerial or satellite image

seg: binary label of building (1) and non-building (0) pixels

The georeferncing information stored in geojson format is named according the basename of the relevant sample, with- out the “_img” or “_seg” component,

e.g. “11APR17162634-502717212090_P001_0017_1.geojson”

Technical Validation

To ensure accuracy and consistency across the team of analysts in a variety of challenging contexts, a continuous QA/QC process was implemented. Steps included:

  • Analyst self-review

    • The analyst reviewed their work to ensure building features were accurately digitized, no buildings were omitted, and non-building features were excluded from each labeled image tile.

  • Buddy-system or expert analyst review

    • A second analyst reviewed labeled image tiles to ensure accuracy and consistency. Edits were made where appropriate.

    • If recurring mistakes were noted, additional instruction was given to the analyst by an expert analyst.

    • Additional examples of frequently misinterpreted/mislabeled features were added to a living document for reference and future instruction.

  • Final review

    • Prior to model training, an expert analyst performed a binary review of labeled image tiles. Any labeled tiles that were below standard were omitted from the training dataset. No additional edits were made.

As part of our process of training image analysts, we created a set of training images with expert made “gold-standard” building label that represent a wide variety of imagery and architectural and settlement pattern contexts. By comparing the variation between the expert made labels and those created by a image analyst under training, we could tailor individual instruction. Figures 13 and 14 show examples of acceptable and unacceptable variation, respectively, and, by extension, quality standards which were used for all of the labels in the ORNL dataset.

Fig. 13
figure 13

Examples of acceptable variation in labeled training data developed by five different analysts shown in unique colors. (a,d) show base image. (b,e) show labels on imagery. (c,f) show labels only. ©Maxar 2023.

Full size image
Fig. 14
figure 14

Examples of unacceptable variation in labeled training data developed by five different analysts shown in unique colors. (a,d) show base image. (b,e) show labels on imagery. (c,f) show labels only. ©Maxar 2023.

Full size image

Statement on authorization for public release

The images included in this dataset were part of MAXAR imagery collection shared with National Geospatial Intelligence Agency (NGA) under NextView agreement. We have obtained the approval (A706) to release this set of images to the public. We also included copyright statement (c 2025 Maxar) into metadata as requested by NGA as part of approval.