Introduction

Gas leaks are a contributor to global warming due to methane being a powerful greenhouse gas, especially in terms of immediate short term impact1,2. Methane is widely used as a key component of natural gas, with leaks potentially having great economic repercussions3,4. Previously, the standard detection protocol relied upon detectors that work at a single point, while these can give accurate readings of gas concentration they do not provide an imaging component, making detection of the source of leaks difficult5. We present a stand-off method of gas detection where video of the gas location is produced in real-time, thereby developing a tool for much faster and efficient leak identification.

Stand-off detection of gases can be performed via optical absorption, these techniques include tunable diode-laser spectroscopy (TDLAS)6 and differential absorption Lidar (DIAL)7,8. In general such systems are focused upon accurately measuring the concentration of the gas rather than displaying a live image of the gas distribution, and also require a level of calibration8,9,10. While the system discussed in this paper does not provide precise concentration measurements, it does display a live image of a gas distribution, allowing the source of leaks to be located. Our system, and the other optical absorption detection, techniques can be termed “active” in that they illuminate with a light source of a wavelength tuned to an absorption band of the gas. As an alternative “passive” systems are also possible but these are dependent on environmental factors such as natural background illumination, temperature, wind speed and wind direction in order to identify the gas source10,11,12,13,14,15.

Perhaps the most obvious comparison to our system is that of optical gas imaging (OGI) hardware. These systems are widely used, commercially available and effective at highlighting gas leak locations. The current best-standard OGI devices are passive sensors, operating at wavelengths in the mid- or long-wave infrared, wherein they use specific filtering in order to highlight temperature differences where gas is present16,17. This results in a thermal image in which the presence of gas is depicted by a black region or cloud, a result of absorption at the filtered wavelength18. This can often result in a need for cooling the sensor which can drive up the prices for such systems10. The major drawback to the system is that because the filters used are not specific enough to individual gas absorption lines it can mean that identification between different gas species with similar absorption properties can be difficult16,17. This lack of specificity does however, allow more than one gas species to be imaged using only one system, which adds to the versatility of such systems17. Similarly, it can be difficult to measure concentration of the gas imaged due to these issues17.

Previous developments of an imaging system were able to give some indication of gas distribution from a static location19, but there were difficulties in matching the position within the infrared detection camera and a visible camera to enable to operator to pin-point the relative position of the gas distribution in a moving system. Reducing parallax in the system and developments in the image processing to account for motion artefacts have enabled a system more resilient to motion to be produced20. The detection system is entirely self contained via the addition of an embedded mini-PC, touch screen and rechargeable lithium ion-based power supply. The system can additionally be mounted to a drone for use within the field21.

One particular application envisaged is the inspection of gas pipelines. However, pipelines can lay along or below a wide variety of surfaces, with potentially different absorption and scattering properties. In order for the system to be a viable technology for detecting gas emissions, it must be able to image gas against a wide variety of common background materials and scenes. Traditional techniques for surveying the position of a gas leak can be conducted on emulated surfaces such as grass or paving, something the system must be robust against22. Pipelines themselves are metallic in nature, so metal surfaces must be able to return adequate signal.

The key focus of our research is to test the validity of gas leak localisation when conducting a drone-mounted survey. The desire to drone-mount primarily comes from a user safety perspective, methane gas leaks can be hazardous due to their associated explosion risk23. In addition to improving user safety, a drone offers the promise of a higher degree of automation and accessibility, potentially leading to an overall increase in regularity of routine facility inspections leading to improved leak detection timescales23,24. In addition to being able to follow pre-determined routes, drone swarms can be deployed to quickly cover large areas in fine detail24,25. Drones offer an increased degree of flexibility in comparison to helicopter based instruments, due to decreased size, cost of running, improved manoeuvrability in confined or complex spaces, lower flight height, and decreased interference from down drafts caused by the helicopter rotors26,27. Drones can also be deployed with ease in less built up areas where access via vehicle is limited28.

An issue that presents itself with drone usage is the associated limited flight time leading to relatively short survey duration26. Whilst having less impact than helicopters, drones do still generate some downwards flow of air which can disrupt and disperse gas below the detection limit29. In order to circumvent this issue instruments are often pointed at an angle ahead of the drone where the downwards flow does not disrupt the gas cloud30,31. Depending on mounting type it can be difficult to fully dampen the impact of drone vibrations23. Even in a relatively stable hover mode, vibrations caused by the rotors can still result in motion artefacts and hence reduced detection efficiency23. The introduction of compensatory technology such as a gimbal has been shown to aid in reducing the impact of these vibrations32. Despite these issues, there are reports of gas detection instruments being successfully integrated onto drone systems30,33.

We introduce our gas imaging device and present the developments made to the system and the image processing procedure used for methane detection. There have been significant improvements in the gas detection performance and these are discussed. To emulate the end-user application field trials have been performed outside of the laboratory, where the device was mounted to a UAV and tested by looking for methane filled balloons using real-time image processing. The system was also tested against a variety of common surfaces at a range of distance to investigate how universally applicable it would be in real world scenarios.

Gas imaging hardware

There are four main components to the imaging system, a \(256\times 320\) InGaAs short-wave infrared (SWIR) focal plane array (FPA) running with an exposure time of 20ms, a visible camera, and two \(\approx {10}{\textrm{mW}}\) InGaAs laser diodes. The diodes are tuned to the absorption wavelength of methane at \({1.653}{{\upmu \hbox {m}}}\) and diffused through \({20}^{\circ }\) engineered diffusers, which provide an expanded, uniform, square illumination profile. The camera also features a band-pass filter centred on \({1.6532}{{\upmu \hbox {m}}}\) and has a full width at half maximum (FWHM) of 18.8nm in order to block unwanted background light. The system itself has a mass of \({3.15}{\textrm{kg}}\), and a size of \({284}{\textrm{mm}}\times {198}{\textrm{mm}}\times {125}{\textrm{mm}}.\) The embedded lithium ion battery source provides more than 2 hours of continuous run time. An image of the system and simplified schematic are presented in Fig. 1. The illumination light will pass through a scene before being back-reflected by a background surface towards the entrance aperture. Upon reaching the entrance aperture a cold mirror will separate incoming light into its visible and SWIR components towards the respective camera detectors.

Fig. 1
figure 1

Configuration of the gas imaging instrument. (a) Photograph of the completed instrument from above showing the internal configuration of the components. (b) The illumination and detection apertures. (c) A schematic representation highlighting the main components, namely the webcam, SWIR FPA, SWIR diodes, cold mirror, engineered diffusers, embedded mini-PC, diode control circuitry and touch control screen.

Full size image

An on-resonance image will be taken, where methane present in the scene will absorb the illumination light and not be back-scattered. Immediately after this, an off-resonance image will be taken, where the illumination light is slightly detuned via current and temperature control such that it is no longer absorbed by the gas. The fact that the diodes are temperature controlled, both with tuned temperatures of around 22-\(25^{\circ }\)C, means that the system is overall relatively robust to external temperature changes. The system is designed such that there is air flow throughout to prevent internal temperature from rising too high. The detuning is achieved by decreasing the input voltage to the diodes such that no significant absorption was observed. This introduces a slight power discrepancy between the two frames, however, as the off resonance has marginally less laser power this acts to slightly suppress the background when subtraction is performed. For efficiency the direction of tuning, and hence subtraction, is swapped with each iteration to increase the frame rate. The overall system is controlled by an in-built mini-PC which can be operated using an embedded touch display. The mini-PC also supports a HDMI output which can allow for integration to larger displays as well as to the streaming channel of the drone. This allows the feed to be monitored in real time whilst the system is airborne.

Image processing

The two captured images are smoothed using a Gaussian filter and convolved with an \(8\times 8\) array of ones in order to smooth the response. A correlation is performed in the system to overlap the consecutive frames before subtraction. The correlation used for the data in this paper involves passing the image through the subsequent frame vertically and then horizontally by 20 pixels each time and looking for a local minima in difference between the two frames. Other techniques such as phase correlation have also been experimented with, however this method was both fast and robust for our purposes20. The images are then subtracted to produce a differential signal image. The differential image is always reported positive, as the sign of the subtraction is swapped for every odd frame. It follows that any negative signal in the differential image is background noise and so can be removed. The differential signal is, as such, subject to the constraint that if the differential signal is negative, it can be set to zero. The differential image can now be used to generate a binary mask, where a value of 1 denotes a potential presence of gas at the given pixel. The following issue is that, although much of the noise is removed at this point, some background noise is positive and will appear in the binary mask. Steps must now be taken in order to effectively tell the difference between real gas signal and noise. A further threshold constraint cannot be utilised here because at larger distances the noise and true signal are often comparable in magnitude. Instead the signal is distinguished from noise in terms of spatial extent and persistence in time.

False signals caused by background variation rarely appear in the same location in consecutive frames, whereas a real signal mostly will. As such the mask generated in the previous frame is multiplied by that of the current frame performing a pixel-wise AND operation. In each of the data sets analysed in the following a harsh persistency filter was used for consistency, this involved ensuring that data was located in the same position for six consecutive frames to be considered a true signal. This is very effective in stationary systems observing gas cells or balloons, but is less consistent where there is moving or evolving signal such as a gas leak or a moving scene such as that acquired from a drone in motion. As a result there will be a dulling in the sensitivity of the drone data, which may lend itself to an increased uncertainty. Though the vibrational motion of the drone’s rotors in hover mode is too small to have much impact, slight gusts of winds and other movement of the drone itself may temporarily result in this dulling. To filter by spatial extent erosion and dilation are used on the mask after the persistency filter is applied. The size and extent of the erosion and dilation can be varied depending on usage, an erosion kernel of size 3x3 was selected. The kernel itself is binary and comprised of 5 ones and 4 zeroes, with the zeroes being positioned in the corners of the kernel. The kernel moves along the image by centring on each pixel and exploring the nearest neighbours. The kernel acts based upon an OR operation in which, for the erode, if a one in the overlaying kernel overlaps with any zero in the mask then the pixel being investigated is set to zero. For the dilate step if a one in the kernel overlaps any one in the mask then the investigated pixel is set to one. The edges of the image are examined by applying a temporary border of ones to the image to allow the process to occur. This erosion kernel cannot be too large as at increased ranges the signal can be comprised of relatively few pixels, however, as the persistency filter has already been applied a small kernel works sufficiently well to reduce false positives in the mask.

Fig. 2
figure 2

Visual timeline of the image processing pipeline. Smoothing consists firstly of a Gaussian blur followed by a general flattening via convolutions with a \(8\times 8\) array of ones. A mask is generated from the differential image and filtered and eroded before being reapplied to the differential image. The masked differential SWIR image is added to the green channel of the visible RGB feed. This conversion can lose some of the magnitude data but will not subtract from the positional data, which is the primary function of our imaging system.

Full size image

After filtering, the mask is applied to the original differential image to extract the gas positional data. The masked differential image is then formatted from 16-bit to 8-bit and added to the green channel of the captured webcam feed of the scene. This provides a false colouration for methane and reveals the location of the gas signature in the frame. A timeline of the image processing is presented in Fig. 2. The raw SWIR and visible images can be saved for post processing later. Additionally, for the frames presented throughout this paper a further Gaussian filter and linear multiplicative gain is applied to the signal to enhance clarity. In the general use case of the camera this linear gain can be controlled in real time on the touch screen. These aesthetic steps are not included in further signal and noise processing.

In post processing an average signal strength is calculated by multiplying the differential signal by the mask, summing over all pixels, and dividing the result by the sum of the mask over all pixels. An overall average signal for an observation is found by averaging the signal strength over all frames in the observation. In order to quantify uncertainty on this measurement a standard deviation value is calculated for each average signal. This standard deviation will encapsulate variations in signal strength. To allow the persistency filter to take effect the initial few frames are not used when calculating this average. This process is used to characterise the system.

The fact that the diodes are tuned to a specific absorption wavelength of methane means that it has a high level of specificity, escaping a key shortcoming of other, passive, OGI cameras. The effectiveness of this is highlighted in Fig. 3, wherein we detect no signal from other similar gas species.

Fig. 3
figure 3

Image taken using the gas imaging device of two clear bottles, one containing butane and the other containing methane. It can clearly be observed that only the methane bottle returns the bright green gas signature, highlighting the specificity of the absorption band selected for investigation. This is further highlighted in the insert which shows the processed differential SWIR image, where signal is only present for in the region of methane gas.

Full size image

Drone campaign

The system was attached to the underside of a drone (DJI Matrice 600-Pro), an HDMI connection between the drone and the device meant that a live stream of the camera data could be relayed to the mobile device controlling the flight, an image of the combined system can be observed in Fig. 4. We opted not to implement a more sophisticated motion compensating gimbal system due to the increased mass and the custom nature of our system’s mounting. The maximum possible battery life of the drone without a load is estimated to be around 30 minutes, significantly less than the maximum battery life of the system, making the gas imaging system not the limiting factor on acquisition duration. Four gas balloons were observed from a fixed horizontal range of 10m from the balloons and an altitude of 3m. This altitude was selected as it was deemed a reasonable height from the ground where the effects of rotor down-wash do not interfere strongly with the balloons. This height also ensures that the horizontal range of the system remains reasonable, meaning that the down-wash impact was diminished further. A height of 3m, which provides an approximate horizontal detection range of 10m, should be acceptable for gas pipeline observations. Of the four balloons three contained nitrogen, which cannot be detected by the system, with the other containing methane. A series of images were captured of these balloons from the drone as the evening progressed.

The drone mounted combined system was trialled during times at which sunlight levels were low, namely during sunset. The lux levels were recorded on a light exposure meter (RS Pro ILM-201L). The lux ranged from roughly 400Lux to as low as around 10Lux, corresponding to values for sunset and into the night34. The results acquired around sunset were favourable when compared to those acquired during high sunlight levels, as it was previously discovered that in conditions of moderate sunlight the system suffers a large loss of sensitivity. The reason for this is that the SWIR camera will quickly saturate due to sunlight passing our filter, however narrowing our filter greatly impacts the field of view of the detector, as such we gather our data below 400Lux20. Additionally, a benefit of working at night is that we can avoid variation in the absorption profile of methane as a result of temperature. Higher temperatures can lead to a decrease in absorption from the gas, which can negatively impact quantitative evaluation of the concentration of gas present, though for significant impact in accuracy the temperature would have to vary considerably35,36. Sunlight levels were also not so low that there was a lack of light for visible camera operation, such as during nighttime observations.

Fig. 4
figure 4

The drone-mounted methane imaging system. Left image depicts the system in flight, while imaging 4 target balloons. Right image is an example frame observed from the gas imaging system from a vertical height of 3m during a fly in shot from a horizontal distance of 22m to 10m at a background light level of 52Lux. The methane filled balloon is highlighted in red in contrast to the three containing nitrogen. The methane is coloured red here to distinguish more clearly from the grass background. Due to low light intensity a \(3\times\) gain is applied to the visible feed to increase clarity. This is an example frame from a video attached as supplementary material S1.

Full size image

Background testing

Due to battery life and location constraints for drone flight, testing could only be performed with grass as the background surface. For end-user applications, testing was performed against a wide range of background surfaces each with different scattering profiles. In order to characterise performance against these profiles the gas imaging instrument was set up on tall tripod and taken to a variety of different locations with different surfaces to emulate the data from the drone, examples are shown in Fig. 5. The surfaces explored were grass, paving stone, tarmac, steel and aluminium. Additionally, an \(18\%\) grey calibration card, commonly utilised in photography, was used as a reference frame for reproducibility. The tripod was then moved away from the balloon until it reached a distance comparable to that obtained on the drone. The strength of the gas signal as a function of distance was then plotted for each of the explored surfaces to allow for direct comparison. These observations were conducted over the course of two evenings, with grass and paving stone investigated on the first evening and tarmac, card, steel, and aluminium investigated on the second. The distance to the balloon from the aperture of the device was measured each time and the signal levels calculated, these are plotted in Fig. 6. For the card, steel, and aluminium the balloon was placed atop a sheet of the respective material and it was checked by eye that the light the camera detects has passed through the balloon and reflected off of the test material as opposed to underlying tarmac. In some cases for increased distances, this involved introducing a tilt in the test material by propping it up. In order to avoid the impact of sunlight issues the data was collected after the sun had set. As this is a stationary system, the correlation aspect of the processing was not implemented to optimise system speed.

Fig. 5
figure 5

Simplified schematic for surface investigation accompanied by example images taken by the device for (a) steel at a distance of 3.1m, (b) aluminium at a distance of 5m, (c) \(18\%\) reflective photography calibration card at a distance of 7.1m, and (d) tarmac at a distance of 9m. Note that a gain of 0.7 was applied to the visible channel to enhance clarity of the gas signal in these frames.

Full size image

The main issue the system faced was unwanted sunlight signal, which under some weather conditions could saturate the camera. This interference arises from the fact that our current filter does not block enough of the incident sunlight and as such more was detected by the camera than intended. The details of this are explored in more detail in previous publications20,21, but is summarised briefly here. In order to reduce incident sunlight on our SWIR camera the filter in front of the camera must be a narrow band-pass, however reducing the FWHM of the filter also decreases the angular field of view as a consequence of the tilt-tuning of band-pass interference filters20,37. As back scattered light is incident from a variety of angles, narrow filters in this scenario result in a reduction in field of view, limiting the full frame capabilities of the system. The combined drone mounted system was also susceptible to a higher level of noise in addition to that caused by sunlight. The movement of the drone caused small amounts of noise, mainly in regions of the image with non-uniform texture such as grass, much of this could be addressed with the computational filtering as outlined in this work. However, this could lead to an overall decrease in sensitivity. Overall, the effects of sunlight upon the gas detection protocol is complex and whilst some attempts have been made to reduce its impact on the data, this is a continuing area of research.

Fig. 6
figure 6

Linear fits of the returned gas signal from the balloon as a function of distance (to the power negative two) on tarmac, card, aluminium, steel, grass, and paving surfaces.

Full size image

As shown in Fig. 6 we can see that the signal follows an inverse square relation with distance, which is to be expected as a result of the imaging mechanism. The light leaving the diodes must be diffused, travel through the gas, scatter off the background and be collected by the camera. This causes much of the light intensity to be lost. This loss follows an inverse square relation with distance and as such it is expected the return signal from the gas would also follow this trend. What is also evident from this data is that in general no surface performs significantly worse than another. Whilst initial return signals are varied, with the strongest returns from paving and steel and the weakest from tarmac, all six surfaces begin to perform similarly at further distances and, most importantly, all return clearly identifiable signal as the distance begins to approach 10m. As such it can be observed that the system can be used in a variety of real world scenarios with no great detriment to overall response. We observed an increased uncertainty on the surface return signals closer to the camera than those further away. It is believed that the reason for this was a result of fluctuation in size of the signal mask. The mask is inherently larger for larger signals and as such fluctuations around the circumference of the mask lead to larger variation in the signal value than with a smaller circumference.

As the gas cells used in these trials were unquantified and used primarily for demonstration of the stability of the system, we also have performed a trial on a gas cell containing a measured amount of gas to place the system response in wider context. The cell used contains an approximate maximum concentration of 5000 parts per million metres (ppm.metre). The cell was placed in front of a wall and the camera moved away from the cell as in the surface tests. The results of this experiment are presented in Fig. 7. The data was processed using the same methodology as for the surfaces. This data suggests that the system is capable of detecting 5000 ppm.metre of gas up to a range of around 13.6m. It can be seen that, as in the surface data, the return signal increases in an inverse-square relation with distance as a result we imagine that the sensitivity to concentration would behave in a similar trend, meaning that lower concentration could be observed at a closer range and higher concentrations at further distances. In order to fully characterise this response one would need to survey a wide range of cells at different concentrations, which is outside the scope of this paper.

Fig. 7
figure 7

Return signal from a methane cell containing 5000 ppm.metre of gas. It can be seen to follow a similar trend to the surface data. The signal and noise become similar in size for the data points at 16.3m and 17.3m, what this results in is a gas signature that will occasionally flicker in and out. As such we can say that the threshold for reliable detection of a gas signature is approximately 13.6m where this is not the case.

Full size image

Discussion and conclusions

We have presented a real-time methane imaging system with wide applicability in the field. By highlighting the system’s ability to be drone mounted we demonstrate its flexibility as a device that can be used handheld by a user or controlled remotely to access different dangerous or difficult locations. We additionally indicate the systems resilience to different background surfaces, concluding that at ranges of up to 10m return signal is always obtained such that gas location can be identified. This level of robustness in regards to the back-scatter surface profile also further highlights the potential widespread use of the device in pipeline surveys, where such pipelines could be located under streets or alongside grass. We provide additional context for the systems performance by characterising the signal response as a function of distance for a cell containing 5000 ppm.metre of gas, demonstrating sensitivity up to approximately 13.6m. Further work with the system will seek to improve performance in direct sunlight, further increasing the wide applicability of the system as a monitoring device.

Methods

The two images were captured using a SWIR camera (Goldeye G-008 InGaAs SWIR FPA) running with an exposure time of 20ms with a Kowa LM25HC-SW 25mm C-mount lens attached, and a visible camera (Logitech c270 webcam) running at 30 frames per second with a 4mm lens. The two \(\approx {10}{\textrm{mW}}\) InGaAs laser diodes (Eblana Photonics EP1653-7-DM-TP39-01) were tuned to the absorption wavelength of methane at \({1.653}{{\upmu \hbox {m}}}\). Each diode was controlled using a custom circuit board mounted with a Thorlabs MLD203CLN current controller chip and a MTD415T temperature controller chip. The base current and temperature was set and calibrated using Thorlabs propriety software and finely tuned using the analogue input pin of the current control chip via an Arduino Nano and an Analogue Systems AD5696 DAC. The laser diode outputs were collimated using 30mm lenses and passed through Thorlabs ED1-S20-MD \({20}^{\circ }\) engineered square diffusers. The back-scattered diode light passed through a cold mirror (50.8mm Thorlabs FM203 mirror) before reaching the respective cameras. A band-pass filter (Spectrogon) centred on \({1.6532}{{\upmu \hbox {m}}}\) featuring a full width at half maximum (FWHM) of 18.8nm was used to reduce background signal. A lithium ion battery (Enix Energies MGL2806) charged using a custom circuit board provided a portable power supply. The system was controlled using a mini-PC (Lattepanda Alpha), with accompanying Lattepanda 7” touch display. All components were mounted within an enclosure using custom 3D printed parts and laser cut acrylic base and cover.