Lossy Compression Techniques for Scientific Data

Lossy compression techniques reduce the size of large scientific datasets by selectively discarding or coarsening data elements that contribute least to the overall information content. Unlike lossless methods, which guarantee exact reconstruction, lossy approaches trade exactness for substantially higher compression ratios, typically guided by user‐specified error tolerances. Principal strategies include transform‐based methods that convert spatial or temporal data into alternative domains (for example, wavelet transforms) for efficient quantisation; prediction‐based schemes that exploit correlations within structured or unstructured meshes; and hybrid methods that combine quantisation with domain‐specific filters. Recent advances emphasise rigorous error‐control frameworks, domain‐informed metrics for preserving key derived quantities, and adaptive algorithms that partition data into blocks or sequences with tailored compression parameters. Applications span climate simulation archives, high‐resolution imaging in microscopy and astronomy, unstructured mesh data in computational fluid dynamics, and checkpoint‐restart files in exascale supercomputing. By balancing compression ratio against scientifically acceptable distortion, these techniques are reshaping data management, enabling longer simulations, larger ensembles and more extensive collaborative analyses across global research infrastructures.

Research from Nature Portfolio

Recent studies have formalised an information‐theoretic measure of “real information content” to distinguish meaningful bits from numerical noise in atmospheric and climate forecast datasets. By rounding away bits with negligible real information and combining this quantisation with standard lossless algorithms, one study achieved a 17-fold reduction in file size for 64-bit floats while preserving 99 % of the information deemed essential for forecasting applications. When extended to four‐dimensional spatio-temporal compression, overall factors exceeded 60× without compromising predictive skill. A proposed data-compression Turing test quantifies end-user detectability of errors, offering an objective criterion to optimise compressibility against information loss for weather and climate data products.

Lossy Compression Techniques for Scientific Data publication trend

The graph below shows the total number of articles in lossy compression techniques for scientific data across all publications each year (not limited to Nature Index journals).

Technical terms

Lossy Compression: A data reduction method that allows controlled loss of fidelity to achieve higher compression ratios than lossless techniques.

Quantization: The process of mapping a continuous or high‐precision value to a discrete set of levels, introducing a known rounding error.

Error-Bounded Compression: An approach that guarantees the maximum deviation between original and reconstructed data does not exceed a user-defined threshold.

Derived Quantity of Interest (QoI): A secondary metric or feature computed from the primary data, whose accurate preservation is critical for scientific analysis.

Four-Dimensional Compression: Compression that jointly exploits spatial and temporal correlations in multi-frame datasets to enhance efficiency.

References

1. Compressing atmospheric data into its real information content. Nature Computational Science (2021).
2. The compression–error trade-off for large gridded data sets. Geoscientific Model Development (2017).
3. Assessing the Effects of Data Compression in Simulations Using Physically Motivated Metrics. Scientific Programming (2014).
4. A Prediction‐Traversal Approach for Compressing Scientific Data on Unstructured Meshes with Bounded Error. Computer Graphics Forum (2024).
5. WaveRange: wavelet-based data compression for three-dimensional numerical simulations on regular grids. Journal of Visualization (2022).
6. Error-Bounded Learned Scientific Data Compression with Preservation of Derived Quantities. Applied Sciences (2022).