Non-invasive jaundice detection using spectral-band expansion from RGB images and direct hyperspectral images

Abstract

The gold standard for diagnosing jaundice has been measuring serum bilirubin level through venous blood sampling. Despite its high accuracy, this method is invasive, requires clinical infrastructure, and thus may delay jaundice diagnosis. Here, we use two spectral-image techniques, image spectral-band expansion and hyperspectral imaging (HSI) to analyze the spectral features of patients’ sclera and skin for comparison with serum bilirubin levels. Image spectral-band expansion is compatible with standard RGB cameras or smartphones and thus widely available at relatively low costs, while HSI captures and processes image information across a wider range of wavelengths at higher spectral resolution for each pixel in an image. Ocular RGB-based analysis showed strong predictive performance with R² = 0.9880 and a Pearson correlation coefficient of 0.9945. HSI revealed distinctive spectral signatures in jaundiced skin not visible in RGB images. Jaundiced skin exhibits notably reduced redness at 600–740 nm, but elevated reflectance emerges around 750–850 nm, followed by three key crossover points at approximately 850 nm, 950 nm, and 980 nm versus normal skin. These near-infrared spectral features indicate that extending spectroscopic jaundice diagnosis beyond 740 nm could enhance diagnostic accuracy.

Introduction

Jaundice, a condition characterized by yellowish discoloration of the skin and sclera, arises from elevated serum bilirubin levels and is a frequent clinical manifestation in adult patients with various diseases such as hepatic dysfunction, hemolysis, and biliary obstruction, as well as in neonates and individuals with inherited metabolic disorders^1,2,3,4. Timely and accurate monitoring of bilirubin levels is essential for assessing disease progression, guiding treatment decisions, and identifying complications such as hepatic failure^3,5,6 in patients with chronic liver diseases or acute hepatic injury and risk of kernicterus in neonates^7,8. Venous blood sampling to quantify total bilirubin (T-Bil) and direct (D-Bil) levels has been the gold standard for diagnosing jaundice^9,10,11 and demonstrates high accuracy and specificity^2,12,13. Many studies have also explored population-specific bilirubin metabolism variations and reference thresholds^14,15,16,17. Despite their reliability, they are invasive, require clinical infrastructure, and do not support continuous or point-of-care assessment, and hence unsuitable for repeated use in neonates or home-based monitoring^18,19,20.

As the degree of yellowish discoloration of the skin and sclera correlates with severity of jaundice, it is plausible that the degree of jaundice might be assessed with optic-based methods to provide real-time, non-invasive, and repetitive evaluation of jaundice in clinical and home-based settings. Therefore, optical tools such as Bilicheck and JM-103 were developed to estimate bilirubin non-invasively via skin reflectance^21,22,23 for reducing invasiveness and providing convenience. Transcutaneous Bilirubinometry (TcB) devices are rapid and painless, making them ideal for neonatal screening. However, performance can be affected by skin pigmentation, device calibration, and anatomical location, necessitating occasional confirmation via blood sampling^24,25. Diffuse reflectance spectroscopy (DRS) techniques²⁶ have also been researched to analyze backscattered skin spectra to infer concentrations of chromophores like bilirubin and hemoglobin^27,28. Compared to TcB, DRS offers deeper tissue information and multi-parametric sensing. Although promising in wearable designs and point-of-care settings, its accuracy still requires validation across larger cohorts^26,29. Another seemingly convenient method associated with smartphone-based jaundice detection tools, such as the Dr. Jaundice app, uses camera images and color calibration cards to estimate bilirubin levels^30,31. These innovations democratize screening access, especially in rural or under-resourced communities. Nonetheless, accuracy may be influenced by lighting conditions, device variability, and skin tone, calling for improved algorithms and cross-device calibration protocols^23,31,32. Each of these methods contributes uniquely to the landscape of jaundice assessment, and many are complementary rather than mutually exclusive. Exploring their respective strengths and limitations is key to building a responsive, accurate, and context-sensitive jaundice detection strategy. Thus, more research using the analysis of optical images could pave the way for non-invasive, convenient, and rapid jaundice diagnostics towards point-of-care assessment.

The yellowish discoloration of skin and sclera observed in jaundiced patients indicates that RGB images could enable diagnosis of the condition, but varying lighting conditions and varying skin tone make diagnoses in practice unreliable. Therefore, we applied two spectral-image techniques with more spectral bands to analyze the spectral features associated with jaundice symptoms and compare with the conventional venous blood sampling method. One of the spectral-image techniques is image spectral-band expansion^{33,34,35,36,37}, particularly using machine learning methods to convert standard RGB images into hyperspectral images containing 13 spectral channels. A great advantage is its compatibility with standard cameras or smartphones, giving significant cost reduction compared to using traditional hyperspectral sensors. We further selected hyperspectral imaging as a technique that captures and processes image information across a wider range of wavelengths and separates the images into more (typically tens to hundreds) spectral bands for each pixel in an image^38,39,40. The spectral emission characteristics of an image pixel as measured by HSI cameras is a convolution of the lighting source’s spectral characteristic, the camera’s spectral sensitivity characteristic and the object’s spectral reflectivity and its underlying material characteristic^41,42. All HSI concepts represent a trade-off between spatial, spectral, and time resolution, spectral sensitivity range, as well as data volume generated for the hyperspectral image stack. For jaundice diagnosis described in this paper, we selected a starring HSI approach⁴³ in a relatively small (~64 cm³) and easy to use form factor that records a hyperspectral image cube of 28 independent spectral bands across a spectral range of 400 nm to 1050 nm with VGA spatial resolution in ~2 s, which were further derived to 141-channel hyperspectral bands ranging from 400 nm to 1000 nm.

Our studies reveal that our JaundiceAI-Mobile model exhibits strong predictive performance in estimating jaundice index from ocular color images, achieving an R² of 0.9880 and a Pearson correlation coefficient of 0.9945 when compared to the jaundice index obtained from the hospital. Aggregated HSI spectra, stratified by equivalent jaundice levels, revealed consistent patterns that align with established spectral features of jaundiced skin in both 13-channel multispectral and 141-channel hyperspectral datasets. Across two lighting conditions (halogen lamp and fluorescent tube), jaundiced reflectance spectra consistently showed lower intensity than normal ones below 550 nm, and higher intensity between 560 nm and 590 nm, corresponding to the yellow–orange region of the visible spectrum. These patterns confirm the diagnostic relevance of visible light reflectance. Beyond the visible range, HSI data revealed spectral characteristics extending into the near-infrared (NIR) range. For people with jaundice, a notable reduction in redness is observed in the spectral range of 600–740 nm, causing jaundiced skin to appear less reddish than healthy skin, but elevated reflectance emerges around 750–850 nm, followed by three key crossover points between jaundiced and non-jaundiced reflectance curves approximately at the wavelengths of 850 nm, 950 nm, and 980 nm. Such differences cannot be directly observed in the RGB images taken by usual cameras. These spectral transitions indicate that extending into the NIR range may enhance the accuracy of jaundice detection. Moreover, the consistency between RGB and HSI-derived results suggests a promising path toward integrating convenience and precision—such as leveraging the 13-channel color pattern to approximate the spectral resolution of HSI.

Results

Both the color images and HSI images echo the same jaundice traits detected by different cameras with different spectral resolutions. We will describe them separately.

Dual normalization for Mobile Jaundice AI

RGB scleral images from phone cameras are collected from patient directories. Each image undergoes a dual normalization process. The first normalization is performed during global pre-processing, in which three pixels from the top 95% brightness range and three from the bottom 5% are averaged to establish reference white and black values, respectively. The resulting normalized image is then used for metadata extraction and luminance feature computation. The second normalization is applied during target sampling, where two reference white pixels and two reference black pixels are selected together with three to five scleral pixels, while explicitly excluding eyelashes and blood vessels. The averaged reference values define a Reference Gray (RG) baseline. Thirteen scleral channels are computed by subtracting RG from the averaged scleral values:

$$\mathrm{Representative\_Sclera}=\mathrm{AveragedSclera}-\mathrm{RG}$$

(1)

Figure 1 demonstrates normalization effectiveness across different lighting environments, comparing original images (left column) captured under halogen lamp (top) and fluorescent tube (bottom) illumination with their normalized counterparts (right column). This standardization preserves clinically relevant scleral features while unifying image characteristics for accurate jaundice detection.

Fig. 1: Image normalization under different lighting conditions for Mobile Jaundice AI.

Normalized images acquired under halogen lamp illumination (top row) and fluorescent tube illumination (bottom row). For each lighting condition, the original color image (left) is shown alongside its normalized counterpart (right).

Each image yields a structured record including patient ID, lighting type, a jaundice index, and the corresponding 13-dimensional feature vector. A total of 90 records are generated for training the JaundiceAI-Mobile model.

The 13 channels expansion

Expanding the input dimensionality enables feature engineering, particularly when leveraging YO (Yellow-Orange) and BC (Blue-Cyan) group structures for self-/group-attention-based AI/ML training as indicated in Fig. 2. The integration of 13 distinct channels coupled with an adaptive learning rate mechanism contributes to model stability during abrupt transitions in feature space, promoting smoother convergence and increased resilience during training. The circular wavelength-oriented 13-channel provides more anchors to align with HSI wavelength for super resolution learning.

Fig. 2: Circular wavelengths used as the foundation for 13-channel color expansion.

The 1931 CIE chromaticity diagram illustrating the circular wavelength mapping, with regions corresponding to different perceived colors.

The RGB channels are transformed through the CIE L*a*b* color space based on major colors found from 1931 CIE chromaticity diagram in Fig. 2⁴⁴ and manifested in Fig. 3a.

Fig. 3: Bridging color channels to hyperspectral bands for AI/ML super-resolution.

a Eleven primary color channels augmented with two virtual channels (yellow average and yellow weighted). b Mapping of the 13 color channels to their corresponding hyperspectral imaging (HSI) wavelength bands.

The normalized image is then projected into 13 channels based on reference colors within the CIE L*a*b* space. For each pixel, the intensity in a given channel is computed from the Euclidean distance between its CIE L*a*b* value and the corresponding reference color as shown in Fig. 3a. A zero-distance yields a maximum intensity of 255, indicating perfect correspondence. This transformation effectively reconstructs the original three channels into a 13-channel representation, enhancing semantic differentiation across the color spectrum.

The 13-channel colors estimated from CIE L*a*b* colorspace are then mapped to their nearest corresponding HSI (band_ID, wavelengths) as shown in Fig. 3b with red dots. The figure shows only the left part of the entire 141 bands used in this study.

Mobile color image cases analysis

Two highly contrast scenarios are presented for comparison purpose, each case accompanies with halogen lighting image and fluorescent-tube lighting image, both of their Z-profiles are output below their corresponding eyes images.

For the case analysis from the color image, patient ID 018 has no jaundice (index 0.25 < 1.0), her halogen lighting eye/sclera image is placed at left and fluorescent-tube lighting image at right in Fig. 4a, their corresponding processed Z-profile output is shown at Fig. 4b. The Y-axis represents the normalized response relative to the reference gray (ref.gray) quantified with an 8-bit resolution (0 to 255). The X-axis is the wavelength (λ) of the 13 spectral channels used for data acquisition. The data for each illumination source are plotted as polylines to distinguish their characteristics: The reddish polyline represents data acquired under halogen illumination (lower color temperature). The bluish polyline represents data acquired under fluorescent-tube illumination (higher color temperature). The reference gray (ref.gray), a selected white label (chromaticity color palette) located outside the eye area in the image, is used as the normalizing standard. All subsequent figures in this study adhere to this color convention for illumination sources.

Fig. 4: Multispectral wavelength polylines from scleral ROIs of a jaundice-free subject.

a Input images of Patient ID 018 acquired under halogen illumination (left) and fluorescent-tube illumination (right). b Processed Z-profile derived from (a), showing the 13-channel spectral responses for Patient 018.

Under normal conditions, reflectance in the Yellow-Orange to Orange (YO–O) spectral channels remains low—approaching zero under halogen illumination and reaching up to ~80 under fluorescent-tube lighting. Conversely, the Blue to Cyan (B–C) channels exhibit elevated responses, with spectral troughs near zero or slightly negative (e.g., −20). These patterns contrast sharply with those observed under severe jaundice (see Fig. 5) or relative to reference gray values, underscoring the discriminative capability of this spectral decomposition.

Fig. 5: Multispectral wavelength polylines from scleral ROIs of a severely jaundiced patient.

a Input images of Patient ID 003 acquired under halogen illumination (left) and fluorescent-tube illumination (right). b Processed Z-profile derived from (a), showing the 13-channel spectral responses for Patient 003.

In contrast, patient ID 003 has very severe jaundice (index 37.23 > 20.0), same Input/Output arrangement as previous Fig. 4. Please see her sclera images at Fig. 5a and its corresponding 13-channel Z-profiles output in Fig. 5b below.

Compared with individuals without jaundice symptoms or reference gray values (see Fig. 4), subjects with severe jaundice exhibit elevated responses in the Yellow-Orange to Orange (YO–O) channels, with peaks reaching up to +120. Simultaneously, a pronounced reduction is observed in the Blue to Cyan channels, where troughs descend to approximately −80. These spectral deviations underscore the contrast between jaundiced and non-jaundiced conditions within the expanded color channel representation.

Comparison between severe jaundice and jaundice-free

Overlaying the severe case of Patient 003 (Jaundice Index = 37.23) from Fig. 5b as solid curves onto the healthy subject Patient 018 (Jaundice Index = 0.25) from Fig. 4b, shown as dashed curves, demonstrates that color images alone reveal substantial spectral discrepancies between severe and normal individuals, as presented in Fig. 6. These divergences—particularly the locations where the solid and dashed polylines cross—serve as key features for characterizing jaundice and underscore the diagnostic relevance of spectral crossover behavior.

Fig. 6: Spectral comparison between severe jaundice and normal cases.

Comparison of the severe jaundice patient (solid lines) and the normal subject (dashed lines). Cyan-bluish curves correspond to fluorescent-tube illumination, while warm orange curves correspond to halogen illumination. Red circles indicate the spectral crossing points between severe and normal cases under halogen lighting.

Mobile AI/ML model

The AI/ML model, as shown in Fig. 7, is created to fit for the data collected and transformed. The activation function tanh() is chosen because the data is normalized by Eq. (1) which can produce either of positive or negative inputs for predicting jaundice index.

Fig. 7: Block diagram and Python code snippet of the JaundiceAI-Mobile model architecture.

The tensor shape is denoted as (batch size = None, rows = 11, columns = 48 filters). The row dimension of 11 results from a kernel size of 3 convolving over 13-channel input data, yielding 13 − 3 + 1 = 11. The flatten layer collapses the spatial dimensions into a 528-element vector (11 × 48), followed by a final output layer that predicts a single Jaundice Index value.

The tanh() fits right into the algorithm above: boost jaundice index if certain channels (yellow to orange) while penalties/reduces jaundice index if some other channels (e.g., blue to green) are negative. Additional details of the model are provided in Supplementary Data S1.

Hyperspectral imaging and data processing

Raw interferograms, comprising 200 images per *_raw.bin file (488 × 648 pixels each), were collected under halogen and fluorescent-tube lighting sources, with HSI mirroring color imaging’s illumination.

For target regions of interest (RoIs), this study identified palm center (e.g., 7 × 7 pixels) correlating with jaundice stages. Skin with creases or not, forming three skin types, Homogeneous skin or creases or Mixing them, were compared for their efficacy in revealing jaundice. Palmar Creases (Dermatoglyphics) required optimal imaging with a fully extended hand to minimize blood interference. Palm creases are identified via Canny Edge Detection, focusing on edges exceeding the top half strength. Early jaundice in darker-skinned individuals may first manifest in palm creases due to unconjugated bilirubin’s affinity for elastin and collagen. This is attributed to creases being rich in collagen fibers, possessing a thinner epidermis, being less vascularized (allowing more bilirubin deposition), and being less pigmented.

*_raw.bin file preparation

The HSI jaundice detection analyzes the extended palm under halogen light. Processing begins by locating *_raw.bin files within each patient’s HSI_palm directory. These files contain time-lapse palm image stacks (interferogram). The palmTrack4Zprofiles.exe tool processes each raw palm image stack, identifying patient and lighting type, track palm center, process collected raw Z-profile into wavelength responses. Please consult the 3 videos (Movie A, B and C) in the Supplementary Information for its palm tracking details. The palm center (cross-point, marked by , of wrist-to-middle finger and thumb-to-pinky lines, aggregating four landmarks for stability) is tracked using moving average for its stability. Local normalization, using an “Integral Image”, is then applied to enhance Canny edge features for creases.

From each normalized palm image, a 7 × 7 pixel patches are extracted, averaging color and intensity. Three patch types are extracted: smooth skin (H), mixed skin/creases (M), and creases-only (Dermatoglyphics). These averages are stored as raw Z-Profiles, providing 200-point spectral fingerprints. Raw Z-Profiles are then converted from the time-space domain to the frequency domain using the Discrete Fast Fourier Transform (DFFT) with associated pre- and post-processes. This yields a wavelength response (spectrum), more useful for analyzing skin’s specific color responses. Background noise mitigation presents challenges. Traditional normalized reflectance:

$$\mathrm{normalized\_reflectance}=(\mathrm{Measured}-\mathrm{Dark})/(\mathrm{White}-\mathrm{Dark})$$

(2)

using 5% darkest/brightest image areas for Reference Black/White, failed due to the palm being brighter than background and unreliable NIR responses from wall patches. The hand spectrum’s maximum value normalizes its shape (Fig. 9a, right column), yield a subjective relative reflectance, representing the relative light reflected at each wavelength. Repeated for Homogeneous, Mixed, and Dermatoglyphics patch types, the final spectral Z-Profiles are plotted on the right, with raw Z-Profiles plotted on the left and original magnitudes in the middle column shown on the output chart. Finally, key information—patient ID, lighting type, jaundice index, and the full 141 spectral values for each skin type is recorded in a .csv file used to train model for jaundice detection.

Acquire spectrum Z-profile of 3 skin types from interferogram

The steps to obtain the spectrum from the raw Z-profile are as follows. Each element [i] in raw Z-profile[i]_types represent an average of an adequate visual area of 7 × 7-pixel patch from time-domain image[i], where i spans 0 to 199 along the X-axis. Data for three skin types presented in Figs. 9 and 10 are—homogeneous (top row), mixed (middle row), and dermatoglyphics (bottom row). Raw Z-profiles undergo sequential processing. Cubic interpolation linearizes interferogram values to ensure a linear phase, critical for accurate Fast Fourier Transform (FFT). The Z-profile is adjusted for even sampling in optical path difference (OPD) and centered on the initial centerburst. Apodization suppresses signal artifacts, followed by zero-padding to enhance spectral resolution. Discrete Fast Fourier Transform (DFFT) converts the processed Z-profile from a complex time-domain signal to frequency components, yielding frequency domain intensities, data is cleaned by removing 100 elements at the left side lobe and 15 at the right from 256 values and generate 141 wavelengths range 400 to 1000 nm. These intensities are normalized to enable consistent comparisons across patients, regardless of HSI camera or environmental variations. Finally, Frequency Intensity are converted to wavelength intensity (WL_Intensity) for spectral visualization.

The middle column displays WL_Intensity for each skin type, with the Y-axis retaining original intensity and the X-axis mapping image IDs to wavelengths in nanometers. Normalized data from this column are presented in the right column and stored in a corresponding .csv file, which contains three records, structured as follows: patient ID, Lighting Take#, ground truth Jaundice Index, skinType, and Z-Profiles (141 normalized intensity) for analysis and later AI/ML model training. A collection of 30 .csv files, gathered from 47 patients in this trial, is subsequently used to produce our final HSI analysis result.

Sampling 3 skin types by tracking

The following snapshot shown in Fig. 8 from our generated video illustrates a balanced exposure interferogram contains patient’s palm, the palm center is tracked reliably throughout the entire 200-image stack, and so do the palm creases are retrieved to make the 3 skin types for analysis, please refer to the 3 palm center and crease tracking movies (Supplementary Movie A, B and C) in Supplementary Information for details. However, the palm creases are confined within the yellow triangle indicated in Fig. 8.

Fig. 8: Palm-center and crease tracking for three skin types to construct raw HSI Z-profiles (interferograms).

Five keypoints are detected by the AI-based tracking module palmTrack4Zprofiles.exe. Two cyan lines intersect to define the palm center, marked by a red target symbol (a “+” within a circle). Palm-crease detection, shown as green edges, is confined within the yellow triangular region.

Figures 9a and 10a below, from left to right represent the transformation from raw Z-Profile (at left), to wavelength magnitude (in the middle), and finally normalized wavelength intensity (at right). The row from top is ‘H’omogeneous, the middle row is the ‘M’ixed, and bottom row is the ‘D’ermatoglyphics skin type.

Fig. 9: Normalized Z-profiles and color-to-HSI wavelength linkage for a healthy subject.

a Processing pipeline for three palm skin types, converting raw interferograms to normalized HSI Z-profiles via DFFT for a non-jaundiced subject (Jaundice Index = 0.25, Patient ID 018) under halogen illumination. b Magnified central chart from (a) as an example, illustrating the linkage between HSI bands and the 13 color channels.

Fig. 10: Normalized Z-profiles and color-to-HSI wavelength linkage for a severely jaundiced subject.

a Processing pipeline for three palm skin types, converting raw interferograms to normalized HSI Z-profiles via DFFT for a severely jaundiced subject (Jaundice Index = 21.11, Patient ID 032) under halogen illumination. b Magnified central chart from (a) as an example, illustrating the linkage between HSI bands and the 13 color channels.

Hyperspectral images quality statistics

Because of excessive noise presented in individual *_raw.bin files, aggregating these files based on similar jaundice levels is needed to reveal a clearer trend. Individual data are aggregated based on a predefined Table 1 below to mitigate noises present in individual Z-profiles, enabling the identification of group trends for further analysis.

Table 1 Common classification of jaundice levels based on jaundice index ranges

For the statistics of HSI data, the methodology grouped data derived from jaundice levels from Table 1, alongside a subset of usable HSI *_raw.bin files (30 out of 47 patient files, excluding those deemed over-/under-exposed, blurry, or otherwise unsuitable). Table 2 summarizes usable files per group.

Table 2 Number of valid *_raw.bin files grouped by jaundice levels

Grouping was necessitated by excessive noise present in individual files, rendering the application of AI/ML models infeasible at this stage.

Cases analysis for HSI

This study selects patient 018 from the 12 people listed in “No Jaundice” group in Table 2 as the first example, then compared with patient 032 (severe jaundice, index = 21.11).

The Jaundice-free cases analysis for HSI is described in the following. Figure 9a shows patient 018 under Halogen lighting with Jaundice index = 0.25 (no jaundice: index 0.25 < 1.0) with 3 by 3 charts as described above. The corresponding 13-channel color values are marked in blue in the HSI wavelength response charts as shown in Fig. 9b, which not only shows what human see is very limited, but also illustrates for the potential to extend low resolution RGB[-IR] spectrum into wider range of high-resolution HSI spectrum through a Super-Resolution AI model to transcend the usage of smartphones.

In this analysis, the yellow-orange polyline is utilized to represent the spectral data acquired from the patient presenting with jaundice, reflecting the characteristic yellow-orange pigment. Conversely, the bluish polyline denotes data from a normal subject without evidence of jaundice. This color convention is consistently applied throughout the figures to distinguish between the jaundiced and normal physiological states.

A collection of 30 .csv files, gathered from 47 patients in this trial, is subsequently used to produce our final HSI analysis result.

For the severe jaundice cases analysis with HSI, the Z-profile of patient 032 under halogen lighting (take H1-2) deviates significantly from the healthy individuals. This pronounced difference reinforces the utility of our method in identifying severe hyperbilirubinemia. The HSI process again start palm center tracking through the 200 interferogram images as previous case did, and collect the 3 types of skin in their corresponding raw Z-profiles as shown in the left column of Fig. 10a. The correspondence between the 13-channel colors and 141-channel HSI measurements is illustrated in Fig. 10b, mirroring the relationship shown in Fig. 9b that underpins the super-resolution framework for smartphone AI/ML model.

The processed results from raw Z-profiles are stored in its *_raw(21.11).png (index 21.11 > 20.0, very severe jaundice) as shown in Fig. 10.

The 13 highlighted points across 141 hyperspectral bands guide the RGB[-IR] Super-Resolution model in learning HSI spectral patterns. Only normalized spectra are comparable across files. The right column shows the normalized data, saved as HSI032H1-2*_raw(21.11).csv for further analysis and machine learning.

Here is a sample comparison of normal people from Fig. 9a vs. severe jaundice patient from Fig. 10a as shown in Fig. 11a. The homogeneous and mixed skin regions demonstrated no significant spectral differentiation in this case; consequently, only the mixed skin type is subsequently referenced. The absence of distinguishable signals between these two types is primarily attributed to their shared origin within a localized 7 × 7 pixel area of the palm center, particularly when distinct creases are insufficient or not sharply resolved within the small 49-pixel Region of Interest (RoI).

Fig. 11: Overlay of wide-range HSI Z-profiles from a healthy subject and a severely jaundiced patient.

a Comparison of mixed-type skin HSI responses between Patient ID 018 (cyan-bluish polyline, Jaundice Index = 0.25) and Patient ID 032 (yellow-orange polyline, Jaundice Index = 21.11). Severe jaundice is characterized by elevated yellow–orange wavelength responses, highlighted in the red-orange box, with a core increase typically observed in the 560–590 nm range, and reduced reflectance for wavelengths λ < 550 nm, as indicated by the purple dashed box. b Comparison of palm-crease HSI responses between Patient ID 018 (0.25) and Patient ID 032 (21.11). Severe jaundice exhibits an overall lower spectral response, except for two pronounced peaks near 800 nm and 850 nm.

However, Fig. 11a indicates that within the mixed skin type, a significant spectral difference is observed between healthy individuals and jaundice patients. This difference echoes established jaundice symptoms, exhibiting lower reflectance when λ < 550 nm and a wider range of elevated reflectance between the 550−600 nm wavelength, contrasting with the core elevated reflectance typically observed at 560−590 nm.

Nevertheless, jaundiced palm creases show dramatic darker from healthy person as shown in Fig. 11b, where it has generally lower reflectance than healthy person’s response, except between early NIR range till 855 nm.

Palm creases has variable searching distance where the pixels are collected at 2 × (7 × 7) pixels outward from palm center and identified within the defined palm triangular boundary. When the 98 edge pixels are reached only the top half Canny Edge strength 49 pixels are kept for analysis. Figure 11b demonstrates a salient spectral discrepancy. Specifically, the gray line with cyan markers represents a healthy individual, while the yellow line with red markers corresponds to a severe jaundice patient with a high index value of 21.11. The distinct dermatoglyphic difference highlights a promising area for investigation. Currently, no specific literature directly examining palm creases (dermatoglyphics) via hyperspectral imaging for jaundice detection is available, indicating a key area of focus for future study. Hand creases may prove crucial for discerning both direct and indirect (unconjugated) jaundice.

The comparison of the crossing points between these two extreme cases (normal vs. severe) in Fig. 11a for mixed skin type and Fig. 11b for dermatoglyphic skin type also reflects the HSI behavior observed later in the more comprehensive group comparisons. This consistency further supports the robustness of the spectral patterns identified across different skin types and jaundice levels.

AI predicted jaundice index vs. blood test

Although three sources of uncertainty—blood test reference, image quality, and inter-individual variability in bilirubin expression—could affect accuracy, the proposed model (fluorescent-tube lighting data trained version with help from ambient sunlight spectrum) demonstrated robust performance despite limitations in image fidelity during this trial. It achieved an R² of 0.9880 (RMSE = 0.1535) and a Pearson correlation coefficient of 0.9945 using 90 original images. The model was trained over 1000 epochs with SMOTE applied to address class imbalance. Despite minor blur in some scleral regions, the approach proves viable.

Evaluation results under various lighting conditions are presented below, beginning with halogen lighting in Fig. 12a. Halogen lighting data yielded a performance of R² = 0.9561 (RMSE = 0.2988), corresponding to a Pearson correlation coefficient (PCC) of 0.9778 for the model trained over 380 epochs, as shown in Fig. 12a.

Fig. 12: Prediction performance of JaundiceAI-Mobile under different lighting conditions.

a Model trained with halogen-illuminated data (R² = 0.9561, R = 0.9778). b Model trained with fluorescent-tube-illuminated data (R² = 0.9880, R = 0.9940). c Model trained with combined halogen and fluorescent data (R² = 0.9656, R = 0.9826).

In contrast, under fluorescent-tube lighting conditions, the same JaundiceAI-Mobile architecture achieved even higher accuracy with R² = 0.9880, as illustrated in Fig. 12b. Fluorescent-tube lighting images were acquired under mixed illumination—office light combined with daylight—while halogen lighting images were captured under halogen illumination alone. The fluorescent-tube dataset exhibited smaller error spikes in color predictions compared to the halogen set.

As expected, when all lighting conditions were combined, the overall model performance fell between the two, yielding R² = 0.9656, as shown in Fig. 12c.

Although the hybrid dataset—combining halogen and fluorescent-tube lighting data—was twice the size of the individual datasets, the same model architecture did not surpass the performance of the model trained solely on fluorescent-tube lighting data. The hybrid-trained model achieved R² = 0.9656, RMSE = 0.1853, R = 0.9826 and PCC(r) = 0.9842, with training halted at epoch 607 due to a plateau without further performance improvement.

For a more comprehensive evaluation of model performance and characteristics, readers are referred to Section II, “Supplemental Document for Jaundice AI Metrics,” in the Supplementary Information.

Briefly, Subsection 1 (“R² score versus PCC”) evaluates continuous regression performance for predicting the jaundice index (mg/dL). Subsection 2 (“Common AI/ML metrics”) then describes the underlying evaluation setup (Python internal matrices and confusion matrices) and reports performance at multiple clinical decision levels: (i) binary classification (normal vs. jaundiced), (ii) dual-class severity assessment, where both F1 and F2 scores, along with other metrics, are reported but F2 is emphasized due to the higher clinical cost of false negatives, with per-class and Macro-F2 summarized, and (iii) ordinal multi-class classification, reflecting disease progression from normal through mild, moderate, severe, and very severe stages.

Finally, Subsection 3 (“Mobile AI/ML Model”) analyzes internal model representations to enhance interpretability of the otherwise black-box architecture and includes an assessment of potential overfitting.

The main discrepancy between the ground truth jaundice index from blood test and the predicted jaundice index likely stems from limitations in the current snapshot acquisition method, leaving room for future optimization.

Compare with existing solution BiliScreen⁴⁵, or other similar methods, Table 3 summarizes the benchmark of bilirubin index(mg/dl) Prediction.

Table 3 Comparison of similar methods

HSI characterization

Hyperspectral AI/ML model training and inference were deliberately excluded at this stage due to excessive noise in individual *_raw.bin files from the trial. HSI model would have with similar structure with 13-channel color version, except its entire size would be log scale increased, especially its hidden layers can be much more complicated than color version. Stars from the input spans from (RGB header + 13 values) to (HSI header + 141 values).

Aggregate Z-profiles to find the trend

Averaging the individually normalized Z-profiles within each group effectively suppresses random noise, allowing underlying spectral patterns to emerge. This process aligns group signatures more closely with the expected ground truth and highlights meaningful inter-group differences. Analysis revealed no notable distinctions between localized Homogeneous and Mixed palm skin types. However, variable (Dermatoglyphics) palm skin pixel patches exhibited visible differences during this trial phase, warranting further investigation in subsequent phases to ascertain their dominant influence on the jaundice index. Enhanced hyperspectral imaging (HSI) quality with separated direct and indirect bilirubin indexes ground truth are essential in the next phase to clarify these ambiguities. The variable (Dermatoglyphics) patch focuses on hand creases within the triangular region defined by the wrist, index, and pinky finger bases. Nonetheless, over- and under-exposure in HSI images frequently obscures palm creases. Conversely, localized (Homogeneous, Mixed) palm skin patches, centered on the palm, display limited deviation.

Averaging data across these classified groups are summarized in Table 4. Group curves are plotted in Fig. 13 for ‘M’ixed skin type and in Fig. 14 for Dermatoglyphics. They amplified the visibility of spectral patterns and accentuated inter-group differences.

Fig. 13: HSI Z-profile spectral comparison across four jaundice groups for mixed-type palm centers under halogen illumination.

The halogen spectral distribution is shown in the background, with stronger intensity toward the early near-infrared region and diminishing intensity toward shorter wavelengths. Spectral crossing points between normal and severe jaundice groups are highlighted by circles, including crossings near 550 nm and 600 nm (yellow circles), as well as additional crossings in the near-infrared region around 740 nm, 850 nm, 950 nm, and 980 nm (green circles).

Fig. 14: HSI Z-profile spectral comparison across four jaundice groups for palmar dermatoglyphics.

Although severe jaundice generally exhibits lower overall reflectance, a relative increase is observed during the transition into the near-infrared region, particularly between approximately 690 nm and 855 nm.

Table 4 Valid data to be aggregated for this study

Spectral characteristics of jaundice from palm center skin

Observed spectral changes in jaundiced skin (‘M’ixed skin type) reveal both elevated and depressed reflectance at specific wavelength ranges. These special characteristics are described in Fig. 13 as follows.

From a spectral-response perspective, these effects can be understood by examining how bilirubin alters skin reflectance across different wavelength regions, this phenomenon is evident in two key regions. The Yellow-Orange Region (around 560 and 590 nm) directly corresponds to human visual perception, aligning with the visible yellowing of skin indicative of jaundice. This observation corresponds to the absorption spectrum of bilirubin well, which shows almost no absorption for wavelength longer than 550 nm⁴⁶. Additionally, elevated reflectance is observed in the early part of the Near-Infrared Region (around 750–850 nm), a range undetectable by the human eye and requiring Hyperspectral Imaging (HSI) for analysis of bilirubin-related spectral shifts. Refer to the blue-circled areas in the accompanying figure for detailed crossing points.

Conversely, reduced reflectance is noted in other spectral bands. The Blue-Green Region (λ < 550 nm) is particularly significant, as phototherapy for neonatal jaundice commonly utilizes light within this range for bilirubin degradation. This reduction of reflectance for blue-green region also matches the absorption spectrum of bilirubin, which exhibits strong absorption in the spectral range between 350 nm and 550 nm⁴⁶. Furthermore, a perceptible decrease in redness is observed in jaundiced skin within the Red to NIR Transition (600–740 nm), causing affected skin to appear less reddish than healthy skin. See the blue-circled areas for further insights into crossing points and segment variations.

In addition to the consistent crossing points in Fig. 11a (normal vs. severe) that align with those observed in Fig. 13 for the mixed-skin-type group within the visible spectrum, we also identified three notable crossing points in the Near-InfraRed region. Specifically, the reflectance curves for jaundiced and non-jaundiced subjects intersect at approximately 850 nm, 950 nm, and 980 nm, as highlighted in Fig. 13. These features further reinforce the spectral distinctions associated with jaundice across both visible and NIR wavelengths. These indicate alternating contrast regions where jaundiced skin reflectance falls below, exceeds, then again falls below that of healthy skin. These jaundice characteristics exhibited in Fig. 13, suggests that it would be beneficial to utilize HSI for jaundice detection.

Spectral characteristics with dermatoglyphics characters

In contrast to established jaundice detection targets, palm creases (dermatoglyphics) present distinct spectral characteristics due to their heterogeneous skin composition. Figure 14 demonstrates that severe jaundice exhibits elevated reflectance intensity exclusively within the 690–855 nm range compared to healthy palm creases. This spectral signature enables straightforward segmentation using two crossing points, though visual discrimination remains little to impossible within this near-infrared transition wavelength range.

Conversely, within the visible light spectrum (400–700 nm), jaundiced patients exhibit darker palm creases than healthy individuals. Group-wise analysis reveals that jaundiced palm creases demonstrate reduced spectral intensity beyond 860 nm compared to normal subjects.

Consistency check between 13-channel and HSI result

A comparison of the jaundice extrema of color sclera under halogen lighting is presented in Fig. 6 (13-channel) and Fig. 13 (HSI). Despite differences in precision, both methods exhibit consistent spectral patterns and anchor points in the visible light range (400–700 nm). Notably, both color image and HSI methods for normal and severe jaundice measurements display consistent spectral crossing points near 550, 560, 590, and 600 nm. Further, the segment of 550–560 nm and segment of 590–600 nm are sticky segments, where jaundice has only slightly higher intensity but not salient. These trends highlight the utility of the 560–590 nm window for jaundice discrimination in both RGB and HSI modalities. The consistency result of jaundice wave patterns, crossing points and common 13 anchor wavelengths provide the foundation for enabling Super-Resolution for smartphones.

Discussion

To ensure consistency from a purely jaundice-driven standpoint—regardless of imaging modalities (RGB versus HSI interferogram), camera models, or tissue types (sclera versus palmar skin)—Fig. 6 overlays the severe-jaundice spectral response of Patient 003 (solid curves, from Fig. 5b) with that of the healthy subject Patient 018 (dashed curves, from Fig. 4b). The overlaid spectra show clear and systematic discrepancies between severe and normal cases, demonstrating a strong visual differentiation that aligns with jaundice severity.

A similar pattern emerges in the HSI group analysis presented in Fig. 13, which is further supported by the case-level extrema comparison in Fig. 11a. Although the micro-level polylines in Fig. 11a appear jaggier than the smoother macro-level curves in Fig. 13 for mixed-type skin, both views highlight the same underlying spectral behavior. An analogous relationship is observed between the micro-view in Fig. 11b and the macro-view in Fig. 14 for dermatoglyphic skin. Across both skin types, the individual-level and group-level trends remain consistent, and their agreement is reinforced by matching spectral crossing points.

Furthermore, when comparing the normal and severe groups, both display a consistent crossing point near 550 nm. Severe jaundice exhibits a reduced response below approximately 550 nm, an elevated response from 560 to 590 nm (yellow-orange for jaundice patient), and only modest differences beyond 590 nm through 600 nm.

Importantly, the correspondence between RGB-derived and HSI-derived features remains strong. The halogen-illuminated coarse (orange) polylines in Fig. 6 show two crossing points between healthy and severe cases at 555 nm and 581 nm. These align closely with the finer-resolution HSI group curves in Fig. 13, which show analogous crossings near 560 nm and 590 nm under halogen illumination. This agreement indicates that, despite differences in modality and tissue sampling, the spectral signatures of jaundice are robust and reproducible across measurement systems.

The strong performance of the JaundiceAI-Mobile models in predicting the Jaundice Index from color images of patients’ eyes is largely attributable to a rigorously designed processing pipeline that integrates 1. dual normalization procedures (Ref. White for pre-processing and Ref. Gray for sampling), 2. targeted feature engineering (the 13 channels), 3. self-/group-attention mechanisms, 4. comprehensive hyperparameter optimization, and 5. both SMOTE and supplementary augmentation strategies. Together, these components form a robust framework that effectively addresses the challenges inherent to real-world, image-based jaundice estimation. Consequently, under fluorescent-tube illumination, the model exhibits excellent diagnostic capability, achieving an R² of 0.9880, R = 0.9940 and a PCC(r) = 0.9945 in this study.

As the dataset expands, the PCC of JaundiceAI may decline slightly due to the increased likelihood of encountering outlier conditions. This highlights the need for improved pre-filtering of non-jaundice-related scleral or palmar discoloration, as well as greater standardization of object positioning and lighting control. Such measures will be essential for ensuring that only diagnostically relevant sclera and palm features are captured for accurate Jaundice Index prediction.

The optical methods investigated here offer several advantages: they are non-invasive, timely, low-cost—utilizing standard cameras or smartphones—and suitable for convenient detection, particularly for individuals in remote areas with limited access to hospitals. However, there are some drawbacks that require further improvement. First, certain image regions may be overexposed to light, resulting in loss of color and intensity data. This issue can be mitigated through appropriate camera setup adjustments. Second, the sclera of the eye is a promising site for jaundice detection. However, due to its small area, imaging the sclera can be easily obstructed by eyebrows, blurred by blinking, or affected by head movement. The use of a fixture to stabilize the subject’s chin or facial position could facilitate clearer and more consistent scleral imaging. Similarly, imaging the palm may be affected by hand movement, leading to unstable results. This instability can be corrected through software methods such as the “Sampling 3 skin types by tracking” approach described in this paper. Such involuntary movements may cause pixel displacement in hyperspectral imaging (HSI), especially during longer capture durations—for example, approximately two seconds in this study. Therefore, implementing a setup that stabilizes the target area during image acquisition is essential for improving accuracy and reliability.

In conclusion, our studies demonstrate that the optical ways in this work have high potential for jaundice detection, giving strong predictive performance in estimating jaundice index from ocular color images, achieving an R² of 0.988 (R = 0.9940) and a Pearson correlation coefficient of 0.9945, compared to the data obtained from the hospital. The HSI revealed spectral characteristics beyond the RGB images taken by usual cameras. For jaundiced skin, a notable reduction in redness is observed in the spectral range of 600–740 nm, but elevated reflectance emerges around 750–850 nm, followed by three key crossover points between jaundiced and non-jaundiced reflectance curves approximately at the wavelengths of 850 nm, 950 nm, and 980 nm, indicating that the NIR range of HSI images may enhance the accuracy of jaundice detection.

Methods

Imaging setup for eye and palm detection

Figure 15a shows the setup for capturing hyperspectral images (HSI) & phone images (HSI & phone images) of the palm. The camera is mounted on a tripod. The subject extends their right arm horizontally forward, with fingers slightly spread and the palm positioned parallel to the arm. The hand and fingers are oriented straight ahead, enabling the camera to capture the entire palm effectively.

Fig. 15: Setup and camera for capturing hyperspectral images & phone images

a Setup of capturing the image for the palm. b Photo of the HSI camera. c Optical configuration of HSI camera.

Due to the ~2-s capture time required by the hyperspectral imaging (HSI) camera, involuntary movement during acquisition causes pixel displacement, so a software was developed to obtain the stationary images for palms (refer to the section of “Sampling 3 skin types by tracking”). On the other hand, for image spectral-band expansion, only RGB images, which allow rapid capture, were utilized.

Images for the eyes were taken with a setup similar to Fig. 15a.

Two lighting conditions were employed during imaging. The first uses standard ceiling lighting (fluorescent tube), while the second employs a halogen lamp, selected for its broader spectral range. To minimize specular or harsh reflections, the shade of halogen lamp is directed upward toward the ceiling, allowing light to diffuse gently across the scene. Only one light source was used at a time—when the halogen lamp was active, the ceiling light was turned off, and vice versa.

Figure 15b shows a photograph of the hyperspectral imaging (HSI) camera used in this study. Its detailed optical configuration is illustrated in Fig. 15c. The camera consists of an objective lens assembly, two liquid-crystal (LC) layers sandwiched between transparent electrodes, two orthogonal polarizers, and a CMOS sensor array driven by control electronics. These components are arranged in the order shown in Fig. 15c. The operating principle of the HSI camera is described in a later paragraph.

Image acquisition

Images are obtained through mobile devices and HSI camera.

Color image acquisition for this study utilized mobile devices, capturing photographs sized 4K×3K. Images were obtained in .heic format from iPhone 8 devices and .jpg format from Android phones. The Android phone’s output was calibrated to the sRGB gamut. While iPhone 8 offers 16-bit dynamic range capability, color images were standardized to an 8-bit depth per channel to ensure consistency across cameras. Neither camera type employed lossless compression formats, such as ProRAW or DNG (Digital Negative), which are typically preferred for medical applications. However, the cameras’ efficient image compression proved sufficient for the current study.

The HSI camera was used to take HSI images for Jaundice diagnosis. Here we selected a starring HSI concept, that operates based on the principle of Fourier transform spectroscopy using a liquid crystal variable retarder (LCVR) as a polarization interferometer⁴³. At the core of the system is a liquid crystal cell that introduces a voltage-controlled optical path delay between two orthogonal polarization components of incoming light. The process begins by linearly polarizing incident light and passing it through the LCVR, where the birefringent properties of the liquid crystal introduce a wavelength-dependent phase shift between the ordinary and extraordinary rays. This phase shift alters the polarization state of the light in a way that depends on its spectral content. A second polarizer, oriented parallel to the first, converts this polarization modulation into measurable intensity variations, effectively creating an interferogram—an intensity pattern that varies as a function of the applied voltage and hence the optical path delay. These interferograms are recorded simultaneously at every pixel of a panchromatic CMOS image sensor as the voltage applied to the LCVR is varied over time. The resulting sequence of images encodes spectral information at each pixel location across the camera’s sensitivity range between 400 nm to 1050 nm.

To extract the spectral information, a Fourier transform is applied to each pixel in the stack of recorded interferograms. This computational step reconstructs the full hyperspectral data cube, providing a spectrum at each spatial point in the image.

Data collection

Serum bilirubin levels measured during hospital admissions were retrieved and used as the reference standard.

To avoid spectral artifacts due to movements of patient’s hands, we spatially aligned the images in the interferogram image stack by translating and rotating images so that selected features of the patient’s hand overlap spatially throughout the interferogram. While this procedure does not correct for hand deformation, it corrects the more prominent variations in hand positioning that occur over the 2 s HSI acquisition time. After alignment of the interferogram images, Fourier transform was applied, so that the skin’s spectral characteristic can be evaluated in the wavelengths space rather than the phase delay space. This technique is not suitable for the sclera region of the eye unfortunately because the sclera is too small and could be easily blurred due to the blink of eyes.

For data from mobile camera, this study utilized 90 left-face images, each with visible sclera, acquired from 47 patients. These images were evenly distributed between halogen and fluorescent-tube lighting sources. Acquisition was performed from a 45-degree angle of the patient’s left face, with the patient looking forward, to ensure maximal scleral exposure. Both .heic and .jpg image types were processed at an 8-bit depth. While neither format employed a lossless compression default, no discernible issues related to image quality degradation were observed at the full image level. Challenges in image quality primarily arose from environmental factors, such as eye socket shadows from overhead (e.g., fluorescent-tube) lighting or flares in the scleral area caused by direct (e.g., halogen) illumination. Despite these external noise sources, effective methodologies were developed to mitigate such color data quality issues specifically within the scleral region.

Acquire spectrum Z-profile

The core part of FTIR camera process is DFFT Transformation, which transform time domain of interferogram data into frequency domain of spectrum data. The output from complexZ_profile = numpy.fft.fft(rawZ_profile) will be a complex array representing the spectrum. Then the magnitude of complexZ_profile is retrieved by Z_profile = numpy.abs(complexZ_profile), which is essentially does the following conversion on an interferogram through the DFT formula:

$$\mathrm{Spectrum}\left[k\right]=\mathrm{abs}\left({\sum }_{{\rm{n}}=0}^{{\rm{N}}-1}{\mathrm{Interferogram}[{\rm{n}}]{\rm{e}}}^{-{\rm{i}}\frac{2{\rm{\pi }}}{{\rm{N}}}kn}\right)$$

(3)

Where:

• Interferogram[n] refers to the palm center intensity in *_raw.bin that is tracked, which is collected into 1D Array of rawZ_profile as shown at left column of Fig. 9a or Fig. 10a.

• Spectrum[k] refers to the final output of numpy.abs(complexZ_profile) as shown in the middle column of Fig. 9a or Fig. 10a.

The output array Spectrum will contain the frequency components, k is corresponded to the spectral information of wavenumbers. Then finally, we map the wavenumber back to wavelength by using

$$\hat{{\rm{v}}}=\frac{1}{\lambda }$$

(4)

and calibrate them.

Ethical approval

The study was approved by the Institutional Review Board (IRB) of National Taiwan University Hospital (IRB No. 202211082RINC). This study used a hyperspectral camera to obtain still images of the body surface of included patients after informed consent, and assessed the correlations between the images and serum bilirubin levels obtained during the course of clinical care for the patients. Taking photos of body parts would not have influenced the serum bilirubin levels. Therefore, our study was purely observational and did not “assign human subjects to one or more health-related interventions to evaluate the effects on health or biological outcomes” and hence did not qualify as a clinical trial.