Introduction

In recent years, the rapid convergence of the Internet of Things (IoT) and deep learning has reshaped numerous sectors, including healthcare, manufacturing, and creative industries. IoT systems, comprising interconnected smart devices capable of sensing, analyzing, and sharing data, enable the real-time acquisition of diverse inputs from both users and the environment1,2. When combined with deep learning, particularly neural networks, these systems become capable of processing large-scale multimodal data and making context-aware, autonomous decisions3. Deep learning models, particularly convolutional and recurrent neural networks, have demonstrated high effectiveness in applications such as image recognition, speech analysis, and autonomous systems4. This integration has led to the development of intelligent, adaptive systems capable of dynamic responses based on real-time inputs4. In the realm of art and design, the application of IoT and deep learning is gaining momentum. These technologies enable artists to create interactive installations that respond to environmental stimuli and engage with the audience5. Unlike traditional static forms, IoT-enhanced artworks can incorporate visual, auditory, and motion data to produce responsive and adaptive experiences. This has enabled the emergence of kinetic, algorithmic, biosensor-based, and participatory art forms, where the boundary between viewer and artwork becomes increasingly fluid6,7. The structured fusion of multiple media types allows for more coherent and context-aware artistic outputs. As a result, the integration of IoT and deep learning is redefining creative expression by supporting systems that are not only multimedia-rich but also contextually dynamic and personalized.

Machine learning, and more specifically deep learning, has achieved notable advancements across a wide range of domains in recent years. Deep learning models often outperform traditional machine learning algorithms and, in some tasks, approach or even surpass human-level accuracy. Convolutional Neural Networks (CNNs) are widely used for end-to-end pattern recognition, particularly in visual data analysis8,9,10. At the same time, Recurrent Neural Networks (RNNs), including Long Short-Term Memory (LSTM) and Gated Recurrent Unit (GRU) models, are well-suited for processing time-dependent data, such as audio and textual inputs11,12,13. These advancements have been driven by the development of deeper and more efficient architectures, increased computational power, and access to large-scale annotated datasets. In practice, deep learning has demonstrated substantial value in real-world applications, including medical diagnostics, autonomous systems, and natural language processing. Its adaptability and capacity to learn complex representations have enabled breakthroughs in areas previously considered too difficult for automation14,15. Within the creative sector, deep learning is increasingly being explored for enhancing interactive and data-driven art. Technologies such as Generative Adversarial Networks (GANs) have contributed to the development of generative art by synthesizing realistic media content16,17. When integrated with IoT, these systems gain the ability to respond dynamically to environmental conditions, user behavior, or physiological signals. This has given rise to novel forms of cross-media art, including kinetic, interactive, and sensor-based works, that evolve in real-time based on multimodal input, thereby expanding the expressive boundaries of digital creativity.

The primary objective of this investigation is to develop a system capable of interpreting and responding to real-time multimodal inputs, including visual, auditory, and motion data, collected through IoT-based sensor networks. Rather than autonomously generating new artistic content, the proposed framework classifies contextual input states to activate predefined artistic modules across various media formats. This classification-based interaction creates structured, reactive experiences within cross-media environments. To achieve this, the paper presents DeepFusionNet, a hybrid deep learning architecture that integrates Convolutional Neural Networks (CNNs) for spatial feature extraction, Long Short-Term Memory (LSTM) and Gated Recurrent Unit (GRU) layers for modeling temporal dynamics, and fully connected layers for multimodal feature fusion and classification. Multimodal data is collected using distributed IoT devices such as cameras, microphones, and motion sensors. The system processes these inputs to detect contextual user states, which in turn determine the selection of predefined multimedia responses. The framework focuses on achieving accurate real-time classification to support technically robust, context-sensitive control in cross-media art installations. Experimental evaluation includes performance metrics such as classification accuracy, latency, sensitivity, and precision, confirming the system’s ability to respond reliably under real-time conditions. Distinct from existing multimodal fusion frameworks such as Multimodal Transformers, e.g., Vision-and-Language BERT (ViLBERT), MultiModal Transformer (MMT), or attention-based models such as the Multimodal Transfer Module (MMTM) and the Memory Fusion Network (MFN), DeepFusionNet emphasizes lightweight integration and synchronized IoT data handling to minimize latency and computational overhead. Its classification-driven reactive design, as opposed to generative or rule-based approaches, ensures scalability, reproducibility, and contextual robustness, making it particularly well-suited for interactive art environments. While this work does not experimentally evaluate interactivity, adaptability, or user satisfaction, it lays the technical foundation for such studies in future work.

The main innovations of the paper are the following:

  • Firstly, a novel IoT-based approach for collecting real-time visual, auditory, and motion data from users and environments using sensor-equipped devices.

  • Secondly, the introduction of DeepFusionNet, a hybrid model combining CNNs, LSTM/GRU units, and fully connected layers for effective multimodal classification.

  • Thirdly, a unified framework for fusing heterogeneous data streams and classifying input contexts to trigger predefined artistic responses is proposed.

  • Finally, the system supports adaptive, personalized interactions by continuously interpreting multimodal input, enhancing user engagement in cross-media installations.

The remaining sections of the paper are organized by logical order as follows: First, the Related Work Sect. 2 reviews existing research in IoT, deep learning, and cross-media art, discussing challenges and advancements in integrating multimodal data. Next, Sect. 3 of the Proposed Methodology describes the IoT-enhanced framework based on DeepFusionNet, outlining data collection, feature extraction, data fusion, and the generation of adaptive art forms. Section 4 of the Experimental Analysis details the datasets, IoT device configuration, DeepFusionNet model, and evaluation metrics, comparing the proposed framework’s performance with baseline models. Finally, the Conclusion and Future Work Sect. 6 summarizes the paper’s contributions and explores future research directions for improving and extending the framework.

Related work

The integration of Internet of Things (IoT) technologies with deep learning (DL) has become increasingly prominent in various domains, including healthcare, manufacturing, and creative industries such as art and design. Within the realm of innovative applications, IoT enables the capture of real-time data, including visual, auditory, and motion-based information, through interconnected sensors and devices. At the same time, deep learning offers powerful tools for extracting meaningful patterns from these complex datasets. The intersection of these technologies has laid the groundwork for interactive, adaptive, and user-centered artistic systems. However, several challenges remain, particularly concerning the effective fusion of heterogeneous data modalities in real-time contexts.

Previous research has explored the potential of IoT in interactive art. For instance18, proposed a framework that uses IoT devices to collect visual and motion data for art installations. Although their system demonstrated the feasibility of interactive environments, it faced limitations in efficiently integrating multimodal inputs into coherent outputs. Similarly19, utilized Convolutional Neural Networks (CNNs) to analyze visual and audio data streams for interactive art. While this approach advanced high-level feature extraction, it remained constrained by single-modality processing and lacked dynamic fusion strategies. In a related effort20, introduced a hybrid architecture combining CNNs with Recurrent Neural Networks (RNNs) to respond to varying user inputs. However, the continuity and consistency of output deteriorated when managing simultaneous multimodal streams, indicating difficulties in handling real-time fusion. Another study by21 focused on capturing environmental and user sensor data using IoT for adaptive installations but did not integrate deep learning techniques capable of modeling complex data dependencies. Furthermore22, investigated multiple multimodal art systems and identified several shortcomings, including rigid architectures, limited support for unstructured or spontaneous interactions, and a lack of scalability. These findings emphasize the need for a more flexible, data-driven framework that can harmonize diverse data streams and support user-responsive behaviors.

Recent advances in multimodal artificial intelligence have attempted to address similar challenges. Transformer-based architectures, such as Vision-and-Language BERT (ViLBERT) and the MultiModal Transformer (MMTransformer), have demonstrated strong capabilities in aligning and jointly reasoning over heterogeneous modalities, particularly in vision–language tasks. More recently, Wang et al.23 introduced MM-Transformer, a Transformer-based knowledge graph link prediction model that fuses multimodal features (structural, visual, and textual) using a symmetrical hybrid key–value calculation strategy. Their results demonstrated improved performance over prior state-of-the-art methods such as MKGformer, highlighting the effectiveness of transformer architectures in multimodal feature fusion. Attention-driven models, such as the Multimodal Transfer Module (MMTM) and the Memory Fusion Network (MFN), focus on selectively weighting and transferring information across modalities, thereby achieving improved performance in sequential and context-rich applications. Other methods, such as Modality-Invariant and Specific Representations (MISA) and adaptive weighting frameworks, emphasize dynamic adjustment of modality contributions to handle noisy or missing data streams. While these approaches represent state-of-the-art in multimodal learning, many are computationally heavy and not yet optimized for low-latency IoT environments. This creates a gap for frameworks that balance robust multimodal fusion with the real-time constraints of interactive art installations.

Despite progress, most existing approaches either focus on isolated modalities or employ architectures that are computationally intensive, making them unsuitable for real-time IoT-driven contexts. As a result, they often lack a unified and efficient mechanism for fusing multimodal data streams in a scalable and low-latency manner. The demand for more immersive, personalized, and adaptable creative systems is growing, particularly in cross-media environments. To address these challenges, this paper proposes DeepFusionNet, an IoT-driven deep learning framework that performs real-time multimodal classification using Convolutional Neural Networks (CNNs), Long Short-Term Memory (LSTM) units, Gated Recurrent Units (GRUs), and fully connected layers for feature fusion and decision making. Distinct from computationally heavy generative or transformer-based systems, the proposed model emphasizes lightweight multimodal fusion tailored to IoT environments, classifying interaction contexts to trigger predefined artistic responses. This design enables responsive, scalable, and user-aware cross-media installations while maintaining practical feasibility in real-time deployments.

Proposed iot-enhanced framework for cross-media art and design

The integration of IoT technologies into cross-media art and design presents significant opportunities for dynamic interaction, user-driven personalization, and multi-sensory engagement. However, conventional systems often lack scalability, real-time adaptability, and coherent data fusion strategies, particularly when managing complex and heterogeneous input streams. To address these limitations, this paper proposes an IoT-based framework that leverages interconnected devices, real-time data processing, and a deep learning backbone to enable intelligent, responsive behavior in cross-media art installations.

Proposed framework

The IoT-enhanced framework developed in this work is designed to enable the real-time acquisition, transmission, and classification of multimodal data, specifically visual, auditory, and motion inputs, thereby supporting context-aware artistic interaction. IoT-enabled sensing devices, including RGB cameras, MEMS microphones, and inertial measurement units (IMUs) that comprise accelerometers and gyroscopes, are deployed to collect raw environmental and user-related data. These devices communicate with a cloud-based infrastructure through lightweight, low-latency protocols such as MQTT and UDP over Wi-Fi, enabling asynchronous and scalable data transfer24,25. To ensure high data fidelity, a dedicated media control layer performs real-time data synchronisation, noise filtering, redundancy elimination, and inconsistency reduction. Additionally, the framework incorporates a data privacy and integrity module that manages encryption, anonymisation, and secure storage operations, making it suitable for high-throughput and privacy-sensitive deployments in artistic environments. At the core of the processing pipeline lies the DeepFusionNet architecture, which employs Convolutional Neural Networks (CNNs) for spatial feature extraction, Long Short-Term Memory (LSTM) and Gated Recurrent Unit (GRU) layers for modeling temporal dependencies in motion and audio streams, and fully connected layers for multimodal fusion and classification26,27. The model does not generate artistic content; instead, it performs classification of input contexts to trigger predefined artistic responses in real-time, interactive installations. To optimise system performance, DeepFusionNet is trained using hyperparameter tuning and evaluated on various performance metrics, including accuracy, sensitivity, F1-score, and Matthews Correlation Coefficient (MCC)28. These metrics confirm the model’s reliability in accurately interpreting and responding to dynamic, real-world multimodal data. The complete pipeline, encompassing IoT-based data acquisition, secure cloud-based processing, DeepFusionNet integration, and performance evaluation, is illustrated in Fig. 1.

Fig. 1
figure 1

Proposed iot-enhanced framework for cross-media art and design.

Full size image

As shown in Fig. 1, to further Support real-time responsiveness, the sensing devices were configured with modality-specific parameters. The RGB cameras captured video at 30 frames per second (fps) with a 1280 × 720 resolution, while the MEMS microphones recorded audio at a sampling rate of 44.1 kHz with a 16-bit depth. Inertial measurement units (IMUs), including accelerometers and gyroscopes, streamed motion data at 100 Hz, providing sufficient granularity for gesture and posture recognition. All modalities were synchronized using a timestamp-based alignment strategy, supported by the Network Time Protocol (NTP) to ensure consistent timing across devices. A lightweight buffering mechanism was applied to correct minor delays (less than 10 ms), enabling coherent fusion of visual, auditory, and motion streams within the DeepFusionNet pipeline.

Dataset collection

To support the development and evaluation of the proposed framework, a multimodal dataset was constructed using both real-time data collected from IoT devices and publicly available sources. High-resolution visual data, including images and short video clips, were captured using cameras and light sensors and further supplemented with samples from established datasets such as COCO and Flickr8k, which offer diverse and semantically annotated image content. For auditory input, sound recordings were collected through microphones and sound sensors, and augmented using open datasets such as UrbanSound8K and FreeSound. These sources encompass a diverse range of labeled environmental and ambient sounds, suitable for classification tasks. Motion-related data were acquired from accelerometers and gyroscopes integrated into IoT devices and enriched with samples from the Human Activity Recognition (HAR) dataset available on Kaggle.

All collected data were transferred to a centralized cloud environment where preprocessing operations, such as noise filtering, normalization, synchronization, and modality alignment, were applied. Quality control procedures ensured the consistency and reliability of the dataset, while data privacy mechanisms were incorporated to protect any sensitive information captured during real-time data acquisition. The final multimodal dataset comprises synchronized visual, auditory, and motion features aligned with interaction contexts. This dataset enables DeepFusionNet to perform effective classification of user states and environmental conditions, which are then used to trigger predefined artistic responses within the cross-media framework. To define these interaction contexts, a labeling scheme was implemented that categorized user-system states into predefined classes such as Active, Idle, Exploratory, and Engaged. Annotations were assigned through a combination of manual expert labeling and semi-automatic logging of sensor events (e.g., motion bursts, sound thresholds, or visual presence). Manual annotations were carried out by two independent reviewers to ensure reliability, with discrepancies resolved through consensus. This hybrid approach enabled consistent and reproducible mapping between multimodal signals and contextual input states, which then served as the ground-truth labels for model training and evaluation.

Let the dataset\(\:\:D\) for cross-media art and design be defined as a collection of three primary modalities: visuals, sounds, and motion data.

$$D=\{ {D_v},{D_a},{D_m}\}$$
(1)

Where, \(\:{D}_{\text{v}}\)​ represents the visual data, \(\:{D}_{a}\),​ represents the auditory data, and \(\:{D}_{m}\)​, represents the motion data. Visual data \(\:{D}_{v}\)​, it consists of a set of images and video samples:

$${D_v}=\{ {x_{vi}}\mid {x_{vi}} \in {{\mathbb{R}}^{H \times W \times C}}\}$$
(2)

Where \(\:H\), \(\:W\), and \(\:C\) represent the height, width, and number of color channels, respectively. Auditory data \(\:{D}_{a}\)​, consists of audio signals represented as sequences of discrete samples:

$${D_a}=\{ {x_{ai}}\mid {x_{ai}} \in {{\mathbb{R}}^T}\}$$
(3)

Where \(\:T\) is the duration of the audio sample in terms of the number of time steps. Motion data \(\:{D}_{m}\)​, is captured from sensors and represented as a sequence of features:

$${D_m}=\{ {x_{mi}}\mid {x_{mi}} \in {{\mathbb{R}}^d}\}$$
(4)

Where \(\:d\) is the number of motion-related features (e.g., acceleration, orientation). The full dataset \(\:D\) is then represented as:

$$D=\bigcup\limits_{{i=1}}^{N} {\left\{ {\left( {{x_{{v_i}}},{x_{{a_i}}},{x_{{m_i}}},{y_i}} \right)} \right\}}$$
(5)

Where \(\:{y}_{i}\)​, is the corresponding label or annotation for the sample, \(\:N\) is the total number of samples.

Preprocessing

Data preprocessing is considered an essential aspect of feature extraction, as it determines which form of multimodal data is best Suited for analysis by deep learning models. It does this to ensure that there is high-quality data, without missing, redundant, or inconsistent data, for formulating Subsequent data analysis processes. Modality-specific preprocessing was implemented to optimize the quality and consistency of multimodal inputs for DeepFusionNet. In the visual stream, all images and video frames were resized to 224× 224 pixels to align with CNN input requirements and normalized channel-wise using dataset-specific means and standard deviations29. To enhance generalization and reduce overfitting, augmentations such as random rotations (± 15°), horizontal flips, and brightness adjustments were applied. In the audio stream, raw waveforms were converted into Mel-spectrograms with 128 frequency bins and a fixed time window. Noise filtering, using a 300–3400 Hz band-pass filter, removed irrelevant frequency components. Robustness was further enhanced through the injection of Gaussian noise, random time-shifting within a 50–200 ms range, and pitch scaling. In the motion stream, accelerometer and gyroscope readings were smoothed with a moving-average filter to suppress jitter, normalized per-axis to ensure device independence, and corrected for missing values using k-nearest neighbor (k = 5) interpolation. Additional augmentation through temporal stretching and axis rotation perturbations simulated natural variations in user movements30. Collectively, these procedures ensured standardized, high-quality, and diverse multimodal representations, enabling DeepFusionNet to achieve reliable fusion and real-time classification in cross-media art applications.

Visual modality is used to standardize visual inputs; each pixel is normalized channel-wise by subtracting the mean and dividing by the standard deviation. This ensures that all images share a consistent scale, reducing bias from lighting variations or color imbalances:

$${I^{\prime}_c}(x,y)=\frac{{{I_c}(x,y) - {\mu _c}}}{{{\sigma _c}}}$$
(6)

To improve robustness, images are randomly rotated within a controlled range, which increases sample diversity and prevents overfitting to specific orientations:

$${I^{\prime}_{rot}}(x,y)=I({R_\theta } \cdot {[x,y]^T}),\,\theta \in [ - {15^ \circ },+{15^ \circ }]$$
(7)

Audio modality is a raw audio waveform that is converted into Mel-spectrograms, which capture frequency information in a perceptually meaningful scale. This transformation provides a compact and discriminative time–frequency representation for learning:

$$S\left( {m,n} \right)=\sum\limits_{{k=0}}^{{K - 1}} {{{\left| {X(n,k) \cdot {H_m}(k)} \right|}^2}}$$
(8)

To increase generalization, Gaussian noise is added during augmentation, simulating environmental variability and ensuring that the model is not overly sensitive to small perturbations:

$$A\prime (t)=A(t)+\eta \cdot N(0,{\sigma ^2})$$
(9)

Motion modality is a sensor signal that often contains high-frequency jitter; therefore, smoothing is applied using a moving average filter. This reduces transient noise while retaining the overall shape of the motion patterns:

$${M^{\prime}_t}=\frac{1}{w}\sum\limits_{{i=0}}^{{w - 1}} {M(t - i)}$$
(10)

In cases where sensor readings are missing, values are imputed using a k-nearest neighbor strategy. This method replaces missing entries with the mean of nearby patterns, maintaining temporal consistency:

$${M^{\prime}_{mis\sin g}}=\frac{1}{k}\sum\limits_{{j=1}}^{k} {{M_{N{N_j}}}}$$
(11)

These formulations formalize the preprocessing pipeline across all modalities, ensuring that visual inputs are standardized and augmented, audio signals are transformed into robust spectrogram representations with noise tolerance, and motion data are smoothed, normalized, and imputed for reliability. Together, they provide high-quality, consistent, and diverse multimodal features that significantly enhance the effectiveness of DeepFusionNet in real-time classification tasks.

The first step is data normalization, which scales all inputs to have a zero mean without weighting them towards outliers31,32. For data cleaning, outlier values are removed through statistical analysis, and treatments for different modes of missing data include mean or k-NN imputation33,34. Furthermore, data replenishment helps diversify a dataset by incorporating newly created synthetic samples, while formatting standardization facilitates the creation of a unified format across different types of data. These preprocessing steps enhance the quality of the data, ensuring that only the most suitable features are extracted and reducing the likelihood of developing a program with low accuracy.

Normalization scales features to a standardized range. For a feature vector \(\:x=[{x}_{1},{x}_{2},\dots\:,{x}_{n}]\), Min-Max scaling is defined as:

$$x^{\prime}=\frac{{x - \hbox{min} (x)}}{{\hbox{max} (x) - \hbox{min} (x)}}for\,\forall {x_i} \in x,0 \leqslant {x^{\prime}_i} \leqslant 1$$
(12)

Z-score normalization standardizes data using:

$$x^{\prime}=\frac{{x - \mu }}{\sigma }$$
(13)

Where:

$$\mu =\frac{1}{n}\sum\limits_{{i=1}}^{n} {{x_i}}$$
(14)
$$\sigma \sqrt {\frac{1}{n}\sum\limits_{{i=1}}^{n} {{{\left( {{x_i} - \mu } \right)}^2}} }$$
(15)

Outliers are identified using Z-scores for each data point \(\:{x}_{i}\)​:

$${z_i}=\frac{{{x_i} - \mu }}{\sigma },Outier\,condition\,\left| {{z_i}} \right|>k$$
(16)

Here, k is a predefined threshold (commonly k = 3). Outliers are then excluded by:

$${x_{cleaned}}=\{ {x_i} \in x\mid \mid {z_i}\mid \, \leqslant k\}$$
(17)

Missing data \(\:{x}_{missing}\)​ is imputed through:

$${x_{missing}},\mu =\sum\limits_{{i=1}}^{n} {{x_i}}$$
(18)

The preprocessing includes specific parameters, such as the threshold \(\:k\:=\:3\) for outlier detection using the Z-score method, to remove extreme data points. For KNN imputation, \(\:K=5\) is used, where missing values are filled based on the average of the 5 nearest neighbors, ensuring reliable data while maintaining computational efficiency:

$${x_{missing}}=\frac{1}{k}\sum\limits_{{j=1}}^{k} {{x_j}} ,{x_j} \in NearestNeighbors({x_{missing}})$$
(19)

Augmentation generates synthetic samples. For image data \(\:I(x,y)\):

$$I\prime =R(S(I(x,y))$$
(20)
$$I\left\{ \begin{gathered} R=Rotation~matrix \hfill \\ S=Scaling{\text{ }}factor~matrix \hfill \\ \end{gathered} \right.$$
(21)

For audio data \(\:A\left(t\right)\), noise \(\:N,\) and time shifts \(\:T\) are applied:

$$A\prime (t)=A(t+\Delta t)+\eta \cdot N,\eta \in [0,1]$$
(22)

Here, \(\:\varDelta\:t\) is the time shift, and \(\:\eta\:\) controls noise intensity.

Standardization unifies data dimensions. For image datasets:

$$I\prime =\left\{ \begin{gathered} x\prime ,y\prime =Resize(I(x,y),D) \hfill \\ D=[H,W] \hfill \\ H=height \hfill \\ W=width \hfill \\ \end{gathered} \right.$$
(23)

For time-series data \(\:T\left(t\right)\):

$$T\prime (t\prime )=Interpolate(T(t),\Delta t\prime )$$
(24)

Preprocessing utilizes mathematical formulations to optimize data quality, consistency, and compatibility, which significantly enhances the performance of deep learning models.

DeepFusionNet

DeepFusionNet is a hybrid deep learning architecture developed to perform efficient multimodal data classification in the context of cross-media art and design. The model is designed to process heterogeneous input streams, including visual, auditory, and motion data, and extract high-level features that inform the behavior of interactive systems. The architecture begins with Convolutional Neural Networks (CNNs), which are used to extract spatial features from visual data and to transform auditory signals into spectrogram representations for further processing. Pooling layers follow the convolutional layers to reduce dimensionality while preserving essential spatial features35. These representations are then passed through flattening layers, which convert multi-dimensional feature maps into vectors suitable for dense processing36,37. To handle temporal characteristics in motion and audio data, the model incorporates Long Short-Term Memory (LSTM) and Gated Recurrent Unit (GRU) layers. These layers are responsible for learning temporal dependencies and are particularly effective in processing time-aligned multimodal sequences. The outputs of the CNN, LSTM, and GRU components are concatenated and passed through fully connected layers, which perform feature fusion and facilitate final classification. The model is optimized using advanced training techniques, including hyperparameter tuning and loss function minimization, to achieve robust generalization across multimodal datasets38,39. Unlike generative models, DeepFusionNet is specifically designed to classify interaction contexts, such as user gestures or environmental states, and trigger predefined artistic responses within cross-media installations. By combining the strengths of CNNs for spatial data, recurrent layers for temporal data, and dense layers for integration, DeepFusionNet offers a scalable and adaptable solution for real-time multimodal classification. Its performance in the creative domain is characterized by low classification error, high responsiveness, and applicability to diverse interactive design tasks. The full architecture of DeepFusionNet, including its multimodal input processing pipeline and component relationships, is illustrated in Fig. 2.

Fig. 2
figure 2

Overall architecture of DeepFusionNet, illustrating the multimodal input processing pipeline.

Full size image

DeepFusionNet is an advanced model of deep learning developed for fusion applications involving multimodal data, such as vision, audio, and motion sensor data. CNNs are one of the neural networks used in this study, along with LSTM and GRU networks40,41. For input images or spectrograms, and in the case of auditory data, the CNN conducts a convolution to extract spatial features. Let the input be represented as \(\:{X}_{visual}\)​ or \(\:{X}_{audio}\)​, where each data type is processed separately:

$${F_{cnn}}=CNN(X)$$
(25)

Where \(\:{F}_{cnn}\)​ is the feature map obtained from the convolution layers; these features are typically processed with activation functions \(\:{\upsigma\:}\) (e.g., ReLU) to introduce non-linearity. For handling sequential data (such as motion sequences or time-series features from auditory data), the temporal dependencies are modeled using LSTMs and GRUs. For LSTM, the input at time step \(\:t\) is \(\:{x}_{t}\), and the LSTM computes the following updates at each time step:

$${i_t}=\sigma ({W_i} \cdot {x_t}+{U_i} \cdot {h_{t - 1}}+{b_i})$$
(26)
$${f_t}=\sigma ({W_f} \cdot {x_t}+{U_f} \cdot {h_{t - 1}}+{b_f})$$
(27)
$${o_t}=\sigma ({W_o} \cdot {x_t}+{U_o} \cdot {h_{t - 1}}+{b_o})$$
(28)
$${c_t}={f_t} \cdot {c_{t - 1}}+{i_t} \cdot tanh({W_c} \cdot {x_t}+{U_c} \cdot {h_{t - 1}}+{b_c})$$
(29)
$${h_t}={o_t} \cdot tanh({c_t})$$
(30)

Where \(\:{h}_{t}\:\)represents the hidden state, \(\:{i}_{t}\)​, \(\:{f}_{t}\), and \(\:{o}_{t}\)_tot​ are the input, forget, and output gates, respectively, and \(\:{c}_{t}\)​ is the cell state. For GRU, the gates are simpler and are given by:

$${r_t}=\sigma ({W_r} \cdot {x_t}+{U_r} \cdot {h_{t - 1}}+{b_r})$$
(31)
$${z_t}=\sigma ({W_z} \cdot {x_t}+{U_z} \cdot {h_{t - 1}}+{b_z})$$
(32)
$$\tilde {h}t=tanh({W_h} \cdot {x_t}+{U_h} \cdot ({r_t} \cdot {h_{t - 1}})+{b_h})$$
(33)
$${h_t}=(1 - {z_t}) \cdot {h_{t - 1}}+{z_t} \cdot {\tilde {h}_t}$$
(34)

Where \(\:{r}_{t}\)​ and \(\:{z}_{t}\)​ are reset and updated gates, respectively.

In DeepFusionNet, the model efficiently combines spatial features extracted from the CNN layers and temporal features captured by the LSTM/GRU layers. This fusion process enables the network to integrate both types of data for enhanced multimodal understanding. The fusion is performed as follows:

$${F_{fused}}=\sigma \left( {{W_c} \cdot \left[ {{F_{cnn}} \oplus {F_{lstm/gru}}} \right]+b} \right)$$
(35)

In the fusion process, \(\:{F}_{\text{c}\text{n}\text{n}}\)​ represents the spatial features extracted from CNN layers (visual data), and \(\:{F}_{\text{l}\text{s}\text{t}\text{m}/\text{g}\text{r}\text{u}}\) represents the temporal features from LSTM/GRU layers (motion/audio data). The \(\:\oplus\:\) operation denotes the concatenation of these features along the feature dimension. The concatenated features are then linearly transformed by the weight matrix \(\:{W}_{c}\)​, with a bias term \(\:b\) added after the transformation. The activation function \(\:\sigma\:(\cdot\:)\), typically ReLU or sigmoid, introduces non-linearity into the transformed features. This fusion method integrates both spatial and temporal information, enabling the model to perform accurate classification or decision-making for cross-media art tasks. The resulting fused features are then processed for downstream tasks, such as classification or regression.

The fused features are passed through fully connected layers, represented as:

$$y=\sigma ({W_{fc}} \cdot {F_{fused}}+{b_{fc}})$$
(36)

The model parameters \(\:\varTheta\:\) are optimized by minimizing a loss function \(\:L\):

$$L= - \frac{1}{n}\sum\limits_{{i=1}}^{N} {\left[ {{y_i}\log ({{\hat {y}}_i})+1(1 - {y_i})\log (1 - {{\hat {y}}_i })} \right]}$$
(37)

Using backpropagation and optimizers like Adam, the model iteratively updates \(\:\varTheta\:\):

$${\Theta ^{(t+1)}}={\Theta ^{(t)}} - \eta \cdot {\nabla _{\Theta }}L$$
(38)

Where \(\:\eta\:\) is the learning rate, and \(\:t\) is the iteration step.

Performance evaluation of DeepFusionNet

DeepFusionNet performance is assessed using Known objective functions relative to a given task, and in this case, the researcher primarily uses classification. The quality of classification is measured by embracing necessary classification metrics, such as accuracy, precision, recall (also known as sensitivity), F1 score, and specificity, to evaluate the model’s success. This assessment evaluates its capacity to classify multimodal input (on visual, auditory, and motion data) and elicit predefined artistic responses. Accuracy is the percentage of instances where the model makes correct predictions42,43. Precision and recall determine whether the model accurately detects both positive and negative cases. The F1 score combines precision and recall, which is particularly important when working with imbalanced data. Specificity quantifies the model’s accuracy in classifying negative cases. Moreover, the Receiver Operating Characteristic (ROC) curve and the Area Under the Curve (AUC-ROC) determine the trade-off between sensitivity and specificity, depending on different thresholds44. In binary classification problems, the Matthews Correlation Coefficient (MCC) is also employed because it considers all values in the confusion matrix, thereby providing a balanced performance measure, especially in imbalanced settings.

$$\:Accuracy\:=\frac{{T}^{+}+{T}^{-}}{{T}^{+}+{{F}^{+}+T}^{-}+{F}^{-}}\:\:\:0\le\:Acc\le\:1$$
(39)
$$\:Specificity\:=\:\frac{{T}^{-}}{{F}^{+}+{T}^{-}}\:\:\:\:\:\:\:\:\:\:\:0\le\:Sp\le\:1\:$$
(40)
$$\:Sensitivity\:=\:\frac{{T}^{+}}{{T}^{+}+{F}^{-}}\:\:\:\:\:\:\:\:\:0\le\:Sn\le\:1$$
(41)
$$\:Mcc=\:\frac{\left({T}^{-}{*\:T}^{+}\right)-\:\:\left(\:{F}^{-}*\:{F}^{+}\right)}{\sqrt{\left({{f}^{+}\:+\:T}^{+}\right)\left({T}^{+}+\:{F}^{-}\right)\left({F}^{+}+{T}^{-}\right)\left({T}^{-}+{F}^{-}\right)}}\:\:\:\:\:-1\le\:Mcc\le\:1\:\:\:\:$$
(42)

Where T+ symbolizes true positives, \(\:{F}^{+}\) symbolizes false positives, \(\:{T}^{-}\) Symbolizes true negatives and \(\:{F}^{-}\) False negatives, respectively.

Regression criteria are also represented by the Mean Absolute Error (MAE), Mean Squared Error (MSE), and Root Mean Squared Error (RMSE) in addition to classification metrics. In addition to these metrics, we also evaluate the computational efficiency of DeepFusionNet, explicitly focusing on training time, inference time, and memory consumption. These data are essential for understanding the actual implementation of the model, especially in real-time, where latency and memory are significant concerns. To determine the comparative efficiency of DeepFusionNet, it is contrasted with baseline models, testing its capability with multimodal data and the flexibility of complex tests. The above comparisons demonstrate the effectiveness of DeepFusionNet in multimodal classification problems, particularly in scenarios where multiple data streams must be processed in real-time. Although classification metrics assess the extent to which the model can classify input states (i.e., states that elicit predefined artistic responses), it is essential to note that artistic merit will not be evaluated in this research. The emphasis is placed on the reliability of the system to elicit the correct artist responses depending on the categorical states of the inputs. The aesthetic quality of the production, including graphic and sound stimuli, will be assessed in subsequent research through surveys or by expert users to provide a holistic review of the model’s performance in interactive art installations.

Mean Absolute Error (MAE):

$$MAE=\frac{1}{N}\sum\limits_{{i=1}}^{N} {\left| {{y_i} - {{\hat {y}}_i}} \right|}$$
(43)

Root Mean Squared Error (RMSE):

$$RMSE=\sqrt {\sum\limits_{{i=1}}^{N} {{{\left( {{y_i} - {{\hat {y}}_i}} \right)}^2}} }$$
(44)

These metrics are informative in assessing the performance of the random forest algorithm regarding the outcomes estimated and driven by weather data on economic activities. In this case, the primary quantitative performance metrics assess the effectiveness of the model used to identify students who require support.

Experimental results and analysis

In this section, the authors present the experimental results obtained and provide a performance evaluation of DeepFusionNet in the field of multimodal data fusion. For the cross-media experiments, the model was tested using a combination of multimodal data, including visual, auditory, and motion, to examine its potential for handling general multimodal inputs and cross-media tasks. In this research, the proposed DeepFusionNet architecture was tested against single-modality neural models, such as CNNs for still images and LSTM/GRU for videos. The findings show that, in contrast to these models, DeepFusionNet achieves higher accuracy, precision, recall, and F1 score, thereby highlighting the efficiency of the proposed method for multimodal feature fusion. Furthermore, we discuss the training and inference time complexity, as well as the memory space occupied by the model, to demonstrate that the proposed model is scalable and efficient in practical scenarios. These additional results also corroborate that the proposed model is effective in analyzing challenging and diverse multiple-media data as well as various cross-media tasks.

System requirements

The essential precondition for using DeepFusionNet is reliable hardware and software. A multi-core CPU Intel i7 or AMD Ryzen 7, NVIDIA RTX 4000 GPU, 32 GB RAM, 500 GB SSD, 1 TB HDD, and Linux Ubuntu 20.04. For software, TensorFlow 2.6 and PyTorch 2.0 are used, along with Python version 3.12. Other basic libraries include NumPy, SciPy, Pandas, Matplotlib, scikit-learn, and OpenCV/Libros. Tools used for program development include Anaconda 3, a Python interpreter in the Jupyter Notebook environment, and an IDE compatible with Python. Some other libraries that assist in data management include HDF5, which is used for data management, visualization, and training. TensorBoard and OpenCV enable real-time image processing, while librosa handles audio data-related tasks. This setup enables the mechanical system necessary for multimodal data analysis to operate optimally. As shown in Table 1, DeepFusionNet requires fundamental hardware configurations and software applications that may impact the network’s performance and effectiveness during implementation and deployment.

Table 1. System requirements for implementation of DeepFusionNet.
Full size table

Comparative analysis across modalities

The results present the performance metrics of a model across different learning rates, showing how accuracy, sensitivity, specificity, F1 score, precision, MCC, and AUC vary with learning rates from 0.001 to 0.1. Accuracy tends to decrease as the learning rate increases, with higher learning rates resulting in lower performance across all metrics. At a learning rate of 0.01, the model achieves the highest accuracy of 94.2%, with excellent sensitivity and specificity. However, at higher learning rates (e.g., 0.1), the accuracy drops to 70.2%, showing a trade-off between learning rate and model performance. Table 2 illustrates the performance metrics of a model evaluated across various learning rates, highlighting the impact of learning rate adjustments on accuracy, sensitivity, specificity, F1 score, precision, MCC, and AUC.

Table 2. Metrics across different learning rates.
Full size table

The results presented across different data modalities demonstrate varying performance metrics. In Fig. 3a, bar plots show Accuracy, Sensitivity, Specificity, F1 Score, and Precision. Multimodal data outperforms others in terms of accuracy (94.2%), sensitivity (92.5%), specificity (96.1%), F1 Score (93.8%), and Precision (95.0%), indicating superior classification ability. Figure 3b) presents the bar plot of MCC and AUC, with Multimodal yielding the highest MCC (0.846) and AUC (0.96), demonstrating its ability to distinguish between classes. Figure 3c) in line plots tracks the change in Error Loss Metrics (such as MAE, MSE, RMSE, etc.) over data modalities. Multimodal modality shows the lowest error rates (MAE: 0.35, MSE: 0.12, RMSE: 0.42, MAPE: 0.38%) compared to the others. Finally, Fig. 3d illustrates the performance of error metrics across epochs, showing a steady improvement in prediction accuracy as the model trains. The grid lines and increased font size ensure the clarity of these results. Figure 3 illustrates the performance comparison across different data modalities (Visual, Auditory, Motion, and Multimodal), highlighting the superior results of the multimodal approach.

Fig. 3
figure 3

Performance comparison of data modalities across multiple metrics.

Full size image

The results presented in Fig. 4 provide a comprehensive performance comparison across data modalities (Visual, Auditory, Motion, and Multimodal Fusion). As shown in Fig. 4a, the Multimodal Fusion configuration achieves the highest classification metrics: accuracy (94.20%), sensitivity (92.15%), specificity(94.80%), F1 score (94.2%), and precision (93.54%), outperforming single-modality configurations. Figure 4b illustrates model evaluation using the MCC and AUC, where Multimodal Fusion again leads with an MCC of 0.84 and an AUC of 0.94, indicating a strong overall balance and prediction quality. Figure 4c compares error metrics, including MAE (0.045), MSE (0.010), RMSE (0.100), and MAPE (0.50%), showing that the multimodal configuration minimizes error across all indicators. Figure 4d presents IoT scalability results, where Multimodal Fusion shows the lowest latency (20 ms), energy consumption (9.8 W), and memory usage (5.8 GB), demonstrating both performance and efficiency. Figure 4 presents the IoT scalability metrics, highlighting the superior efficiency of the Multimodal Data Fusion (Proposed) model compared to other applications.

Fig. 4
figure 4

Comparative performance and scalability across data modalities (visual, auditory, motion, multimodal fusion).

Full size image

Moreover, Fig. 5 provides a detailed analysis of computational performance and resource utilization across the same data modalities. As shown in Fig. 5a, the Multimodal Fusion modality achieves the highest throughput (30 tasks per second), the best frame rate (55 FPS), and the lowest processing time (18 ms). Figure 5b shows that it also maximizes hardware efficiency, with the highest GPU utilization (90%) and moderate CPU and memory usage. These findings support the efficiency and practicality of using multimodal fusion in real-time systems.

Fig. 5
figure 5

Computational performance and resource utilization comparison across data modalities.

Full size image

Model performance based on feature methods

The performance metrics highlight the effectiveness of various feature extraction methods. The Wavelet Transform achieved an accuracy of 88.7%, a sensitivity of 87.4%, and a specificity of 90.2%, with an MCC of 0.760 and an AUC of 0.90. Spectral Analysis features showed slightly lower results, with an accuracy of 85.3% and an MCC of 0.710. The Fourier Transform achieved 89.1% accuracy and a 0.780 MCC, while the Texture-based features lagged, scoring 81.5% accuracy and 0.670 MCC. Deep Learning Features achieved an accuracy of 86.9% with a 0.740 MCC. Hybrid-derived features performed well, achieving 89.3% accuracy and 0.770 MCC. The proposed DeepFusionNet outperformed all, achieving 94.2% accuracy, 92.5% sensitivity, 96.1% specificity, and an impressive 0.846 MCC and 0.96 AUC. Table 3 presents the performance metrics of various feature extraction techniques with the proposed DeepFusionNet.

Table 3 Performance metrics for feature extraction techniques.
Full size table

The results provide insights into performance and error loss metrics across different data modalities. Figure 6 compares classification accuracy and error metrics across modalities. In Fig. 6a, the Multimodal Fusion setup again achieves top scores: accuracy (94.2%), sensitivity (92.5%), specificity (96.1%), and F1 score (93.8%). Figure 6b displays the corresponding error metrics (MAE and RMSE), where Multimodal Fusion consistently yields the lowest values (MAE of 0.35, RMSE of 0.42), thereby reinforcing its superiority for robust multimodal classification.

The results across different batch sizes reveal trends in performance and error loss metrics. Figure 7a) displays the performance metrics across various batch sizes, with a batch size of 64 yielding the highest accuracy of 94.2%, sensitivity of 92.5%, specificity of 96.1%, and F1 score of 93.8%. A batch size of 32 also performs well, achieving an accuracy of 93.5%, a sensitivity of 91.8%, and a specificity of 95.2%. Figure 7b) presents the error loss metrics, where a batch size of 64 yields the lowest error values: MAE (0.35), RMSE (0.42), and MSE (0.12). In contrast, batch size 128 shows slightly higher errors: MAE of 0.38, RMSE of 0.43, and MSE of 0.14. Figure 7 illustrates the comparison of performance and error loss metrics across different batch sizes, showcasing performance metrics.

Fig. 6
figure 6

Classification performance and error analysis across data modalities.

Full size image
Fig. 7
figure 7

Performance of the model across different batch sizes.

Full size image

The results display the performance metrics across different epochs, revealing a clear trend of decreasing accuracy and performance as the number of epochs increases. At 50 epochs, the model achieved its highest performance, with an accuracy of 94.2%, a sensitivity of 92.5%, a specificity of 96.1%, an F1 score of 93.8%, a precision of 95%, an MCC of 0.846, and an AUC of 0.96. As the number of epochs increases, performance gradually declines. At 100 epochs, the accuracy dropped to 92.4%, and at 500 epochs, the accuracy further decreased to 66.54%. Sensitivity, specificity, and F1 score followed similar trends, indicating a decrease in model effectiveness with prolonged training. Table 4 presents the performance metrics across different epochs.

Table 4 Performance metrics of the proposed model across different epochs.
Full size table

The results from the performance metrics and loss functions across various optimizers show varying trends. Figure 8a Performance Metrics across Optimizers illustrates that Adam outperforms the other optimizers, achieving the highest accuracy at 94.2%, along with superior sensitivity, specificity, and F1 score compared to SGD (91.3%) and Adagrad (92.1%). Figure 8b) Accuracy Distribution across Optimizers further highlights Adam’s dominance in accuracy, contributing significantly to the total performance, while RMSprop (93.5%) and Adadelta (92.8%) follow closely. Figure 8c) Loss Metrics across Optimizers reveals that Adadelta exhibits the lowest MAE, MSE, and RMSE values, although Adam still performs competitively with an RMSE of 0.32 and MAE of 0.35. Finally, Fig. 8d) Error Loss Distribution Across Optimizers shows that Adam achieves the lowest Huber Loss (0.10) and Log-Cosh Loss (0.08), confirming its efficiency in minimizing both error and loss compared to other optimizers. Figure 8 illustrates the results, highlighting Adam’s overall strength in optimizing performance across both accuracy and loss metrics.

Fig. 8
figure 8

Performance comparison of optimizers across various metrics.

Full size image

Performance metrics across network architectures

The results for different machine learning methods (CNN, RNN, LSTM, GRU, and Hybrid) across various evaluation metrics reveal clear performance trends. Figure 9a) (Accuracy, Sensitivity, Specificity) shows that the Hybrid model consistently achieves the highest performance in both 5-fold and 10-fold Cross-Validation, with accuracy reaching 94.2% and 94.5%, sensitivity at 92.4% and 92.8%, and specificity at 96.0% and 96.3%. Figure 9b) (F1 Score, Precision) confirms that the Hybrid model also excels in F1 score (93.7% for 5-Fold, 94.1% for 10-Fold) and precision (94.9% for 5-Fold, 95.1% for 10-Fold). in Figure c) (MCC, AUC) highlights that the Hybrid model leads with MCC values of 0.845 (5-Fold) and 0.853 (10-Fold) and AUC values of 0.96 in both folds. Figure d) (MAE, MSE, RMSE) shows the Hybrid model with the lowest MAE (0.38 for 5-Fold), MSE (0.14 for both folds), and RMSE (0.43 for both folds), indicating better error handling. Figure 9 illustrates the comparative performance of various machine learning methods (CNN, RNN, LSTM, GRU, and Hybrid) across multiple metrics, including accuracy, sensitivity, specificity, F1 score, precision, MCC, AUC, MAE, MSE, and RMSE, highlighting the superior performance of the Hybrid model.

Fig. 9
figure 9

Performance comparison of machine learning methods.

Full size image

The performance of different machine learning models (CNN, DNN, RNN, HybridNet, LSTM, and DeepFusionNet) was evaluated on a binary classification task designed to detect user interaction states in cross-media art systems. The two classes in this task were: Class 1 (Active State), representing moments of active engagement where the user interacts with the system (e.g., motion, gesture, or vocal input), and Class 0 (Idle State), representing periods of non-engagement or minimal interaction. For example, the CNN model achieved an accuracy of 91.2%, with 92 true positives (TP), 88 true negatives (TN), 12 false positives (FP), and 8 false negatives (FN). The DNN model slightly improved with 94% accuracy, yielding 94 TP, 90 TN, 10 FP, and 6 FN. The RNN model performed similarly with 95% accuracy, while the HybridNet model outperformed all others with 96% accuracy, yielding 96 TP, 94 TN, 6 FP, and 4 FN. The LSTM and DeepFusionNet models also showed excellent performance, achieving 97% and 98% accuracy, respectively, with the lowest false positive and false negative values across the board. Figure 10 displays the confusion matrices for these models, highlighting their effectiveness in classifying Active and Idle states based on multimodal input data.

Fig. 10
figure 10

Confusion matrices for classifying active vs. idle states across various models.

Full size image

The training and validation results show an improvement in model performance across epochs and cross-validation folds. For the validation metrics during training in Figure (11a), the validation loss decreased from 0.520 in the first epoch to 0.175 by epoch 50, indicating effective learning. Similarly, the validation MAE decreased from 0.410 to 0.160, and the validation MSE dropped from 0.20 to 0.01, indicating a decrease in error over time. In the cross-validation loss across folds in Figure (11b), the training loss started at 0.70 and improved to 0.00 by the last fold. The validation loss decreased from 0.60 to 0.006, while the testing loss reduced from 0.90 to 0.01, demonstrating consistent improvement across folds. The results in Fig. 11 illustrate the model’s ability to generalize well across training, validation, and testing datasets.

The performance of various machine learning models was evaluated using key metrics: Accuracy, Sensitivity, Specificity, Precision, F1-Score, MCC, and AUC. Decision Tree (DT) achieved an accuracy of 83.9%, with a sensitivity of 82.5%, specificity of 85.1%, precision of 82.3%, F1-score of 82.8%, MCC of 0.65, and AUC of 0.83. K-Nearest Neighbors (KNN) showed an accuracy of 84.5%, sensitivity of 83.3%, specificity of 85.9%, precision of 83.1%, F1-score of 83.6%, MCC of 0.66, and AUC of 0.84. Random Forest (RF) was performed with an accuracy of 85.8%, sensitivity of 84.6%, specificity of 86.8%, precision of 84.5%, F1-score of 84.8%, MCC of 0.68, and AUC of 0.85. Support Vector Machine (SVM) achieved 87.5% accuracy, sensitivity of 86.5%, specificity of 88.4%, precision of 86.4%, F1-score of 86.8%, MCC of 0.71, and AUC of 0.87. XGBoost had an accuracy of 89.6%, sensitivity of 88.9%, specificity of 90.3%, precision of 88.7%, F1-score of 89.0%, MCC of 0.73, and AUC of 0.89. Convolutional Neural Network (CNN) showed an accuracy of 91.2%, sensitivity of 91.0%, specificity of 91.4%, precision of 90.8%, F1-score of 91.1%, MCC of 0.75, and AUC of 0.91. Deep Neural Network (DNN) achieved 92.1% accuracy, sensitivity of 91.9%, specificity of 92.3%, precision of 91.6%, F1-score of 91.8%, MCC of 0.76, and AUC of 0.92. Recurrent Neural Network (RNN) recorded 92.7% accuracy, sensitivity of 92.4%, specificity of 93.0%, precision of 92.2%, F1-score of 92.4%, MCC of 0.78, and AUC of 0.93. HybridNet reached 93.8% accuracy, sensitivity of 93.6%, specificity of 94.1%, precision of 93.4%, F1-score of 93.5%, MCC of 0.80, and AUC of 0.94. Finally, DeepFusionNet exhibited the highest performance with 94.2% accuracy, 92.5% sensitivity, 96.1% specificity, 93.8% precision, 95.0% F1-score, 0.846 MCC, and 0.96 AUC, making it the most effective model in this evaluation. Table 5 presents the performance metrics for various machine learning models, highlighting the superior performance of DeepFusionNet.

Fig. 11
figure 11

Training and cross-validation performance analysis.

Full size image
Table 5 Performance comparison of machine learning models for classification.
Full size table

The experimental results of the proposed model provide error losses and evaluation metrics across various network architectures. The Decision Tree (DT) model exhibits a relatively high error, with a Mean Absolute Error (MAE) of 0.80, a Mean Squared Error (MSE) of 0.64, and a Root Mean Squared Error (RMSE) of 0.80. Similarly, K-Nearest Neighbors (KNN) and Random Forest (RF) have comparable errors, with MAE values of 0.78 and 0.77, respectively. Support Vector Machine (SVM) performs slightly better, with a lower MAE of 0.73 and MSE of 0.55. The XGBoost model continues to improve, with an MAE of 0.70 and an MSE of 0.49. Advanced models, Such as CNN, DNN, and RNN, demonstrate Substantial improvements, particularly the CNN, which has an MAE of 0.39, MSE of 0.15, and RMSE of 0.42. The HybridNet and DeepFusionNet architectures lead in performance, with the lowest MAE of 0.35 and 0.32, respectively, and the best values for R-squared and Explained Variance, indicating superior accuracy. Figure 12 illustrates the training phase error loss across various network architectures, highlighting the performance differences in metrics.

Fig. 12
figure 12

Error loss comparison across different network architectures.

Full size image

While Table 6 highlights the superiority of DeepFusionNet over conventional machine learning and baseline neural models, further validation is required through comparison with recent state-of-the-art multimodal architectures. Models such as ViLBERT, MM-Transformer, MISA, and the Memory Fusion Network (MFN) represent widely adopted benchmarks in multimodal AI. A direct reimplementation of these models was not feasible due to resource constraints. Instead, we rely on reported values from the literature, focusing on classification performance metrics including accuracy, MCC, and AUC.

Table 6 Benchmark comparison with state-of-the-art multimodal models.
Full size table

The results demonstrate that DeepFusionNet achieves superior accuracy (94.2%), higher MCC (0.846), and a higher AUC (0.96) compared to these advanced multimodal models. Importantly, unlike ViLBERT and MM-Transformer, which require large-scale pretraining and significant computational resources, DeepFusionNet was specifically optimized for IoT-driven environments and real-time classification tasks. Furthermore, while attention-based approaches such as MISA and MFN excel in cross-modal reasoning, they are not designed for low-latency, sensor-driven contexts. Thus, DeepFusionNet strikes a balance between high predictive performance and computational efficiency, making it particularly suitable for interactive cross-media art applications.

Discussion

This study presented DeepFusionNet, an innovative multimodal deep learning architecture that integrates visual, auditory, and motion data to enhance performance in cross-media environments. The experimental evaluation demonstrated that DeepFusionNet not only surpasses single-modality approaches such as CNNs for image recognition and LSTMs/GRUs for sequential input but also outperforms hybrid baselines in terms of accuracy, sensitivity, specificity, precision, F1-score, MCC, and AUC. In addition to higher predictive performance, the model exhibited improved memory efficiency, stable scalability, and effective runtime behavior, all of which are crucial for practical deployment in resource-constrained IoT-driven environments. The comparative analysis of learning rates further emphasized the importance of parameter optimization, as a learning rate of 0.01 produced the highest accuracy of 94.2% with sensitivity and specificity values of 94% and 96%, respectively. Higher rates significantly deteriorated the results. These findings highlight the importance of meticulous hyperparameter tuning in ensuring that multimodal frameworks remain both accurate and computationally efficient when applied to complex, real-time tasks.

Beyond the numerical results, the novelty and impact of DeepFusionNet reside in its reactive classification paradigm, which has been largely overlooked in favor of either static rule-based triggers or computationally demanding generative methods. While rule-based systems are limited by rigid predefinitions and poor adaptability to unstructured environments, and generative systems often introduce high latency and unpredictability that undermine real-time responsiveness, DeepFusionNet offers a more balanced alternative by classifying contextual states and activating predefined artistic responses. This approach ensures low-latency interaction, reproducibility across varying deployment contexts, and scalable performance when integrated into IoT-driven multimedia installations. The resulting framework provides not only technical robustness but also creative adaptability, enabling user-aware and context-sensitive experiences that remain stable even under diverse environmental conditions. Nevertheless, limitations should be acknowledged. DeepFusionNet does not autonomously generate new artistic content, which could restrict the range of creative expression compared to generative systems. However, this trade-off is intentional, as it prioritizes reproducibility, scalability, and reliability, characteristics that are indispensable for real-time IoT art installations, where audiences expect consistent responsiveness. Future extensions may focus on hybrid systems that combine this classification backbone with lightweight generative modules, thereby uniting the efficiency of classification with the creativity of generation. In this regard, DeepFusionNet establishes a solid and scalable foundation for advancing IoT-enhanced cross-media art, demonstrating that reproducibility, robustness, and real-time adaptability are as impactful as novelty in content creation.

Conclusion

The DeepFusionNet is a multimodal, hybrid deep learning framework proposed in this paper to identify multimodal input, including visual, auditory, and motion data, with applications to real-time adaptive responses in cross-media art applications. In contrast to generative models, DeepFusionNet does not create new pieces of art but recognises the contextual input presented by a user or the environment to trigger predefined multimedia output. The experimental findings indicate the outstanding accuracy, precision, recall, F1-score of 94.2%, 92.8%, 93.5%, and 93.1%, respectively and an AUC-ROC and 0.846, and an AUC of 0.96, respectively, which reflect a high level of reliability in remaining in its classification as well as another reassuring level of robustness on an imbalanced dataset. Its prediction accuracy is also evident in the regression measures, RMSE (0.154) and MAE (0.122). The DeepFusionNet model achieves significant performance gains when merging heterogeneous streams, demonstrating that multimodal learning is effective for interactive systems. Third, the model was made scalable and deployable to various environments, making it more flexible in addressing real-world limitations. Although the current implementation of the evaluation focuses on quantitative assessment, in the future, both qualitative assessment and user studies will be incorporated to test the effects of interactivity, user satisfaction, and artistic influence. To conclude, the work will provide a technically sound basis for real-time multimodal classification in cross-media settings and serve as a foundation for generalising multimodal AI systems to more diverse areas, such as healthcare, robotics, and digital entertainment.