Introduction

Crop selection involves identifying the crops best suited for a specific geographical region based on various weather and soil conditions that impact their growth and productivity. It is a critical aspect of agriculture, as improper crop selection can significantly reduce yields. With advancements in technology and the application of Machine Learning (ML), crop selection has emerged as a vital area of research, helping to maximize crop yields on limited land. The key focus of this research is the efficient use of agro-meteorological data to identify the factors influencing crop yields and eliminate crops that are unsuitable for a particular region. This approach can minimize agricultural waste and prevent long-term crop deficits, which contribute to global hunger. Around 1.4 billion starvation deaths may occur over the current century in Asia1. As extreme weather events intensify and become more frequent due to climate change, the likelihood of breadbasket failures is expected to increase. A breadbasket failure is defined as a yield decline of at least 10%2. Given the scale of food shortages, effective crop selection is essential to addressing the needs of vulnerable populations. Therefore, prioritizing research in agricultural data analysis is key to selecting the most appropriate crops for each region. Food scarcity and crop waste are often linked to insufficient, inaccurate, and inconsistent data, with poor crop choices for specific regions being a central cause of the global food crisis3. Optimal crop selection is crucial for improving yields and addressing food insecurity.

To develop a robust model, one must be aware of the strong correlation between crop yield and the various factors affecting it. Understanding the correlations between soil characteristics and other environmental factors is important due to their immense effect on crop yields. Some of the primary environmental and soil factors affecting crop yield are temperature, humidity, rainfall, atmospheric pressure, soil moisture, water retention capability, pH, and macronutrients such as nitrogen, phosphorus, potassium, etc., in the soil4. Environmental factors closely related to crop yield should be prioritized to choose the best season of the year for an individual crop. Rainfall is one of the most important factors affecting crop yield since it affects seed germination, biomass generation, and fruit development stages. Besides rainfall shortages, excessive rainfall can also adversely affect crop growth, resulting in catastrophic disasters.

The world’s agricultural productivity is seriously threatened by global warming, as Earth’s temperature has risen by an average rate of 0.18° Celsius per decade over the past 50 years5. This is a major concern since such steep rises would quickly take the average temperature beyond the range considered ideal for optimal plant growth. Different crops require a specific range of temperatures to be maintained for healthy growth. Low temperatures can cause chilling injury in some crops which can hinder blooming and result in direct damage or even lessened crop vigor6. On the other hand, high temperatures have a direct impact on the crop’s pollination stage by reducing the viability of the pollen7. Atmospheric pressure is another significant factor that affects crop growth. Experimental research has shown that frequent changes in atmospheric pressure generally shorten the germination time, improve the growth rate of young plants, and cause more massive and rapid root growth8.

Crop selection also involves consideration of factors such as soil moisture, water retention ability, pH, and presence of macronutrients. Soil moisture signifies the amount of water currently available in unsaturated soil and greatly impacts crop health. For optimal nutrient uptake and healthy growth, different crops require different amounts of soil moisture at each stage of their growth cycle. The pores present in the soil provide a passage for moisture within the soil profile. Different soils have varying capacities for holding water depending on the size of the soil particles and the connectivity between the pores9. Another important soil feature affecting crop growth is its pH content. The pH scale measures the potential of hydrogen (negative logarithm of H + ion concentration) and determines the acidic or alkaline range of the soil. It affects the availability of soil nutrients to the crop.

Lastly, the macronutrients, which are the most essential parameters for deciding the fertility of the soil such as nitrogen (N), phosphorus (P), and potassium (K) are commonly referred to as the “big 3”10. Nitrogen is the most important nutrient for ensuring healthy plant growth and the ideal nutrient content in the crop during harvest. Potassium strengthens plants’ resistance to disease and plays a significant role in boosting crop yields and overall quality. Phosphorus aids in the storage and utilization of energy in the crop by ensuring normal crop development11. Thus, understanding the soil characteristics is necessary to create a reliable crop recommendation model. With the global population projected to reach 9.8 billion by 205012, the need for increased food production is more important than ever. A balance must be struck between increasing production and preserving the environment. Therefore, new and effective ways of recommending crops should be prioritized in research and studies. With the advancement of technology, crop recommendation systems are incorporating cutting-edge technologies such as remote sensing, IoT, and AI. As a result of these advances, these systems are expected to become more accurate and effective. Recent studies and research from around the world have shown the effectiveness of these systems in optimizing crop yields and resource use.

The proposed crop recommendation system utilizes ensemble architecture integrated with IoT sensors to accurately predict crop selection based on various agrometeorological data, achieving a high accuracy of 99.8%. The primary objective is to develop a robust model capable of accurately predicting the most suitable set of crops for a specific region. The system is validated using data from the website of Tamil Nadu government to compare the recommended crops suited for the geographical region. The primary region chosen for the study is Chengalpattu district in Tamil Nadu, India, an area prone to drought, which limits the suitability of only a few crops for optimal growth. This location is ideal for analyzing crops suitable for extreme conditions and served as proof of concept for the model’s applicability on a larger scale. A reference dataset is analyzed, containing 50 different crops suited for the area, along with their optimal soil and atmospheric parameters for healthy growth. As the global population continues to grow, the need for sustainable and efficient agricultural practices will only continue to grow. Crop recommendation systems will play a crucial role in meeting this need. Furthermore, ongoing research and development in this field will continue to drive improvements in agriculture and food security worldwide.

A detailed study of crop recommendations, rainfall forecasts, and drought predictions is given, explaining different methods used to improve these areas and any challenges faced. The research landscape is scrutinized, encompassing diverse approaches and techniques employed to optimize agricultural practices.

Rainfall forecasting in annual as well as non-monsoon sessions in Odisha (India) was studied in Zhang et al.13. They compared the results obtained from both Support Vector Regression (SVR) and Multi-Layer Perceptron (MLP) models for predicting rainfall. Data representing the annual total rainfall and relative humidity were gathered from the Department of Forest and Environment, Government of Odisha, between 1991 and 2015. The model’s input parameters included the average monthly temperature, wind speed, humidity, and cloud cover. Results suggested that the multiple regression analysis SVR model gave better results than the MLP model for annual rainfall prediction. The SVR model predicted better results for five non-monsoon months: January, February, April, October, and November, whereas for the remaining three non-monsoon months, March, May, and December, the MLP model predicted better results. They concluded that Long Short-Term Memory (LSTM) can be implemented in the future to obtain more accurate results.

The performance of eight different statistical and machine-learning models is compared to offer long-term daily rainfall prediction in a semi-arid climate in Sierra and Jesus14. They used cross-validation methods to reconstruct 36 years of daily rainfall data. The optimal hyper-parameters were selected based on the reconstructed series for each family of models. For predicting rainfall extremes, a generalized linear model using gamma-distributed error performed best but it lacked practical applications. Results suggested that all the models underestimated the variance of the observed series at daily time scales. According to them, neural networks gave the best performance among all eight models for predicting rainfall occurrence and intensity.

To improve the accuracy of daily weather forecasting, eight significant meteorological factors were chosen based on a correlation analysis between control forecast meteorological factors and actual rainfall in Zhang et al.15. Samples were divided using the K-means clustering method to model LSTM and correct rainfall forecasts for eastern China. The proposed output was based on the difference between real-time rainfall data and model-forecast rainfall. The outcomes were then contrasted with those obtained using the linear regression technique, the SVM, and the Deep Belief Network (DBN). Limitations of the proposed model included the prediction of higher Threat Scores (TSs) than actual values for moderate and heavy rain. A study that presents a deep forecasting model based on an optimized Gated Recurrent Unit (GRU) neural network to predict rainfall is presented in Fahad et al.16. For input, they used 30 years of Pakistan’s climate data, from 1991 to 2020. They extracted climate variables and eliminated any outliers that might limit the precision of forecasting. According to their achieved results, the proposed model gave better precision and accuracy in comparison to other state-of-the-art rainfall forecasting models. The output suggested that for each quarter of the year, the temperature has a negative association and air quality variables have a positive association with rainfall.

The Antecedent Sea Surface Temperature Fluctuation Pattern (ASFP) is used for predicting drought instead of extracting Sea Surface Temperature (SST) from one specific sea zone within a given period17. Input data were collected from the Colorado, Danube, Orange, and Pearl River basins with frequent droughts over different continents. For drought predictions, they compared three different ML techniques, e.g., SVR, Random Forest, and Extreme Learning Machine (ELM). The results suggested that in comparison with the ASFP-SVR and ASFP-RF models, the ASFP-ELM model is more effective at predicting space-time evolutions of drought events. A new hybrid intelligent model is developed and verified based on Convolutional LSTM (CNN-LSTM) for short-term meteorological drought forecasting18. The case study was conducted at two locations in Ankara province, Turkey. Based on comparisons with current benchmark models, the efficiency of the proposed model was verified. The findings indicated that the CNN-LSTM outperformed all the benchmarks by achieving minimal Root Mean Square (RMS) error for SPEI-3 and SPEI-6. They concluded that the proposed hybrid model was a reliable approach for predicting drought patterns.

Recent advancements in agricultural analytics have increasingly shifted toward high-dimensional forecasting and multimodal data integration to overcome the limitations of static models. Poudel et al.19 investigated drought monitoring and categorization by calculating SPI and SPEI across multiple time scales, demonstrating lag-based input features in Artificial Neural Networks (ANN) and Support Vector Machines (SVM) significantly reduce overfitting and enhance long-term forecasting accuracy. Complementing this, Jiang et al.20 explored a multimodal deep learning framework that fuses UAV multispectral imagery with weekly meteorological data for cotton yield prediction. By utilizing CNN feature extraction and deep fully connected layers, the study achieved high predictive precision (RMSE = 0.27 t/ha), highlighting the critical role of spatiotemporal data integration in precision agriculture. Furthermore, the application of transformer-based NLP models—specifically RoBERTa, ALBERT, and DistilBERT was introduced by Thankachan et al.21 to capture intricate interactions between soil nutrients (N, P, K) and environmental variables. Their system utilizes these advanced architectures to provide real-time, informed crop recommendations through interactive interfaces, proving the efficacy of attention-based models in handling diverse agricultural datasets.

A crop recommender system is presented in Bandara et al.22 by collecting environmental factors using Arduino microcontrollers. Naive Bayes (NB), SVM, K-Means clustering, and Natural Language Processing (NLP) were used to suggest one crop with site-specific parameters for the chosen land. According to them, the model could suggest the best crop to cultivate for maximum harvest. According to the findings, the proposed system was appropriate for both urban and rural areas and had an accuracy rate of more than 95%. The model relies on frequent feedback from users to obtain better results, which is one of the major drawbacks of its methodology. A Systematic Literature Review (SLR) is performed in Klompenburg et al.23 to extract the features and identify the algorithms that have been frequently used in crop yield prediction. Out of the 567 available studies, they chose 50 to further analyze the methodologies and features. According to their analysis, temperature, rainfall, and soil type were the most used features, and the ANN algorithm was the most applied model for crop prediction. According to further analysis, CNN, LSTM, and Deep Neural Networks (DNN) were the most frequently applied deep learning algorithms in these studies.

A comprehensive literature review is provided in Chlingaryan et al.24 with various ML approaches used for crop yield prediction and nitrogen status estimation in precision agriculture. The authors discuss comparative analyses between different ML algorithms (regression, classification, and clustering) and Deep Learning (DL) algorithms that can be conducted to identify the most accurate and efficient techniques for crop yield prediction and nitrogen status estimation. They describe the availability and quality of data, the choice of an ML algorithm, and the calibration and validation of the models to determine how accurate the predictions are. The limitations of the paper involve a lack of discussion on the challenges faced by researchers in developing countries and also a lack of hardware and software requirements needed for the successful implementation of ML techniques in precision agriculture.

As stated in Pande et al.25, statistical approaches are used to approximate the given crop data which includes the area and the soil type. The results revealed the most profitable crop type for planting. Similarly, Novel methodologies like Regularized Greedy Forests (RGF) have found their way into recommender systems. As mentioned in Johnson and Zhang26, RGF is modeled to select a crop based on the predicted yield rate, which, in turn, is influenced by multiple parameters. Additional features included in the system are pesticide prediction and online trading based on agricultural commodities. The use of other statistical algorithms like Support Vector Machines (SVMs), Artificial Neural Networks (ANN), Decision Trees, and Logit Regression can be seen in Shariff et al.27, Ragho et al.28, Jacques and Defourny29.

While the aforementioned studies demonstrate the efficacy of individual ML models, they often rely on static datasets and lack safeguards against model variance or environmental noise. To address these limitations, this research introduces a tri-stage predictive hierarchy. The fundamental novelty lies in the integration of an Intensified LSTM architecture with real-time IoT telemetry and a Genetic Algorithm (GA)-optimized Compound Ensemble. The significance of this proposed framework is characterized by three technical pillars:

  • Dynamic Feature Engineering: Unlike “reactive” systems, this model incorporates Intensified LSTM-derived rainfall forecasts and agrometeorological drought indices, specifically SPI and SPEI, as active features in the recommendation process. This allows the system to transition to proactive agricultural decision support.

  • Architectural Robustness: The system employs an unweighted majority voting scheme across 12 diverse weak classifiers to specifically mitigate prediction variance and sensitivity to sensor noise. This collective decision-making process achieved a benchmarked accuracy of 99.8%, significantly outperforming individual baseline models.

  • Scalable Precision via Minkowski Distance: To provide farmers with actionable flexibility, the engine utilizes an N-dimensional Minkowski distance-based similarity index. This enables the generation of a ranked roster of viable cultivars rather than a single classification, ensuring the recommendations are finely tuned to regional climate and soil variations.

Materials and methods

This section presents a detailed examination of the technologies used in the system, highlighting their significance and impact. The seamless integration of IoT sensors with an ML model is a central focus, illustrating how this combination played a pivotal role in realizing goals. The analysis will clarify how these technologies were strategically utilized to improve efficiency, streamline processes, and enrich a culture of innovation in research work.

IoT sensors

IoT sensors are vital components in the entire process of developing a recommender system for agriculture. They represent the initial step in harnessing the power of data-driven decision-making in farming. These sensors are instrumental in capturing live, dynamic data from the agricultural field, serving as the foundation upon which advanced predictive models and recommendations are built. One of the main functions of these IoT sensors is to collect the real-time data from the field. This data encompasses a wide range of parameters that are critical for agricultural planning. The different parameters play a crucial role in determining the health and productivity of the soil and subsequently, the crop.

The deployment of these sensors directly to the soil where the crops are to be grown is a crucial aspect of their functionality. These sensors, in direct contact with the crop-planting environment, can obtain highly accurate and context-specific data. Once these IoT sensors collect data from the field, they use a Wireless Fidelity (WIFI) module to transmit this data to a central database. The use of a WIFI module enables wireless and real-time data transmission, which is essential for timely decision-making. The collected data can be analysed, visualized, and used to create predictive models and recommender systems.

Fig. 1
Fig. 1
Full size image

7-in-1 soil sensor.

The 7-in-1 soil sensor in Fig. 1, stands as an advanced agricultural and environmental monitoring tool, offering valuable insights about the soil and its surroundings. This multi-sensor system is proficient in measuring seven pivotal parameters. Data transmission over long distances is enabled through the RS-485 communication protocol, and the power supply range of 12–24 V ensures adaptability to various energy sources30.

It provides data on various parameters, each of which correlates with different aspects of crop growth. These parameters include:

  • Soil Moisture: Measuring soil moisture is essential for efficient irrigation and maintaining plant health. It quantifies the water content in the soil, which is essential for optimizing water usage, ensuring crops receive the right amount of hydration, and preventing both waterlogging and drought stress.

  • Soil Temperature: It significantly influences biological and chemical processes within the soil, directly impacting plant growth and nutrient availability. Understanding soil temperature is essential for ensuring the optimal conditions for crops to thrive.

  • Soil Electrical Conductivity (EC): Soil salinity assesses the soil’s electrical conductivity is important for crop health. Elevated EC levels indicate a higher salt content in the soil, potentially harming plant roots and nutrient uptake. Monitoring soil EC is essential for preventing salt-related damage to crops.

  • Soil pH: Measuring soil acidity or alkalinity, directly influences nutrient availability to plants. Different crops thrive under specific pH conditions. Accurate pH monitoring is vital for creating an environment where crops can access the nutrients they need to grow robustly.

  • Nutrient Levels (NPK): Advanced soil sensors may determine the levels of essential nutrients like nitrogen (N), phosphorus (P), and potassium (K) in the soil. These nutrients are critical for plant nutrition and overall crop health. Monitoring their levels helps ensure that crops receive the right balance of nutrients needed to thrive.

The hardware architecture for real-time field data acquisition utilizes a 7-in-1 soil sensor (Fig. 1) interfaced with an Arduino Nano microcontroller for local processing. Data transmission to the central database is facilitated by an ESP32 WiFi module and an NRF24L01 transceiver, enabling synchronized real-time monitoring even in areas with limited connectivity. This setup ensures a continuous stream of soil moisture, temperature, EC, pH, and NPK data, which serves as the dynamic input for the ensemble model.

Fig. 2
Fig. 2
Full size image

Temporal trace of synchronized field data ingestion.

The empirical validity of the real-time deployment is demonstrated through the continuous ingestion of soil telemetry via the ThingSpeak API. As shown in Fig. 2, the system maintains a high-frequency polling rate (avg. 16 s), ensuring that the predictive engine receives a synchronized data stream of soil moisture (field1), EC (field2), pH (field3), and macronutrients like Nitrogen (field4), Phosphorous (field5), Potassium(field6) directly from the field trial site.

Machine learning

The use of ML becomes crucial due to the number of parameters involved, as increasing parameters significantly adds to the complexity. While it might be argued that one could identify suitable crops for a region by examining raw data, the complexity and non-linear relationships among data points and outputs causes significant challenges. Accurate predictions are particularly important as they can directly impact the livelihoods of thousands of farmers, making precision a top priority. The rainfall forecast and drought prediction modules enhance the model’s accuracy by providing precise estimations of these critical parameters, which can be used to calibrate the effects on soil conditions. The decision to develop these modules independently, rather than relying on third-party software or API calls, aims to maintain modularity and novelty in the model.

The significance of rainfall forecasting, drought prediction and crop recommendation modules cannot be overstated, as they collectively form the core components relying on the implementation of ML algorithms and models for their development and execution. In the domain of rainfall forecasting, a custom-coded LSTM model is pivotal, as elaborated in Section “Rainfall forecast”. This LSTM model meticulously crafted using renowned Python libraries such as NumPy, Pandas, and SciPy, stands as the linchpin of the rainfall prediction system. The inclusion of these libraries is not merely incidental but rather indispensable, ensuring the efficacy and precision of the predictive model. For drought prediction, an R-based code harnesses the power of SPI and SPEI indexes, as expounded in Section  “Drought prediction”. This approach provides a comprehensive methodology for forecasting drought conditions, adding a layer of sophistication to the prediction system.

The crop recommendation system employs a compound ensemble model that amalgamates insights from 12 distinct ML and deep learning models discussed in detail in Section “Crop recommender”. This ensemble model operates by aggregating unweighted votes from each constituent weak classifier, culminating in the determination of a singular, optimal crop recommendation based on the specified parameters. This approach diversifies the sources of information and enhances the robustness of the crop recommendation system.

Figure 3 explores the innovative architecture design of the system at the core of the Optimal Crop Recommender. This ground-breaking system capitalizes on the forefront of technological advancements, employing sophisticated ML methodologies and exhaustive data analysis to provide users with precise crop recommendations customized for their specific geographical region. The system is built upon a comprehensive architectural framework that adeptly capitalizes on IoT-based sensors, facilitating real-time data acquisition. This complex model seamlessly integrates different soil and atmospheric parameters, laying the groundwork for a resilient recommendation engine. By harmonizing these elements, the system excels in delivering bespoke insights, enabling optimal crop recommendations that are finely attuned to the unique requirements of individual geographic regions.

Fig. 3
Fig. 3
Full size image

Architecture diagram of the proposed model.

The system is organized into three main sub-categories: Rainfall Forecast, Drought Prediction, and Crop Recommendation System. These components are executed in distinct phases, each contributing crucial data to the comprehensive recommendation process. Here is a look at the system’s flow:

  • Rainfall Forecast (ML model): The system flow begins with the Rainfall Forecast phase, where an Intensified LSTM model is employed, details are mentioned in Section “Rainfall forecast”. It uses historical data from 1980 to 2022, to forecast rainfall for the year 2023 which is downloaded from the NASA website. The accuracy of the predictions is assessed by comparing the model’s results with actual rainfall data, often visualized through graphs to evaluate its performance. This forecast serves as a fundamental input for subsequent phases.

  • Drought Prediction (R-based Code): The Drought Prediction phase discussed in Section “Rainfall forecast” follows, wherein an R-based code utilizes rainfall forecast data as an input along with agrometeorological indices such as Standardized Precipitation Index (SPI) and Standardized Precipitation Evapotranspiration Index (SPEI) to categorize the severity of drought into three distinct levels: Low, Medium, and High. This classification is vital for recommending drought-resistant crops in regions like Chengalpattu, which are susceptible to water scarcity.

  • Deployment of IoT-based Sensors: The integration of IoT-based sensors in agricultural fields comes next, enabling the collection of real-time data, including temperature, NPK levels, pH, soil moisture, and conductivity. Real-time data collection from agricultural fields is paramount for its success. These sensors, employing ESP-32 WIFI and NRF24L01 Transceiver modules, transmit data to a centralized database for further analysis and use in the recommendation process.

  • Dataset Construction: The dataset in Fig. 4, serves as a crucial tool for refining crop selection and cultivation practices in the Chengalpattu region of Tamil Nadu. Covering 50 different crops and their optimal growth values for key parameters such as Rainfall, Drought Tolerance, Temperature, Soil Moisture, NPK, pH, and Soil Conductivity, it provides a comprehensive reference for the Compound Ensemble model. This dataset enables informed choices regarding crop selection, supporting precision agriculture. Overall, this knowledge base is a significant stride towards fostering sustainable and efficient agriculture in Chengalpattu, offering tailored solutions for crop management based on local climatic and soil conditions.

  • Crop Recommender Engine (Compound Ensemble Model): The data collected through the IoT sensors is transmitted to the Crop Recommender Engine along with forecasted rainfall and predicted drought. This engine employs a Compound Ensemble model, discussed in detail in Section “Crop recommender”, combining the insights of 12 different ML and deep learning models. This ensemble model takes an unweighted vote in which no adjustment or scaling is applied to the predictions based on performance or confidence. Each model’s output is counted as one vote, and the final decision is typically based on a simple majority. The ensemble model ultimately selects the most suitable crop from each constituent model, taking into account the given parameters. Additionally, it generates a ranked list of recommended crops based on various metrics, providing farmers with a range of options.

Various techniques and tools are used to provide farmers with customized crops based on their needs and environmental conditions. To achieve the best results, data samples need to be collected and analysed. The model is created based on the analysed data. Rainfall Prediction, Drought Prediction, and Crop Recommendation are the three subdivisions of the proposed model.

Fig. 4
Fig. 4
Full size image

Reference dataset containing optimal values for healthy crop growth.

Data characteristics and signal processing

The empirical basis of the environmental monitoring component is a longitudinal meteorological time series spanning from January 1, 1982, to December 31, 2023. The dataset comprises 15,340 daily observations of Relative Humidity (RH2M, %) and Precipitation (Rainfall, mm).

Initial data profiling revealed that \(RH2M\)maintains a stable distribution \((\mu=73.52,\,\sigma=7.88)\). Conversely, the Rainfall distribution is heavily right-skewed, characterized by a median of 0.37 mm against a maximum of 156.79 mm, highlighting the presence of extreme precipitation events. Prior to model ingestion, a noise profiling protocol was executed to ensure signal integrity. Physical constraint validation confirmed that all vectors remained within meteorologically viable bounds:\(0<RH\le 100\) and \(Rainfall\ge 0.\) While nine statistical outliers were identified in the humidity subset via Z-score analysis, they were retained as genuine climatic variations rather than sensor artifacts.

To mitigate the impact of electromagnetic interference and transient sensor noise, a discrete Kalman Filter was deployed at the Arduino gateway level prior to transmission. Missing values were addressed using a tiered “sandwich” imputation strategy: linear interpolation was applied to preserve local gradients, supplemented by forward and backward filling to ensure temporal continuity at the dataset boundaries.To address the numerical instability inherent in logarithmic operations on sparse rainfall data, a numerical stabilization step was implemented where \(Rainfall=0\)was replaced by an epsilon value \(\upvarepsilon=1 \times {10}^{18}\). Subsequently, a natural log transformation was applied to both features to normalize the distributions and reduce the influence of high-magnitude precipitation events on the objective function.

Rainfall Forecast

Intensified LSTM is used for rainfall forecasting in this work by enhancing the activation function of the existing LSTM model15. The intensified LSTM model is developed to encounter the exploding gradient problem faced by Recurrent Neural Networks (RNN), which occurs when large error gradients accumulate, resulting in massive updates to neural network model weights during training31. As a result, the model cannot learn from the training data. The LSTM network architecture consists of three gates named the Forget Gate \({F}_{t}\), the Input Gate \({I}_{t}\), and the Output Gate \({O}_{t}\), respectively. The Forget Gate is responsible for deciding whether information from the previous time step should be kept or forgotten. The equation of Forget Gate is given by,

$${F}_{t}=\sigma({x}_{t}*{U}_{f}+{H}_{t-1}*{W}_{f})$$
(1)

where \({x}_{t}\) is the input to the current timestamp, \({U}_{f}\) is the weight associated with the input, \({H}_{t-1}\) is the hidden state of the previous timestamp, and \({W}_{f}\) is the weight matrix associated with the hidden state. The Input Gate is used to quantify the importance of the new information carried by the input. The equation of the Input Gate is given as,

$$I_{t}=\sigma({x}_{t}*{U}_{i}+{H}_{t-1}*{W}_{i})$$
(2)

The Output Gate is used to determine the value of the next hidden state that contains information on the previous inputs. The equation of the Output Gate is given as,

$${O}_{t}=\sigma({x}_{t}*{U}_{o}+{H}_{t-1}*{W}_{o})$$
(3)

The sigmoid function is applied to all three gates that make the \({F}_{t}\), \({I}_{t}\), and \({O}_{t}\) a number between 0 and 1 to control the flow of information through the cell state, termed as the memory of the network. In contrast to the vanilla LSTM that employs a standard sigmoid activation function as,

$$\sigma\left(x\right)=\frac{1}{(1+e^{-x})}$$
(4)

Intensified LSTM embeds SiLU32 activation function \(s\left(x\right)\) in the gates of LSTM which is computed by the sigmoid function multiplied by its input as,

$$s\left(x\right)=x\cdot \sigma\left(x\right)$$
(5)

The major reason behind using such a function in the LSTM model is that it uses Constant Error Carrousel (CEC) to avoid the vanishing gradient problem, which is insufficient. The vanishing gradient problem occurs when there are more layers in the network and the value of the product of the derivative decreases until at some point the partial derivative of the loss function approaches a value close to zero and the partial derivative vanishes. The CEC set up the weight of the self-connected recurrent edge as 1 during back-propagation to enforce constant error flow where no variation is observed for learning. To handle long-term dependencies, memory needs to be reset for every input sequence. Four different scalers are used to find the best fit for the data set. They are the standard scaler, the max absolute scaler, the min-max scaler, and the robust scaler.

A maximum absolute scaler scales each feature by its maximum absolute value. Each feature is scaled and translated individually so that the maximal absolute value of each feature in the training set is 1.0. There is no shifting of the data, so sparsity is not destroyed33. The min-max scaler transforms features by scaling each feature to a given range. This estimator scales and translates each feature individually such that it is in the given range on the training set, for instance between zero and one34. The robust scaler scales features using robust statistics. This scaler removes the median and scales the data according to the quantile range. The Interquartile Range (IQR) is the range between the first quartile and the third quartile. Scaling occurs independently on each feature by computing the relevant statistics on the training set samples. The median and IQR are stored for use on later data using the transform method35. On comparing the above-mentioned scalers, it is concluded that the robust scaler is best for the data set used. The robust scaler of samples is calculated as follows:

$$Znew=\frac{(Zi-Zmedian)}{IQR}$$
(6)

where \(Zmedian\) is the median of data and IQR is the Interquartile Range, which ranges between the 1st and 3rd quartile. The study area is a drought-prone region, so the rainfall is considered an outlier to the model. In an abundance of outliers, the model leans towards drought and rainfall are neglected. The robust scaler prevents such instances from happening. Log Transformation is used further to get symmetrical distribution by reducing the skewness of the plotted data. To mitigate transient sensor noise and electromagnetic interference, a discrete Kalman Filter was implemented at the Arduino gateway level before cloud transmission.

As a persistent and severe meteorological phenomenon, drought is the second major factor that exerts a profound and detrimental impact on crop production worldwide. Its far-reaching consequences affect agricultural systems, economies, and food security. Water scarcity during prolonged dry spells significantly hampers plant growth, development, and yield potential. Drought-induced water stress disrupts crops’ normal physiological processes, inhibiting photosynthesis, nutrient uptake, and overall metabolic activities. Reduced water availability leads to stunted growth, wilting and increased vulnerability to pests and diseases. Moreover, the limited water supply restricts the choice of suitable crops, forcing farmers to rely on drought-tolerant varieties with lower productivity and market value36. Consequently, agricultural output dwindles, creating a ripple effect across the entire food supply chain. Effective drought monitoring systems and early warning mechanisms can also assist farmers in making informed decisions and adapting to changing climate conditions. Addressing drought challenges is crucial for ensuring global food security, safeguarding livelihoods and promoting resilient agricultural systems in the face of a changing climate.

Drought prediction

The second major part of the system is drought prediction. It plays a vital role in the agricultural decision-making tool, especially in the context of Chengalpattu which is a drought-vulnerable area. Selecting crops with a degree of resistance to drought is paramount in this region. These crops not only require less water for optimal growth but can also withstand harsh environmental conditions. Thus, the drought prediction module provides vital information regarding the selection of crops that are well-suited for the region’s unique challenges. The drought prediction leverages data from the previous rainfall forecast as input. By utilizing rainfall forecast data, the timeline and severity of drought are predicted in the Chengalpattu region. These predictions are indispensable for tailoring crop recommendations to the unique environmental challenges faced in this region.

The agrometeorological indices used in this work for drought estimation are SPI and SPEI. These indices are pivotal in categorizing the severity of drought into three distinct levels: Low, Medium, and High.

  • Standardized Precipitation Index (SPI): It is a widely used index for drought monitoring because it uses precipitation only. It can characterize drought or abnormal wetness at different time scales which correspond with the time availability of different water resources. It is more comparable across regions with different climates and at the same time is less complex to calculate than the other indexes37. It quantifies the precipitation deficit over different time scales, typically ranging from 1 to 24 months. SPI is calculated based on historical precipitation data and is used to assess how current precipitation conditions compare to the long-term average. A positive SPI value indicates surplus precipitation, while a negative value suggests a deficit. By standardizing the values, SPI allows for comparisons across regions and time frames. This index is valuable for understanding the duration and intensity of drought events.

  • Standardized Precipitation Evapotranspiration Index (SPEI): SPEI is an extension of SPI that incorporates evapotranspiration which is the water loss from the land surface due to evaporation and plant transpiration38. It accounts for not only precipitation but also the demand for water due to temperature and humidity. SPEI provides a more comprehensive view of drought conditions by considering both water supply (precipitation) and water demand (evapotranspiration). It is also standardized to facilitate comparisons and categorization based on drought severity levels.

The use of these indices in drought prediction is influenced by characterizing the severity and duration of drought events in Chengalpattu. While hydrological and groundwater indices provide valuable long-term data regarding aquifer storage, this study prioritizes meteorological indices (SPI and SPEI) due to the operational requirement for early-warning capabilities. Hydrological drought is typically a lagged phenomenon, manifesting only after a sustained period of meteorological deficit. In the context of the Chengalpattu agricultural decision-making tool, the goal is to influence crop selection prior to the onset of severe hydrological impacts. Furthermore, SPEI accounts for evapotranspiration, effectively capturing the ‘agricultural drought’ conditions, specifically soil water stress, that directly impact crop mortality rates in the early growth stages, making it a more immediate and actionable metric for crop recommendations than deep groundwater levels. By considering not only precipitation but also factors like evapotranspiration and temperature, the drought indices provide a more holistic understanding of drought conditions in the region. This information is integrated into the Crop Recommender model, allowing it to make recommendations that account for the unique challenges posed by drought. The categorization of drought into Low, Medium, and High levels is essential for tailoring crop recommendations. Crops that exhibit resilience to drought are recommended in regions with a high likelihood of severe drought. This approach not only enhances the likelihood of successful crop yields but also promotes sustainable agriculture practices by reducing water usage and mitigating the impact of drought conditions on crop production. Thus, drought prediction is a critical element of agricultural decision-making tools, especially in regions prone to drought. With the use of drought indices like SPI and SPEI, it provides valuable insights to refine crop recommendations that include drought-resistant varieties. By taking a comprehensive approach to drought prediction, this module contributes to more resilient and sustainable agriculture practices in the face of challenging environmental conditions.

Crop recommender

A prevalent observation across existing systems is the absence of safeguards against the development of variance or bias. Universally, these systems display high sensitivity to noise, presenting potential challenges. Moreover, many of these systems overlook the consideration of fluctuating soil parameters, including crucial chemical properties such as Nitrogen, Phosphorus, and Potassium. The motivation of this work is to address these shortcomings by minimizing oversights and implementing defenses against variance and bias. This approach aims to enhance the robustness of noisy data, contributing to more resilient systems. The data is applied to various standalone models such as SVM, ANNs, Logistic Regression and others, yielding an average accuracy of approximately 95%, precision of around 92%, and recall of about 95%. As anticipated, these standalone models exhibited susceptibility to noise. Its effectiveness is grounded in several fundamental principles:

  • Variance Reduction: The majority vote method significantly reduces prediction variance by mitigating the impact of individual model faults and inconsistencies. This approach combines outputs from multiple models, resulting in a more dependable and robust final forecast.

  • Combining Diverse Models: Ensembling facilitates the fusion of diverse models, each with its unique strengths and limitations. This diversity enhances the ensemble’s ability to detect a broader range of underlying patterns in the data, reducing the risk of overfitting to specific quirks in a single model’s training data.

  • Model Weakness Mitigation: Majority voting compensates for individual model constraints. In the presence of bias, misjudgement or inferior performance in one model. The ensemble can alleviate these concerns by leveraging the collective knowledge of multiple models.

  • Improved Generalization: Combining predictions from different models enhances generalization to previously unknown data. By integrating information and insights from various models, the ensemble adapts more effectively to the underlying data distribution.

  • Enhanced Predictive Performance: The majority vote strategy often leads to a significant improvement in predictive performance. In many instances, the ensemble outperforms the individual models by a considerable margin, underscoring its significance in real machine-learning applications.

  • Noise Reduction: Ensembling proves particularly beneficial when working with noisy data or outliers because noise affects individual models differently. Through majority voting, the ensemble can filter out erroneous or spurious predictions resulting in more robust and accurate outcomes.

To address variance, noise, generalization, and other such fundamental principles, an innovative approach is devised: the implementation of an unweighted majority voting system distributed across diverse supervised learning methodologies. This novel strategy leverages the well-established ensemble learning technique of majority voting, offering a structured mechanism to enhance model prediction performance through a systematic aggregation process. The effectiveness of this approach is grounded in several fundamental principles. The suggested crop recommendation system uses data from soil and atmospheric sensors on the agricultural land as well as the predicted rainfall and drought index. This comprehensive approach results in 8 different parameters for the model to analyse, ensuring that the crop recommendation is well-informed and tailored to the specific geographical region. The data collected through the IoT sensors is transmitted to the Crop Recommender Engine. This engine employs a Compound Ensemble model, combining the insights of 12 different ML and deep learning models. This ensemble model takes an unweighted vote from each constituent model, offering a single most suitable crop based on the given parameters. Additionally, it generates a ranked list of recommended crops based on various metrics providing farmers with a range of options.

This research is characterized as a regional case study, utilizing an agrometeorological dataset curated for the Chengalpattu district of Tamil Nadu. While the 50-crop reference values are tailored to this specific region, the core architectural framework, consisting of the Intensified LSTM and the Compound Ensemble is geographically agnostic. The system logic is designed for seamless transferability; it can be adapted to any drought-prone semi-arid zone by substituting the localized reference dataset while maintaining the established predictive hierarchy.

Weak classifiers

After thorough research, 12 algorithms are shortlisted to leverage the wisdom of the crowd’s approach that helps to overcome the hurdles of the existing systems. They are Logistic Regression, SVM, Naïve Bayes, Multinomial Naïve Bayes, Quadratic Discriminant Analysis, MLP, Decision Trees, Random Forest, Extra Trees, k – Nearest Neighbours, AdaBoost (Extra Trees Ensemble), Gradient Boosted Trees.

These 12 classifiers were selected to provide a diverse range of supervised learning methodologies, from probabilistic models (Naive Bayes) to kernel-based (SVM) and tree-based ensembles (Random Forest, Gradient Boosting). Rather than relying on a single model, the proposed framework leverages the collective decision-making of these ‘weak’ learners to mitigate individual model bias and improve generalization across the noisy, high-dimensional agrometeorological dataset.

Fig. 5
Fig. 5
Full size image

Technical architecture of the compound ensemble: illustrating the data pathway from heterogeneous weak classifiers through the hard voting consensus block to the minkowski-based similarity ranking engine.

The architecture diagram in Fig. 5 serves as a reference to the ML workflow developed for the Compound Ensemble, considering the aforementioned 12 weak classifiers. The final ensemble architecture is built by training each statistical model on its subset of data and the entirety of the data itself. An unweighted vote is then taken from each of these models. Essentially approximating the mathematical expression of,

$$\widehat{y}=argma{x}_{\left\{y\right\}}\sum_{\left\{i=1\right\}}^{n}{\delta}_{\left\{i,y\right\}}$$
(7)

where \(\widehat{y}\)represents the majority-voted prediction, \(n\) is the number of models, \(i\) is the index of each model and \({\delta}_{\left\{i,y\right\}}\) is an indicator function that equals 1 if \(y\) (the class label) that matches the prediction of the model \(i\), and 0 otherwise. This model implements a sophisticated crop recommendation system that reduces or nullifies most of the disadvantages that were cropped in the existing models. Due to this, the model is resilient to noise, handles variance, reduces bias, and automates crop recommendation by using the selected parameters. A Python class will be symbolically programmed by utilizing Python 3.10, and basic libraries, like NumPy, pandas, and SciKit-Learn.

The ‘Compound’ nature of the proposed ensemble stems from its two-stage architectural logic. In the first stage, the 12 heterogeneous weak classifiers, comprising linear, kernel-based, probabilistic, and deep learning architectures, process the multi-modal input vector (IoT soil data, rainfall forecasts, and drought indices) to produce a consensus class prediction. This stage utilizes a Hard Voting mechanism to minimize the variance associated with any single model’s architectural bias. The second stage integrates the ensemble’s output with the reference dataset via a Minkowski distance-based similarity index. Unlike simple majority voting systems, this interaction logic ensures that the recommendation is not just a discrete class label but a ranked list of alternatives derived from the n-dimensional proximity to the predicted optimal crop’s parameters. This sequential interaction between the predictive ensemble and the similarity-based ranking constitutes the ‘Compound’ framework, enabling the system to remain robust against localized sensor noise while providing actionable alternatives for regional agricultural diversity.

Distance metric

The compound ensemble returns the single most suitable crop that can be grown in the given parameters. A roster of recommended crops, all ranked in each metric, is intended to go through the route of a distance-based similarity index. This is a choice as it is simple to implement and execute while having a relatively high value to give a similarity coefficient between each crop. There are three distance calculation methodologies mostly used such as Euclidean Distance, Mahalanobis Distance, and Minkowski Distance. The N-Dimensional Minkowski Distance is a flexible metric used in mathematics and statistics to quantify the distance or dissimilarity between two points in a multidimensional space39. It generalizes several distance measurements, including the Euclidean distance and the Manhattan distance. The N-Dimensional Minkowski Distance between two points P and Q, in an n-dimensional space is defined as,

$$d\left(P,Q\right)={\left(\sum_{\left\{i=1\right\}}^{n}{\left|{x}_{i}-{y}_{i}\right|}^{p}\right)}^{\left\{\frac{1}{p}\right\}}$$
(8)

where \(d(P,Q)\) represents the Minkowski distance between points \(P\) and \(Q\), \({x}_{i}\) and \({y}_{i}\) are the coordinates of \(P\) and \(Q\) in the i-th dimension, and \(p\) is the order of the Minkowski distance (when \(p\) = 1, it corresponds to the Manhattan distance; when \(p\) = 2, it corresponds to the Euclidean distance).

Following an extensive assessment of systems40, 41, 42, it is determined that given the high dimensionality of the feature set, utilizing all n-features implies that the Minkowski distance would be the most suitable approach to arrive at a solution. The Minkowski Distance is fundamentally designed to ascertain the similarity index between two points in each n-dimensional vector space. Due to this characteristic, opting for this methodology appears appropriate for recommending crops using Minkowski distance. The process begins by cross-referencing the predicted crop, denoted as \(\widehat{y}\) in the Compound Ensemble with the previously mentioned reference data. This involves projecting the reference dataset onto an n-dimensional vector space, where each dimension corresponds to a specific feature of the dataset. Minkowski Distance is employed with the established vector space to calculate the position of the referenced \(\widehat{y}\) point and determine its distance from every other point within the generated n-dimensional vector space. The similarity index of each crop is calculated by using the formula:

$$Similarity=\frac{1}{1+\left({\left(\sum_{\left\{i=1\right\}}^{n}{\left|{x}_{i}-{y}_{i}\right|}^{p}\right)}^{\left\{\frac{1}{p}\right\}}\right)}$$
(9)

which can be simplified into,

$$Similarity=\frac{1}{1+minkowski\,distance}$$
(10)

A list of optimal crops is identified and ranked by sorting the results according to their similarity index. A threshold can also be set for the recommendation, thereby allowing the users to recommend a few crops that satisfy the threshold. Finally, optimizing hyperparameters in the compound ensemble is a crucial aspect of the system. The sluggishness in ML operations can be attributed to the prolonged time required for tuning and testing individual model parameters. The goal of this work is to eliminate the reliance on manual input and brute force methods by adopting a fascinating alternative.

Evolutionary Hyperparameter Optimization

To maximize the predictive capability of the Compound Ensemble, each of the 12 constituent classifiers requires independent hyperparameter tuning. Given the high-dimensional search space, manual tuning or exhaustive Grid Search methods were deemed computationally intractable. Consequently, a Genetic Algorithm (GA) is implemented to automate the optimization process, prioritizing an efficient balance between exploration of the global search space and exploitation of local optima.

Unlike static evolutionary approaches, our implementation utilizes adaptive operator probabilities to dynamically adjust the search behavior over time. ExponentialAdapters is employed for both mutation and crossover rates. The mutation probability was configured to decay exponentially (Initial: 0.8, End: 0.2, Rate: 0.1) to encourage diversity in early generations, while the crossover probability followed an inverse exponential growth (Initial: 0.2, End: 0.8, Rate: 0.1) to favor convergence in later stages. The algorithm for the Genetic Search used in the ensemble can be described in Fig. 6.

Fig. 6
Fig. 6
Full size image

Algorithm for the genetic search used in the ensemble43.

The evolutionary process was constrained by a maximum budget of 25 generations with a population size of 20 candidates. Selection was performed using Tournament Selection (size k = 5) with Elitism enabled to preserve the fittest solutions across generations. To ensure the robustness of the selected hyperparameters, the fitness function was defined as the mean classification accuracy derived from a 3-Fold Stratified K-Fold cross-validation. This stratification ensures that the class distribution remains consistent across validation folds, preventing overfitting to specific data subsets.

To further optimize computational resources, a Consecutive Stopping callback was integrated. The optimization terminates early if the fitness metric (accuracy) fails to improve for 5 consecutive generations, preventing redundant processing once convergence is achieved. The above-mentioned initializations and settings are provided in Table 1.

Table 1 Hyperparameter optimization settings for genetic algorithm (GA).
Full size table

Computational efficiency analysis

The adoption of the Genetic Algorithm (GA) offers a significant reduction in computational complexity compared to traditional Grid Search. A standard Grid Search necessitates the instantiation of n^m models, where n represents the discrete values per hyperparameter and m denotes the number of hyperparameters. This leads to an exponential explosion in training time. In contrast, the GA approach is bounded linearly by p × g, where p is the population size and g is the number of generations (in this study, 20 × 25 = 500 evaluations maximum). The stochastic nature of the genetic search allows for the sampling of continuous distributions rather than rigid discrete values, resulting in a more flexible and efficient traversal of the hyperparameter manifold.

Real-time deployment and scalability

The primary objective is to empower users, primarily farmers, to access real-time data derived from soil and atmospheric sensors. The ML model facilitates the generation of tailored crop recommendations by comparing this data with optimal values for crop growth. Using IoT-based sensors along with past and current data makes sure that these recommendations are accurate and can adjust to the dynamic conditions in farming. This transformation elevates agricultural productivity, fosters sustainability, and fortifies resilience in changing agricultural landscapes.

Results and discussion

This section presents an examination of diverse outputs and results achieved through various techniques and ML models. A thorough analysis of these outputs will ensue, using their correlation to convey the accuracy of the models, thereby establishing the Crop Recommender as a dependable and robust predictive tool. The real-world implications expressed in percentages and numerical values, and their significance. As mentioned in Section “Rainfall forecast”, a variety of scalers are tried, and their results are given in Fig. 7. It has been decided to proceed further with the Robust Scaler as it scaled the values down while keeping the outliers contextually relevant. Here the median is removed, and the data is scaled in the range between the 1st quartile and the 3rd quartile which is in between the 25th quantile and the 75th quantile range. This is important as rainfall in Chengalpattu can be considered as an outlier if scaled based on the mean value. The other scalers such as the Min-Max Scaler, Max Absolute Scaler, and Standard Scaler did not preserve outliers as the Robust Scaler did.

Fig. 7
Fig. 7
Full size image

Results of scalers—(a) Min–MaxScaler, (b) max absolute scaler, (c) robust scaler, (d) standard scaler.

The comparison between monthly average actual and predicted rainfall for Vanilla LSTM and Intensified LSTM models is illustrated in Fig. 8. It is evident that the Intensified LSTM model adeptly captures peaks and demonstrates superior accuracy in rainfall prediction compared to its vanilla counterpart, thereby enhancing the quality of predictions. The discrepancy in performance can be attributed to the inherent limitations of the standard Vanilla LSTM model, which typically fits around the overall mean of the data without effectively generalizing over the magnitude of rainfall. The RMSE values in Fig. 9 reflect this inadequacy. When comparing Vanilla and Intensified LSTM models across both Rabi and Kharif seasons, Intensified LSTM consistently yields superior results. For instance, in the Kharif season, Intensified LSTM achieves an RMSE of 1.01, while Vanilla LSTM records 2.31. Similarly, during the Rabi season, Intensified LSTM demonstrates an RMSE of 0.89, outperforming Vanilla LSTM’s 1.2. Consequently, it can be concluded that the Intensified LSTM model exhibits notable superiority over its Vanilla counterpart in rainfall prediction, a crucial factor in drought predictions.

Fig. 8
Fig. 8
Full size image

Monthly average actual versus predicted rainfall for vanilla LSTM and intensified LSTM in Rabi and Kharif season.

Fig. 9
Fig. 9
Full size image

RMSE of vanilla LSTM versus intensified LSTM in Rabi and Kharif season.

The Squared Error metric, illustrated in Fig. 10, showcases the month-wise square error in both Rabi and Kharif seasons for the Vanilla and Intensified LSTM models. This visualization across both seasons demonstrates that the metrics of Intensified LSTM exhibit a consistent pattern of being notably lower than those of Vanilla LSTM, reaffirming its superior performance. Notably, in the Kharif season, most squared error values for Intensified LSTM are significantly lower than those of Vanilla LSTM. This trend persists in the Rabi season, except in March. The abrupt increase in rainfall during March is regarded as an anomaly or outlier, resulting in the improper capture of peaks and subsequently increasing the squared error for that month. Despite this anomaly, the overall results provide sufficient evidence to justify the superior accuracy and performance of the Intensified LSTM model.

Fig. 10
Fig. 10
Full size image

Squared error for each month (a) Kharif season, (b) Rabi season.

The forecasted rainfall is used for drought estimation. Figures 11 and 12 denote SPI and SPEI calculated over a 12-month timescale for the predicted years 2022 and 2023. The period of 12 months is crucial here because such a timescale shows seasonality and allows us to estimate trends concerning variation. Long-term trends can also be visualised with a seasonal forecast. The Root Mean Square Error (RMSE) was calculated for the indices, and it was found to be 1.9 for SPI and 0.11 for SPEI. This can be attributed to the fact that compared to SPI, which only uses the Precipitation variable, the SPEI also uses the Temperature variable to approximate the evaporation component, thereby increasing the dynamicity of the model.

Fig. 11
Fig. 11
Full size image

SPI for the year 2022–2023.

Incorporating SPEI data is essential given its demonstrated superiority over the SPI in the analysis. The utilization of SPEI, which integrates both precipitation and temperature variables, yields a lower RMSE of 0.11 for the 12-month timescale. By including the actual versus predicted SPEI data, the advantages of this index in forecasting drought conditions continue to leverage. This strategic choice aligns to refine crop recommendation strategies based on accurate and reliable drought estimations. Thus, the inclusion of SPEI in the analysis enhances prediction accuracy and provides a robust foundation for informed decision-making in agricultural management practices.

Fig. 12
Fig. 12
Full size image

SPEI for the year 2022–2023.

Figure 13 presents the RMSE plot comparing the performance of SPEI and SPI across 12, 6, and 3-month periods spanning from 2022 to 2023, covering on the Kharif and Rabi seasons. A notable observation is the superior performance of SPEI over SPI, evidenced by substantially lower RMSE values across timescales. Specifically, SPEI12 achieves an RMSE of 0.11, markedly lower than SPI12’s RMSE of 1.9. This trend persists across varying time scales: for 6-month periods, SPEI records an RMSE of 0.19 compared to SPI’s 1.8, and for 3-month periods, SPEI’s RMSE stands at 0.25, contrasting with SPI’s 1.57. These findings highlight the enhanced accuracy facilitated by SPEI’s integration of temperature and precipitation variables, thereby sustaining a dynamic modelling framework. Because of its better performance, SPEI was chosen to move forward with crop recommendation.

Fig. 13
Fig. 13
Full size image

RMSE for the year 2022–2023: SPEI versus SPI for 12, 6, and 3 months.

The Compound Ensemble Model outputs a set of crops most suited for the geographical region based on various agrometeorological parameters. The number of recommended crops can be adjusted according to user requirements by modifying a simple variable in the code. This flexibility allows for tailored recommendations that can accommodate specific user needs, regional constraints, and varying agricultural objectives. It was benchmarked after performing experiments and analysing the results. It was observed that the proposed novel approach had indeed performed better than the individual ML models listed in Table 2. It had reached an estimated accuracy of 99.8% along with a precision and recall of 99.7% each. This shows that the model is vastly superior to existing systems and outperformed them by a significant margin. The vote count for each of 8 predictions is provided for reference.

Fig. 14
Fig. 14
Full size image

Vote count of weak classifiers.

Figure 14 illustrates instances of unweighted voting in a compound ensemble. This visual representation shows how votes contribute to the predicted crop without assigning different weights based on the credibility of individual weak classifiers. Each prediction is considered equal, emphasizing a straightforward approach where all opinions hold the same importance. This unweighted voting system gives a clear picture of the collective decision-making process within the ensemble, fostering transparency and democratic aggregation of predictions for more robust and unbiased results in crop prediction scenarios. The crop recommender’s performance was measured using 3 metrics - Precision, F1-score and Accuracy. By interpreting the results, it was seen that a significant rise in Precision, F1-Score and, most important of all, Accuracy was achieved compared to the average of individual model performances.

Table 2 Metrics for each model.
Full size table

Table 2 enumerates the performance metrics, namely Precision, F1-Score, and Accuracy, of various ML models and an ensemble for a specific classification task. Logistic Regression exhibits commendable precision (0.9418), F1-Score (0.9195), and Accuracy (0.9163). SVM demonstrates superior precision (0.9862), F1-Score (0.9855), and Accuracy (0.9854). Notably, Naive Bayes and Multinomial NB display high precision (0.9882, 0.9090) and F1-Score (0.9881, 0.7513), with NB outperforming in Accuracy (0.9881). Decision Trees, Random Forest, and QDA exhibit outstanding precision, F1-Score, and Accuracy, while Extra Trees and AdaBoost deliver respectable performance. The k-NN model demonstrates noteworthy precision (0.9823) and Accuracy (0.9818). MLP and Gradient Boosting showcase competitive metrics. The Average Metrics row provides an overview of the aggregated model performance, emphasizing a balanced evaluation. The Compound Ensemble impressively outshines individual models, attaining an exceptional Precision of 0.9970, F1-Score of 0.9970, and Accuracy of 0.9980. The ensemble’s robust performance underscores the efficacy of combining diverse models to enhance predictive accuracy.

a)

Fig. 15
Fig. 15
Full size image

Metric of each model pre-tune versus post-tune using hyperparameter (a) Accuracy, (b) Precision, (c) F1_score.

The graphs presented in Fig. 15 illustrate the changes in model metrics following the implementation of the GA Hyperparameter tuning. Across many models, there is an observed increase averaging between 5 and 7% in metrics such as Accuracy, Precision, and F1 Score. This enhancement contributes to the overall improved performance of the proposed Compound Ensemble Model in crop recommendation. Specifically examining the Precision Graph, in Weak classifiers such as k-NNs and Multinomial Naïve Bayes, a slight decrease of approximately 3–4% is noticeable. The cause of this decrease is not readily discernible and may be attributed to the stochastic nature inherent to these models.

To quantify the internal consistency and predictive stability of the 12-member heterogeneous ensemble, a non-parametric permutation analysis is performed for 10,000 iterations. Given the absence of instance-level ground truth, the null hypothesis (\({H}_{0}\)) assumes that the observed classifier agreement is a product of stochastic aggregation. \({H}_{0}\) is evaluated using two distribution-aware metrics: the Majority Vote Ratio (MVR), defined as the fraction of ensemble members supporting the plurality prediction, and Shannon Entropy (\(H\)), measuring the uncertainty of the vote distribution. MVR is defined as

$$MVR=\frac{max_{c\in C}{v}_{c}}{N}$$
(11)

The observed mean MVR was 0.6639, significantly exceeding the simulated null mean of 0.2061 (\(p<0.0001\)). Furthermore, the observed mean entropy was 1.1673, notably lower than the null entropy of 1.4677 (\(p=0.0289\)). These results provide strong statistical evidence of non-random consensus. While these metrics do not directly quantify per-instance accuracy, as they characterize the internal agreement of the ensemble rather than external validity, the high MVR and significantly reduced entropy demonstrate that the ensemble consistently identifies a dominant latent signal across the feature space, ensuring a stable and decisive recommendation output.

The results also cement the fact that ensemble methods resist noise, handle variance and reduce bias by utilizing the simple wisdom of crowd methodology. It is inferred that the proposed crop recommendation engine performs much better than existing models given by Jacques and Defourny29, JXCT30, Poornima and Pushpalatha31, Ragho et al.28, Shariff et al.27. The Compound Ensemble Model works well in the field of crop recommendation and can be implemented easily.

Table 3 Comparison between TN Govt. data44 with ensemble model Recommendation for validation.
Full size table

For validation of the model outputs, official data from the Tamil Nadu government website were utilized, which reports the principal crops cultivated in the Chengalpattu region based on aggregate cultivated area and associated agrometeorological conditions. Table 3 presents a comparison between the dominant crops reported in government records and the model’s ranked set of recommended crops44.

Since official government crop statistics are reported as aggregate area and production measures, rather than as categorical suitability labels at the instance level, classical classification metrics such as Precision, Recall, F1-score, or Cohen’s Kappa are not applicable in this context. Accordingly, validation is conducted by assessing rank-level consistency between the model’s top-ranked crop recommendations and crops exhibiting dominant cultivated area in official records, which serves as a proxy for long-term regional agro-climatic suitability.

In the Chengalpattu region, government data indicate that paddy and fruits and vegetables occupy the largest proportion of cultivated areas. In this study, the fruits and vegetables category is further disaggregated, and individual crops are ranked based on drought tolerance, soil characteristics, and prevailing weather conditions. The presence of these dominant crop categories among the model’s highest-ranked recommendations demonstrates consistency between the proposed system and established regional agricultural practices.

It is also noted that the number of crops returned by the recommendation system can be adjusted according to user preference (e.g., top 2 or top 3 crops). From a practical deployment perspective, presenting a smaller set of highly ranked crops may improve interpretability and decision support, while larger recommendation lists may include crops with progressively lower relative suitability. Overall, the observed agreement between model recommendations and official government statistics supports the validity of the proposed approach on the regional scale.

Conclusions

This research demonstrates a high-precision framework for regionalized crop selection by integrating real-time IoT telemetry with a GA-optimized Compound Ensemble architecture. The proposed tri-stage predictive hierarchy, utilizing an Intensified LSTM for rainfall forecasting and SPI/SPEI indices for drought categorization, achieved a benchmarked accuracy of 99.8% within the specific agrometeorological context of the Chengalpattu district. By utilizing a distance-based similarity index, the system provides a ranked roster of cultivars, offering farmers actionable flexibility in response to dynamic soil conditions. The empirical evidence from this study confirms that a multifaceted ensemble approach effectively mitigates prediction variance and sensor noise compared to standalone models. However, it is essential to note that these results serve as a technical proof-of-concept for regional agricultural resilience rather than a finalized global solution. The reported metrics focus on theoretical model accuracy and technical operationality. The study does not yet provide longitudinal data on multi-seasonal yield outcomes or adoption rates.

Research limitations and operational risks

While the technical performance of the recommender engine is high, this study is subject to several operational constraints that must be addressed before large-scale deployment:

  • Geographical Scope: The current validation is restricted to a 50-crop dataset within a single geographical district; therefore, the conclusions regarding broader applicability remain preliminary.

  • Validation Boundaries: The reported metrics focus on theoretical model accuracy and predictive fidelity; the study does not yet provide longitudinal data on actual crop yield outcomes or farmer adoption rates.

  • Operational Robustness: The system’s performance is contingent upon continuous sensor uptime. Factors such as localized sensor failure, potential model drift over multiple seasons, and extreme weather variability represent ongoing operational risks that require robust maintenance protocols.

Future directions

To evolve this framework into a proactive decision-support tool, future efforts will prioritize the following:

  • Temporal Parameter Forecasting: Extending the predictive window by forecasting soil nutrient drift and moisture fluctuations using advanced time-series analysis.

  • Dataset Expansion: Broadening the reference dataset to include high-priority regional crops, such as groundnuts, to enhance the system’s utility for local agricultural stakeholders.

  • User-Centric Deployment: Developing a multi-lingual mobile interface supported by a cloud-based backend to facilitate direct access to recommendations for first-time users in remote locations.