Multi-Modal Deep Learning for Crop Yield Prediction Network: Static and Temporal Feature Space

Abstract

In agriculture, crop yield prediction is the process of making yield estimates using data from crops, soil, and weather. Although ML models have been utilized before, they frequently depend on features that were manually created. So, a DL model like a 1D Convolutional Neural Network (1DCNN) can be employed. However, it struggles to learn temporal relations amongtime-series data. Therefore, a Deep learning-based Crop Yield prediction Network (DeepCropYNet) was designed using Long Short-Term Memory (LSTM) and Temporal Convolutional Network (TCN). However, this model struggles to learn significant features from complex datasets that involve multimodal inputs like time-series and image data.Thus, this paper proposes a Deep learning-based Multi-Modal CropYNet (DeepMMCropYNet) for crop yield prediction, which utilizes both time-series and image data related to crop yields. First, the dataset is pre-processed using a normalization technique to remove missing values and outliers. Then, the DeepMMCropYNet is trained using the pre-processed data to predict crop yields. This model comprises two branches: (i) LSTM-TCN for time-series data and (ii) multi-dimensional CNN for soil image data. This multi-dimensional CNN model comprises static and temporal feature extraction modules. The static module learns the static features from the soil images using 18 parallel 1DCNNs. The temporal module employs 16 parallel 2-dimensional CNNs (2DCNNs) to extract temporal features from soil images. The outputs of these modules are fused by the lateral connections. Moreover, each branch applies an attention strategy to assign the feature weights and find significant features. The features of each branch are then merged and given to a Fully Connected (FC) layer followed by an output layer to get a final prediction result of different crop yields.By comparing the DeepMMCropYNet model to previous models, the experimental findings demonstrate that it outperforms them in terms of Mean Square Error (MSE), Mean Absolute Error (MAE), Root Mean Square Error (RMSE), and Correlation Coefficient (R2). when it comes to predicting various crop yields.