Introduction

In reality, people's use of natural language as a way of exchanging information has progressed to the point that it has become essential for technological advancement1. The interest in using technology to decipher natural language in order to make crucial information available to the public has grown tremendously in recent years. In response to this need, natural language processing (NLP) has been a primary focus of natural language computations2. The waveform or speech signal contains the overall message as well as the speaker's features and the language of interaction. The structure of sound units indicates the language. The goal of automatic language identification (LID) is to identify a language from a spoken speech. This domain of NLP performs significant research in the area of language identification3,4.

Almost all Ethiopian languages lack resources, placing them among the language groups that do not benefit from the most recent advances in spoken language technology5. The current effort focuses on Ethiopian Semitic (Ethio-Semitic) languages. According to recent linguistics research, Semitic languages are classified as the parent category of the Afro-Asiatic category. This categorization is supported by a large number of famous linguists1,2,3,4,5. There are up to 200 dialects spoken in Ethiopia, with around 86 languages divided into the Semitic, Cushitic, Omotic, and Nilo-Saharan groups6. The Ethio-Semitic language family includes Amharic, Tigrigna, Geez, Harari, Gafat, Soddo, and Gurage7.

However, for our research, we employed Amharic, Geez, Gurage, and Tigrigna because they all have a sizable number of native speakers. These languages are used for a variety of reasons in Ethiopia. Amharic is the working language of both the Amhara regional state and the Federal Government. Tigrigna is the official language of the 9 million people in the Tigray regional state and its also Eritrea's official language. Ge'ez is Ethiopia's ancient language, and it is still used for religious rites by Jews, Beta Israel, and Ethiopian Christians. The Ethiopic or Ge'ez script is used to write the Gurage languages5,6. In this paper, we have employed a unique deep learning method for Ethiopian LID. Recurrent neural networks (RNN) are excellent at handling sequential data, such as that found in audio and music7. RNN incorporation in this work allows the preceding layer's output data to be considered. We used the most crucial and strong audio to identify sound. Mel-frequency cepstral coefficients (MFCC) is a method for extracting information which we used to aid the proposed system to reliably identify the selected languages8.

Background of deep learning approaches

Overview of recurrent neural network

Recurrent neural network (RNN) is effective in modeling temporal relationships in the audio signals in order to build long-range feature representations throughout the input sequence for language detection. RNN was first proposed by Elman in 19909. RNN is a model in which the output is a function of not only the current input but also the previous output, which is encoded into a hidden state. RNN has a memory of the preceding timestep, which it uses to encode the information in the sequence9,10,11. As a result, by taking into consideration the entire temporal sequence, RNN can model multivariate time-series data and can provide a class prediction.

Long short-term memory

Memory blocks are peculiar items found in the recurrent hidden layer of long short-term memory (LSTM). The memory blocks feature exceptional multiplicative units known as gates that control the flow of information as well as memory cells with self-connections that record the network's temporal state. The input gate regulates the flow of input activations into the memory cell, while the output gate controls how cell activations exit the cell and enter the network. The forget gate scales the internal state of the cell before adding it as input via the cell's self-recurrent link, adaptively forgetting or resetting the cell's memory11,12.

Bidirectional long short term memory

Bidirectional long short-term memory (BLSTM) is a kind of bidirectional recurrent neural network (BRNN). It was originally proposed in9 and comprises of a combination of two hidden layers that expose separate directions to the same output. The output layer can gather information from both previous (backwards) and forward (future) states at the same time. Each LSTM is made up of increasingly complex and often linked subnets called "memory cells." These cells (gates) enable information to have long-distance dependencies. As a result, this adds fresh values to the activation as it progresses through the layers, avoiding the vanishing gradient problem that makes LSTM comparable to residual neural networks, or ResNets13,14.

Numerous research on language identification (LID) have been undertaken in the past which used conventional machine learning and deep learning approaches. The following are the study's contributions:

  • There is no publicly available dataset for the LID of Ethio-Semitic languages. In this study, we created the first audio dataset to address this issue. As a result, this dataset may be utilized for future research and to compare similar works.

  • In addition, we presented an efficient RNN model with appropriate hyper-parameters.

  • Finally, we achieved state-of-the-art LID accuracy for Ethio-Semitic languages using the proposed method.

The rest of the paper is structured as follows: Section "Background of the deep learning approaches" presents a detailed description of the related work, including the gaps and limitations used in this study. Section "Related work" provides a detailed description of the proposed methodology and the dataset analyzed in this study. The experimental findings and analysis are provided in Section "Methodology". Finally, we present the concluding remarks in Section "Results and Discussion".

Related work

In an attempt to identify the gap in knowledge in this study area, we examined past research which used the conventional machine learning and deep learning approaches. This enabled us identify the gap in knowledge, existing datasets and sizes, methodologies, and models employed by various authors. Wondimu and Tekeba3 used Gaussian mixture model (GMM) in a LID system for four Ethiopian languages (Amharic, Oromiffa, Guragegna, and Tigrigna). In the study, the MFCC feature extraction approach was employed, while GMM was used for classification. There is no fixed duration or segment size for audio splitting in the study. For the four languages, the average accuracy of the utterance dependent LID test was 93%; the average accuracy of the utterance independent test was around 70%; and the average accuracy of the speaker independent test, which was only evaluated on the utterance dependent scenario, was around 91%.

Athiyaa et al.4 suggested using the Gaussian Mixture Model (GMM) and Mel-Frequency Cepstral Coefficients (MFCCs) aspects of the speech waveform to discriminate between two independent audio signals in Tamil and Telugu, two South Indian languages. The suggested approach uses MFCC features taken from speech waveforms to train the Gaussian Mixture Model (GMM). With an additional blend feature, the proposed spoken language recognition method improves the accuracy for both languages.

Gonzalez-dominguez et al.12 research study showed how LSTM RNNs effectively exploits temporal correlations in audio data by learning appropriate features for language recognition. The suggested method is compared to the i-vector and feed-forward Deep Neural Network (DNN) baseline methods on the NIST Language Recognition Evaluation 2009 dataset. Despite having less magnitude values of the parameters, their results demonstrate that LSTM RNNs outperformed the DNN system. Additionally, the fusion of the various technologies enables significant speed gains of up to 28%.

Bartz et al. 15 method used a multi-head self-attention layer, an encoder, and the statistics pooling layer. The encoder effectively includes information from the raw waveforms using 1D kernels and an LSTM layer. The multi-head self-attention layer uses the outputs from the LSTM layer to apply self-attention mechanisms with different heads to these characteristics. Through this approach, the model is capable of providing more weight to the features that are more relevant and less weight to the features that are less essential. According to their investigation, they found that their method outperformed the 373 h of audio data for eight distinct Indian languages and that it greatly outperformed the benchmark model in terms of F1-score, achieving a score of 95.90%. Their study shows that anytime raw waveform models were applied, the performance increased by 1.7%.

Deshwal et al.16 used a hybrid robust feature extraction technique for spoken language identification (LID) systems. In the feature extraction stage, different techniques are applied individually, such as Mel frequency cepstral coefficients (MFCCs), perceptual linear prediction features (PLP), and relative perceptual linear prediction features (RASTA-PLP). The feed forward back-propagation neural network (FFBPNN) was used in the language identification or classification phase. With an overall accuracy of 94.6%, the MFCC-RASTA-PLP hybrid feature extraction approach shows promising results when compared to the other hybrid feature extraction techniques.

Anjana and Poorna17 used Linear Discriminant Analysis (LDA) and Support Vector Machine (SVM) as classifiers to extract languages from speech data. The MFCC and Formant features of Speech were used in the study to compare and contrast the two feature types with the two models separately. With just MFCC and formant frequencies, the accuracy of SVM and LDA was 78%, 86%, and 80.02 & 93.06%, respectively. The LDA model, which integrates the two features, performs better, achieving 93.88% accuracy compared to SVM's 84%. While other authors only examined one or two different languages, in this paper, we examined four languages. It is noteworthy to state that there is no existing dataset for Ethio-Semitic languages in the body of existing knowledge online. In addition, authors have used methods which have not been used before such as RNN, Long Short Term Memory (LSTM) and Bidirectional Long Short Term Memory (BLSTM) for the identification of Ethio-Semitic languages.

Methodology

The methods to achieve successful results and achieve the goals of this research are described in this section. The sections of the study include subsections such as data collection and annotation, audio preprocessing, data splitting, feature extraction, model training, and model evaluation. Furthermore, an endeavor is made to elaborate on the study's unique feature, the identification of particular Ethio-Semitic languages.

Data collection

There is no publicly available dataset of audio data from Ethio-Semitic languages. Even though there is no prepared data for the selected languages, we created our own dataset by recording directly from native speakers of each language in order to execute the automatic language identification task. With a record time of 12 min per person for each language, we used 10 people for each language. Each corpus is composed of a variety of audio recordings containing different ages, genders, and accents. Each language's recording lasted for about 120 min (2 h), thereby totalling 480 min (8 h). The speech signal has a sampling rate of 44.1 kHz, and each sample is stored as a 16 bit value. This was recorded using a smart phone.

Proposed system architecture

This section describes the steps undertaken to build the proposed LID model using the proposed architecture in Fig. 1. It primarily focuses on the corpus preparation, overall model system architecture, preprocessing methods, preparation of machine readable datasets (CSV), and model evaluation procedures. Figure 1 depicts the proposed LID model’s architecture where the model consists of a number of steps such as data preprocessing, framing, feature extraction, and model evaluation.

Figure 1
figure 1

Proposed system architecture.

Full size image

Data Pre-processing

After data collection, the first step in the LID system is to pre-process the raw speech data to make it suitable for the study. It is the term used to describe any type of processing performed on raw data to get it ready for the next processing step. It is the primary step that converts the data into a format that can be processed more quickly and efficiently.

Data cleaning

Audio data cleaning is mandatory before any processing steps. When we collect the data by recording, most of the time the speakers speak mixed languages like Amharic with English or Ge’ez with Amharic, etc. Therefore we remove those mixed audios by using pydub package to prepare pure data for the proposed system. The audio files were saved in the form of WAV files.

Silent Removal

We used librosa trim function in order to remove the silence from the actual speech wav files and the normal sound is greater than 10 dB18. Decibel (dB) measures sound intensity (amplitude). We utilized a 10 dB threshold to eliminate the silence from speech. As a result, any speech that falls below this threshold is erased from the whole sound file18. We used 10 dB as we reviewed numerous literature which used the same level of sound. This is a standard value as the sound must be greater than the noise as given by Signal-to-Noise ratio (-10 dB ≤ SNR < + 10 dB). Also any speech which is greater or equal to 10 dB provides an adequate quality for speech communication.

Resampling

Even though our data was initially sampled at 44,100 Hz and had a large data size, in a comparison of 44,100 Hz and 22,050 Hz, the former has much greater data size. Because of the large amount of the data, the dataset was initially down-sampled to 22,050 Hz from 44,100 Hz. All audio files were converted from stereo (two channel) to single-channel (mono) since mono channel is superior for language classification and a sampling rate of 22.05 kHz is sufficient for basic information19,20. Each audio recording in stereo channels contains two channels, and is less essential for language classification than mono channels21,22. Librosa's default settings include a sampling rate of 22050 Hz, a bit depth of 16 bits, and 1 channel (mono).

Framing

Framing is the technique of splitting a continuous stream of speech samples into units of fixed length that enables it to process the signal block-by-block. Speech signals are slowly time-varying or non-stationary. Since sound waves are non-stationary signals, their acquired data are constantly shifting over time. Consequently, it is not possible to extract the speech features at once23. The speech signal is broken down into frames, which are small period subdivisions in second or millisecond8. Consecutive frames usually overlap by 30 to 50% in order to avoid windowing from obliterating any essential voice signal information24. Different segment lengths can be applied, namely: 30 s, 20 s, 10 s, 5 s and 3 s25. We selected intermediate lengths which are, 5 s and 10 s to segment or for framing using audio segment library for each language sound.

Feature extraction

The method of converting a raw speech signal into a collection of acoustic feature vectors that carry speaker-specific information is referred to as feature extraction26. By using Librosa package, we extract useful components from audio data27,28. There are different feature extraction techniques like Chorma Stft, Mel-frequency cepstral coefficients (MFCC), mel-spectrogram, Spectral Contrast, and Tonnetz etc.27,28. There are numerous features of an audio speech segment that can vary from language to language. These can be included into various LID system designs, each with a unique level of complexity and outcome29. Several feature extraction techniques exist for language classification30. The proposed system employs an acoustic-phonetics feature type with feature extraction technique namely MFCC have been used for feature extraction which is mainly used for spoken language identification.

Mel-frequency cepstral coefficients feature

The most common feature extraction technique to represent speech signal data and to express acoustic signals as cepstral coefficients for various applications is Mel-frequency cepstral coefficients (MFCCs)16. MFCC is a well-known feature used to describe speech signals. Since the technique for computing MFCC is based on short-term analysis, a MFCC vector is generated from each frame. These are based on speech processing performed by the human ear and the speech signal's cepstrum30,31. Based on the study in21, we took the number of MFCC coefficients as 20, since the most important information were found from 13 to 20 coefficients8,21,32. The process to extract MFCC features is shown in Fig. 2.

Figure 2
figure 2

MFCC block diagram.

Full size image

A. Windowing

Windowing is the first step to extract MFCC features from framed signals and the effect of windowing is used to convert an infinite-duration signal into a finite-duration signal33. The signal should be attenuated to zero or very similar to zero in order to reduce the discontinuity of the voice signal at the start and end of each frame. Therefore, windowing each frame of the signal to increase the correlation of the Mel Frequency Cepstrum Coefficients (MFCC) can be used to minimize the difference24.

B. Fast fourier transform

Fast Fourier Transform (FFT) is used to convert the data from the frequency domain to the spatial domain. The samples in each frame are converted into frequency domain. A fast algorithm for performing the discrete Fourier transform is the Fourier transformation (DFT) 33,34.

C. Mel-scale

In this stage, the projected spectra from the previous phase are mapped on the Mel scale to produce an approximation of the energy present at each point using a triangle overlapping window, also known as a triangular filter bank8,21,22,33,34.

D. Discrete cosine transform

This is the final stage in the process of converting a given sequence of finite duration data into a discrete vector by calculating coefficients from the provided log Mel spectrum. Discrete Cosine Transform (DCT) is preferred for the coefficient calculation since its outputs can have significant energy contents. Finally, the result of applying DCT is referred to as the MFCC vector 8,23,30. The discrete cosine transform provides cepstral coefficients and is based on the linear transformation principle (MFCCs)16. The sample MFCC vector feature is shown in Fig. 3.

Figure 3
figure 3

Sample of MFCC vector feature.

Full size image

Model training

Long-Short Term Memory (LSTM) and Bidirectional Long-Short Term Memory (BLSTM) models were used to develop the proposed system. The MFCC features are fed into the proposed recurrent neural networks as input. The acquired audio dataset was used to train the models.

Proposed LSTM model

The LSTM model is better for sequential data, so therefore, we used the sequential feature value which is MFCC. We fed the MFCC feature vector form directly to the LSTM model36. So, using the specified feature extraction technique, we ran one experiment using this model. Figure 4 depicts the proposed LSTM model.

Figure 4
figure 4

Proposed LSTM model.

Full size image
Proposed BLSTM model

Experimentally, the BLSTM model, like the LSTM model, is better for sequential data. We used the sequential feature, which is similar to the MFCC feature. We fed the MFCC feature vectors directly into the BLSTM model. So we run one experiment with the MFCC feature in this model and used 64 and 128 filters, respectively. Figure 5 depicts the proposed BLSTM model.

Figure 5
figure 5

Proposed BLSTM model.

Full size image

Model evaluation metrics

There are two types of classification problems depending on the number of classes: binary classification, where there are only two classes, and multi-class classification, where there are more than two classes37,38. The confusion matrix is the visual representation of actual and predicted class values. The actual class indicates the real classification of each language and the predicted class also the prediction values of our trained models. Confusion matrix evaluates the accuracy of an algorithm to arrange samples in the appropriate classes2. This study used multi-class classification to evaluate the proposed models with four languages and we used evaluation metrics to evaluate the performance which include accuracy, precision, recall, and F1 score given by Eqs. (1)–(4).

$$\mathrm{Accuracy}=\frac{\mathrm{TP}+\mathrm{TN}}{\mathrm{TP}+\mathrm{FN}+\mathrm{FP}+\mathrm{TN}}$$
(1)
$$\mathrm{Precision}=\frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FP}}$$
(2)
$$\mathrm{Recall}=\frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FN}}$$
(3)
$${\mathrm{F}}_{\mathrm{Score}}=2*\left(\frac{\mathrm{Precision}*\mathrm{Recall}}{\mathrm{Precision}+\mathrm{Recall}}\right)$$
(4)

where, TP, TN, FP and FN represent true positive, true negative, false positive and false negative respectively.

Experimental setup

After feature extraction, we split the dataset into training, validation, and testing data to train, validate and evaluate the model. From the Scikit learn library, we used the train test split method to divide the data into 70% train set, 15% test set, and 15% validation set. The models used and experiments performed in this study are presented in this section along with the results used to evaluate how well the system performed in relation to other systems. To accomplish this, we first evaluated the dataset we used to build our model, then explained the results of that implementation, and finally evaluated the results using the four metrics in Eqs. (1)–(4). Bayesian hyper-parameter tuning algorithm was used to set values for hyper-parameters in our neural network (NN) model. This helps us to find a vector of hyper-parameters that works well with the problem domain in order to determine which one would perform effectively. Hyperparameters including batch size, learning rate, and dropout rates were optimized. We used the hyper-parameters presented in Table 1.

Table 1 Hyper-parameter selection summary.
Full size table

Human ethics statements

Authors confirm that all experiments were performed in accordance with the Helsinki declaration guidelines and regulations. Authors confirm that informed consent was obtained from all participants. Authors confirm that all experimental protocols were approved by the institutional licensing/research ethics committee of the author’s institution which include members of staff of the College of Informatics, University of Gondar, Ethiopia, namely: Tsehaye Wasihun, Yigezu Agonafir, Solomon Zewdie, Sied Hassen, Ibrahim Gashaw.

Results and discussion

This section presents the implemented models as well as the experiments, results produced to evaluate the models performance and the comparison of the various deep learning methods. To evaluate our model, we analyzed the dataset using the deep learning approaches that we utilized to create our model. The results of the implementation and the metrics used to evaluate the results produced are presented in this section. When we examined the performance with durations of 5 s and 10 s, we observed that the performance with durations of 5 s outperforms that of 10 s. The results of the experimentations when LSTM with MFCC is used as a feature is presented in Table 2, while that which used BLSTM with MFCC as a feature is presented in Table 3.

Table 2 Experimental result using LSTM with MFCC as feature.
Full size table
Table 3 Experimental result using BLSTM with MFCC as feature.
Full size table

For 5 s, Guragigna achieved a precision, recall, and f1-score of 90, 92, and 91, respectively, and for 10 s, it achieved a precision, recall, f1-score of 82, 86, and 84. As a result, it can be deduced that the length of 5 s was superior to that of 10 s.

As shown in Table 3, the developed BLSTM model using MFCC as a feature achieved a recall of between 80 and 95% lasting 10 s and between 90 and 97% lasting 5 s. As a consequence, when the results of 5 s and 10 s are compared, the findings presented in Table 3 surpass the later. As a result, it can be concluded that 5 s was preferred over 10 s.

As illustrated in Fig. 6, BLSTM outperforms LSTM with respect to the LID models performance when employing the MFCC features. As a result, we can infer that the BLSTM is a more crucial algorithm than the LSTM for the building of LID models. With an average of 93.5%, 93.25%, and 93.25% for Precision, Recall, and F1-score, respectively, we achieved better results as compared to existing systems. Generally, for the two models that we proposed, the result of 5 s was greater than 10 s. The reason behind this is that, it is essential to evaluate speech for a short time frame to execute a stable acoustic feature.

Figure 6
figure 6

comparison of BLSTM and LSTM using MFCC features.

Full size image

Confusion matrix of the proposed models

When the models use the confusion matrix to make predictions for each language, we observed how it becomes confused. We discussed the proposed models' confusion matrix with a specific feature, the MFCC feature. Figure 7 describes the confusion matrix of which the model is examined using the test data set by calculating the number of correctly and incorrectly classified test samples within each class.

Figure 7
figure 7

Confusion matrix of BLSTM model (a) 5 s (b) 10 s.

Full size image

Figure 7(a, b) displays the percentage of sample data that are classified as True Positive (TP), False Positive (FP), True Negative (TN), and False Negative (FN) in each of the four classes. Geez is taken as an illustration for the confusion matrix. It has the true positive value of 210, false-positive value 14, true negative value of 627, and false negative value of 7. Therefore, 210 samples are classified into the Amharic category, 1 sample is misclassified as Amharic, 4 samples are misclassified as Guragigna, 5 samples are misclassified as Tigrigna.

Furthermore, Fig. 8 clearly illustrates the accuracy and cross-entropy (loss) performance evaluation of the LSTM model during the training and validation phases. At epoch 35, the accuracy for training and validation is 92.50% and 89.16%, respectively. Similarly, the LSTM architecture's training and validation losses are 0.0673 and 0.3820, respectively.

Figure 8
figure 8

Evaluation metrics of LSTM model with duration of 5 s. (a) accuracy (b) loss.

Full size image

Figure 9 presents a visual display of the accuracy and cross-entropy (loss) performance evaluation of the BLSTM classifier during the training and validation periods. At epoch 35, the observed training and validation accuracy are 94.5% and 90.33%, respectively. Also, the BLSTM architecture's training and validation losses are 0.0187 and 0.4231, respectively. The BLSTM architecture surpassed the LSTM architecture in terms of both training and validation accuracy scores. A comparative analysis of the proposed model with existing models is presented in Table 4.

Figure 9
figure 9

Evaluation metrics of BLSTM model with duration of 5 s (a) accuracy (b) loss.

Full size image
Table 4 Comparison of the proposed model with existing language identification models.
Full size table

Conclusion and future work

Conclusion

This paper presented the development of a language identification (LID) system for Ethio-Semitic languages using Recurrent Neural Networks (RNN). We developed our corpus in order to implement the proposed system because there is no existing large-scale corpus for the four investigated Ethio-Semitic languages for LID purposes. Despite the fact that there are many Ethio-Semitic languages, we chose only four to develop the LID model. The four Ethio-Semitic languages are Amharic, Ge'ez, Guragigna, and Tigrigna. The dataset consists of data obtained directly from native speakers. We used RNN in this work to conduct experiments. We utilized Google Co-laboratory (COLAB) on a CPU with a GPU (Tesla T4, 8 GB) and 12 GB of RAM to do all of the tests. The studies were carried out to examine the results achieved utilizing two extended versions of RNN, LSTM, and BLSTM algorithms, with MFCC Acoustic features for durations of 5 and 10 s. The results achieved show that the most suitable approach for the Ethio-Semitic language dataset was the BLSTM algorithm with the MFCC feature running for 5 s.

Future work

The study's findings can be used in a variety of fields where spoken language is required. We couldn't readily investigate the suggested system further since there were no resources for other Ethio-Semitic languages. Due to a lack of annotated data for Ethio-Semitic languages, we were forced to limit our investigation to four languages.

For upcoming work, the following are recommended:

  • Future studies should try to investigate more Ethio-Semitic languages so as to develop a more robust model for Ethio-Semitic languages.

  • This paper used only MFCC as a feature. In future work, we recommend that more features should be used to increase the performance of the model.