Towards Effective Detection of Recent DDoS Attacks: A Deep Learning Approach

School of Computer Science and Technology, Huazhong University of Science and Technology, Wuhan 430074, China National Engineering Research Center for Big Data Technology and System, Huazhong University of Science and Technology, Wuhan 430074, China Cluster and Grid Computing Lab, Huazhong University of Science and Technology, Wuhan 430074, China Services Computing Technology and System Lab, Huazhong University of Science and Technology, Wuhan 430074, China Big Data Security Engineering Research Center, Huazhong University of Science and Technology, Wuhan 430074, China School of Cyber Science and Engineering, Huazhong University of Science and Technology, Wuhan 430074, China Shenzhen Huazhong University of Science and Technology Research Institute, Shenzhen 518057, China

Introduction
Recent advancements in communication technologies such as long-term evolution, smart-grids, 5G, and the Internet of ings (IoT) have caused a tremendous increase in data transmission because of many communicating entities. e number of devices might touch 29.3 billion by 2023, three times the current global population [1]. Among these devices, the IoT devices appear to be the most helpful amplification platforms to launch cyberattacks because of their large number and lack of security best practices [2,3]. e DDoS attack is a primary threat currently observed by most service providers worldwide. In DDoS attacks, a malicious actor first explores some vulnerable systems over the Internet to control and use them to generate massive traffic. en, it floods the target system with the generated traffic, thereby disrupting the normal operations of the target system. e first most severe DDoS attack was recorded in September 2016, when massive IP traffic having a volume of 620 Gbps originated from hundreds of thousands of IoT devices. An even greater DDoS attack at 1.1 Tbps was reported the same year. e DDoS attacks have grown severely over the past few years and have reached 2.3 Tbps in 2020 [4], resulting in extreme downtime of online services. e massive data generated by the current network requires a communication channel to handle and transmit it efficiently, securely, and reliable. erefore, as cybersecurity events become more sophisticated and frequent, an IDS must detect and tackle those anomalies as quickly as possible. To do so, researchers have devised various intrusion detection techniques in the literature. For instance, researchers propose solutions such as a signature-based intrusion detection system (IDS) to discriminate the DDoS traffic from a normal one. However, these schemes are unable to classify new attacks. Motivated by the challenges described above, the research community has paid particular attention to applying state-ofthe-art machine learning (ML) models for detecting DDoS traffic. ML provides various advanced tools and techniques to predict cyberattacks using a quick and automated approach. Nevertheless, one of the primary limitations of classical ML techniques is the amount of data they can handle. In contrast, deep learning (DL) can handle and learn patterns from a vast amount of data. IDSs based on DL have been proven to be very efficient and effective in identifying anomalies within IP network traffic. However, existing DL-based IDSs have been trained and evaluated using out-of-date datasets such as the KDD99 or NSL-KDD dataset, publicly available from 1999 to 2009 [1]. ese datasets do not reflect the recently reported DDoS attacks in the current network environment [5][6][7]. Additionally, the existing architectures are more compute-intensive, which can be improved significantly by optimizing the feature selection to reduce the number of features for traffic prediction, enhancing the detection performance.
Building on this observation, we propose CyDDoS, an integrated IDS framework, which combines an ensemble of feature engineering algorithms with a deep neural network to effectively detect the most recent DDoS attacks while incurring the least processing overhead. Specifically, it uses an ensemble of feature selection based on a supervised feature ranking technique to enable dataset dimensionality reduction and lower computation complexity. e significant contributions of this paper are as follows: (1) We proposed a lightweight IDS framework, combining the ensemble of feature selection algorithm with DNN classifier, to improve the intrusion detection performance while reducing the system processing overhead. e model is trained in 5 seconds, reducing 88% of features from the original dataset while providing a detection performance superior to 99.6% in most considered metrics.
(2) To reduce the processing overhead, several ensemble groups are evaluated, using different numbers of classifiers, to determine the feature list that can produce the best prediction performance. (3) In the context of feature ensemble, we employed two approaches that combine decisions from multiple ranking algorithms into one. e decision is made based on a threshold that defines the minimum number of classifiers that must agree on a feature or based on the top ten feature weights. (4) e trial-and-error approach is used to optimize the DNN hyperparameter and improve the detection performance. (5) We argue the efficiency of our proposed DNN model against existing DL techniques using the CiCD-DoS2019 dataset. e dataset covers the most recent DDoS attacks reported. e remainder of the paper is organized as follows: we first highlight existing IDSs literature, followed by the description of our proposed system architecture. en, we present the results obtained, and, finally, the last section draws the conclusion.

Related Work.
e research community has presented several detections and mitigation solutions for DoS attacks since their first occurrence in 1999 [8]. In this section, we briefly describe the existing strategies that use ML for anomaly detection. A large number of studies propose network anomaly detection schemes based on the classical ML models. However, Doriguzzi-Corin et al. [9] state that adopting classical ML-based techniques in the "real world" is limited due to network intrusion challenges such as traffic variability and high error costs. erefore, recent works have shifted their focus towards studying the effectiveness of DL models in IDSs because they can handle large-scale data and learn more complex patterns within that data.
For the reasons mentioned above, we focus only on DLbased DDoS detection systems. Recent studies such as [10,11] and [12] propose a network intrusion detection solution based on DL. ese studies use the KDD99 (1999) or NSL-KDD (2009) dataset to train and validate their proposed models. Yuan et al. [13] presented DeepDefense, an anomaly detection method based on DNN. e model can learn the network's traffic sequence to trace attacks related activity. DeepDefense leverages Convolutional Neural Network (CNN) and Recurrent Neural Network (RNN), which has proved to provide huge performance improvement in terms of accuracy when trained with a large dataset. e authors used ISCX2012 dataset to evaluate the model's performance, and the results demonstrate a considerable reduction in terms of error rate compared to the classic ML method. Since the datasets used in the said studies are outdated, they do not represent the features and characteristics of recently reported DDoS attacks [5,6]. erefore, the previously mentioned intrusion detection schemes cannot be reliably used to detect new threats in networks such as IoT.
Elsayed et al. [14] developed an IDS for the Software-Defined Networking (SDN) environment. e authors combined RNN with autoencoder (AE) to improve the DDoS detection accuracy. It uses the CICDDoS2019 dataset for training and validation. e authors proposed a pretraining based on an unsupervised approach and the finetuning stage using supervised learning. e AE is an unsupervised model which produces a compressed representation of the input samples to reduce the processing overhead. e proposed scheme achieves a substantial improvement in terms of accuracy compared to existing systems. However, the research did not discuss the performance details such as model training time or samples classified for a given amount of time. Similarly, "Kitsune" [15] uses an unsupervised approach based on AE for intrusion detection. Kitsune is a lightweight real-time detection model that uses an ensemble of autoencoders to collectively predict normal and abnormal traffic patterns. e experiments conducted using a Raspberry PI system showed that this solution can achieve performance similar to offline anomaly detection solutions. A lightweight approach based on MultiModal Deep AutoEncoder (M2-DAE) and a soft-output classifier are proposed [16]. is framework is optimized for the IoT environment and can identify both known and unknown attacks.
Doshi et al. [17] developed a DL-based design for DDoS detection in an IoT setting. e authors used networkspecific IoT behavioral features, such as reduced terminals and constant arrival time between packets, for feature selection to improve detection accuracy. Similarly, Ge et al. [18] deployed an intrusion detection system for the IoT environment based on a customized deep learning method.
is proposal adopted the feedforward neural network (FNN) technique with an embedding layer to perform multiclass prediction of attacks. Furthermore, they developed a second feedforward neural network model that implemented the transfer learning technique to encode a high-dimensional categorical feature and perform a binary classifier. Both classifiers achieved good detection performance.
Doriguzzi-Corin et al. [9] proposed LUCID, a lightweight online DL-based IDS. It uses CNN for making a binary classification on three benchmark datasets. Although the datasets allow for offline evaluation, they use a network traffic processing technique that generates a spatial representation of data to support online attack detection. According to the authors, it can reduce the processing time by 40X compared to its counterparts when deployed in a resource-constrained environment while matching the high accuracy of recent research. Nevertheless, this solution requires a high retraining time to update the model's weights and biases using new traffic samples, exposing the network to new DDoS attacks during this period.
Wang et al. [19] proposed an intrusion detection framework to address network and host intrusions. Network intrusion detection leverages the Stacked Denoising AutoEncoder-Extreme Learning Machine (SDAE-ELM) to address the slow training time and poor detection performance of existing neural network approaches.
e host intrusion detection model is based on DBN-Softmax, aimed to improve the detection effectiveness of host intrusion.
Small-batch gradient descent is used during the network training and optimization to improve both models' training efficiency and classification performance. Experiments were conducted based on several datasets to validate the higher performance of this proposal against other classic machine learning approaches. SDAE is a powerful deep learning technique to compress datasets. However, most of the datasets used in this study are outdated, and the average detection performance for binary classification still needs further improvements. e dataset used to evaluate an IDS model is critical for determining the model's reliability in detecting potential DDoS attacks. Most of the existing DL-based strategies are unable to detect recent attacks reported by the research community due to the limitations of datasets being used for training and evaluation [5,6]. On the contrary, some recent studies do consider datasets containing data related to recent DDoS attacks. However, they pose a high processing overhead for training the models and making predictions. e computational complexity of the detection system can be significantly improved if the number of features being used for training and detection can be reduced without affecting the performance of the IDS. To solve the said issues, we propose a DL-based IDS, which attains better DDoS attack detection accuracy and requires less computing capacity compared to its counterparts.

Proposed Methodology.
e detection approach employs a discriminative deep neural network mechanism that makes the binary classification of the traffic sample. Figure 1 illustrates the general solution block diagram, consisting of four steps: preprocessing, feature engineering, building model and testing accuracy, and classification.

Data Preprocessing.
Preprocessing is the first step that takes the IP traffic data as input and outputs cleaned data. Data preprocessing is considered to be a critical task in machine learning as it may significantly improve the efficiency and effectiveness of the training process. It includes some essential tasks such as removal of irrelevant data, handling missing values, label conversion, categorification, and data normalization.

Removing Irrelevant
Features. e preprocessing step starts by removing irrelevant features. Features such as "Unnamed: 0," "Flow ID," and "Inbound" do not add any relevant information to the problem at hand. Features such as "Timestamp" IP address and port number information are related to IP socket or feature ID, which may cause the model to overfit. Hence, we keep only relevant features and remove the irrelevant ones manually. After identifying relevant features, the next task in the preprocessing step is to handle the missing values.

Handling Missing Records.
ere are several strategies to tackle the missing values. e proposed approach replaces the missing values by the median of the corresponding feature because it provides a better representation of the Security and Communication Networks majority value within the feature. Additionally, the median provides a robust estimation when the data is skewed or contains outlier. Equations (1) and (2) depict the formula for calculating the median of a series of values: (1) where n denotes the number of samples on the dataset.
Label Conversion. is paper considers binary classification of network traffic as normal traffic or DDoS attack. Mathematically, this function can be denoted as where y denotes the data's label.
Categorify. Columns that have discrete string values are converted to integer category index. Category embedding is used to process the discrete variable. Categorical variable embeddings were proposed recently in [20] to improve the processing of discrete variables in neural networks (NN). Compared to onehot encoding, its benefits include reduced memory requirements and higher processing speed. Entity embedding also allows a neural network model to better generalize for sparse data and unknown statistics. Our model processes every numerical column with cardinality (amount of distinct level or unique value) below nine thousand as categorical variables.
Normalization. Continuous values on the dataset have an enormous difference range, leading to higher prediction errors. Dataset normalization helps reduce classification errors significantly and allows the model to converge faster. We used the standardization method (z-score) that scales the features to have a mean of 0 and a standard deviation of 1. is approach enables the model to be less sensitive to outliers. e standardization is given by where σ denotes standard deviation, x i denotes the feature value, N is the number of training samples, and µ represents the mean and is given by Feature Engineering. e CICDDoS2019 dataset is composed of 88 features extracted from IP packets captured using the CICFlowmeter tool. However, not all features can equally contribute to DDoS attack detection. erefore, feature extraction is an essential step in ML as it reduces dataset dimensionality issues. In the context of this paper, we employ the ensemble technique using a different number of feature selection techniques to extract features that contribute the most to identifying DDoS attacks. e ensemble concept has been used for several years in the machine learning field, which combines the results of several models to outperform the model's prediction performance based on a single machine learning technique. Ensemble of feature selection integrates several feature rankings algorithms to produce the final list of features used by the predictive model. e feature ranking techniques adopted are decision tree (DT), random forest (RF), Gradient Boost (GBoost), Extreme Gradient Boosting (XGBoost), Light Gradient Boosted Machine (LightGBM), and CatBoost. e DT creates a simple representation of the dataset using the tree structure, where the branch represents the observed data, and nodes or leaves represent the class label. RF implements an ensemble of multiple DT. Typically, it uses the majority voting approach to compute the classification results as a combination of the results of an individual tree. GBoost algorithm creates ensemble over boosted DT through minimization of error gradient. Each predictor is fitted on the residual error introduced by the previous predictor to improve the overall model performance. XGBoost, LightGBM, and CatBoost are variant implementations of GBoost aimed to tackle some inefficiency of the original algorithm.
To determine the optimal number of feature ranking algorithms that allow our deep learning model to achieve the best classification results, we created four feature ensemble groups, with each one consisting of three to six feature ranking algorithms. We refer to each group as "ensemble group x," where x has the value of 1, 2, 3, or 4, as illustrated in Table 1.
Different approaches can be employed within an ensemble of feature raking to create the final list of features from the lists of features proposed by each ranking algorithm. To select a minimal restricted list, we first applied the  recursive feature elimination (RFE) on each ranking algorithm to find the minimal number of features that enable them to achieve the best predictive performance. RFE is a pragmatic approach that selects features based on their impact on the model performance. It removes features one at a time and computes the impact of the remaining feature until it identifies the optimal minimum feature list. is approach generates one list of features per ranking algorithm within an ensemble group. e final feature list per ensemble group was selected using a threshold, determining how many feature rankers must agree on a specific feature. Considering that the ensemble group consists of three to six ranking algorithms, we defined a nonlinear threshold, calculated by subtracting n algorithm from the list, where n is calculated as the integer division of the total ranking algorithm within an ensemble group by three, according to Table 1. We refer to the list of features generated based on the number of ranking algorithms that agreed on it as an "ensemble list." e list of features produced by each ensemble group is used to train the DL model. However, according to Table 2, ensemble group 1 and ensemble group 2 generated a list consisting of only five features, which resulted in a poor model classification performance. e model trained based on these feature sets achieved a low classification of 94% in most considered metrics. e reduced list of features was generated because each feature ranking algorithm within the ensemble group follows its own strategy to select or rank features from a dataset, and it is not easy to have several different algorithms to agree on the same features set. is issue occurred on the ensemble group having five and six ranking algorithms.
A scenario might have occurred in the experiments described above, where a feature obtaining high evaluation weight by multiple models gets penalized because the number of models that agree upon it is below the defined threshold. We adopted the "Top 10 Weight" strategy to select additional features according to their total average weight within the ensemble group to address this issue. Feature weight value indicates how much a model believes it contributes to the overall prediction performance. For feature ranking techniques such as LightGBM and CatBoost, which evaluate the contribution of features based on criteria other than the weight, we converted this value to the weight by dividing it by the sum of all values assigned to each model. e average feature weight within the ensemble group is used to sort the feature list before selecting the top highest raked features. Table 2 illustrates the feature produced by each of the two strategies adopted. e feature order does not have any significance.
Each ensemble group generates two lists of features. ese lists are used to train three DL models. e first and second models are trained based on the features illustrated in the table above. A third model is trained using a feature list created by merging the two lists above.

Deep Neural Network Architecture.
e proposed DLbased intrusion detection model comprises an input, output, and two hidden layers, as depicted in Figure 2. e input layer has 296 neurons. e first and second hidden layers consist of 200 and 100 neurons, respectively. e network computes the weighted sum from neurons on the input layer, adds bias, and stores the results at each neuron on the first hidden layer. CyDDoS adopts the Rectified Linear Units (ReLu) activation function as it performs better than other activation functions for the said problem. e ReLu function lies between the linear layers to turn all negative numbers into zero and is given by where , and x denotes the activation value. e model then calculates the weighted sum plus bias from 200 size vectors in the first hidden layer and stores it on each neuron in the second hidden layer. Finally, the 100 size vector arrives at the output layer, containing two units of fully connected SoftMax, representing the probability of the traffic sample belonging to the DDoS or normal category. e output layer makes a prediction based on Softmax, which is given by [21] σ(z) j � e z j K k�1 e z k for i � 1, . . . , K , where z is the sample to be predicted and K is the number of classes. CyDDoS uses the Adaptive Moment Estimator (ADAM) as an optimizer. A loss function measures the difference between the actual value and the foreseen value. e model's accuracy is higher if the loss is lower and vice versa. CyDDoS utilizes the cross-entropy loss function to compare the predicted values with actual values and adjust the hyperparameters based on those values. e loss function is given by [22,23] where y andŷ are the target and predicted values, respectively, N is the number of training records, and K is the number of classes. Lastly, another essential aspect of any deep learning model is the learning rate (LR), which defines how fast the deep neural network learns. Choosing the correct LR (not too high, not too low) is imperative as it helps the model to converge faster and with higher accuracy. CyDDoS employs the cyclical learning rate strategy [24], where boundary values are defined for minimum and maximum values at each layer. Cyclical learning rates allow a deep learning model to achieve improved classification accuracy. Table 3 presents a summary of the parameters used in CyDDoS model's definition. e DNN is used to train the model because its main advantages over classic machine learning are its ability to extract different abstraction levels at different processing layers without requiring human intervention [5]. Multiple layers allow a deep learning model to automatically discover complex correlations and mapping from the input to output data [25].

Experimental Setup.
To evaluate the proposed system, we conduct experiments on an Intel-based system equipped with Core (TM) i9-9900X CPU @3.50 G and 128 GB of RAM running Ubuntu 18.04 -LTS. Also, the host system is equipped with Nvidia TITAN RTX GPU, having 24 GB of RAM. In order to reduce the uncertainty of the final results, we repeated the experiment ten times. We use PyTorch v1.7 and fastai v2. PyTorch is an open-source ML framework developed by Facebook's AI Research lab. It includes an optimized tensor capable of running on both CPUs and GPUs [26]. fastai is a higher-level framework built on top of the PyTorch framework to provide cutting-edge features for building DL models. e primary purpose of fastai is to provide the research community an abstract, easy-to-use, and high-performance framework while preserving the lowlevel components for flexibility to develop DL-based systems [27].

Dataset Description.
Our study uses the CICDDoS2019 dataset [28,29], developed and published by the "Canadian Institute for Cybersecurity" in collaboration with the "University of New Brunswick." ese institutions have provided high-quality datasets used by the research community for several years [30]. It is available in both CSV and PCAP formats. e dataset contains over 113 thousand normal traffic samples and over 70 million malicious flows, distributed over 13 classes of recent DDoS attacks. e dataset has been developed due to the limitations of previous datasets used by existing studies. It captures data related to application-level DDoS attacks that use TCP and UDP at the transport layer, categorized as reflectionbased and exploitation-based taxonomy.
Reflection-Based Attacks. Here, the attacker hides its identity and exhausts the victim resources using a legitimate thirdparty node as a reflector server. e attacker first sends packets to the reflector server, using the victim's spoofed IP address as the source address. e victim receives the replay of this packet, and its ability to process it is reduced as many reflectors server is used.   Training data contains 12 types of DDoS attacks, while the test dataset comprises only seven types of attacks. e test dataset does not include the same number of attack types as the training dataset. However, it contains the port scan attack, which is not part of the training set to study the system detection capability for unseen attacks. e dataset is highly unbalanced; malicious traffic flow is much higher than the normal ones. We utilize an undersampling technique that randomly removes records from the dominant class to ensure unbiased learning and classification. A balanced dataset is reliable and ensures that the learning model extracts the correct features representation from the input traffic patterns. CyDDoS makes binary classification; it classifies the incoming traffic as normal traffic or DDoS attack. It uses the 80/20 split of the dataset for training and validation purposes; however, it chooses the validation samples randomly. Finally, we employ the test dataset to evaluate the model performance to assure a truthful detection rate as the model does not have to access it during the learning process. Table 5 illustrates the sample distribution used in our experiments.

Model Training.
e base deep learning architecture used to train several models is first created. is model is trained using 34 features obtained by merging all feature lists produced by the ensemble of feature ranking. Extensive experiments are conducted to learn the appropriate values for different parameters. e model achieved the best results using two hidden layers with 200 and 100 neurons and a batch size of 1024 samples when trained for five epochs, with a duration of 10 seconds. Increasing hiding layer parameters provides a very negligible improvement in accuracy compared to the processing overhead. Table 5 illustrates the flow distribution of traffic samples from the dataset used for training, validation, and testing purpose.

Results and Discussion
Before discussing the obtained results, we first briefly explain the metrics considered for assessing the appropriateness of our proposed system in detecting DDoS attacks compared to the existing state-of-the-art systems.

Performance Metrics.
We use different performance metrics to evaluate the appropriateness of our proposed design compared to existing techniques. e confusion matrix is a commonly used metric to measure an IDS's effectiveness [5]. Many other essential evaluation metrics are derived from it. Also, we consider metrics such as precision, accuracy, recall, and F1-Score to argue about the effectiveness of our model against existing ML-based DDoS detection schemes.
Accuracy is calculated as the ratio of correctly foreseen DDoS and normal flow over total flows: Precision can be computed as a ratio of true positives over the sum of true positives and false positives: Recall is calculated as the ratio of true positives over the sum of true positives and false negatives:  F1-Score is determined as the balanced average between precision and recall: where true positive (TP) and true negative (TN) denote traffic correctly predicted, while false positive (FP) and false negative (FN) represent misclassified traffic.
Receiving Operating Characteristics (ROC) is a graph used to demonstrate the classification model performance at every classification threshold; it plots the False Positive Rate (FPR) and True Positive Rate (TPR). FPR is assigned to abscissa and represents specificity, while TPR is assigned to ordinate to represent sensitivity. Equations (5) and (6) indicate the formula to calculate FPR and TPR, respectively. From the equation, one can see that TPR uses the same equation as recall. ROC curve provides an improved balanced evaluation method because it considers the tradeoff between positive and negative samples [33]: In addition to the metrics mentioned above, area under the ROC curve (AUC) represents the entire 2-dimensional area under the ROC curve that indicates the accuracy of the proposed system. AUC is calculated using the following equation [34]:

Select the Best Minimal List of Features.
A set of twelve additional experiments are conducted to select the best set of minimal features list that allows our DL model to achieve the highest classification performance. Table 6 illustrates the accuracy, precision, recall, and F1-Score evaluation results of each feature set. e results reveal that the model trained based on the Top 10 Weight list, produced by five classifiers (ensemble group 2), provides the best classification performance. From the results, our framework is able to reduce 89% of features from the original dataset while giving a classification superior to 99.6% on most of the considered metrics. is model is trained in five seconds and is selected as the best model used for further analysis. e ROC/AUC graph provides a visual illustration of a balance between the classifier's sensitivity and specificity; a higher AUC value indicates better performance in terms of prediction. Figure 3 shows the CyDDoS's ROC, where the xaxis and y-axis correspond to wrong and correct classification rates, respectively. According to the figure, our proposed solution obtains results near the optimal point, where false positives are zero and true positives are one. is means that CyDDoS can correctly classify about 99.8% of DDoS and normal classes.

Precision-Recall Curves.
Precision-recall curves usually measure the classifier's quality, primarily when the dataset is imbalanced. It depicts the tradeoff between precision (relevancy) plotted on the y-axis and recall (completeness) metrics plotted on the x-axis. Figure 4 reveals that CyDDoS obtains a large area under the curve, indicating that it achieves a high score in both recall and precision. e red dot line indicates the average precision rate, which is 0.999.

Detection Performance.
is experiment calculates the prediction performance based on the different number of samples processed per second on CPU and GPU devices. To the best of our knowledge, no approach was evaluated based on the same testing environment to allow a fair comparison.
e evaluation is performed based on test samples from the CiCDDoS2019 dataset. Figure 5 shows the time required to predict the different magnitude of the dataset in the range of 2 to 100. Results demonstrate that the CPU provides better performance than GPU for a small dataset, having an average inference time of 9.6 milliseconds per batch. Furthermore, Figure 6 compares the number of samples per second processed by the proposed model using CPU and GPU based on different batch sizes. According to the figure, the detection performance significantly increases with larger batch size.
is parameter defines the number of traffic samples processed by CyDDoS simultaneously at each iteration. As the batch size increases, more memory read is required, while the number of iterations needed to process all samples from the test dataset decreases. Performance improvement introduced by GPU is noticed from a batch size of 128 samples. e results show that, using a batch size of 16,384, our model can process around 519,885 samples per second using GPU and 371,346 samples per second when CPU is disabled.

Comparing CyDDoS Performance with Some Classic ML
Algorithms. To validate the efficiency of CyDDoS over classical ML algorithms using the same benchmarking dataset, we executed a new set of experiments to train the four following algorithms: random forest, Logistic Regression, Perceptron, and Gaussian Naïve Bayes. Figure 7 compares our proposed system's accuracy, precision, recall, and F1-Score against the classical ML techniques. As we can see, CyDDoS performs better than any classic technique regarding the considered metrics. Among the classical ML techniques, random forest performs best because it is an ensemble of decision tree algorithm that combines many decision tree models' predictions into a single model.

Comparison with DL-Based Systems.
In addition to classic ML techniques, we compare our proposed model against existing DL-based approaches. Moreover, we consider ROC in addition to precision, accuracy, recall, and F1-Score. Figure 8 Security and Communication Networks     clearly shows that CyDDoS outperforms existing state-of-theart DL-based techniques regarding all considered metrics. DDoSNet [14] discussed in the second section performs almost near our model in terms of precision, as it combines a deep learning technique called Recurrent Neural Network (RNN) with an autoencoder; however, the CyDDoS achieves 1% higher ROC. ROC is a preferable method to compare classifier performance as it represents the classifier's performance over the entire class distribution range [35]. Metaclassification using decision jungle [36] combines different classifiers for binary and multiclass classification. Decision jungle acts as the metalearner that ensembles multiple learners to achieve optimal prediction performance. is proposed approach achieves a precision of 0.99. Kalman backpropagation neural network [37] is proposed to detect DDoS in 5G-enabled IoT networks.
is model achieved the highest score of 0.9749 in the recall. Finally, convolution neural network-based IDS [38] obtained the highest value of 0.954 in accuracy metrics.

Conclusion
DDoS is a predominant threat to the reliability of online services and is experienced frequently by most service providers worldwide. An effective DDoS classification mechanism is required to prevent resource outages as a result of a DDoS attack. However, very few studies consider an up-to-date dataset containing data related to recent DDoS attacks. Furthermore, the existing solution requires a high processing capacity for model training and making predictions.
is work presents an integrated DL-based IDS framework that effectively uses ensemble feature selection to improve DDoS attack detection performance with low processing overhead. Several ensemble groups are evaluated based on different numbers of classifiers to determine the minimal feature list that can produce the best prediction performance. Our proposed model can reduce 89% of features from the CICDDoS2019 dataset and is trained in five seconds. Overall, the experimental results reveal that our proposed design outperforms its competitors, attaining accuracy of 99.6% and the ROC value nearly to 1, based on the test dataset. CyDDoS can be deployed as an SDN application and be integrated with the controller to provide a highly accurate prediction of DDoS attacks.
CyDDoS performed a detection only of DDoS attacks, which is one limitation of our framework. As future work, we will improve the proposed IDS by incorporating a controller to block DDoS traffic and allows the normal traffic to be forwarded. Additionally, the model will be enhanced by introducing adjustments to allow a multiclass detection of the various attacks category. e adoption of DL models in certain critical AI applications, where the reliability and interpretation of the model's internal behavior are crucial, is limited due to its black-box nature. To address this issue, as future work, we will use the eXplainable Artificial Intelligence (XAI) [39] technique to investigate the trustworthiness and interpretability of CyDDoS and make improvements if required.


Joint Link Rate Selection and Channel State Change Detection in Block-Fading Channels

In this work, we consider the problem of transmission rate selection for a discrete time point-to-point block fading wireless communication link. The wireless channel remains constant within the channel coherence time but can change rapidly across blocks. The goal is to design a link rate selection strategy that can identify the best transmission rate quickly and adaptively in quasi-static channels. This problem can be cast into the stochastic bandit framework, and the unawareness of time-stamps where channel changes necessitates running change-point detection simultaneously with stochastic bandit algorithms to improve adaptivity. We present a joint channel change-point detection and link rate selection algorithm based on Thompson Sampling (CD-TS) and show it can achieve a sublinear regret with respect to the number of time steps $T$ when the channel coherence time is larger than a threshold. We then improve the CD-TS algorithm by considering the fact that higher transmission rate has higher packet-loss probability. Finally, we validate the performance of the proposed algorithms through numerical simulations.

I. INTRODUCTION
Selecting a proper transmission rate that adapts channel conditions is important efficient communications in the current wireless systems, i.e., devices using IEEE802.11 standards [1], [2]. Considering the fact that precise channel state information (CSI) feedback is supported by few 802.11 devices [3] but ACK/NACKs of the transmission outcomes are always available to the transmitter, rate sampling approaches [4], [5] that select transmission rate based on historical acknowledgments have been widely used. Online sequential decision making frameworks such as stochastic bandit provide efficient solutions to optimize rate sampling methods [6] when the channel statistics remains constant. However, the use of high frequency spectrum (e.g., mmWave) in the 5G wireless communication systems [7] causes CSI to change more frequently than in conventional spectrums. This motivates us to incorporate changepoint detection algorithms into rate sampling link rate selection strategy in order to improve the cumulative link throughput.
Employing stochastic bandit algorithms for link rate selection in slow and fast fading channels have been widely studied previously. Gupta [6]. The algorithm is then applied to the joint selection of modulation and transmission rate [8]. By utilizing the correlations between transmission success probabilities and transmission rates, [9]- [11] designed modified TS algorithms that can further decrease the expected cumulative throughput regret in higher confidence intervals. The aforementioned techniques are useful under the assumption that channel statistics remain constant, which is true for both slow are fast fading scenarios. However, when channels experience abrupt changes in block fading scenarios and the channel statistics become non-stationary, those algorithms may suffer from slow adaptation. Although TS with a sliding window is proposed in [8] to overcome this issue and is shown to achieve a high cumulative throughput empirically, theoretic evaluations on regret performance are not well addressed.
Designing fast adaptive stochastic bandit algorithms in quasistatic environments are studied in [12], [13]. It is shown that by running a simple CD algorithm to control the clearance of historical data, the expected regret of Upper Confidence Bound (UCB) algorithm in the quasi-static environment can be improved. However, these algorithms are designed for general bandit problems and can be improved when applied to link rate selection scenarios.
Employing change-point detection algorithms to design efficient transmission schemes for block-fading and quasi-static channels has recently been considered in [14]. The key is to estimate channel change points accurately from the high dimensional received signals, and then re-estimate the new CSI and re-design beamforming strategies for channel throughput maximization. Notice that their algorithm relies on the sparse structure of the received signal in angular domain subspace, which requires the receiver to possess large-scale uniform antenna arrays. Therefore, the algorithm cannot be applied directly to mobile devices with a small number of antennas.
To overcome the aforementioned challenges, in this work, we investigate the problem of link rate selection in block fading quasi-static channels. The distribution of channel change points and the probabilities of successful transmission are unknown. The goal is to design a transmission strategy that attempts to maximize the cumulative channel throughput by using historical transmission outcomes. We present an efficient change-point and online decision making algorithm, and then derive the corresponding expected regret upper bound. We then improve the empirical performance of the algorithm by constraining the TS posteriors to lay in a set that can characterize the relationship between transmission success probabilities and transmission rates.
The rest of the paper is organized as follows: in Section II we introduce the system model and formulate the overall optimization problem. Section III proposes a joint channel change-point detection and Thompson Sampling based link rate selection algorithm (CD-TS). We then provide theoretic analysis of the expected regret upper bound. Section IV improves the proposed CD-TS by constraining the sampling posteriors. Section V validates the proposed algorithms via numerical simulations and Section VI draws the conclusion.

II. PROBLEM FORMULATION
We consider a communication rate selection problem for a discrete-time point-to-point wireless channel and let t ∈ {1, · · · , T } denote the index of the current slot. The communication channel between the transmitter and the receiver is quantized into Q < ∞ states, and let h(t) ∈ Q = {1, · · · , Q} denote the current quantized channel state. The channel h(t) experiences block fading and during slot [0, T ], there exists M (M ≥ 1) channel state change points, whose timestamps are denoted by {ν 1 , ν 2 , · · · , ν M }. The channel state h(t) remains the same during the interval between two change points, i.e., h(t 1 ) = h(t 2 ), ∀t 1 , t 2 ∈ [ν m , ν m+1 − 1] but changes rapidly before and after the change point, i.e., h(ν m −1) = h(ν m ), ∀m. The difference between two nearby change points is lower bounded by the channel coherence time L, i.e., ν i+1 − ν i ≥ L, where L 1 is predetermined by the physical channel and mobility characteristics.
In each slot t, the transmitter chooses a transmission rate r i(t) ∈ R = {r 1 , · · · , r R } indexed by i(t). The wireless channel is erroneous and let X(t) ∈ {0, 1} be the indicator function of whether the transmission in slot t succeeds: if X(t) = 1, all of the r i(t) transmitted packets will be successfully received by the end of slot t; otherwise, all r i(t) packets will be lost due to decoding error [6], [9]. The transmission success probabilities depend on the current transmission and channel state, since using higher transmission rate will lead to higher packet-loss probabilities. Therefore, we assume that the transmission success probability follows a Bernoulli distribution with parameter E[X(t)] = θ h(t),i(t) . With no loss of generality, assume that transmission rates are arranged in an increasing order, i.e., r 1 < · · · < r R , and therefore 1 > θ q,1 > · · · > θ q,R > 0. The distribution of X(t) is independent of transmission outcome of other slots X(t ), ∀t = t. At the end of slot t, the receiver sends X(t) to the transmitter without error.
The goal of this research is to maximize the cumulative throughput by designing a link rate selection strategy π that chooses i(t) based on historical transmissions. The problem is organized as follows: (1) Maximizing (1) can be cast in the stochastic bandit framework. However, traditional bandit algorithms assume the distribution of reward remains constant across all the slots. Therefore, adopting these algorithms directly to the quasi-static channel link rate selection may cause large regret, because historical data cannot reflect the current best rate after the channel changes. To overcome this issue, we propose to run a change-point detection (CD) algorithm simultaneously with stochastic bandit to control the clearance of historic data and therefore, improve adaptivity in block-fading channels.
AssumeT = {τ 1 ,τ 2 , · · · } is the detected channel change points sequence and let c(t) = max m {τ m |τ m ≤ t} denote the time-stamp of the most recently detected change point before slot t. We use N i (t), s i (t) and f i (t) to denote the recorded number of total transmissions, success and failure times using Channel change points detection algorithms to calculate {τ m } are proposed in Section III-B. The two parts of the algorithm are synthesized in Section III-C.

A. Thompson Sampling for link rate selection
Consider that in slot t, selecting rate i has succeeded for s i (t) and failed for f i (t) times from slot c(t) + 1 to t − 1. If the channel has not changed after c(t), the posterior distribution of transmission success probability λ i = θ h(t),i follows a Beta distribution parameterized by s i (t) and f i (t). The probability density function of such Beta distribution is: where The Thompson Sampling algorithm for link rate selection is as follows: first sampleλ i ∼ Beta(s i (t) + 1, f i (t) + 1) and then choose rate i(t) = arg max iλi r i for transmission.

B. Change-Point Detection
Recall that CSI changes lead to variations in transmission success probabilities of every transmission rate. Therefore, we attempt to detect past channel changes by comparing the empirical transmission success probability of the latest w transmission outcomes and the second latest w transmission outcomes of a transmission rate. We require there is at least a specific transmission rate indexed by i cd that is selected every F slots, so that the change-point detection algorithm can always be run when needed. In this work, we select i cd to be the rate with the highest historical empirical throughput during slot To efficiently implement the change-point detection, we use W i (k) to record the k-th transmission outcome of using rate i. If rate i(t) has been selected more than 2w times, i.e., N i (t) > 2w, we run a CD algorithm by comparing the mean of the latest w outcomes of using rate i(t) (denoted by M 1 ) and the second latest w transmission outcomes (denoted by M 2 ), where M 1 and M 2 can be computed by: If the absolute difference between the two mean values is larger than a predetermined threshold b, i.e., |M 2 − M 1 | > b, we conclude that a channel change point occurs recently and append t to the detected change points setT .

C. Algorithm Design
We then synthesize our proposed algorithm CD-TS by combining the aforementioned CD and TS parts together.
First we initialize the historical transmission records N i (0) = 0, s i (0) = 0, f i (t) = 0, ∀i ∈ R and set c(1) = 0. Then in each slot t, we perform a joint link rate selection and channel change-point detection as follows: • Transmission Rate Selection: If t = c(t) + kF, ∀k ∈ N + , then we should transmit with rate indexed by Ni(c(t)+F ) r i for efficient channel change-point detection; Otherwise, we select transmission rate based on Thompson Sampling. We sampleλ i ∼ Beta(s i (t) + 1, f i (t) + 1) and treat it as an estimation of transmission success probability using rate i. Then we transmit with i(t) = arg max iλi r i , observe the transmission outcome X(t) and update N i (t+

D. Regret Analysis
We evaluate the performance of the proposed CD-TS algorithm via the cumulative expected throughput regret against the optimum link rate selection algorithm when the precise CSI is known. Denote i * (h) arg max i r i θ h,i as the index of optimal rate when the channel state is h. By definition, the cumulative regret in non-stationary environment denoted by R T , is the difference between the expected cumulative throughput of using for {i(t)} T t=1 chosen by our algorithm and the optimum rate using {i * (h(t))} T t=1 , i.e., A lower regret (4) implies a higher expected cumulative throughput and indicates the algorithm adapts faster to the block fading channel. To compute the expected regret of the proposed algorithm, we first introduce the following lemma, the proof is provided in Appendix A: Lemma 1: Consider the channel remains stable during slot 1 ∼ T , i.e., h(t) = h. Let ∆ i = r i * (h) θ h,i * (h) − r i θ h,i be the expected throughput difference between choosing rate i and the optimum rate i * (h), the expected regret of the proposed algorithm for the this time invariant channel, denoted byR T can be upper bounded by the following inequality for any ∈ (0, 1]:R where coefficient B is a constant unrelated to T , and D KL (p, q) = p log p q +(1−p) log 1−p 1−q is the KL divergence. With Lemma 1, we proceed to our main theorem, whose proof is provided in B:

IV. IMPROVING CUMULATIVE THROUGHPUT WITH CONSTRAINED THOMPSON SAMPLING
In this section, we improve the performance of CD-TS by replacing TS with Constrained Thompson Sampling (CoTS) algorithm proposed in [9]. CoTS can achieve a lower regret performance by utilizing the fact that choosing higher transmission rate will have lower success probability. Recall that θ h,i is the transmission success probability if the channel state h(t) = h and r(t) = r i . To exploit the relationship between {θ h,i } N i=1 for better exploration, we first define set Θ, where the element of each vector λ ∈ Θ is monotonically decreasing: Thus, for any fixed channel state h(t) = h, the transmission success probability θ h,: should belong to set Θ due to the fact that transmitting at higher rate is more likely to get packetlosses. Then we sampleλ(t) = [λ 1 , · · · ,λ N ] jointly from: where Z is a normalizing constant.
To implement CoTS during link rate selection, we first sampleλ i ∼ Beta(s i (t) + 1, f i (t) + 1) independently for each rate. Ifλ ∈ Θ we stop and proceed to select the rate i(t) = arg max iλi r i for transmission. Otherwise we keep samplingλ i ∼ Beta(s i (t) + 1, f i (t) + 1) untilλ ∈ Θ.

V. SIMULATION RESULTS
In this section, we provide numerical simulations to validate the performance gain of our proposed algorithms. We consider link rate selection settings in IEEE802.11a/g systems [15], where we can choose from 8 possible transmission rates from set R = [6,9,12,18,24,36,48,54] (in Mbps) [6]. Suppose the channel is quantized into 3 states and the transmission success probabilities are shown in TABLE I. The cumulative regret is computed for T =3000 slots with coherence time L ≥ 750 and M = 3 change points, and the time-stamps of the channel change points are marked in the figure. The expected regret is computed by taking the average of 100 runs.  Fig. 1 plots the expected cumulative regret and Fig. 2 plots the cumulative throughput of the proposed CD-TS, CD-CoTS, CD-UCB [12] and pure Thompson Sampling [9] algorithm with no change-point detection. The expected throughput/regret are computed by taking the average over 100 runs. From the Fig. 1, the proposed CD-TS and CD-CoTS achieve small regret by incorporating change-point detection into stochastic bandit framework. This is because when the channel changes, the proposed CD-TS and CD-CoTS algorithms can efficiently detect the changes, and they can fit quickly to newly changed channel state by throwing away historical data, while the pure TS algorithm without change-point detection algorithm does not clear up historical data and therefore experiences linear regret once the channel changes. Compared with CD-UCB algorithm, the proposed CD-TS and CD-CoTS algorithms can further reduce the cumulative throughput regret. The decrease in cumulative regret implies that the proposed CD-TS and CD-CoTS algorithms can achieve higher cumulative throughput as depicted in Fig. 2. Compared with the CD-TS algorithm, CD-CoTS algorithm achieves a smaller cumulative regret and higher cumulative throughput empirically by taking the non-increasing characteristics of transmission success probabilities into account. In this paper, we study rate sampling based link rate selection problem in block fading channel. We propose a CD-TS algorithm that detects abrupt channel change points by using a sliding window and then choose the best transmission rate based on Thompson Sampling (CD-TS). We show that, theoretically, the proposed CD-TS can achieve a sub-linear regret with respect to the total number of time steps when the channel coherence time is above a certain threshold. We then proceed to improve CD-TS by sampling packet-loss probabilities of different rates from a constrained posterior set, in which higher transmission rates have higher packet-loss probabilities. Simulation results show that, by detecting channel change points and performing adaptive link rate selection jointly, the proposed CD-TS achieves a smaller cumulative regret and higher cumulative throughput. The proposed CD-CoTS algorithm further improves CD-TS empirically by taking the distribution of transmission success probabilities into account.
Future work includes theoretic analysis of the CD-CoTS algorithm and extending both CD-TS/CD-CoTS to multi-users scenario with block fading channels. Moreover, currently the selection of threshold b and window size w depend on time length T and the number of change points M . It will be of interest to design algorithms that can achieve sub-linear regret without knowledge of T and M .

APPENDIX A PROOF OF LEMMA 1
For time invariant channel with fixed channel state h, let ω be a sample path andT = {τ 1 (ω), · · · } are the time slots of detected channel change points on the sample path. For simplicity, denote i * = i * (h) = arg max i r i θ h,i to be the optimum transmission rate. LetR T (ω) be the cumulative expected throughput regret of sample path ω for the timeinvariant channel, i.e., Since there is no channel change points before slot T , event F = {ω :τ 1 (ω) < T } denotes the CD algorithm raises at least one false alarm before T . By conditional expectation, we can decompose R T into: We proceed to bound each item on the RHS respectively. To upper bound the first item Pr(F)E[R T (ω)|F], we introduce Lemma 2 to upper bound the probability of false alarm p FA in each slot. The proof of which is a direct application of Hoeffding bound and is omitted due to space limitations.
Lemma 2: In each slot t, the probability that a false alarm raises can be upper bounded by: Since we can run the change-point detection algorithm for at most T times from slot 1 to T , by union bound we can upper bound Pr(F) by: Plugging the above equations into the first term on the RHS of (8), we have: Next we proceed to upper bound the second item Pr(F c )E[R T (ω)|F c ]. To do this, let policyπ be algorithm without a change-point detection, i.e., in each slot select i(t) based on Thompson Sampling and transmit with rate i cd every F slots. Notice that for ω ∈ F c , transmission strategy obtained by the proposed CD-TS algorithm andπ is exactly the same. Therefore: where (a) is obtained because regret is non-negative. Then according to [6,Theorem 1], the regret caused by TS R TS ) can be upper bounded by: where and D KL (p, q) = p log p q +(1−p) log 1−p 1−q , ∆ i = r i * (h) θ h,i * (h) − r i θ h,i are the KL divergence and difference in expected throughput, respectively. The regret caused by frequent sample i cd can be simply upper bounded by: Plugging the above items into (12) and then into (8) yields Lemma 1.

APPENDIX B PROOF OF THEOREM 1
The change-point detection algorithm guarantees after the latest change point has been detected for 2wF slots, rate i cd can be selected at least 2w times for runnning change point detection algorithm. Denote M m := {ω :τ m > v m + 2wF } to be the event that the proposed algorithm does not detect change point m after 2wF slots. Then by properties of conditional expectation, the regret can be upper bounded by: where δ min min h,h ,i |θ h,i − θ h ,i | is the minimum absolute difference of transmission success probability. The proof is provided in Appendix D Let T m = ν m − ν m−1 be the duration of the m-th stationary period. Notice that for eachτ 1 , the expected regret E[Rτ 1+1:T (ω)|M c 1 ] = R T −τ1 . Repeating the manipulation in (14) for R T −τ1 and finally we have: PluggingR Tm from Lemma 1 into the equation, we have: Next, we discuss parameter tuning problems to achieve a sub-linear expected regret. We simply set b = 1 2 δ min for easy implementations. To guarantee 2M T exp − w(δmin−b) 2 2 ≤ C 1 and 2M T 2 exp − wb 2 2 ≤ C 2 , we require w to satisfy: For simplicity, we choose C 2 = 2 √ M T and let w = 12 δ 2 min log M T so that the first and second item on the RHS of (17) are sub-linear with respect to T . Achieving a total sub-linear regretR T then requires M T /F and M wF to be sub-linear. To achieve this, we set F = (M T ) α (α > 0) so that M T /F is sublinear to T . If the channel coherence time L satisfies: L/2 ≥ 2wF = 24 may happen. By using the union bound we have: where the last inequality is obtained by Hoeffding inequality.

APPENDIX D PROOF OF LEMMA 3
Suppose the channel in the m-th stationary period is h and denote δ = |θ h,i cd − θ h ,i cd |. If event M m happens, i.e., |M 1 − M 2 | > b. Therefore, since change point m is not detected, it can be conducted that either event |M 1 − θ i cd ,h | > δ−b 2 or event |M 2 − θ h,i cd | > δ−b 2 must happen. By using the union bound and Hoeffding inequality we have: Recall that δ min = min h,h ,i |θ h,i − θ h ,i | be the minimum absolute difference of transmission success probability, by taking the minimum on both sides of (20), we have:


extfieldsofstudy:
Engineering
Computer Science

text:
Automated Methods for Detection and Classification Pneumonia based on X-Ray Images Using Deep Learning

Recently, researchers, specialists, and companies around the world are rolling out deep learning and image processing-based systems that can fastly process hundreds of X-Ray and computed tomography (CT) images to accelerate the diagnosis of pneumonia such as SARS, COVID-19, and aid in its containment. Medical images analysis is one of the most promising research areas, it provides facilities for diagnosis and making decisions of a number of diseases such as MERS, COVID-19. In this paper, we present a comparison of recent Deep Convolutional Neural Network (DCNN) architectures for automatic binary classification of pneumonia images based fined tuned versions of (VGG16, VGG19, DenseNet201, Inception_ResNet_V2, Inception_V3, Resnet50, MobileNet_V2 and Xception). The proposed work has been tested using chest X-Ray&CT dataset which contains 5856 images (4273 pneumonia and 1583 normal). As result we can conclude that fine-tuned version of Resnet50, MobileNet_V2 and Inception_Resnet_V2 show highly satisfactory performance with rate of increase in training and validation accuracy (more than 96% of accuracy). Unlike CNN, Xception, VGG16, VGG19, Inception_V3 and DenseNet201 display low performance (more than 84% accuracy).

I. Introduction
Throughout history, epidemics and chronic diseases have claimed the lives of many people and caused major crises that have taken a long time to overcome. To describe a disease within populations that arise over a specific period of time, two words are used epidemic and outbreak (Obrane et al., 2017;CDCP, 2012). Indeed, we can define epidemic as the occurrence of more cases of illnesses, injury or other health condition than expected in a given area or among a specific group of persons during a particular period. Mostly, the cases are pretending to have a common cause (CDCP, 2012). The outbreak is distinguished from an epidemic as more localized, or the term less likely to evoke public panic.
Past epidemics include pneumonia. The pneumonia is an infection of the lungs most often caused by a virus or bacteria. More specifically, the infection affects the pulmonary alveoli, the tiny balloonshaped sacs at the end of the bronchioles (see figure 1). It usually affects only one of the 5 lobes of the lung (3 lobes in the right lung and 2 in the left), hence the term lobar pneumonia. When pneumonia also reaches the bronchial tubes, it is called "Bronchopneumonia". It is the most important cause of death in the world for children younger than 5 years (about 12.8% of annual deaths) (PERCH, 2019; Liu et al., 2016). It is also a leading cause of morbidity and mortality in adults worldwide and in particular in China Prina et al., 2015;Welte et al., 2012).Pneumonia is the third leading cause of death in Japan with a higher mortality rate for the elderly, particularly among individuals ≥80 years old (Kondo et al., 2017). Excluding lung cancer, in Portugal, Pneumonia is the huge cause of respiratory death (Hespanhol and Barbara, 2017).

Fig. 1: Pneumonia Diagram
Several Coronavirus have passed over the species barrier to cause deadly pneumonia in humans since the beginning of the 21st century. To know the pathogenesis of these deadly epidemics, the specialists have to examine the structure of the viruses and the mechanism of infection. This allows them to help elucidate and provide information for the development of effective treatment and possibly vaccines . Here, we will provide a brief summary of the epidemiology and history of type of Coronavirus in particular: SARS, (Drosten et al., 2003;Ksiazek et al., 2003) is an acute respiratory illness caused by a coronavirus, characterized by fever, coughing, breathing difficulty, and usually pneumonia. SARS It appeared first time in China exactly in the province of Guangdong in 2002 and spread to the world through airtravel routes. Approximately 8098 people were affected, causing in 774 deaths (Walls et al., 2020;Roussel et al., 2020;Wit et al., 2016) with a lethal rate about 10% . It is suggested to originate from bats (Walls et al., 2020;Hu et al., 2017). SARS symptoms are usually same to flu symptoms: fever, chills, muscle aches, headache and occasionally diarrhoea. After about one-week, other symptoms appear like fever of 38°C or higher, dry cough, breath shortness   . Other cases were identified in Jordan (Hijawi, 2013), Qatar (Farooq et al., 2020) then spread to the world. MERS is a zoonotic virus that can be transmitted between animals and people. Indeed, the World Health Organization has confirmed that humans are affected because they were on contact with affected dromedary camels (WHO, 2019; Mohd et al., 2016;Azhar et al., 2014). Studies have shown that how the virus is transmitted from animals to humans is not yet understood, and also the human-to-human transmission is very limited unless there is close contact (WHO, 2018;Ki, 2015;Oboho et al., 2015). The different MERS symptoms are as follows: Fever, Cough (Dry, Productive), Shortness of breath, Diarrhea, Myalgia, Headache, Nausea, Vomiting, Abdominal pain, Chest pain, Sore throat, Haemoptysis (Farooq et al., 2020;Wang et al., 2019;Baharoon et Memish, 2019;Al-Omari et al., 2019;WHO, 2018;Aguanno et al., 2018). The world is currently experiencing a dangerous viral epidemic caused by a virus that has killed tens of thousands of people. This new virus called Covid-19 was identified in Wuhan, China in December 2019 Lippi et al. 2020;Amrane et al., 2020;Cortegiani et al., 2020;Tian et al., 2020;Tang et al., 2020;Faralli et al., 2020;Wilder-Smith et al., 2020;Driggin et al. 2020;Cheng et al., 2020;Pung at al., 2020;El Zowalaty and Järhult, 2020). It belongs to the Corona family of viruses, but it is more deadly and dangerous than the rest of the coronaviruses WHO, 2020). First cases of the disease have been related to a live animal seafood market in Wuhan, denoting to a zoonotic origin of the epidemic (Kandel et al., 2020;Sun et al., 2020;El Zowalaty and Järhult, 2020;Wilder-Smith et al., 2020;Roosa et al., 2020;CNHC, 2020). The routes of transmission, treatments, and results of Covid-19 continually receiving much research attention in the world . Indeed, researchers have identified three main modes of the virus transmission: close person-to-person contact, aerosol transmission and transmission by touch. Lu et al., 2020;Liu et al., 2020). What makes the Corona virus very dangerous is that it can have up to two weeks of incubation without symptoms. We can cite the symptoms of Covid-19: high fever, dry cough, tiredness, shortness of breath, aches and pains, sore throat and very few people will report diarrhoea, nausea or a runny nose (Simcock et al., 2020;WHO, 2020).
As the number of patients infected by this disease increases, it turns out to be increasingly hard for radiologists to finish the diagnostic process in the constrained accessible time (Zhang et al., 2011). Medical images analysis is one of the most promising research areas, it provides facilities for diagnosis and making decisions of a number of diseases such as MERS, COVID-19. Recently, many efforts and more attention are paid to imaging modalities and DL in pneumonia. Therefore, interpretation of these images requires expertise and necessitates a several algorithms in order to enhance, accelerate and make an accurate diagnosis. Following this context, DL algorithms (Bhandary et al., 2020) have obtained better performance in the detection of pneumonia, and demonstrated high accuracy compared with previous state of the art methods. Motivated by the fastest and accurate detection rate of pneumonia using deep learning (DL), our work is going to present a comparison of recent Deep Convolutional Neural Network (DCNN) architectures for automatic binary classification of X-Ray images between normal and pneumonia in order to answer to the following research question: Is there any Deep Learning techniques which distinctly outperforms other DCNN techniques?
The contributions of our paper are as followings: (1) We design fined tuned versions of (VGG16, VGG19, DenseNet201, Inception_ResNet_V2, Inception_V3, Xception, Resnet50, and MobileNet_V2), (2) To avoid overfitting in our models, we used weight decay and regularizers that are provided under keras and have the name L2. Our models have been tested on the chest X-Ray & CT dataset  for binary classification.
The remainder of this paper is organized as follow. Section 2 deals with some related work. In Section 3, we describe our proposed methods. Section 4 presents some results obtained using our proposed models and interpreting the results. The conclusion is given in the last section.

II. Related works
To make a very good diagnosis and to detect the source of the problems of the diseases and in an oppurtum time remains a major challenge for the doctors in order to reduce the sufferings of the patients. Indeed, the use of image processing and deep learning algorithms in the analysis and processing of biomedical images has given very satisfactory results. In this section, a brief review of some important contributions from the existing literature is presented.
Pneumonia remains one of the diseases that is increasingly becoming research hotspots in recent years. Indeed, Toğaçar et al. (Toğaçar et al., 2019) used the lung X-ray images. They employed the convolutional neural network as feature extractor and used some existing convolutional neural network models like AlexNet, VGG-16 and VGG-19 to realize a specific task. The number of deep features was reduced using of deep features was reduced for each deep model. Then a step of classification was done using decision tree, KNN, linear discriminant analysis, linear regression, and SVM. The obtained results showed that the deep features provided robust and consistent features for pneumonia detection.
In (Liang and Zeng, 2020), Liang and Zeng have proposed a new deep learning framework to classify child pneumonia image bay combining residual thought and dilated convolution. To overcome the over-fitting and the degradation problems of the depth model and the problem of loss of feature space information caused by the increment in depth of the model the proposed method used the residual structure and dilated convolution respectively.
The work presented in (Jaiswal et al., 2019) proposes a deep learning-based method to identify and to localize the pneumonia in Chest X-Rays images. The identification model is based on Mask-RCNN that can incorporate global and local features for pixel-wise segmentation. Good performances evaluated on chest radiograph dataset showed the effectiveness and robustness of the model.
The investigation of post-stroke pneumonia prediction models using advanced machine learning algorithms, specifically deep learning approaches has been presented in (Ge et al., 2019). Indeed, authors have used the classical methods (logistic regression, support vector machines, extreme gradient boosting) and have implemented methods based on multiple layer perceptron neural networks and recurrent neural network to make use of the temporal sequence information in electronic health record systems. The obtained results showed that the deep learning-based predictive model achieves the optimal performance compared to many classic machine learning methods.
In (Sirazitdinov et al., 2019), authors have proposed an automated detection and localization method of pneumonia on chest x-ray images using machine learning solutions. They presented two convolutional neural networks (RetinaNet and Mask R-CNN). The proposed method was validated on a dataset of 26,684 images from Kaggle Pneumonia Detection Challenge and the obtained results were satisfactory.
Bhandary et al. (Bhandary et al., 2020) have proposed a Deep Learning framework for examining lung pneumonia and cancer. Indeed, they proposed two different deep learning techniques: the first one was Modified AlexNet. It was intended to classify chest X-Ray images into normal and pneumonia class using Support Vector Machine and its performance were validated with other pretrained deep learning (AlexNet, VGG16, VGG19 and ResNet50). Whereas the second one implemented a fusion of handcrafted and learned features in the MAN for improving classification accuracy during lung cancer assessment.
The work presented in (Behzadi-khormouji et al., 2020) presents a method that can detect consolidations in chest x-ray radiographs using Deep Learning especially Convolutional Neural Networks assisting radiologist for better diagnosis. Authors have used a Deep Convolutional Neural Network pre-trained with ImageNet data to improve the accuracy of the models. Then, to enhance the generalization of the models, they proposed three-step pre-processing approach.
When we use DL and image processing, especially in medical imaging, a natural question arises: how we use this DL, and which is the best architecture of DL will be used? Following the context of X-Ray images classification, we present a comparison of recent Deep Convolutional Neural Network (DCNN) architectures for automatic binary classification based fined tuned versions of (CNN, VGG16, VGG19, DenseNet201, Inception_ResNet_V2, Inception_V3, Xception, Resnet50, and MobileNet_V2).

III. Proposed contributions
Deep Learning (DL) methods have recently demonstrated huge potential with state-of-the-art performance on image processing and computer vision (Alom et al., 2018). These techniques have been applied in various modalities of medical imaging with high performance in segmentation, detection, and classification (Litjens et al., 2017). Some DL methods incorporate skin cancer detection, breast cancer detection, and classification, lung cancer detection, etc. Despite the fact that these methods have shown huge achievement in terms of success in medical imaging, they require a large amount of data, which is as yet not available in this field of applications. Following the context of no availability of medical imaging dataset, our work is going to fine-tuned (Transfer learning (Bhandary et al. 2020))) the top layer of the following DL architectures (CNN, VGG16, VGG19, DenseNet201, Inception_ResNet_V2, Inception_V3, Xception, Resnet50, and MobileNet_V2) and compare between them their performances.

III.1. Proposed baseline CNN architecture
Generally, a CNN model consists of five layers which are: input layer, convolutional layers, pooling layers, full-connection layers, and output layer ( Figure 2). Moreover, it is known that CNN model is can be trained end-to-end in order to allow the feature extraction and selection, finally, classification or prediction. How the network understands an image or process the image is difficult, but it is known that features obtained in different layers of a network works better than human built features (Ouhda et al., 2019). The proposed baseline CNN for our experiment has the following architecture:  Input layer: In our experiment the inputs are X-Ray images. The parameters are defining the image dimension (244x244).  Convolutional layers: a convolution is a linear operation that consists on the multiplication of a set of weights with the input. It's designed for two-dimensional input; the multiplication is performed between a two-dimensional array of weights (filters) and an array of input data. In the proposed architecture we have 3 layers with a filter of size 3x3 and zero padding.  Pooling Layers: Pooling layers is a technique to down sample feature maps by summarizing the presence of features in patches of the feature map. There is two type of pooling methods that are average pooling and max pooling. In the proposed architecture we used Max pooling in order to calculate the maximum value in each patch for every feature map. The max pooling is set to 2x2 with a stride of 2.  ReLU layers: we have used 4 ReLU layer for each convolutional layer  Inner-product layers or fully connected layers: Treats the input data as a simple vector and produce an output as a single vector. We have one inner-product layer in this model. The last one, a fully connected output layer with sigmoid activation.

III.2. Deep Learning architectures
Deep learning architectures is highly used for the diagnosis of pneumonia since 2016 (Bhandary et al. 2020;, the most investigated DL techniques are VGG16, VGG19, Inception_V3, DenseNet201, Xception, Resnet50, Inception ResnetV_2, and MobileNet_V2. We have chosen these 8 techniques due to the high result and accuracies they offer.

VGG16 and VGG19
Proposed on 2014 by Simonyan and Zisserman VGG (Visual Geometry Group) is a convolution neural net (CNN) architecture and used to win ILSVR(ImageNet) competition in 2014 (K. Simonyan and A. Zisserman, 2014). The major characteristic of this architecture is instead of having a large number of hyperparameters, they concentrated on simple 3×3 size kernels in convolutional layers and 2×2 size in max pooling layers. In the end, it has 2 FC (Fully Connected layers) trailed by a softmax for output. The most familiar VGG models are VGG16 and VGG19 which include 16 and 19 layers, respectively. The difference between VGG-16 and VGG-19 is that VGG-19 has one more layer in each of the three convolutional blocks .)

Inception_V3
Inception models are a type on Convolutional Neural Networks developed by Szegedy on 2014 (Szegedy et al., 2014). The inception models differ from the ordinary CNN in the structure where the inception models are inception blocks, that means lapping the same input tensor with multiple filters and concatenating their results. Inception_V3 is new version of inception model presented for the first time on 2015 (Szegedy et al., 2015). It is an improved version of inception_V1 and inception_V2 with more parameters. Indeed, it has block of parallel convolutional layers with 3 different sizes of filters (1x1, 3x3, 5x5). Additionally, 3×3 max pooling is also performed. The outputs are concatenated and sent to the next inception module.

Resnet 50
Resnet50 is a deep residual network developed by (He at al., 2016) and is a subclass of convolutional neural networks used for image classification. It is the winner of ILSVRC 2015. The principal innovation is the introducing of the new architecture network-in-network using residual layers. The Renset50 consists of five steps each with a convolution and Identity block and each convolution block has 3 convolution layers and each identity block also has 3 convolution layers. Resnet50 has 50 residual networks and accepts images of 224 × 224 pixels.

Inception_Resnet_V2
Inception-ResNet-v2 is a convolutional neural network that is trained on more than a million images from the ImageNet database (Szegedy et al., 2016). It is a hybrid technique combining the inception structure and the residual connection. The model accepts images of 299×299 image, and its output is a list of estimated class probabilities. The advantages of Inception ResnetV2 are converting inception modules to Residual Inception blocks, adding more Inception modules and adding a new type of Inception module (Inception-A) after the Stem module.

Densenet201
DenseNet201 (Dense Convolutional Network) is a convolutional neural network that is 201 layers deep and accepts an image input size of 224 × 224 (Huang et al., 2018). DenseNet201 is an improvement of ResNet that includes dense connections among layers. It connects each layer to every other layer in a feed-forward fashion. Unlike traditional convolutional networks with L layers that have L connections, DensNet201 has L(L+1)/ 2 direct connections. Indeed, compared to traditional networks, DenseNet has can improve the performance by increasing the computation requirement, reducing the number of parameters, encouraging feature reuse and reinforcing feature propagation.

MobileNet_V2
MobileNet_V2 (Sandler et al., 2018) is a convolutional neural network and an improved version of MobileNetV1 and is made of only 54 layers and has an image input size of 224 × 224. Its main characteristic is instead of performing a 2D convolution with a single kernel, instead of performing a 2D convolution with a single kernel uses depthwise separable convolutions that consists in applying two 1D convolutions with two kernels. That means, less memory and parameters required for training and small and efficient model. We can distinguish two types of blocks: first one is residual block with stride of 1, second one is block with stride of 2 for downsizing. For each block, there are layers: the first layer is 1×1 convolution with ReLU6, the second layer is the depthwise convolution and the third layer is another 1×1 convolution but without any non-linearity.

Xception
Xception, presented by Chollet (Chollet, 2017), is a convolutional neural network that is 71 layers deep. It is an improved version of Inception architecture and involves depthwise deparable convolutions. More precisely, Xception replaces the standard Inception modules with depthwise separable convolutions. It showed good results compared to VGG16, Resnet and Inception in classical classification problems. Xception has an image input size of 299x299.

IV. Experimental results and analysis
We implemented our contributions for automatic binary classification on a new publicly available image dataset (chest X-Ray & CT dataset). As it can be observed in Figure 3 which shows the diagram of the main proposed contributions, the entire contributions are mainly divided into three steps: data acquisition, data pre-processing and classification. The following sections give out in detail the steps of our contribution.

IV.1. Dataset
This present work introduces a publicly available image dataset which contains X-Ray and computed tomography (CT) images. This dataset, named chest X-Ray & CT dataset available in this link, is composed of 5856 images (jpeg format) and has two categories (4273 pneumonia and 1583 normal). As can be seen from Figure

IV.2. Data pre-processing and data splitting
The next stage is to pre-process input images using different pre-processing techniques. The motivation behind image pre-processing is to improve the quality of visual information of each input image (eliminate or decrease noise present in the original input image, improve image quality through increased contrast, delete the low or high frequencies, etc). In this study, we used intensity normalization and Contrast Limited Adaptive Histogram Equalization (CLAHE).
Intensity normalization is an interesting pre-processing step in image processing applications (Kassani et al., 2019). In our models, we normalized input images ( Figure 5. (a)) to the standard normal distribution using min-max normalization (Equation1). Furthermore, before feeding input images into the proposed models, CLAHE is a necessary step to improve the contrast in images (Kharel et al., 2017;Makandar and Halalli, 2015). Figure 5 illustrates an example of using these techniques.
(1) For data splitting, we used in this experiment 60% of the images for training and 40% of the images for validation. We ensure that the images chosen for validation are not used during training in order to perform successfully the binary classification task. Moreover, we observed that our dataset is imbalanced, indeed 75% of the images represents the pneumonia class. In order to overcome this issue, we resampled our dataset by using data augmentation. In fact, we generated from each single input image 2 new images with different augmentation techniques. Therefore, the total number of images in normal class was increased by 2 times.

IV.3. Data augmentation
Data augmentation is used for the training process after dataset pre-processing and splitting and has the goal to avoid the risk of over fitting. Moreover, the strategies we used include geometric transforms such as rescaling, rotations, shifts, shears, zooms and flips (Table 2).

IV.4. Training and classification dataset
After data pre-processing, splitting and data augmentation techniques used, our training dataset size is increased and ready to be passed to the feature extraction step with the proposed models in order to extract the appropriate and pertinent features. The extracted features from each proposed model are flattened together to create the vectorized feature maps. The generated feature vector is passed to a multilayer perceptron to classify each image into corresponding classes. Finally, the performance of the proposed method is evaluated on test images using the trained model. We repeat each experiment three times and report their average results.

IV.5. Experimental setup
Toward an automatic binary classification based on a publicly available image dataset (Chest X-Ray dataset), our experimentations were carried out based on following experimental parameters: All the images of the dataset were resized to 224x224 pixels except those of Inception_V3 model that were resized to 299x299. To train our models, we set the batch size to 32 with the number of epochs set to 300. The training and validation samples are respectively set to 159 and 109. Adam with β1=0.9, β2=0.999 is used for optimization, and learning rate set to 0. 00001 and decreased it to 0.000001. Moreover, we used weight decay to reduce overfitting of our models. The regularizers are provided under keras and have the name L2. The implementation of the proposed models is done using computer with Processor: Intel (R) core (TM) i7-7700 CPU @ 3.60 GHZ and 8 Go in RAM running on a Microsoft Windows 10 Professional (64-bit). For simulation, python 3 is used and Keras/tensorflow as deep learning backend. Our training and validation steps are running using NVIDIA Tesla P40 with 24 Go RAM.

IV.6. Evaluation criteria
After extracting the appropriate feature, the last step is to classify the attained data and assign it to a specific class (Blum et al., 2001). Among the different classification performance properties, and since our data is balanced our study uses the following benchmark metrics, including accuracy, sensitivity, specificity, precision and F1 score (Blum et al., 2001;Bhandary et al. 2020). These popular parameters are defined as follows: (

IV.7. Results and discussion
In this section we present the result for the binary classification for the chest X-Ray & CT images with the following architectures (Baseline CNN, Fine tuning the top layers of VGG16, VGG19, Inception_V3, Xception, Resnet50, Inception_Resnet_V2, DenseNet201, and MobileNet_V2). In addition, to check the performance and robustness of each proposed model, several experiments are conducted on chest X-Ray dataset. The results are presented separately (Fig.6 -15) using training and validation curves of accuracy and loss. Other experiments are tabulated below (Table.3).

IV.7.1. Classification results of the proposed architectures
This subsection presents and discusses of the classification results of chest X-Ray & CT images. Before to discuss these results, let us define the two most parameters used in the state of the art of deep learning and computer vision: Train curve is calculated from the training dataset that provides an idea of how well the model is learning, while the validation curve or test curve is calculated from a hold-out validation dataset that provides an idea of how well the model is generalizing. Moreover, the training and validation loss is defined as a summation of the errors made for each example in validation or training sets. In contrast to accuracy, loss is not a percentage. To summarize, a model that generalizes well is a model that is neither overfit nor underfit, is the best model. Furthermore, the confusion matrix shows the detailed representation of what happens with images after classification (Bhandary et al., 2020). According to the figure above, it is observed that the accuracy curve of train data is rapidly increasing from epoch 0 to epoch 6 where the accuracy is equal to 83.17% then it starts to be stable until epoch 300 where the accuracy is equal to 86.28% Same for the accuracy curve of test data with an accuracy of 84.84 for epoch 300.
For the loss curve of train data is rapidly decreasing from epoch 0 to epoch 6 where the loss is 43.77% then it starts to be stable until the end of training (epoch 300) where the loss is equal to 36.56%. Same for loss curve of data test with a loss of 37.85% for epoch 300.
From the confusion matrix, it is observed that the first images class (Normal), the model was able to identify 1634 images correctly in the normal class, but 95 were labelled as Pneumonia. Likewise, for the second images class (Pneumonia), the model was able to identify 1277 images correctly, but 252 images were labelled as Normal. The following figure presents accuracy and loss curve and confusion matrix of VGG16. Indeed, from the epoch 0 to epoch 11, the accuracy curve of train data is quickly increasing where is equal to 81.05% then it converges to a value of 87.51%. Same for the accuracy curve of test data with an accuracy of 86.32% for epoch 300.
A rapid decreasing of loss curve can be observed for train data from epoch 0 to epoch 25 where the loss is equal to 1.43% then a kind of stability can be observed up to the value 1. 15%. The same goes for the loss curve of test data where the loss is equal to 2.21 for epoch 300.
From the confusion matrix, we can observe for images class (Normal) the model was able to predict 1517 images correctly in the normal class, but 212 were labelled as Pneumonia. For the images class (Pneumonia), the model was able to identify 1466 images correctly, but 263 images were labelled as Normal.

Fig. 8. Accuracy and loss curve and confusion matrix of VGG19
As it is showed in the figure above, the curve of train data (test data) can be divided into two intervals: the first one starts from epoch 0 to epoch 13 (from epoch 0 to epoch 10) where we can observe a quick increase of accuracy where the accuracy is equal to 81.01% (83.05%). The second interval the accuracy starts to be stable and converges to 87.42% (86.89%).
For the loss curve of train and test data, we can observe a good fit. Indeed, from epoch 0 to epoch 18 the loss is rapidly decreasing where is equal to 1.27% then it starts to be stable until the epoch 300 where is equal to 1.31%.
As observed in the normal class, the VVG19 model was able to predict 1390 images correctly and 339 images as Pneumonia. The model also was able to classify 1582 images as Pneumonia and 147 images as Normal for the Pneumonia class.

 Inception_V3
The next figure shows accuracy and loss curve and confusion matrix of Inception_V3. In fact, for the train and test accuracy and from epoch 0 to epoch 7, we can observe that the accuracy is increasing until the value 91.03%. After epoch 7 the accuracy starts to be stable where is equal to 97.01% and 95.94% for training data and test data respectively. A good fit can be observed for the loss curve of train data in either the quick increasing interval from epoch 0 to epoch 32 where the loss is 3.98% or in the other interval where the decreasing is slow and converges to 1.76%. As shown in the confusion matrix, for the Pneumonia class, the model was able to identify 1650 images as Pneumonia and 79 images as Normal. It can be seen that for images class (Normal), Inception_V3 model was able to predict 1621 images as Normal and 108 as Pneumonia.

 ResNet50
The figure bellow illustrates the results attained by Resnet50. In fact, from epoch 0 to epoch 24 the accuracy values are increasing expeditiously either for train data or test data where the maximum value is 97.36%. After that, the values start to be stable where the value is 99.23% and 96.23% for training data and test data respectively. We can observe a good fit for the loss curve of train and test data, Indeed, from epoch 0 to epoch 21 the loss is briskly decreasing where is equal to 6.89% then it starts to be stable until the epoch 300 where is equal to 0.85%. The confusion matrix indicates that, for images class (Normal), 1703 images were predicted correctly as Normal and 26 were labelled as Pneumonia. Same for the second images class (Pneumonia), the model was able to identify 1638 images correctly, but 91 images were labelled as Normal. From the figure bellow, we can observe that from epoch 0 to epoch 18 the train and test accuracy the accuracy is increasing until the value 95.51%. After epoch 18 the accuracy starts to be stable where is equal to 99.11% and 96.41% for training data and test data respectively. For the loss curve, we can see an excellent fit for training and test data where the values are 3.99% (epoch 24) and 1.17% (epoch 300). For the Pneumonia class, the Inception_ResNet_V2 model was able to identify 1618 images correctly as Pneumonia and 111 images as Normal. On other hand, for Normal class, 1705 were correctly classified as Normal and 24 images as Pneumonia. The obtained accuracy curve of train data is speedily increasing until the value 93.49%. After epoch 16 the accuracy starts to be stable where is equal to 97.16% and 94.91% for train data and test data respectively. A good fit can be checked, for the loss curve of train and test data. In fact, from epoch 0 to epoch 17 the loss is quickly decreasing where is equal to 3.99% then it starts to be stable until the epoch 300 where is equal to 1.91%. The confusion matrix depicts that for the first images class (Normal) the model was able to predict 1712 images correctly in the normal class, but 17 were labelled as Pneumonia. The model also was able to identify 1527 images correctly, but 202 images were labelled as Normal for the second images class (Pneumonia). As illustrated by the previous figure, we can observe that from epoch 0 to epoch 16 the train and test accuracy the accuracy is increasing until the value where the accuracy is equal to 96.38%. After epoch 16 the accuracy starts to be stable where is equal to 98.27% and 96.64% for train data and test data respectively. For the loss curve of train data, an excellent fit is observed. Until the epoch 44 the value of loss is expeditiously decreasing where the value is 1.49% then converges to 0.24%. Regarding to the confusion matrix above, for the first images class (Normal) the model was able to identify 1696 images correctly in the normal class, but 33 were classified as Pneumonia. Likewise, in the Pneumonia class, 1634 images were labelled correctly as Pneumonia and 95 were identified as Normal. It is noted that the accuracy of train data is fastly increasing from epoch 0 to epoch 10 where the accuracy is equal to 93.10% then it starts to be stable until epoch 300 where the accuracy is equal to 95.45%. For the test data, a quick increasing can be observed from epoch 0 to epoch 12 where the value is 86.87% then it starts to decrease till 69.03% for epoch 300.
For the loss curve of train and test data, the values are rapidly decreasing from epoch 0 to epoch 26 where the value is 1.10%. After epoch 26 the value converges to 0.44% and 0.69% for train and test data respectively. When we see this confusion matrix, we can observe that for the first images class (Normal), the model was able to predict 1656 images correctly in the normal class, but 73 were labelled as Pneumonia. The model also was able to identify 1219 images correctly, but 510 images were labelled as Normal for the second images class (Pneumonia).

Discussion
We investigated in this study the binary classification (Normal and pneumonia) based on X-Ray images using transfer learning of recent deep learning architectures, in order to identify the best performing architecture based on the several parameters defined in equation2. First, we individually compare the deep learning architectures by measuring the accuracies. After that, we compare the accuracy and loss results given by each DL architecture to discern the outperforming architecture (Fig  14, 15). Table 3 illustrates a comparison between our different deep learning models used in our experiment in terms of parameters defined in equation2. Fig 19 (a) and (b), the plots of training and validation accuracy increase to a point of stability. It is observed that fine-tuned version of Inception_Resnet_V2, Inception_V3, Resnet50, Densnet201 and Mobilenet_V2 show highly satisfactory performance with rate of increase in training and validation accuracy with each epoch. They outperform the baseline CNN, Xception, VGG16 and VGG16 that demonstrate low performance. In fact, from epoch 20, they start to be stable until the end of training where the training and validation accuracy of baseline CNN, VGG16 and VGG16 are equals to 85%. However, Xception reaches 83 % in validation accuracy and 95% in training accuracy. In this case, the predictive model produced by Xception algorithm does not adapt well to the training Set (Overfiting). Besides, the plots of training and validation loss (Fig 20 (a) and (b)) decrease to a point of stability for each proposed model. As it can be seen, fine-tuned version of our models shows highly satisfactory performance with rate of decrease in training and validation loss with each epoch.

For each model in
Results for our multi-experiment classification, based on different fine-tuned versions of recent deep learning architectures, are tabulated in Table 3. The table depicts in detail classification performances across each experiment. From the results, it's notable that the accuracy when we use baseline CNN, Xception, VGG16 and VGG19 is low compared with other DL architectures, since these last models help to obtain respectively 84.18%, 83.14%, 86.26 % and 85.94 % of accuracy. Unlike, the highest accuracies are reported by Inception_V3, DensNet201, MobileNet_V2, Inception_Resnet_V2, and Resnet50. Therefore, these models attain better classification accuracy since they achieve respectively 94.59 %, 93.66 %, 96.27 %, 96.09 % and 96.61%.

V. Conclusions & Future works
We presented in this work automated methods used to classify the chest X-Ray into pneumonia and the normal class using nine Deep Learning architectures (a baseline CNN, VGG16, VGG19, DenseNet201, Inception_ResNet_V2, Inception_V3, Xception, Resnet50, and MobileNet_V2). The main goal is to answer to the following research question: Is there any Deep Learning techniques which distinctly outperforms other DCNN techniques? Toward this end, the experiments were conducted using chest X-Ray & CT dataset which contains 5856 images (4273 pneumonia and 1583 normal) and performances were evaluated using various performance metrics. Furthermore, the obtained results show that the Resnet50, MobileNet_V2 and Inception_Resnet_V2 gave highly performance (accuracy is more than 96%) against other architectures cited in this work (accuracy is around 84%).
Ongoing work intends to develop a full system for pneumonia via deep learning detection, segmentation, and classification. In addition, the performance may be improved using more datasets, more sophisticated feature extraction techniques based on deep learning such as You-Only-Look-Once (YOLO), and U-Net that was developed for biomedical image segmentation.