Multiscale Attention Guided Network for COVID-19 Diagnosis Using Chest X-Ray Images

Coronavirus disease 2019 (COVID-19) is one of the most destructive pandemic after millennium, forcing the world to tackle a health crisis. Automated lung infections classification using chest X-ray (CXR) images could strengthen diagnostic capability when handling COVID-19. However, classifying COVID-19 from pneumonia cases using CXR image is a difficult task because of shared spatial characteristics, high feature variation and contrast diversity between cases. Moreover, massive data collection is impractical for a newly emerged disease, which limited the performance of data thirsty deep learning models. To address these challenges, Multiscale Attention Guided deep network with Soft Distance regularization (MAG-SD) is proposed to automatically classify COVID-19 from pneumonia CXR images. In MAG-SD, MA-Net is used to produce prediction vector and attention from multiscale feature maps. To improve the robustness of trained model and relieve the shortage of training data, attention guided augmentations along with a soft distance regularization are posed, which aims at generating meaningful augmentations and reduce noise. Our multiscale attention model achieves better classification performance on our pneumonia CXR image dataset. Plentiful experiments are proposed for MAG-SD which demonstrates its unique advantage in pneumonia classification over cutting-edge models. The code is available at https://github.com/JasonLeeGHub/MAG-SD.

I. INTRODUCTION
T HE coronavirus disease 2019 (COVID-19) caused by severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) is one of the most devastating infectious diseases after millennium [1]. This new type of coronavirus is announced in late December, 2019, then spread globally in 2020. It has been declared as a pandemic by World Health Organization (WHO) according to its high contagiosity and unprecedented pressure bought to public healthcare system [2]. The current gold-standard for screening COVID-19 is polymerase chain reaction (PCR) laboratory test, however, the test capacity is extremely limited and requires professional equipment [3]. [4] also reports that PCR tests suffers from high false negative rate.
Radiological images collected by X-ray and computed tomography (CT) are important complements to PCR tests. The virus leads to pneumonia, which is an inflammatory condition of the lung's air sacs [5]. Radiological signs show ground-glass opacity, airspace opacities and later consolidation with bilateral, peripheral, and lower zone predominant distributions [6]. Comparing with CT imaging, CXR diagnosis provides a low-cost and time-saving diagnosis method [7]. Besides, underdeveloped regions can hardly have sufficient CT scanners, making CT based COVID-19 screening impossible. X-rays are the most common diagnostic imaging equipment available even in rural regions, which means X-ray diagnosis can cover larger susceptible population [8].
Diagnosis accuracy of COVID-19 and radiography based infection localization are critical for treatment planning and follow-up evaluations [9]. However, pressure of pandemic forces physicians to evaluate in limited time, which raises misdiagnosis rate implicitly [10]. As a result, accurate and robust classification methods are required. This is challenging as COVID-19 is a new type of disease which has low amount of data comparing with available datasets, such as image data published by [11] or [12]. In addition, the COVID-19 shares characteristic with other types of pneumonia, which requires the method focus on both global and local features [13]. Moreover, varied parameter settings causes imparities when collecting X-ray image from different devices.
Massive radiological data and rapid developing computational power give artificial intelligence (AI) a chance to assist clinical diagnosis. Recently, classification of COVID-19 from radiological images have been explored. Wang and Wong [14] present a COVID-Net operated on CXR images to classify COVID-19 from pneumonia and normal cases. COVID-19 cases are extracted from online COVID-19 datasets published by [15] and [16]. Non-COVID-19 image includes 1591 pneumonia images and 1203 normal images released by National Institutes of Health Clinical Center [17]. The experimental results showed that classification method with residual projection-expansionprojection-extension (PEPX) design pattern achieves 93.3% accuracy, which is better than general deep models such as VGG-19 (83.0% Accuracy) and ResNet-50 (90.6% Accuracy). The authors illustrate the locations focused by their model to visualize its decision making process. Ghoshal and Tucker [18] present a Bayesian CNN to make diagnosis through model uncertainty. It is trained on 68 COVID-19 cases from [15] and Non-COVID-19 cases from Kaggle's Chest X-ray Images (Pneumonia) [19], which improve the classification accuracy of a standard ResNet50V2 model from 86.0% to 89.8%. The authors further discuss the effectiveness of uncertainty-aware classification by decision visualization.
Zhang et al. [20] design a screening method based on ResNet to detect COVID-19 and find abnormalities from CXR images. Images are evaluated by an abnormity detecting module producing reference score to optimize classification loss. The model is trained on 70 COVID-19 images and 1008 non-COVID-19 images, which reaches 96.0%, 70.7%, 95.2% in Sensitivity, Specificity and AUC respectively.
Generally, current studies operated on CXR images mostly depends on online datasets with limited COVID-19 cases. Insufficient data can hardly evaluate the robustness of the models and restricted their generalizability. Models trained on extremely imbalanced dataset also lead to long-tail distribution problems. Although plenty of works have discussed diagnosing COVID-19 by AI, few works address the problem of imbalanced data and limited size of dataset because of several issues: 1) Models trained by imbalanced data tend to classify all the targets to the dominant class which has overwhelmingly more labels than other classes. 2) Unique labels on X-ray image, such as L/R position labels, easily attract model's attention then mislead the predictions. 3) COVID-19 cases share features with non-COVID cases, which requires a sensitive and robust model to do classification.
These challenges inspired us to treat pneumonia classification as a Fine-Grained Visual Classification (FGVC) problem, which aims at classifying sub-level categories under a basic-level category. FGVC cases are similar apart from some minor differences and also has the problem of lacking training data. Classic Convolutional Neural Networks (CNNs), including VGG [21], ResNet [22] and Inception [23], has difficulties handling this problem. We propose a novel Multiscale Attention Guided deep network with Soft Distance regularization (MAG-SD) for COVID-19 CXR image classification. To balance the quantity of different data, a weakly-supervised method is presented, which requires a few labeled data to do effective augmentations. Multiscale strategy is applied to attention generator, producing detailed scalar matrix for prediction. Our classification model is motivated from the fact that clinical diagnosis of COVID-19 follows a procedure which firstly evaluates the regional appearance, then makes diagnosis exclusively. Thus, we propose a multiscale attention module which estimates both shallow and deep layers. Comparing with using feature maps from only highest level features, the utilization of lower features could increase its ability of finding fine-grained features. Moreover, a soft distance regularization method is integrated to refine classification result by adaptively adjusting classification loss. In a nutshell, contribution of this paper is threefold: 1) We design a novel deep network, MA-Net, to treat COVID-19 diagnosis as a FGVC problem. Multiscale attention is introduced to assess attention maps on multi level features. Composed attention maps are used as guidance for training steps. Attention pooling is proposed to utilize attention maps for classification.
2) We address data shortage by proposing attention guided data augmentation and multi-shot training phase. It includes attention mix-up, attention patching and attention dimming that could enhance and search local feature then generating data. Models are trained on imbalanced COVID-19 datasets and achieve the state-of-the-art.
3) Without introducing other modules or parameters, we formulate a new regularization term utilizing soft distance between predictions, which works as a constraint to limit classifiers from producing contradicted output for one target.
This paper is organized as follows. In Section II, we introduce insightful works which have high relevance with our contribution. Section III presents the proposed method. In Section IV, database and experimental setup are reported in detail, then results are presented and discussed individually. The last section concludes this study and highlights the future work.

II. RELATED WORKS
Related works are introduced in this section, including X-ray appearance for typical pneumonia, fine-grained visual classification, attention mechanism for CNNs and multiscale feature fusion utilized in computer vision.

A. Pneumonia X-Ray Imagery
Chest X-ray is a widely used imaging modality providing high-resolution pictures to visualize the pathological changes of thoracic diseases. Diagnosis could be made according to the visual patterns demonstrated on CXR images [19]. Clinical research from Katz and Leung [24] demonstrated that typical image pattern for bacterial pneumonia includes opacity of single lobe and pleural effusion. Viral pneumonia also has radiological appearance such as pulmonary edema, small area of effusions, consolidation or lobe mass. Reports from [25], [26], demonstrated that the most common pattern on CXR in COVID-19 was consolidation or ground-glass opacity. It is notable that COVID-19 shares some visual feature with viral pneumonia while viral and bacterial pneumonia can hardly be differentiated because of similar spatial appearance.

B. Fine-Grained Visual Classification
Mass application of CNNs revealed its advantage in solving large scale image classification problem [27] and illuminated a promising way to settle FGVC tasks by using CNN models to explore inconspicuous local features. Some models relied on local annotations to train part-based detectors, localizing certain parts before prediction [28], [29]. However, local feature annotation requires expensive human labor, limiting its reproducibility in reality. Recently, approaches only require global labels also emerged whose motivation was to first localize the corresponding parts and then compare their local features [30]. Fu et al. [31] introduced WS-DAN, which was a weakly supervised deep network handling FGVC by posing attention to enhance local feature and guide augmentation. FGVC was also a common problem in medical image because of spatial similarity between infections. Qin et al. [32] proposed a fine-grained classification CNN for different types of lung cancer in PET and CT Images.

C. Attention for CNNs
For visual task, attention usually indicates a scalar matrix representing the relative importance and inner relevance of local feature [33]. This nonuniform representation was produced by special designed modules [34]. Works reported that applying attention on classification oriented CNN could provide an intuitional way to localize target object, helping to identify visual properties through local representation. An attention guided method demonstrated by Gondal et al. [35] reported that attention mechanism is helpful in Diabetic Retinopathy (DR) localization and recognition. Zhang et al. [36] regulated the attention of deep model by training self-attention blocks for skin lesion classification and surpassed the baselines. Generally, attention mechanism guide the models to analyze global and local features simultaneously then generate believable classification results.

D. Multiscale Feature Fusion
Extracting hybrid feature maps from multi-resolution input image is a common strategy in computer vision since the the era of hand-engineered features. CNNs have an inherent multiscale feature in pyramidal shape, which is advantageous in producing semantically strong representations if effective feature fusion is operated. Models such as U-Net [37] and V-Net [38] exploited skip connections to associate feature maps across resolutions. FPN [39] leveraged the prediction of multiscale hierarchy by generating multiple prediction. For CXR image, Huang et al. [40] presented weight concatenation method to cooperate global and local feature. Thriving of spatial attention gave inspiration to extract attention from multi-resolution feature map. Sedai et al. [41] proposed A-CNN for chest pathologies localization, which utilized multiscale attention by calculating convex combination from weighted average of the feature maps.

III. METHOD
In this episode, we propose our approach that explore multiscale fine-grain feature adaptively. We first produce an overview for our MAG-SD. Then MA-Net is presented in terms of network architecture with attention modules. A weakly supervised data augmentation module, Attention Guided Augmentation, is introduced to address the shortage of COVID-19 cases. At last, Soft Distance Regularization is proposed to erase noise imported by augmentations.

A. Overview
COVID-19 CXR images are less distinctive comparing with other pneumonia cases, which requires a model to extract features for fine-grained feature of input image. WS-DAN [31], which is competitive in fine-grained image classification has been adopted for this topic. The architecture includes a feature extractor (i.e. ResNet50), an attention generator operated on feature map and an augmentation generator producing localenhanced and noise-blended image. An overview of our MAG-SD is shown in Fig. 1. In primary training route, preprocessed CXR image I 0 is fed into MA-Net for prediction vector P and attention map A. Attention Guided Augmentation is operated on I 0 , using A to produce augmented data I 1 , I 2 , I 3 . In Auxiliary training routes, I 1 , I 2 , I 3 are pushed into MA-Net for prediction vectors p 1 , p 2 , p 3 . All the vectors (i.e. P, p 1 , p 2 , p 3 ) are utilized by Soft Distance Regularization for a proper loss.

B. Multiscale Attention Guided Network (MA-Net)
1) Network Architecture: Fig. 2 presents a demonstration of our proposed MA-Net. As observed, a CNN based encoder is operated on augmented images. Encoder utilizes ResNet50 as backbone, extracting size-different feature maps f 1 , f 2 , f 3 from image I. Multiscale attention generator is used to extract attention map a1, a2, a3 and estimate scale-wised interests. Attention maps are resized for a single output A from features. Then, the output of encoder f 3 and attention map A are assessed by Attention Pooling to generate prediction vector P .
2) Multiscale Attention Generator: Attention mechanism has been used in natural image topics to guide feedforward process [42], [43]. Recently, tentative efforts have been made on deep models such as image classification [44], person perception [45] and sequential decision tasks [46]. Most of the attention models aim at gathering top level information to decide where to attend for the next learning steps. The proposed attention generating model is operated on multiscale feature maps, aiming at extracting attention from different scale. Layers before downsampling are selected as feature map in order to squeeze information out of single resolution feature. For ResNet50 we used, feature maps with 512 * 28 * 28, 1024 * 14 * 14, 2048 * 7 * 7 sizes are chosen. The number of attention map is 32.
The architecture of multiscale attention generator is shown in Fig. 3. f 1 , f 2 and f 3 are feature maps selected from feature extractor. Each of them are processed by 1 * 1 convolution to generate corresponding attention. All the attention maps are downsampled to 7 * 7 and connected residually. The effect of using different number of feature maps is discussed in experiments.
3) Attention Pooling: Attention pooling module mimics the structure proposed by [31], which associates attention output and feature map. Fig. 4 shows the pipeline of the pooling method. Feature map f 3 (2048 * 7 * 7) is extracted from the output of CNN encoder. Multiscale attention map A presented by attention generator is 32 * 7 * 7. Each attention map focuses on diverse location that may contain valuable fine-grained feature. Attention biased features (i.e. part feature map (P F )) are presented by multiplying all the attention maps A, each by each, with  feature map. There are 32 P F s which size equals 2048 * 7 * 7. Global average pooling (GAP) is operated to shrink each P F to 2048 * 1 * 1 in order to describe the activation intensity of attention on feature map. Feature matrix M is produced by concatenating GAP results, producing a vector of 65 536 * 1 * 1. Eq. (1) describes the calculation of P F .
where stands for multiplication of elements between two tensors. f 3 is feature map extracted by CNN. N represents the number of attention maps, which is 32 in our work.
P F j has to go through a downsampling method such as GAP to get description with compressed size, which is 2048 * 1 * 1. Feature matrix M is represented by concatenating all condensed P F j presented in Fig. 4.

C. Attention Guided Augmentation
As mentioned above, attention mechanism emphasizes local feature which affects the classification result. Following the idea, the performance of classification network could be enhanced if attention guided training cases are considered. Weakly supervised methods demonstrated in Fig. 5 present effective augmentations for original image. One normalized attention map (A * ) is randomly chosen for each instance to do individual augmentation.

1) Attention Mixup:
Mixup is an augmentation strategy which generates data by mixing overall image and regional feature  together. As we have a attention map A * j , a detailed region D j could be extracted by doing threshold.
For elements in A * j (l, m), Eq. (2) sets D j (l, m) to 1 if it is greater than threshold θ m ∈ [0, 1]. If not, it will be set to 0. A bounding box surrounding the extracted region is proposed from the raw region. Region coved by the box is enlarged to the same size as input image then merged together with original input I 0 to get augmented input I 1 , which is defined in Eq. (3).
where γ is a parameter range in [0,1] and B stands for the enlarged bounding box. Model could see target precisely by learning local and global feature together.
2) Attention Patching: Encoder could be sensitive to limited part of reception field as valuable spatial features usually distribute in similar position. To encourage the encoder to explore feature from varied part, attention patching is proposed. D mentioned in 1) is patched onto the original image I 0 to propose patched data I 2 , which is demonstrated in Fig. 5. Attention patching enlarges the model's interest region by duplicating the interested area, promoting model to global evaluate its input.
3) Attention Dimming: When training attention generating module for feature map, multiple attention maps may be sensitive to similar region. A responsible fine-grain classification model have to focus on different local features of one target. Attention dimming is proposed to stimulate the attention model searching the whole reception field for valuable information. We obtain a Dimming Mask (DM ) from A * , applying threshold θ d ∈ [0, 1], as represented in Eq. (4).
Augmented image I 3 is generated by applying the mask onto the input, which is illustrated in Fig. 5(c).

D. Soft Distance Regularization
Disturbances are introduced into the original image by using augmentation. (e.g. infection area reduced by attention dimming). To address this problem, we formulate the uncertainty of predictions via the distance between prediction vectors. Intuitively, the distance d could be modeled as Eq. (5).
where x denotes the augmented image, P (I), p(x) represent primary and auxiliary prediction vector respectively. However, the distance between P (I) and p(x) is unstable before the the model well-fitted. Ground truth labels are referenced to stabilize gradients. In Algorithm 1, P (I) is replaced by soft label P (I), filtering out low confidence inferences. Soft distance d (x) can be represented in Eq. (6). The value of θ in Algorithm. 1 is 0.7.
At last, overall loss is modeled by a combination of cross entropy loss and average soft distance, which is demonstrated in Eq. (7).
where L prim ce operates between labels and primary prediction. If two vectors have different prediction for one target, L reg will generate a large value, which reflects the uncertainty of the model on one target. It is also notable that L reg punishes soft distanced , leading the model to generate similar predictions.

IV. EXPERIMENTS
In this section, extensive experiments were conducted to comprehensively assess MAG-SD. Models were trained on datasets with different types of pneumonia and the performance of each proposed method was evaluated. Then the models were compared between other baseline methods using several metrics.

A. Dataset and Experimental Settings
The proposed model was trained and tested on several datasets to evaluate its classification performance and ability of finegrained pneumonia localization. Details of each dataset was shown in Table I. Dataset A was a mutated dataset with 90 COVID-19 from [15] and 168 other pneumonia cases from [17], which directly assessed model's fine-grained classification ability. Dataset B was selected from [47] and [17], aiming at evaluating the model's performance on larger scale. Dataset C was the largest dataset we operated on, which included COVID-19 detection and fine-grained pneumonia classification. Quality of pneumonia localization was evaluated by Localization dataset, which had 13 COVID-19 cases with pixel-wise masks from [48] and 118 non-COVID pneumonia cases with bounding boxes annotations from [17]. In experiments, classic ResNet50 has been adopted as feature extractor. Its layer4 output was chosen as feature map. Attention was extracted from the output of layer2, layer3 and layer4 to ensure multiscale attention. Size of the attention maps were 28 * 28, 14 * 14 and 7 * 7. Both training and testing sets were divided roughly in the same class proportions. 5-fold cross validation was applied to get reliable results.
Models were implemented using Pytorch and trained on two NVIDIA RTX 2080TI GPUs. The optimizer was Stochastic Gradient Descent (SGD) with the momentum of 0.9. For each training, 100 training epochs were deployed, with 10 −6 weight decay, 32 cases per minibatch and 10 −3 learning rate at beginning. Images were resized to 224 * 224 when training and testing.

B. Pre-Processing and Data Augmentation
X-ray images have different appearance according to varied imaging equipment configurations, resolving that the same tissue can be radiologically different. To ensure the intensity distribution of one tissue is similar over the dataset, Z-score normalization was employed when training and testing. Large contrast distribution also introduced extra noise to the dataset, impacting the performance of trained model. Contrast limited adaptive histogram equalization (CLAHE) was proposed to enhance contrast between tissues and restrain noise signal [49].
In image classification, data augmentation has been proved as an effective method to improve robustness and evaluate performance [50]. Augmented data provides more varieties for classification target and remitting the impact of overfitting. Random number of transformations were chosen from a sequence of linear transformation for each training sequence. The list is shown in Table II.

C. Evaluation Metrics
Experiments were evaluated by several metrics. For Classification, Accuracy (ACC), Sensitivity (SEN), Specificity (SPC) and F1 score were employed. For multi-class datasets, mean value between classes were calculated to represent the final performance score of each model. Plots of receiver operating characteristic (ROC) curve and area under the curve (AUC) value were used to compare model functionality. Localization quality was quantified by intersection over union (IOU) which has been widely used in target detect and semantic segmentation task [41]. Accuracy describes the proportion of correctly classified targets,  which is expressed in Eq. (8).
where TP, TN, FP and FN stand for the number of true positive, true negative, false positive and false negative predictions. Sensitivity, also known as true positive rate (TPR), is useful to measure the proportion of true positive predictions over all positive targets, which is defined in Eq. (9).
Specificity, or true negative rate (TNR), is a ratio between the amount of true negative (TN) and false positive (FP) predictions, defined in Eq. (10).
F1 Score considers the performance from both precision and recall, defined in Eq. (11).
IoU represents a value calculated by dividing the overlap of prediction and ground truth by their union. It could be defined straightforward in Eq. (12), where A o and A u denote area of overlap and area of union respectively.

D. Components Validation and Discussion
The methods composed could be concluded into attention modules, attention guided data augmentation and soft distance regularization. Each component was studied by evaluating its improvement in classification performance, which has been quantified by metrics mentioned above. Performance gain was obtained by the following method: the proposed model was first trained on specific dataset with metrics, then, single component was changed or removed and reevaluate on the same dataset.
For all the tested models, Mean value and standard deviation of ACC, SEN, SPC, F1 were recorded. Components validations were reported in Tables IV, V, VI, VII and VIII. Inter-model comparisons could be found in Table III and Fig. 6. Regions interested the attention module were presented in Fig. 7. Parameters in all the experiments were maintained unchanged as possible for condition control. The model were trained on the same size of training set then evaluated on the same size of testing set. 1) Architecture Comparing: Advantages of architecture design has been deeply explored. It has been performed by evaluating classic coarse-grained deep neural networks (i.e. VGG16, ResNet18, ResNet50 and InceptionV3), COVID-19 oriented architectures (i.e. [51] (ResNet), [51] (InceptionV3), COVID-Net-Large), high performance fine-grained image classification structure (i.e. BCNN, BCNN(Attention)) and multiscale feature fusion models (i.e. FPN, U-Net). Statistics analysis between these deep structures helped to explain our advantages in finegrained feature extraction. It can be observed in Table III and Fig. 6 that proposed model had noticeably better performance over others. For our model, accuracy on dataset A, B and C reached 96.94% ± 1.10%, 95.85% ± 1.27%, 87.12% ± 1.55% respectively and performance assessed by AUC are 99.94%, 98.72%, and 95.11%.
Comparing with classic models, our model was specialized for COVID-19 image classification and attention guided training phase had advantage in fine-grained visual classification task. Most of the other COVID-19 oriented models presented better performance than classic models, however, none of them applied attention mechanism or considered fine-grained features, which impacted their accuracy on large scale, multi-class dataset such as Dataset B and C. Comparison between FPN, U-Net and classic models demonstrated that FPN presented results over InceptionV3 in Dataset A and B. In Dataset C, U-Net had higher accuracy than FPN, which exceeded ResNet50. Results indicated that multiscale feature fusion models reached competitive results using relatively simple structures comparing with classic deep models, which left us a hint that multiscale attention might be a possible route to improve.   generated by our model has advantage in AUC value, which is over 0.5% in dataset A, B and C. Architecture differences of our proposed method also influence the performance over datasets. Generally, MAG-SD(proposed) is the most stable model which stays in top-3 in all the datasets, which is a method given consideration to both generalizability and robustness. BCNN was a FGVC model with stable performance on multiple datasets. In order to evaluate the generalization ability of our attention module, multiscale attention and attention pooling were transported to BCNN to train BCNN(Attention). Statistically, BCNN reached 96.00% ± 1.52%, 94.41% ± 1.37%, 84.36% ± 1.84% in Accuracy, which was competitive in all the evaluated models. Attention modules remarkably boosted the performance of BCNN, exceeded our proposed method in Dataset A (SPC) and Dataset B (SEN), which were 96.16% ± 1.74%, 96.61% ± 2.00% respectively.
2) CLAHE Preprocessing: Images collected by different devices were probably distinct in contrast due to configuration variety. CLAHE was employed to relieve the noise brought by contrast distribution. Table IV showed the result that CLAHE obviously improved the performance of proposed model and raised over 1.5% Accuracy on average. Model trained without CLAHE had notable higher standard deviation value. Larger datasets such as Dataset B and C were reported to have more performance gain.

3) Multiscale Attention Generator and Attention Pooling:
Normally, state-of-the-art coarse-grained CNN models suffer from similar global features when dealing FGVC. Under this circumstance, models have to depend on local features, which could be effectively localized by our multiscale attention module. Models trained with attention module (i.e. MAG-SD(0AUG)) and baseline model (i.e. ResNet50) were compared in Table III, and Fig. 6. Results revealed that proposed model surpasses the baseline on dataset A, B and C using most of the benchmarks. In dataset B (SPC), ResNet50 has slight advantage. Comparing with AUC, MAG-SD(0AUG) was 2% over ResNet50. Furthermore, two parts of attention module, attention generating and attention pooling has been investigated separately. Firstly, models were compiled to assess multiscale attention, with 1,2 or 3 size of attention maps considered. Results presented in Table V and Fig. 6 showed the model considering two feature maps achieved the best performance in all three datasets. Possible explanation was that the proposed attention module was too simple to locate valuable fine grained feature on low-level feature maps. Instead of importing meaningful location information, noise was brought into the proposed model. Secondly, we evaluated attention pooling module with models trained with other commonly used pooling methods such as global average pooling (GAP) or global max pooling (GMP). Results on pooling methods were presented in Table VI and Fig. 6, demonstrating that attention pooling surpassed GAP and GMP in all three datasets.

4) Attention Guided Augmentation:
The generated attention maps emphasized local feature that interested the model, which could be used to effectively guide data augmentation in Fig. 5. Models trained with 0, 1, 2 or 3 augmentation were discussed in experiment. In the case of 1 augmentation, attention mixup was selected. 2 augmentations model included attention mixup and attention patching. The results were presented in Table VII and Fig. 6. The table reflects that model with all three augmentations had better performance, however, AUC value showed that in dataset A and B, two augmentations was advantageous. The proposed augmentations emphasized data according to attention map, minimizing negative effect caused by random augmentations. 5) Soft Distance Regularization: Soft distance regularization was presented to relieve augmentation variance. Experiments have been composed to compare it with L2 distance regularization. Table VIII and Fig. 6 illustrated that it surpassed L2 in mean value, but, inferior in standard deviation. Constraint between auxiliary vector and primary vector screen the false prediction introduced by attention guided augmentations. Regularization was calculated between ground truth and auxiliary vector when primary vector cannot provide reliable prediction, keeping the final result away from local minima. L2 compared all predictions directly with ground truth, which has higher stability and sidestepped the disturbances introduced by primary prediction. 6) Attention Based Infection Localization: Technically, attention improved the models by roughly localize the part with high activation intensity. This characteristic of attention inspired us to try MAG-SD on localization topics. The models were trained on the Dataset B we proposed, then test on Localization dataset demonstrated in Fig. 7. It included COVID-19 cases with pixel-wise segmentation and non-COVID-19 cases with bounding box for pneumonia infection. Attention maps A were upsampled from 7 * 7 to 224 * 224. Localization masks for COVID-19 cases were extracted by applying threshold to the attention maps. Bounding boxes for other pneumonia were produced by simply enclosing the localization masks with rectangles. IoU was calculated to evaluate the quality of localization. Image showed that the attention module we proposed could roughly indicate the position of different type of pneumonia with over 0.25 IoU score. Attention map emphasized the influential part from the input image effectively.

E. Distribution Analysis
As we imported multiple fine-grained classes into this topic, it was necessary to report the distribution of our prediction result, which has been shown using confusion matrix in Fig. 8. MAG-SD has been selected to generate the charts to represent the classification result of deep learning models. It could be inferred that the model was suitable for searching definitive features from cases showed in dataset A and B as most of the cases were located on the diagonal line of matrices. In dataset C, classification result between viral pneumonia and bacterial pneumonia was significantly inferior than others, which impacted the global performance of the classification model. These results proved the arguments reviewed in Section II-A, indicated that the CXR visual appearance between viral and bacterial pneumonia was insufficient to make accurate diagnosis.

V. CONCLUSION
We have presented MAG-SD for automatic COVID-19 CXR image classification that reached the state-of-the-art on our dataset. The proposed novel method treated this topic as a fine-grained image classification task, utilizing local features efficiently under the guidance of attention mechanism. Attention maps were generated using multiscale features then used as a reference to data augmentation, helping the model to overcome the lack of COVID-19 cases. The proposed network learned to weight the predictions from both primary and auxiliary training pathways by calculating soft distances between vectors, gaining improvements by screening noise generated by augmentations.
Findings of our exploration were demonstrated and discussed in Section IV. The results indicated the great potential of applying advanced pattern recognition model to clinical diagnosis and epidemic screening. Trained on the clinical knowledge acquired by physicians, our model was capable to extract fine-grained spatial features for COVID-19. Attention was applied in both feature extraction and augmentation stage, which helped to localize pneumonia infection and accrete the data effectively as part of weakly supervised method. Attention module also shows its capability in different models. It could be interesting to design more auxiliary training strategies to guide the model to an optimal solution. Positive feedback on soft distance regularization proved that our method considered auxiliary predictions and eliminated label noise simultaneously, however, hard threshold may limit its adaptability in complicated data.
Although deep learning methods seem promising in clinical diagnosis and pandemic screening, lacking of prior knowledge is always the Achilles' Heel. Supervised learning method, such as MAG-SD we proposed, have to be trained on expensive labeled data. Newly occurred or rare diseases without available data may not be classified properly. Abnormal detecting and clustering model could be proposed as a guidance for supervised models to alleviate the limitations, which is part of our future work.


PowerLinear Activation Functions with application to the first layer of CNNs

Convolutional neural networks (CNNs) have become the state-of-the-art tool for dealing with unsolved problems in computer vision and image processing. Since the convolution operator is a linear operator, several generalizations have been proposed to improve the performance of CNNs. One way to increase the capability of the convolution operator is by applying activation functions on the inner product operator. In this paper, we will introduce PowerLinear activation functions, which are based on the polynomial kernel generalization of the convolution operator. EvenPowLin functions are the main branch of the PowerLinear activation functions. This class of activation functions is saturated neither in the positive input region nor in the negative one. Also, the negative inputs are activated with the same magnitude as the positive inputs. These features made the EvenPowLin activation functions able to be utilized in the first layer of CNN architectures and learn complex features of input images. Additionally, EvenPowLin activation functions are used in CNN models to classify the inversion of grayscale images as accurately as the original grayscale images, which is significantly better than commonly used activation functions.

I. INTRODUCTION
Deep neural networks, in particular, Convolutional Neural Networks (CNNs) [1], have had a crucial role in the performance improvement of computer vision and image processing fields in the last decade. CNNs are a class of deep neural networks that process imagery data and are being widely used both in research and industry. CNN networks utilize the convolutional layer as the main operator, which mimics primary visual cortex cell operation.
Traditionally, a convolution layer in CNN uses the inner product operator as its main computational core to measure the similarity between input image patches and some filters (kernels). However, the inner product is a linear operator and it is not capable of learning complex features. One way to remedy this problem is accompanying convolutional layers with nonlinear activation functions. Another remedy is to replace the ordinary convolution operator with more complex operators, such as similarity measure functions. As an example, Lin et al. introduced "Network In Network"(NIN) in [2], which replaced the inner product operator in convolutional layers with multilayer perceptron (MLP). In the NIN proposed architecture, a multiple fully connected layer followed by nonlinear activation functions is located for each patch in the receptive field of input images.
In 2016, Cohen et al. introduced SimNets architecture that replaced the inner product operator in the convolutional layers with an inner product in feature space [3]. The feature space is essentially controlled by special kernel functions. SimNet architecture was introduced based on two fundamental operators. The first operator is a substitute for the inner product operator, which is a weighted similarity function such as "lp" form functions. The next operator is a replacement of non-linear activation functions and pooling layers. The second operator is defined based on the MEX(log −mean−exp) operator. Cohen et al. proved that the SimNet MLP architecture is associated with kernel machines.
Kernel Pooling was introduced to capture higher-order, nonlinear interactions of features in the kernel form [4]. This method was proposed without additional parameters. It uses Count Sketch to generate a feature map up to p order. Then, the inner product between two features can capture high order feature interactions by using kernel pooling. In this case, the Discrete Fourier Transforms method was applied p times. However, higher-order features should be calculated explicitly, which makes the order grow.
Wang et al. introduced Kervolutional Neural Networks(KNNs) as a generalization of CNNs using the kernel methods [5]. In the architecture of KNN, kernel functions are utilized instead of the inner product operation. For instance, Polynomial Kervolution was defined in line with the polynomial kernel function.
Basically, since the convolution operator is a linear operator, one way to increase the capability of models and learn more complex features of input images is by applying a non-linear activation function on top of the convolution operator. Thus, non-linearity layers can be applied to convolution layers to obtain non-linear feature maps. Currently, activation functions are being used as an inseparable component of neural network architectures. Traditionally, Sigmoid and TanH functions were popular activation functions and were used widely in neural network models. The output of the Sigmoid activation function, also called Logistic Sigmoid, is between 0 and 1. Also, the output of TanH activation function is between −1 and 1. These two functions were de facto activation functions for several decades. However, both Sigmoid and TanH functions have the saturation problem. The saturation of activation functions causes the vanishing gradient problem to happen. This problem made those functions unusable for deep neural networks, i.e., neural networks with more than a few layers.
Rectified Linear Unit (ReLU) solved the saturation problem in the positive input region of activation functions [6]. Introducing this activation function paved the way for deepening the neural network models and adding more layers to networks. Besides, in this function, the saturation problem in the positive input region is eliminated. The output of the ReLU activation function is linear in the positive inputs region and the constant value of zero in the negative inputs region. It should be mention that the saturation problem remains in the negative inputs region of the ReLU activation function. Thus, to solve these problems, several generalizations have been introduced. As an example, the Leaky ReLU activation function [7] had been proposed to prevent the saturation problem in the negative inputs region by a slight slope for inputs with negative values.
In this paper, we introduce a class of activation functions that have non-zero gradient values in both the positive input and the negative input regions. This class of activation functions, which we call PowerLinear activation functions, does not saturate in any region. Theoretically, we propose this class of activation functions based on the generalized convolution operators introduced in [8]. Here, we use the polynomial kernel function for generalizing the dot-product operator in convolution layers. The main type of PowerLinear activation function is called the EvenPowLin activation function, which contains major discussions of this paper. According to our experiments, this function can be applied to the first convolution layer of each CNN. This capability makes some attractive features for convolutional networks.
The interesting behaviour about EvenPowLin activation functions is that they are even functions and produce positive outputs for negative input values. This feature of our proposed functions makes it remarkable among other proposed activation functions. On the other hand, OddPowLin activation functions, another type of PowerLinear activation functions, are odd functions that activate the negative input values with negative values. The proposed activation functions are not saturated in the negative input region.
Additionally, EvenPowLin activation functions provide practical usages. We found an application for EvenPowLin activation functions in making networks that are blind to the colour-inversion of images. For instance, consider a white horse picture on a black background that has been given as a training data image to a network that utilizes commonly used activation functions. The trained model cannot easily diagnose the inversion of the original image (i.e. a black horse image on a white background) as shown in Figure 1. Nonetheless, a network equipped with an EvenPowLin activation function can recognize the inverted image as accurately as the original image.
The paper proceeds as follows. We give some background material in Section II. Next, we introduce the proposed method in Section III. In Section IV, we report the results of our experiments for evaluating the PowerLinear activation functions. We conclude the paper in Section V.

II. BACKGROUND
As mentioned before, the convolution operator is a linear operation. Two generalizations of the convolution operator were introduced by [8] to increase the capability of CNN architectures to learn more complex features. The first generalization was defined as followed: Theorem 1 (Generalization 1): The convolution operator in CNNs can be generalized by substituting the inner product operation x T w between a vectorized input x and a weight vector w by f (k(x, w)), where k is a positive definite kernel function and f is a monotonically increasing function with According to this generalization, monotonically increasing functions of kernel functions can be utilized inside the convolutional layer. Additionally, as far as the inner product operator is a similarity measurement function, it is possible to utilize distance measurement functions instead of the inner product. Therefore, the second proposed generalization of the convolution operator was defined as followed: Theorem 2 (Generalization 2): The convolution operator in CNNs can be generalized by substituting the inner product operation x T w between a vectorzied input x and a weight vector w by f (−d(x, w)), where d is a distance metric and Usual non-linear activation functions can be regarded as monotonically increasing functions that act on more basic forms of generalized convolution operators, e.g. Euclideandistance based convolution operator.

III. POWERLINEAR ACTIVATION FUNCTION
Polynomial kernels, along with Gaussian kernels, are the two types of kernels that were most widely used in kernel methods from its advent. Specifically, polynomial kernels were the default choice of the kernel for digit recognition tasks [9]. Polynomial kernels are defined as where d ∈ N is the degree of the kernel. Theorem 1 introduces a generalization of the convolution operator using positive definite kernel functions. In this section, we will introduce a generalization of the convolution operator using polynomial kernels, which we call polynomialmeasure convolution. Basically, polynomial-measure convolution can be implemented by applying the power activation function σ(x) = x d on the output of an ordinary convolution layer. For odd d the power function σ(x) = x d is an odd function and for even d it is an even function. Note that the odd extension is a monotonically increasing function and therefore a valid activation function according to the framework introduced in [8]. Additionally, the even extension of the power function can be written as: which is equal to the composition of the polynomial kernel x, w 2 and the monotonically increasing function σ : R + → R defined by the formula σ(x) = x d/2 .
One problem with the functionality of even/odd-extended power activation functions is that the gradient explodes outside the range [−1, 1]. Even if we use an initialization algorithm that ensures that the input to the power activation function is in the range [−1, 1] at the start of the training, this property may be violated as the training proceeds. We use two mechanisms to solve this problem. First, we normalize the input data in our experiments. Second, we replace the power function with linear functions outside the range [−1, 1] to avoid exploding gradients. The result of the above line of reasoning is two novel classes of activation functions which Similarly, the OddPowLin activation function of order d is defined as: When an input image patch x is not aligned with a weight vector w, i.e. the magnitude of x, w is small, then both OddPowLin and EvenPowLin activation functions strongly attenuate the dot-product similarity. When x and w have a strong positive correlation, i.e. the magnitude of x, w is large, then both OddPowLin and EvenPowLin produce positive similarity values. The interesting case is when x and w have a strong negative correlation. In this case, the OddPowLin activation function produces a strong negative value, indicating strong dissimilarity, while the EvenPowLin activation function outputs a strong positive value, accepting both the similarity to w and −w. This analysis provides a new explanation about why polynomial kernels have performed well in kernel methods and justify the usage of even activation functions.
PowerLinear activation functions are zero-centered and differentiable at all points. PowerLinear activation functions do not eliminate the activity of any neurons, i.e. direct the activity of neurons with negative outputs toward non-zero valuespositive values for EvenPowLin functions and negative values for OddPowLin functions.
In the following section, we will discuss the experiments on the EvenPowLin activation functions as the activation function applied to the first layer of CNN architectures. Basically, our main discovery in this paper belongs to the EvenPowLin functions. It is because EvenPowLin functions are even functions and activate negative inputs as well. This attribute is unique among other activation functions, which can lead us to practical applications that will be discussed.

IV. EXPERIMENTS
In this section, we will demonstrate some possible applications of EvenPowLin activation functions. We should emphasize that EvenPowLin activation functions will be applied on just the first layer of CNN architectures during the experiments.
First of all, we found that the performance of EvenPowLin activation functions is similar to the performance of the ReLU activation function in terms of classifying normal datasets. Normal datasets are those which are used preparedly and are not inverted, such as Mnist and SVHN datasets.
The second category of our experiments refers to the special functionality that we expected to get better results by applying EvenPowLin activation functions to convolution layers. This functionality is the fact that our proposed functions should make the neural networks persistent against the inversion of grayscale images. In this case, we inverted some grayscale images based on normal image datasets. Fortunately, we noticed a significantly better result than applying other activation functions instead.
We will discuss these two major parts of our experiments in the following.
At first, we examined the general functionality of our proposed activation functions. Basically, our proposed class of activation functions can be used in rudimentary CNN architectures as the activation function applied to the first convolution layer.
As the first case of the experiments, we implemented ResNet-18 architecture. This architecture is a modern CNN architecture and it has been widely used since its creation. ResNet-18 is a convolutional neural network that is considered a deep neural network. We utilized this architecture to classify the Mnist dataset images, SVHN dataset images, and also CIFAR-10 dataset images.
We implemented ResNet-18 architecture by applying the ReLU activation function on the output of each convolution operator layer. After that, we loaded the Mnist dataset and normalized it to classify the input images into digit labels between 0 and 9. As shown in Table I, the accuracy of classification on the test Mnist dataset using ResNet-18 architecture is 99.56%.
In the next experiment, we examined the performance of EvenPowLin activation functions in the same condition of the previous experiment. In our experiments, We decided to utilize only the power 2 of EvenPowLin activation functions and apply this function only on the output of the first convolution layer. The result of using the EvenPowLin activation function with power 2 is shown in   are similar to each other in this case. It should be noticed that the number of epochs in these two experiments was 10 with a batch size equal to 256 and the learning rate was set to 0.01. Similarly, we repeated these experiments for SVHN and CIFAR10 datasets. SVHN dataset is an image dataset including 10 classes. Each class is dedicated to a digit. According to Table I, the accuracy of classifying the test dataset using EvenPowLin activation functions for SVHN is 0.9571, and for the CIFAR10 is 0.8447. We have set the number of epochs for training both of these datasets to 15.
In the next experiments, we will discuss the main functionality of the EvenPowLin activation functions. As mentioned before, EvenPowLin activation functions have the ability to activate equal-value positive and negative inputs with the same output values. To show this point, we just examined grayscale images. In this case, if the output of the input neuron is bigger than 1, it will be activated linearly, which is similar to the ReLU activation function. Similarly, if the output of the input neuron is less than -1, it will be activated linearly with a negative slope. The value of the negative slope is the same as the value of the positive slope in the positive input region. Thus, in grayscale images, white pixels can be activated with the same value as black pixels. From the perspective of EvenPowLin activation functions, the darkness of black pixels and the lightness of white pixels are equal.
In the following experiments, we examined our proposed activation functions on two image datasets, the Mnist dataset, and the SVHN dataset. By applying the EvenPowLin activation function on the output of the first convolution layer in a CNN model to classify normal grayscale images into digit labels, the model can diagnosis the inversion of images accordingly. The inversion of a grayscale image is the negation of all its pixel values. For instance, if the Mnist dataset images are white digits on black background, the inversion of these images will be a black digit on white background.
One of our innovations to make a better result was converting the range of input data pixels between -1 and 1. It seems necessary since our proposed function is zero-centered. After finishing the training process on the normal Mnist train dataset, we inverted the Mnist test dataset. Then, we tested both the normal Mnist test dataset and the inverted test dataset to show the comparison. Basically, we expected that both have the same accuracy because the dark pixels activate with the same value as the light pixels in the training process.
In the beginning, we implemented the ResNet-18 architecture by applying the ReLU function over all the convolution layers. The accuracy of testing the model on normal Mnist test data was 99.48 % and the accuracy of testing the model on inverted Mnist test data was approximately 11.36 %, as shown in Table II. In the next step, we implemented ResNet-18 architecture again and set the EvenPowLin activation function on the output of the first convolution layer and the ReLU activation function on other layers. Then, we selected normal training data of the Mnist dataset as the training dataset. The accuracy of testing our model on the inverted Mnist test came to 99.48 %, with the same accuracy as normal test data reached, as shown in Table III. It is almost 88 % more than the accuracy obtained by the network equipped by only the ReLU activation function.
In the next experiment, we focused on the SVHN dataset. Similar to the Mnist dataset, the task is to classify input digits, which are the house numbers in this case. According to the fact that the images in the SVHN dataset are colorful, to obtain better efficiencies, we converted the images in this dataset to grayscale images. Figure 3 (a) demonstrates the original first test image of the SVHN dataset, Figure 3 (b) shows the grayscale conversion of the image in Figure 3  In this case, it should be mentioned that the grayscale SVHN training dataset has both black digit images on white backgrounds and white digit images on black backgrounds. Naturally, it is because these images were colorful originally, and we converted them to grayscale. Hence, it should be noticed that the ReLU activation function will be applied to both kinds of grayscale images during the training process. In this case, a model that includes the ReLU activation function on the first convolution layer will diagnose white digits on black backgrounds as well as black digits on white backgrounds and they are somehow persistent against the inversion of grayscale images. However, since the number of black digit images on white backgrounds is much more than white digit images on black backgrounds in the training dataset, the model with only the ReLU activation function on each layer will be dependant on this type of grayscale image.
In the final experiment case, We examined the SVHN dataset using the ResNet-18 architecture equipped with the ReLU activation function on the output of all the convolution layers. In terms of classifying normal grayscale test images of the SVHN dataset, the accuracy reached 94.51 %. As described before, we expected using the ReLU function on the top of all convolution layers would be persistent against inversion. Thus, the accuracy of classifying the inverted test images peaked at 92.92 %. Fortunately, with changing the first applied activation function to the EvenPowLin activation function, the accuracy stood at 94.51 %, which is equal to the accuracy that normal SVHN data reached. This property of EvenPowLin activation functions makes them remarkable and superior to commonly used activation functions in this case.

V. CONCLUSIONS
In this paper, we proposed a class of activation functions that can be utilized within the Convolutional Neural Networks. PowerLinear activation functions are based on the Polynomial kernel function, which is a generalization of the convolution operator. This class of activation functions prevents the saturation problem in both the positive input region and the negative input region. EvenPowLin activation functions are the main type of PowerLinear activation functions, which are even functions and activate the negative input values with positive values. We examined EvenPowLin activation functions to classify normal image datasets and inversion of grayscale image datasets. EvenPowLin activation functions perform superior to commonly used activation functions in the case of diagnosing the inversion of grayscale images.


RaspberryPI for mosquito neutralization by power laser

In this article for the first time, comprehensive studies of mosquito neutralization using machine vision and a 1 W power laser are considered. Developed laser installation with Raspberry Pi that changing the direction of the laser with a galvanometer. We developed a program for mosquito tracking in real. The possibility of using deep neural networks, Haar cascades, machine learning for mosquito recognition was considered. We considered in detail the classification problems of mosquitoes in images. A recommendation is given for the implementation of this device based on a microcontroller for subsequent use as part of an unmanned aerial vehicle. Any harmful insects in the fields can be used as objects for control.

Introduction
In the next papers used the developed device for remote mosquito control [15], for pest control [16] and for weed control [17] The relevance of this work is the need to identify the presence of a mosquito in place and determine its position for subsequent destruction because mosquitoes carry a great danger because mosquitoes are carriers of diseases such as malaria, from which more than 200 million people in the world directly infect from mosquitoes. The danger and methods necessary to combat mosquitoes are described in the following articles [1,2,3,4].
In this paper, we propose to use machine vision, neural networks, and a 1-watt laser to neutralize mosquitoes. It is a new direction for control mosquitoes. For the first time, the idea of using a laser to protect against insects was expressed in the early 1980s by American astrophysicist Lowell Wood. But at the same time, due to the complexity of the technical implementation of this project, this idea has not yet found practical application. In the last decade, the rapid development of artificial intelligence technology, machine learning, and computer vision has provided an opportunity for the implementation of many ideas conceived decades earlier. Including the use of modern technology to combat mosquitoes.
The use of traditional methods such as nets, lotions, creams of various kinds, traps is especially popular as individual defense. Especially interesting works are [5,6,7,8]. A common disadvantage of all these methods that they are not autonomy.
As methods against mosquitoes, there are hundreds of different creams, devices, and other devices. But because of the analysis carried out, given that the problem with mosquitoes from year to year it becomes more and more relevant to conclude that no effective solution has yet been found to eliminate this problem. As a result, automating the process of neutralizing mosquitoes gives a favorable forecast for the expediency of using the method proposed in this paper.
In the past decade, the following methods have been proposed for mosquito control. In the article [9], the implementation of an acoustic insect flight detector, which was constructed from a noise reduction system, is presented, the response error was 6.5%. In our study, this detector can be used as an auxiliary method for determining the approximate area for the camera for work with mosquitoes. In the research [10], the classification of certain species of flying insects according to the frequency of beating of the wings is presented.
The following study is the most approximate in the topic of our work, where the results of mosquito detection using the camera are presented [11]. In this case, only one method was considered for detecting absolute difference estimation -and the results on determining the position of z are not presented.
It is worth adding that the Bill & Melinda Gates Foundation sponsored research on the implementation of laser mosquito neutralization technology as a promising method for combating malaria. As a result, company Photonic Sentry was created, information about this from the official site of the company was used. For several years of company operation, a demonstration video was presented on the neutralization of mosquitoes without a description of the installation.
The main advantage when using a laser to neutralize mosquitoes over known methods to combat mosquitoes are the following: -ability to use in open areas, even in the presence of wind; -the ability to create a given safe territory, which is protected from mosquitoes, around houses, or when traveling to nature around the camp; -potentially -a large area of coverage; -in the perspective, to protect a group of moving people who have left a safe territory, by mounting on an unmanned aerial vehicle or on the human body.

Materials and method
The equipment that was used in the research: -From box with mosquitoes to laser system distance -300 mm.
The size of a mosquito can vary from 1 mm to 5 mm, this is the main criterion for the method of detection and retrieval of mosquito coordinates. When monitoring the position of the mosquito with ultrasound, it is necessary to use several sensors in different places and processing their information to calculate the location, which theoretically is suitable only for detecting one mosquito, but if there are several mosquitoes, the device will not work correctly. The temperature of the body of a mosquito due to its cold blood is like the temperature of the environment. With a very high resolution of the thermal imager, the mosquito temperature will differ insignificantly against the background -on the order of 0.1 C. The use of sonar has several difficulties when working in open areas where it is necessary to use sonar with a narrow beam and a narrow radiation pattern.
Following the analysis, in the present study, it was decided to use a camera to detect and determine the coordinates of the mosquito. After the camera detects a mosquito, to increase the likelihood of a laser being affected it is necessary to predict the position of the mosquito in advance (by more than 0.2 s).
Therefore, is necessary to research the factors affecting the behavior of the mosquito during flight.
This paper describes the effect of mosquitoes on odor [12]. In the paper [13], mosquito flight formulas are presented, which are calculated considering the presence of CO2. The speed of a mosquito was determined by a deterministic formula: (1) where the maximum and minimum speed of a mosquito, F is a ramp function that takes values between 0 and 1.
In operation, the flight direction and speed of each mosquito are updated every △T time units. The updated position of mosquitoes is calculated by the formula: (2) where (xn, yn) -the position of the mosquito at time step n. Number d is a direction vector that varies between tracking and tracing.
Considering the detection of a mosquito with the help of a camera, tracking it and using mathematical formulas to predict its further flight, the probability of positive control is increased.
To use a pre-trained deep learning network on the Raspberry Pi for the process of monitoring mosquitoes, initially, deep learning methods were considered to implement. But due to the limited RAM on the Raspberry Pi 3 (1 GB) and even the maximum configuration of the Raspberry Pi 4 (4 GB) and due to the low processor speed of 1. Real-time detection with R-CNN, Fast R-CNN, Faster R-CNN, Yolo, RetinaNet has the same problem with speed of recognition. The solution may be to use -NVIDIA Jetson TX1 and TX2 -the special platform for computing neural networks. The main disadvantage, which is the high cost. Therefore, we focused on methods that can be implemented on the Raspberry PI, which allows you to create an economical and compact device.One of the most frequent ways to detect moving objects is the difference of frames, background subtraction, and analysis of the field of optical flow. Details about the advantages and disadvantages of these methods are written in paper [14].
Using the Hara cascade, you can train a model to track an object, the accuracy is determined largely by Description of which are available on the website of the library developer. We have checked all the parameters. In the best way using -tracker = cv2.TrackerCSRT_create. The advantage of tracking is that it runs faster than detection. Since when an object that was detected in the previous frame is tracked, the program knows the initial data -the appearance of the object, the location, speed and direction of its movement. Fig. 1 shows the tracking of a mosquito with various methods, for mosquitoes size 2-4 mm. since with a sharp change of flight path the speed of the camera is not enough to fix the shape of a mosquito. Therefore, this moment all methods lose the mosquito from the field of view. Fig.1.e for the optical method with mosquitoes showed and many noises.
Using the image preprocessing function can significantly increase the readability of images for machine vision.
In Fig. 2 results in photo processing by Thresholding function-OpencV with different parameters. To determine the distance to the object was used the stereo vision OpenCV function. At the same time, the depth value is inversely proportional to the displacement pixel, and the relationship between disparity and display depth of the mosquito, fig.3. The relationship between the camera and the object is described by the following formula:

Experimental research
For experimental research, the installation was developed, Fig.4.  Boxing with mosquitoes stands at a distance of 300 mm from the laser system. The camera finds the mosquito and transmits the data to the galvanometer, which puts the mirrors in the correct position, and then the laser is turned on.
The algorithm working installation is shown in fig.6. To implement tracking in the Python language, various algorithms were written, both with tracking at each moment of only one mosquito and using the multithreading function and transmitting data on the position of mosquitoes using an array to the galvanometer. The success of the experiment can be enhanced by using a more powerful laser, which will make it possible to neutralize more than 2 mosquitoes in one second. Since considering the speed of the galvanometer, which allows you to change the thousandths of a position in seconds, we are limited in speed to neutralizing mosquitoes only by the power of the laser and the power of the central processor. The number of tracking mosquitoes at the same time depends on the processing power of the processor.

Discussion and conclusions
One of the factors limiting the use of laser technology to neutralize mosquitoes is the small allowable lesion area. The possibility of using a telephoto lens to monitor the position of mosquitoes was analyzed.
In fig.7 prototype for remote monitoring of mosquito's present. At the same time, the use of a microcontroller instead of a single-board computer will significantly reduce power consumption by several watts. It is worth considering various programming languages, for example, a program on C will be more efficient, than python language.
As drivers for motors, it is recommended to use drivers based on SMD components. Like a rechargeable battery, for example, Thunder Power Adrenaline Weight not more than 179.69g and capacity 1100mAh This will significantly reduce the dimensions of the device and reduce its weight to 450 grams.
What ultimately increases the efficiency of the device, because it will be possible to use the device using an autonomous aircraft. Nowadays many companies specialize in the production of an apart capable of bearing the burden and fly significant distance without recharging.
The results of these works will be fully disclosed in subsequent research.
In the future necessary to use a laser with focusing. In this research, due to the insignificant distance between the laser and the mosquito box, the pre-configured focus of the laser did not play a big role, since the box width was 70 mm. The lesion area on the near side of the box exceeded the mosquito damage area by 2% by the distant side in the view. But it can be of the general error, in this research we do not focus on this.
It is advisable to consider replacing the audio sensors that were used to demonstrate the ability to neutralize mosquitoes over considerable distances. As a device for determining the distance can be used a laser rangefinder with a high data rate, where for an interface to use SPI. Since the mirrors can change position within milliseconds, monitoring with a laser rangefinder from technical implementation does not carry any difficulties.
Due to the mosquito's flight speed, 1 meter per second, with increasing distance from 15 meters the rotation speed along the z-axis with a servomotor greatly affects the result. Given this, recommend using the algorithm is implemented in such a way that after identification, mosquitoes are assigned numbers, if the mosquito has left the camera's observation area, its coordinate is calculated using the mathematical model and the laser focus is transferred to the calculated coordinates In this case, this will give a significant increase in time. Otherwise since due to the mosquito leaving the observation zone, to determine the exact position, it is necessary to process the entire picture from the camera once again after turning along the z-axis, which is very expensive from time.
The use of neural networks to predict the position of a mosquito will lead to the installation of an additional microcontroller, which will lead to a 2-fold increase in the mass of equipment and the consumption of electrical energy.
In this research the idea of using a galvanometer with a powerful laser to neutralize mosquitoes was first realized. The system can neutralize 2 mosquitos per sec and this result can be easily improved.
Experimentally proved the feasibility of using a laser for the destruction of mosquitoes. Developed a laser unit that changes the direction of the laser with a galvanometer. A program has been developed for the monitoring of mosquitoes in real-time. Calculations and results of experimental studies are given.
Experimental research various techniques for mosquitoes and selected the best option for neutralizing mosquitoes with using in Raspberry PI -library OpenCV with function cv2.TrackerCSRT_createm.

Funding Statement
The authors received no funding from an external source.


Digitally Controlled Steered Dual Beam Pattern Synthesis of a Rectangular Planar Array Antenna in a Range of Azimuth Plane Using Evolutionary Algorithms

This paper presents a pattern synthesis method to generate dual-beam patterns of a rectangular planar array of isotropic antennas in a particular scanning angle using Evolutionary Algorithms. The dual-beam patterns are cosec2 pattern and pencil beam pattern, and both the patterns are steered to an elevation angle of 20 degrees (θ = 20◦). Moreover, each pattern is synthesized in three azimuth planes (φ = 0◦, 5◦, and 10◦). The isotropic elements are uniformly spaced, and nonuniform excitations are applied to achieve the desired patterns. These patterns are obtained by applying the optimum set of common elements amplitude and phases for the cosecant-squared pattern only. The optimum 4-bit discrete amplitudes and 5-bit discrete phases are produced using Differential Evolutionary (DE) Algorithm, Genetic Algorithm (GA), Particle Swarm Optimization (PSO) Algorithm, and Firefly Algorithm (FA). These discrete excitations are helpful to reduce the Dynamic Range Ratio (DRR) and the design complexity of the feed networks. The excitations are also verified in a range of arbitrarily chosen azimuth planes. The patterns are generated in the same steering angle with minor variations of the desired parameters. The outcomes established the superiority of DE over PSO, GA, and FA and the effectiveness of the proposed method.

INTRODUCTION
The array antenna having scanning capability is very useful in wireless communication, mainly in radarrelated applications. However, generating the cosec 2 beam and pencil beam at a range of azimuth planes along with an elevation angle faced a high sidelobe with an enormous ripple problem. Various approaches reported in literature for generating array patterns and dual beam patterns are as follows [1][2][3][4][5][6][7][8][9][10][11].
Multiple beam patterns using simulated annealing algorithm [5] have been generated by Diaz et al. Lei et al. proposed and developed a process for generating a cosecant-squared beam pattern of a linear antenna array by incorporating optimum amplitudes and phases of the isotropic array elements by applying the modified least square method [6]. Both uniform and Gaussian distributions of common amplitudes are applied to generate shaped beam patterns (flattop and a cosecant-squared). Different sets of phases are also used for different beam patterns. For finding these excitations, the Woodward-Lawson technique [7] has been introduced by Durr et al. Chatterjee et al. proposed a method for generating dual beams using optimum sets of radial amplitudes and phases of a concentric ring array antenna. These excitations are achieved by using Gravitational Search Algorithm (GSA) [8]. Chatterjee et al. also proposed a Firefly Algorithm (FA) technique for generating a dual-beam pattern of a concentric ring antenna array. The applied elements are isotropic, and the excitations are used in the rings of the CCAA instate of the elements. The expected patterns are generated by applying specific states or combinations of switches [9]. A dual radiation pattern of the rectangular planar array has been generated at different azimuth planes with zero degrees elevation angle using the evolutionary algorithm by Mandal et al. [10]. A flattop and pencil beam is also generated using common amplitudes distribution and phases for shaped beams at zero degrees elevation by Mandal et al. in 2015 [11]. Kenane et al. proposed a method of synthesizing two different non-uniform antenna arrays Linear and Circular, using Dynamic Invasive Weeds Optimization (IWO), which helps to get deep nulls in the directions of interferences with low sidelobe levels [12].
In this paper, a cosec 2 pattern and a pencil beam pattern from a planar array [1][2][3][4] of 90 isotropic elements are obtained by finding out the optimum set of common elements amplitudes for both the patterns and a group of phases for cosec 2 shaped beam using Evolutionary Algorithms. Here 20 • progressive phase has also been applied to scan the beam patterns on that elevation angle. The patterns have been generated in three predefined azimuth planes using the excitations achieved by DE, GA, PSO, and FA. This approach also verifies that the patterns retain their desired parameters within a range of azimuth planes instead of a single φ plane. These are also proved by selecting some arbitrary φ planes and obtained similar patterns by applying the same excitations for each Evolutionary Algorithm with some minor variations. The received amplitudes (4-bit) and phases (5-bit) are both digitals for providing a lower Dynamic Range Ratio (DRR). These discrete excitations are used to shorten the design complexity of the feed network as DRR is less, so less number of attenuators and phase shifters are required. The comparative performance of these four Evolutionary Algorithms, Differential Evolution algorithms (DE), Genetic Algorithm, Particle Swarm optimization algorithms, and Firefly Algorithm (FA) are also analyzed.

PROBLEM FORMULATIONS
A planar array of 90 isotropic elements is considered. The far-field pattern of the array shown in Figure 1 can be written as [1][2][3][4]: I mn e j[kmdx(sin θ cos φ−sin θo cos φo)+kndy(sin θ sin φ−sin θo sin φo)+αmn] Here, I mn is the excitation amplitude of mn-th element; α mn denotes the phase excitation of mn-th element; M and N denote the number of isotropic elements in X and Y directions, respectively; d x and d y = 0.5λ represent the inter element spacing along with X and Y directions; θ and φ are polar and azimuth angle; θ o and φ o are beam pointing angle; k = 2π λ is the wavenumber. The fitness function for the dual-beam pattern (cosec 2 and Pencil Beam) is defined as: where ∆ is defined as: In Equations (2) and ( ρ is the unknown parameter set responsible for the desired beam patterns for this approach. ρ is defined as follows: peakSLL d 1 and peakSLL d 2 are the desired values of peak SLL for cosec 2 and pencil beam patterns. S 1 and S 2 are sidelobe regions for both patterns. D dB (θ, ϕ) is the desired pattern shown in Figure 2 at   Particle Swarm Optimization (PSO), and DE. This Optimization Algorithm DE [13][14][15][16][17][18][19] can find out true global minima with fewer control parameters and achieve convergence quickly for nonlinear problems. DE is an efficient and effective global optimizer that is population-based. DE algorithm has used a very powerful stochastic search technique. In DE, three major parameters like population size, scaling factor, and crossover rate are crucially responsible for algorithm's performance. The flowchart of DE is shown in Figure 3. To minimize a function f (X), where, where D is the search space dimension or number variables. If the population size is considered as N , the population matrix can be expressed as Here g is the generation and n = 1, 2, 3, . . . , N . Initial population is generated randomly between upper bound (x U i ) and lower bound (x U i ). After random initialization and generation in D dimensional search space, the three main steps involved in the DE Algorithm are Mutation, Recombination, and Selection.
In the Mutation process, three different vectors x g r1n , x g r2n , and x g r3n are selected from each parameter vector. Donor vector v g+1 n can be written as follows. v g+1 F is called Scale Factor, and the value is between 0 and 1. During the Recombination operation the trial vector u g+1 n,i is generated using target vector x g n,i and donor vector v g+1 n,i . Finally, in the Selection process, the objective function values are compared with each target vector x g n,i and trial vector u g+1 n,i . Those who produce the lowest function value, i.e., the best fitness function, are selected for the next generation. These steps are repeated until it reaches the predefined value of generation. The result, i.e., the best solution of the objective function in the current population, is expressed as X best,G .
For solving this optimization problem, the population size is considered as 50, the scale factor (F ) taken as 0.8, and the crossover rate (CR) of DE chosen as 0.2, whereas the scheme used in this nonlinear problem is DE/best/1/bin with maximum iteration number of 2000 [13][14][15][16][17][18][19].

An Overview of Particle Swarm Optimization Algorithm (PSO)
With the inspiration of social behavior of birds flock, fish school, etc., a population-based stochastic optimization technique is developed by Kennedy and Eberhart in 1995. The name of this optimization algorithm is Particle Swarm Optimization. It utilizes a population of potential solutions to probe the search space concurrently. The individuals are called the particles, and the population is called the swarm in PSO. In the search space, each particle is associated with a position and velocity. As the swarm is a set of N particles, it can be expressed as: s = {x 1 , x 2 , . . . , x n }, and the position and velocity of a particle in n-dimensional problem is defined as [20][21][22][23][24][25] The velocity and positions of the particles are swaps according to the iterations, and these swaps are controlled by the execution of the particles itself and that of the other particles. The algorithm can be summarized as follows: Step 1: Initialization The position and velocity of N number of particles are initialized randomly at iteration t using the following equations: for i = 1, 2, 3, . . . , N and j = 1, 2, 3, . . . , n (11) where x min and x max are the lower and upper bounds of the problem. Within the range [0, 1] rand is an uniform random variable.
Step 2: Fitness evolution at the current iteration Compute the fitness at the current generation for each particle in the swarm, i.e., evaluate f (x i ) for i = 1, 2, . . . , N .
Step 3: Compute pbest i and gbest At current iteration, pbest i is the best previous position of the i-th particle, and gbest is the global best position among all the particles present in the swarm using the following equations [20][21][22][23][24][25][26][27] Step 4: Update particles position and velocity Update the velocity and position at iteration t of the particles using the following equations for i = 1, 2, . . . , N and j = 1, 2, . . . , n. Two uniformly distributed random variables rand 1 and rand 2 have a range in between [0, 1]. c 1 and c 2 weighting factors are called cognitive and social parameter. w is known as the inertia weight. It is expressed as: where w min and w max are set to 0.4 and 0.9, and T is the maximum iteration, which restrict the velocity of the particles such that |v ij (t + 1)| ≤ v max , for i = 1, 2, . . . , N and j = 1, 2, . . . , n. This is termed as velocity clamping and is accomplished using the equation given bellow: Step 5 Repeat steps 2-5 until iterations reach their maximum limit. Retain the values of gbest as the final result. The swarm size of PSO is taken as 50 with the randomly chosen initial population. The values of C 1 and C 2 are considered as 2 [20][21][22][23][24][25][26][27]. The time-varying inertia weight (w) is chosen in the way where the merits are decreased linearly from 0.9 to 0.4. For every particle on the dth dimension the maximum admissible velocity is considered as 0.9r d , where r d is the deflection between the maximum and minimum possible utility of the decision variables on this dth dimension. At a maximum iteration of 2000, the termination condition is considered.

Parametric Setup of Genetic Algorithm (GA) and Firefly Algorithm (FA)
In GA, the Population size is considered 50, and the two-point crossover schemes are selected. The probabilities of Crossover and mutation are chosen as 0.08 and 0.01. In the Selection process, the Roulette Wheel scheme is thought about for this proposed problem, and the termination condition is chosen as a maximum iteration of 2000. Other parametric setups of the Genetic Algorithm are selected from the recommendation given in [28][29][30][31][32]. In FA number of fireflies 50 is taken, β 0 = 0.20, γ = 0.25, α = 1, and Search space dimension and Choice of initial population are considered as 48 and random, at a maximum iteration of 2000 [9].

RESULTS
A planar array of 90 uniformly placed isotropic elements has been considered. M number of elements are uniformly placed along X direction, and N number of elements are placed along Y direction. where M = 10 and N = 5 have been chosen. The inter element spacing is considered as 0.5λ, i.e., d x = 0.5λ and d y = 0.5λ shown in Figure 1.
The design parameters of the steered dual-beam patterns along with their corresponding obtained results are shown in Table 1. From Table 1, it has been seen that by applying DE the obtained values of the peak side lobe level for the cosec 2 beam pattern in three different predefined planes are −19.4309 dB, −19.2812 dB, and −19.3472 dB corresponding to their desired value of −20.00 dB. The parameter ripple (∆) is incorporated to measure the total deviation between the obtained (shown in Figure 4) and desired patterns (shown in Figure 2 Similarly, Table 1 shows the obtained and desired values of peak SLL and ripple ∆ using Genetic   Figure 4, Figure 5, Figure 6, and Figure 7 depict the obtained steered dual-beam patterns using DE, GA, PSO, and FA, respectively. The element wise amplitudes and phases of the array elements obtained using DE for generating the beam patterns are shown in Figure 8, and their corresponding values of normalized amplitudes and phases are written in Table 3. These normalized amplitudes have 2 4 , i.e., 16 different levels between 0.625 and 1. Similarly, the phases have 2 5 , i.e., 32 different levels between −180 • and +180 • . Figure 9 and Figure 10 show the 4-bit discrete amplitudes and 5-bit discrete phases using GA and PSO. The exact value of discrete 4-bit amplitudes and 5-bit phases are given in Table 4 and Table 5. In Figures 11, 12, 13, and 14, the beam patterns are generated in three arbitrarily chosen azimuth planes for the same excitations with some minor variation. In each figure, the first arbitrary azimuth angle is chosen as 2.7 degrees (0 • < 2.7 • < 5 • in between the predefined φ plane); the second is 7.8 degrees (within the predefined φ plane 5 • and 10 • ); and the third is 12.1 degrees (> 10 • beyond the predefined azimuth plane). Figure 4 shows the array pattern in predefined azimuth planes whereas Figure 11 shows the pattern in arbitrarily chosen azimuth plane, and in both the cases, the optimum excitations are generated using DE. The obtained cosec 2 beam patterns follow the desired beam pattern shown in Figure 2 within the coverage range of elevation angle (20 • to 50 • ).
Similarly, (Figure 5, Figure 12 ), (Figure 6, Figure 13 ) and (Figure 7, Figure 14 ) have nearly equal values of the design parameters obtained by using the optimum excitations produced by GA, PSO, and FA for predefined azimuth angle and arbitrary plane.
In Table 2, the values of design parameters are mentioned for the arbitrary angles for all four EAs. The obtained values of design parameters for arbitrarily selected φ planes and predefined φ planes are comparable. So for this array synthesis method, it is not essential to consider all the azimuth planes rather some predefined azimuth planes which ensure a range where the pattern retains the desired characteristic. Figure 15 shows the convergence curves of DE, GA, PSO, and FA. From these convergence curves it is clearly observed that DE is superior to GA, PSO, and FA as fitness value of DE is less than the others. The Computations have been done in MATLAB 2015a with core 2 duo processor, 3 GHz, 4 GB RAM. Table 6 below

CONCLUSION
A steered dual-beam planar array antenna has been synthesized in three different azimuth angles. Two different beam patterns (cosec 2 and a pencil beam) are scanned to a particular elevation angle and again synthesized in three predefined azimuth planes using three well known Evolutionary Algorithms. Each Evolutionary Algorithm generates optimum 4-bit discrete amplitudes and 5-bit discrete phases to achieve the desired parameters of the patterns. After selecting and achieving the similar pattern in arbitrary planes, the presented method shows its capability to produce the same beam pattern not only in the predefined azimuth planes rather a range of azimuth planes. The design parameters like peak side lobe level (peak SLL) and ripple ∆ are reduced by finding the optimum set of array excitations using DE, GA, and PSO Algorithm. The Dynamic Range Ratio (DRR) is also reduced by using these digital excitations, which results in the smaller number of attenuators and phase shifters in the feed network. This pattern synthesis method also ensures that the desired patterns retain their desired specifications with some minor variations in a range of azimuth plane on the scanning angle also rather in a particular predefined φ plane with zero elevation angle. An acceptable concurrence between the desired and obtained outcomes validated the proposal, and the introduced method can also be used to synthesize other array geometries.