Algorithm of Helmet Wearing Detection Based on AT-YOLO Deep Mode

: The existing safety helmet detection methods are mainly based on one-stage object detection algorithms with high detection speed to reach the real-time detection requirements, but they can’t accurately detect small objects and objects with obstructions. Therefore, we propose a helmet detection algorithm based on the attention mechanism (AT-YOLO). First of all, a channel attention module is added to the YOLOv3 backbone network, which can adaptively calibrate the channel features of the direction to improve the feature utilization,and a spatial attention module is added to the neck of the YOLOv3 network to capture the correlation between any positions in the feature map so that to increase the receptive field of the network. Secondly, we use DIoU (Distance Intersection over Union) bounding box regression loss function, it not only improving the measurement of bounding box regression loss but also increases the normalized distance loss between the prediction boxes and the target boxes, which makes the network more accurate in detecting small objects and faster in convergence. Finally, we explore the training strategy of the network model, which improves network performance without increasing the inference cost. Experiments show that the mAP of the proposed method reaches 96.5%, and the detection speed can reach 27 fps. Compared with other existing methods, it has better performance in detection accuracy and speed.

Introduction
In recent years, intelligent surveillance has played an increasingly important role in our daily life. As a hotspot of computer vision, object detection provides many ideas for intelligent surveillance. Object detection methods mainly use traditional features in early research. Zhu et al. [1] used Histograms of Oriented Gradients (HoG) to extract image features, combined with the cascade-of-rejectors to accelerate the calculation speed and realize pedestrian detection. Zuo et al. [2] used Haar-wavelets transform to model local texture attributes, successfully compensated for the extra cost of two-dimensional texture features, and realized face detection. However, traditional object detection algorithms have many drawbacks, the feature extraction algorithms have poor generalization ability and low robustness for complex scene images, and the region selection methods using sliding window lacks specific calculation, so its time complexity is too high to solve practical problems.
The development of massively parallel computing technology provides a technical guarantee for deep learning, and deep learning also provides more effective solutions for information hiding [3][4][5][6][7], image classification [8][9][10], image retrieval [11,12], object detection [13], image inpainting [14], and many other fields. So far, the most state-of-the-art detection algorithms are based on deep learning. R-CNN [15] firstly uses the deep learning model to extract image features and generate region proposals with a sliding window, but it has many repeated calculations, and the amount of calculation is too high. Fast R-CNN [16] integrates the classification and regression of bounding boxes into a network to reduce repeated calculations while using the SPP module to generate fixed-size output. Faster R-CNN [17] inputs the features extracted from images into RPN (Region Proposal Networks). The RPN can input feature maps of any size, output the coordinate information and confidence of the object candidate boxes, and then classify the object candidate boxes. Because region selection and classification should be performed step by step, Faster R-CNN belongs to the two-stage object detection model. With the continuous development of deep learning, two-stage detection methods such as Faster R-CNN are affected by factors such as the complexity of the basic network, the number of candidate boxes, the complexity of classification, and regression sub-networks, the amount of calculation continue to increase. YOLO [18] discards the candidate boxes extraction step in the algorithm and directly implements feature extraction, candidate boxes classification, and regression in an end-to-end deep convolutional network. The detection algorithm similar to YOLO is a one-stage detection algorithm, which further simplifies the implementation steps of object detection, makes the network structure simpler, and the detection speed is faster than that of the two-stage network.
As an essential practical application in object detection, safety helmet wearing detection is closely related to our production and life. Many scholars have done a lot of research about it. Mneymneh et al. [19] extracted the features of the worker and the helmet in the image and set up a cascade according to the feature points to judge whether the worker is wearing a helmet. Li et al. [20] used head positioning, color space transformation, and color feature recognition to achieve the detection of wearing helmets based on the detection results of pedestrians. Wu et al. [21] used the improved YOLO-Densebackbone for helmet detection to improve feature resolution. Long et al. [22] used SSD (Single Shot multi-box Detector) [23] to detect wearing helmets. However, the current detection methods for wearing helmets have disadvantages, such as low detection accuracy for small objects and poor generalization ability for multi-scene detection. Chen et al. [24] introduced the K-means++ clustering algorithm to cluster the size of the helmet in the image and then used the improved Faster-RCNN algorithm to detect the helmet wearing.
However, among the current helmet-wearing detection algorithms, the detector based on onestage has a faster detection speed, but its detection accuracy for small targets and dense targets is low, and the generalization ability of multiple scenes is poor. The two-stage detector has the disadvantages of large calculation amount and slow detection speed, which is difficult to meet the real-time requirements of helmet detection.
In order to solve the above problems, we propose an AT-YOLO network model based on the attention mechanism for helmet wearing detection. In this paper, we model the correlation between the channel and the spatial dimension in the feature to enhance the ability of feature representation. At the same time, this paper optimizes the dataset, loss function, and training strategy to improve the network detection performance in all directions while maintaining a high detection speed. The main contributions of this paper include: (1) Construct a helmet dataset with more balanced categories and richer scenarios. Part of the data comes from Safety-Helmet-Wearing-Dataset, and on this basis, site images are collected through web crawler, video capture, and other ways to expand the dataset so as to make the dataset scene richer and categories more balanced. (2) Propose the AT-YOLO object detection algorithm for helmet wearing detection. The mutual dependence of features is simulated in the dimensions of space and channel, respectively, so that the network can obtain better detection results on small objects and occluded images. (3) The DIoU bounding box regression loss function is used. Combining the IoU between the prediction box and the ground truth box and the center point distance as the bounding box regression loss function, the loss function measurement is improved. While improving the accuracy of the network's detection of small objects, it also accelerates the speed of the network convergence. (4) Different training strategies are used to improve network performance in the network training stage. This paper uses several different training strategies to enhance the network's performance without increasing the cost of network inference and providing a valuable reference for other image research.
The rest of this paper is organized as follows. Section 2 reviews the related research on attention mechanisms and target detection algorithm. Section 3 introduces the method proposed in this paper. Section 4 introduces the evaluation experiment of the method in this paper. Section 5 summarizes the work of this paper.

You Only Look Once
YOLO integrates the candidate region extraction and classification and regression tasks of object detection into an end-to-end deep convolutional network. That is, the input image is inferred once, and the positions and categories of all objects and the corresponding confidence probabilities can be obtained. The backbone network of the YOLO network is similar to GoogLeNet [25]. The inception structure is removed to make the backbone network simpler. Input the image into the YOLO model to obtain a feature map with a size of 7 × 7, which divides divide the image into 7 × 7 regions. Each area has the confidence of the target, the position of the bounding boxes, and the category information. The YOLO network is simple, the detection speed is fast, and the background false detection rate is low, but the detection accuracy is not as good as the R-CNN detection method, and YOLO is not accurate enough in object positioning.
YOLOv2 [26] uses a new backbone network called Darknet-19, which is the same as the VGG16 [27] model design principle. The network mainly adopts the 3 × 3 convolution and the 2 × 2 maximum pooling layer. After passing through the maximum pooling layer, the height and width of the feature map are halved, and the number of channels of the feature map is doubled. YOLOv2 still has the advantage of fast speed. However, its backbone network is not deep enough, it is difficult to recognize more abstract image semantic features, and the bounding box predicted by each grid cell is too less, which is not effective in predicting targets with large-scale changes.
YOLOv3 [28] draws on the idea of Resnet [29], introduces the residual structure, and establishes a deeper Darknet-53. And compared to YOLOv2, the downsampling method of the pooling layer is canceled, but the feature map is downsampled by adjusting the step size of the convolutional layer to obtain more fine-grained features. YOLOv3 uses multiple-scale fusion methods to make predictions. Similar to FPN, YOLOv3 integrates feature maps of three scales and simultaneously detects multiple-scale feature maps. The small size of the feature map is used to detect large-size objects, and the large size of the feature map is used to detect small-size objects. Compared with YOLOv2, YOLOv3 uses multiple scales to predict at the same time to make the bounding box more, cover a richer object size, closer to the real object size, and also strengthen the detection effect of small objects.
However, YOLOv3 still has low utilization of features and poor detection performance for small objects and object-intensive images. In the application of actual helmet wearing detection, small objects and dense objects are very common. Therefore, we need a network with better performance to make the helmet-wearing detection system more robust.

Attention Mechanism
Both language and vision problems contain information that is closely related to the research task and some irrelevant information. The attention mechanism can help the algorithm focus on analyzing some vital information while ignoring irrelevant information. In recent years, the attention mechanism has been used in various tasks and achieved good results. Bahdanau et al. [30] are the first to use the attention mechanism to solve machine translation problems. Wang et al. [31] draw on the idea of non-local mean filtering to model the correlation of arbitrary non-local features. Non-local operations would not change the size of features and can be embedded in any network. Hu et al. [32] achieved first place in the 2017 ILSVRC classification task through the attention mechanism and adaptive calibration of the channel direction's characteristic response. DAnet [33] can be considered as a specialized example of Nolocal-Net [31], which solves the problem of scene segmentation by capturing rich contextual relevance through a self-attention mechanism.
Different from previous work, this paper adds attention mechanisms to the task of helmet detection. Based on YOLOv3, we have added a spatial attention module and a channel attention module to capture the correlation between features, enrich context information, and improve the detection ability of feature representation in object detection.

AT-YOLO Network Construction
This paper designs the AT-YOLO deep model to optimize the feature expression ability and feature learning ability of the network. The backbone and neck of the YOLOv3 have been added channel attention modules and a spatial attention module, respectively. The network adaptively captures the correlation between the channel and space of the feature and models the global context relationship, improving the feature representation ability of the object detection algorithm. The AT-YOLO helmet detection framework is shown in Fig. 1.
Channel attention block. The Darknet-53 network used residuals linking to merge the features of different layers to alleviate the problem of gradient disappearance. However, the network does not make good use of the dependence between the channel of the feature map. Inspired by [32], we insert CA-block (Channel Attention block) into each residual block in Darknet. Recalibrate the dependencies between channels by learning global features, selectively enhancing high-contribution information, suppressing low-contribution information, and improving the feature expression ability and feature utilization of the network. The CA-block we designed is shown in Fig. 2.

Figure 2: Channel attention block
Suppose the input feature map F in , first obtain the feature map F R ∈ R C×H×W through two DarkNet convolutions.
The feature map F in obtains a global feature map F g ∈ R C×1×1 after the global average pooling operation. Subsequently, two convolution kernels with a size of 1 × 1 are performed to obtain the information of the global receptive field, and simultaneously, the feature dependence in the channel is learned. The feature vector F C ∈ R 1×1×C was obtained at this time. Then activate the feature vector F C to get F se . F se is used to describe the learned channel weight. Finally, the learned channel weights are used to weight the feature map F R , and then the feature maps of the residual link are combined to obtain the final output.
Position attention block: In object detection, there are many detection objects dense or object occlusion, which leads to object false detection. Therefore, to obtain richer contextual information and enhance the expressive ability of feature maps, this paper adds a spatial attention module PA-block (Position Attention block), as shown in Fig. 3.

Figure 3: Position attention block
Suppose the input local feature map is F in ∈ R C×H×W . First, two new feature maps F 1 and F 2 are obtained through a convolutional layer {F 1 , F 2 } ∈ R C×H×W . Then reshape them to obtain feature maps with the scale of R C×N where N = H × W . Then perform matrix multiplication on the F 1 and F 2 transpose, and the spatial attention feature mapF s ∈ R N×N is obtained after normalization processing with softmax activation function: where F s ji represents the degree of influence of the i-th position on the j-th position. The greater the connection between the two locations, the more similar their feature representations.
Simultaneously, we convolve the feature map F in to obtain a feature map F 3 ∈ R C×H×W and then reshape it to obtain a feature map of size R C×N . Then, matrix multiplication is performed on the deformations of F 3 and F s , and the result is reshaped to obtain a feature map of size R C×H×W . Finally, multiply it with the scale parameter α and perform element addition with the feature map F in to obtain the final output F out ∈ R C×H×W , which is calculated as follows: where the scale parameter α is initialized to 0, and it is learned to give greater weight to different location features during the training process. It can be seen from the above formula that F out each point is the weighted sum of the original feature F in and features across all locations. Therefore, the F out has a global receptive field and selectively aggregates the context information in the location attention feature map.

DIoU Bounding Box Regression Loss Function
In the YOLOv3, the MSE (mean square error) loss function is used for bounding box regression, but the mean square error has problems in evaluating the prediction results. That is, the mean square error loss function is quite sensitive to the object scale. In order to solve the above problems, we modified the bounding box regression loss function part, using DIoU [34] loss function.
Besides, using the IoU of the real boxes and the predicted boxes can better reflect the detection effect of the predicted boxes. When the prediction boxes are completely contained in the real boxes, the relative position between the prediction boxes and the real boxes is ambiguous, and the gradient descent is easy to occur slowly.
Therefore, when training the model, this paper uses DIoU as the bounding box regression loss function, which is defined as follows: IoU means the intersection ratio of the predicted boxes and the real boxes, b, b gt respectively represent the center points of the predicted boxes and the real boxes, ρ represents the calculation of the Euclidean distance between the two central points, and c represents the diagonal distance of the minimum closure distance area that can contain both the predicted box and the real box, as shown in Fig. 4.
The DIoU loss function increases the normalized distance between the predicted box and the real box based on IoU. When the predicted box is completely contained in the real box, it can still provide the moving direction for the bounding box to make the network convergence faster and more accurate.

Training Strategy
Without changing the network structure, different training strategies get different results. Thanks to Zhang et al. [35], we have explored different training strategies to enhance the performance of the model without adding additional calculations to model inference.
Label smoothing. In order to prevent the network from over-confidence about the category probability prediction and causing overfitting, we use a label smoothing strategy. In the classification model, we use a one-hot form of encoding to predict the probability that the sample belongs to each label. However, there would be such a label encoding that the model may be too confident in the prediction to ignore the small number of sample labels in the actual process. Therefore, this paper adopts the label smoothing strategy for learning. The formula for the real probability of label smoothing is: where y is the label probability generated by one-hot encoding, and K is the number of categories.
There are two categories of hat and person, so K = 2. We set ε = 0.01, the probability value p = 0.995 after label smoothing when the true label y = 1 of a certain sample. Label smoothing can avoid the absolutization of the predicted value of the network, which can improve the overall effect.
Multi-scale input. Due to the different resolutions in the captured natural images and the memory limitations of different operating devices, we need to improve the adaptability of the network to different scale image input, so we use multi-scale input for model training.
During training, each iteration of the network randomly selects pictures of different sizes for training. Since the network uses a total of 32 times downsampling, the size of our image input is a multiple of 32, for example: {320, 352,. . ., 608}. Therefore, the smallest input picture size is 320 × 320, and the largest is 608 × 608. There are a total of ten different sizes of the input pictures.
Using this method can force the network to learn predictions of various input scales while preventing the network from overfitting. The speed of the network is faster under a smaller input size, and the smaller the memory of the running device is. In the case of high-resolution image input, it has higher accuracy.

Datasets
Dataset acquisition. Part of the dataset in this paper comes from Safety-Helmet-Wearing-Dataset, which contains 7581 pictures and the label information about whether a person wears a helmet or not. The label contains two detection categories, person and hat, where the hat represents the head of the person wearing a safety helmet, the person represents the person's head without a safety helmet. This dataset contains a large number of crowded scene pictures, but there is a phenomenon of unbalanced categories, as shown in Fig. 5. The number of hat and person instances is 9031 and 111514, respectively, and the category ratio is close to 1:11. Simultaneously, this dataset has fewer detection scenarios, which easily causes the network to overfit, low generalization ability, and poor multi-scene detection ability. We used web crawlers to grab pictures of wearing helmets on the Internet. At the same time, we selected video clips containing hard hat scenes and cropped them into images to expand the dataset, alleviating the imbalance of dataset categories and enhancing the generalization ability of the model. Particularly, we have selected pictures of wearing ordinary hats so that the network can better distinguish the difference between safety helmets and ordinary hats. Data cleaning. A large number of similar pictures in the dataset would affect the model performance. Therefore, we use the deep learning model DenseNet [36], which is pre-trained on ImageNet [37] to extract the features of each image in the dataset and then calculate the Euclidean distance of the image features of each two pictures in the dataset to determine the similarity of the two images. The manually annotate the collected image, following the VOC labeling format, marking the coordinates of the upper left and lower right vertices of the bounding box of the object and the bounding box category, and stored in the XML file. Finally, we obtained 13620 pictures, the number of hat and person instances is 27236 and 118357, respectively, and the category ratio is close to 1:4 (see Fig. 5), which greatly alleviates the category imbalance of the dataset.

Data augmentation.
To avoid possible over-fitting of the network, we use six methods for data augment: random horizontal flipping of images, image cropping, image filling, color dithering, brightness augment, and mixup [38].

Implementation Details
Due to the uneven distribution of the dataset categories, we use random sampling to randomly select 1000 images on the dataset as the test set, 500 images as the validation set, and 12260 images as the training set.
All experiments are completed in Intel (R) Core (TM) i7-7800X CPU@3.50 GHz, 64.00 GB RAM and Nvidia GeForce GTX 1080Ti GPU, the Tensorflow framework is adopted.
SGD is used as training optimizer, the initial learning rate is set to 0.0001, and it is changed every iteration. The momentum coefficient is 0.9, the L2 weight decay coefficient is 0.0005, the batch normalized decay coefficient is 0.99, and the batch size is set to 4. Iterate for 400 epochs and save the model weight with the least loss value of the validation set.

Ablation Experiments for Training Strategy
To compare the impact of different training strategies on the network, we explored the impact of label smoothing, multi-scale input, and data augmentation on network detection performance, as shown in Tab. 1, where Baseline represents the baseline model of YOLOV3, LS represents label smoothing, Multi-Input represents random multi-scale input, and DA represents data augmentation. Compared with YOLOv3, the label smoothing increases mAP by 3.15%, which proved that the label smoothing strategy could effectively prevent network overfitting. The multi-scale increases mAP by 1.62%, which can make the network adapt to more input size images, thus adapting to the memory limitations of different code running devices and preventing network overfitting. The data augmentation increases mAP by 1.97%, the color dithering helps the network recognize different color scenes, and the brightness augmentation helps the network recognize images in dark or bright environments.

Ablation Experiments for Attention Module
In order to explore the influence of the attention mechanism on the network model, we set up the experiment in Tab. 2. We insert CA-block into darknet-53 to make the network focus on channel features with a more significant contribution. After the last module of the backbone network, the spatial attention module PA-block is used to obtain richer context information. To verify the improvement of the model performance by this design, the network designed in this paper is compared with the object detection network YOLOv3 when the input image size is 608 × 608, and the training strategy is the same. The detection accuracy improvement effect is shown in Tab. 2. It can be seen from Tab. 2 that compared with YOLOv3, after adding CA-block, the accuracy is increased by 1.72%, reaching an accuracy of 95.62%, and the detection speed is 27.4 FPS. After adding PA-block, the accuracy is increased by 0.68%, reaching an accuracy of 96.30%, while the detection speed is maintained at 25.7 FPS. We can find that although the computational cost increases after the dual attention module are added, the frame rate of the general video does not exceed 25 FPS, which can meet the requirements of real-time video detection. Moreover, we also use the DIoU bounding box loss function to train the model, which improves the convergence speed of the network, makes the bounding box prediction more accurate, improves the detection accuracy of the network, and makes the model testing accuracy reach 96.50%.
Considering, our detection model is used for video surveillance of construction sites. Since the location of the surveillance camera is generally far away from the surveillance scene, and the pixel size of the worker in the video is small, we designed a comparative experiment on the detection accuracy of the model for small objects (pixel area < 32 2 ).
It can be seen from Tab. 3 that the proposed network can also achieve better detection accuracy when detecting small objects. That's because small objects can be inferred through the contextual information aggregated by the attention mechanism, and feature representation can be highlighted, thereby improving the detection results. We also compared the speed and accuracy of AT-YOLO with an input size of 320 × 320 (AT-YOLO 320), AT-YOLO with an input size of 512 × 512 (AT-YOLO 512), AT-YOLO with an input size of 608 × 608 (AT-YOLO 608), and YOLOv3, then drawn a speed-accuracy curve. As shown in Fig. 6, although AT-YOLO increases the attention mechanism, it is slower than YOLOv3 under the same input size, but under the premise of the same inference speed, AT-YOLO has higher accuracy than YOLOv3. AT-YOLO is more efficient and accurate. with dense and more occluded objects, and (e), (f) with occluded objects, some occluded objects can't be detected by YOLOv3 but can be detected by AT-YOLO. Since AT-YOLO extracts the surrounding information of the object through the spatial attention module, making it easier to infer the occluded object. For (c), (d) with low resolution. AT-YOLO analyzes the interdependence between the channels through the channel attention mechanism, extracts more effective features, and detects low-pixel images excellent. For (g), (h) containing ordinary hats, it is easier for the model to distinguish ordinary hats from safety helmets after adding ordinary hats into the dataset.

Performance Comparison of Different Models
We tested our dataset on different networks, as shown in Tab. 4. The mAP of our method is 9.34% higher than that of YOLOv3 without increase too much inference cost. Compared with the two-stage method (Faster-RCNN with FPN [39] ), our method is better in accuracy and efficiency.

Conclusion
This paper proposes an AT-YOLO helmet wearing detection model. By introducing the attention mechanism into the YOLOv3, the modeling ability of the network on the dependencies between different positions in the image is enhanced to extract more accurate features and effectively improve the model's feature representation ability. At the same time, we combine the optimized training strategy to improve the network performance without increasing the inference cost. To verify the performance of these proposed methods, we produced a dataset and conducted some evaluation tests on it. The experimental results show that the methods proposed in this paper effectively improve the performance of the AT-YOLO network, thereby providing an excellent solution for the helmet-wearing detection system in actual scenarios.
In the next work, we will combine the correlation of time series to optimize the video's object detection to improve the accuracy and speed of detection and make it more suitable for the safety helmet wearing detection system.
Plan of Hunan Province under Grant 2019SK2022, author Y. T, http://kjt.hunan.gov.cn/; in part by the Graduate Science and Technology Innovation Fund Project of Central South University of Forestry and Technology under Grant CX2020107, author Q. Z, https://jwc.csuft.edu.cn/.

Conflicts of Interest:
The authors declare that they have no conflicts of interest to report regarding the present study.


Minimization of information leakage in continuous-variable quantum key distribution

A communication protocol based on a Gaussian modulation of squeezed states in a single quadrature and measured via homodyne detection can completely eliminate information leakage to an eavesdropper in a pure-loss channel. However, the asymmetry of the protocol with respect to the quadratures of light presents security issues and the eavesdropper's information is not necessarily minimized for general asymmetric attacks. Here, we perform asymptotic security analysis of the asymmetric protocol against general asymmetric collective attacks and bound the eavesdropper's information via the Heisenberg uncertainty principle. The bound is not tight and therefore, we symmetrize the protocol in a heralding way, discarding the issues of asymmetry altogether. Our proposed heralding protocol asymptotically eliminates information leakage in a pure-loss channel and minimizes leakage in a noisy channel.

I. INTRODUCTION
Quantum key distribution (QKD) [1,2] is the task of distributing a secret random key between two distant parties. Its provable security relies on quantum physics and it is therefore a promising solution to the vulnerability of current classical cryptosystems. The first QKD protocols were based on discrete-variables (DV), such as BB84 [3], where the fundamental component is a qubit. Recently, QKD has been extended to continuous-variable (CV) [4,5] systems, utilizing the infinite-dimensional space of modes of light, with the benefit of using simpler experimental set-ups and off-the-shelf optical communication devices such as homodyne detectors.
There is a zoo of CV QKD protocols [6][7][8][9][10][11][12]. For instance, either coherent states [6,10] or squeezed states [11,12] may be distributed between the trusted parties, and the ensemble may be symmetric or asymmetric with respect to the quadratures of light. See Ref. [2] for a review. A key difference between CV QKD and DV QKD is that in DV QKD lost photons are discarded, but in CV QKD all states are retained but are noisier.
A CV analogue of BB84 is the protocol based on Gaussian modulated squeezed states with switching (i.e., randomly choosing to squeeze and modulate either the x or p quadrature) and measured via homodyne detection [11], however, even in a pure-loss channel, information is inevitably leaked to an eavesdropper. During the protocol, correlations exist between all three parties, and the trusted parties, Alice and Bob, must suppress their correlations with the eavesdropper, known as Eve, to ensure the final key is secret. This can be done via error correction, privacy amplification, or for an entanglement protocol, via entanglement distillation or purification. Recently, a CV QKD protocol has been devised which takes a different approach. By designing the alphabet of input states in a certain way, information leakage to Eve is completely and deterministically eliminated in a pure-loss channel and minimized in a symmetric noisy channel [13]. Information is encoded in a single quadrature via a Gaussian modulation * matthew.winnel@uqconnect.edu.au of squeezed states, which are squeezed in the modulation direction such that the overall variance of the ensemble is shot noise. The motivation for minimizing information leakage is that this will simplify classical post-processing and ultimately speed up the secret key rate. Another motivation is that zero information leakage in a pure-loss channel is analogous to discarded photons in DV QKD.
It is vital that QKD protocols can be proved secure against the most general eavesdropping attacks allowed by the laws of quantum physics. Indeed, currently composable security for finite key lengths has been proved against general attacks for only several CV QKD protocols, such as the no-switching protocol based on Gaussian modulated coherent states and heterodyne detection [14], and the Gaussian modulated squeezed state protocol with switching and homodyne detection [15]. Recently, a discrete modulation protocol has been proved secure including finite-size effects [16]. In Ref. [13], the authors considered a symmetric Gaussian attack which is not asymptotically optimal since the protocol is asymmetric and the channel parameters in the unmodulated quadrature are unknown.
In this paper we deal with the asymmetry of the asymmetric minimum-leakage protocol from Ref. [13]. We first extend their security analysis to include general asymmetric Gaussian attacks. We show that Eve's information is not necessarily minimized for asymmetric attacks. Further, we introduce a new protocol which is strictly symmetric with respect to the quadratures of light. This forces Eve to implement symmetric attacks thereby avoiding the issues of asymmetry. Conditioned on a homodyne measurement at Alice's station, our protocol heralds squeezed states modulated in a single quadrature. We show that for this new protocol information leakage is asymptotically eliminated for pure-loss channels, and minimized for noisy channels.
The outline of this paper is as follows. In Sec. II, we revise state of the art security for CV QKD, and we introduce the asymptotic key rate formula and the Holevo bound. In Sec. III, we recall the asymmetric protocol from Ref. [13] and extend their analysis to include general asymmetric attacks. In Sec. IV, we introduce a symmetric heralding protocol. In Sec. V, we discuss our results and conclude.

II. SECURITY OF CV QKD PROTOCOLS
A. State-of-the-art security proofs The goal of security analysis is to provide a useful upper bound on Eve's information using correlations between Alice and Bob, leading to an expression for the secret key rate. Unfortunately, proving security is problematic in CV QKD because the Hilbert space is infinite dimensional, the measurement operators are unbounded, and channel parameters must be estimated of noisy quantum channels. Compare this with DV QKD where the Hilbert space is finite dimensional and loss can be dealt with by simply discarding the state. It is known that Gaussian collective attacks are asymptotically optimal for fully Gaussian CV QKD protocols [17], i.e. in the asymptotic case of infinitely long keys. To go beyond asymptotic analysis, only a few Gaussian CV QKD protocols are known to be composably secure against general attacks and including finite-size effects, for instance, the no-switching protocol based on Gaussian modulated coherent states in the x and p quadratures and heterodyne detection [14,18], and the switching protocol based on squeezed states with a Gaussian modulation in the x or p quadratures and homodyne detection such that the overall state prepared by Alice is a Gaussian thermal state [15,19].
We restrict our attention here to the asymptotic regime though we caution that finite-size effects are expected to be significant. The original asymmetric protocol from Ref. [13] with squeezed states is analyzed against collective attacks in the asymptotic limit in Sec. III. Our symmetric heralding protocol, discussed in Sec. IV, is Gaussian and symmetric, however, current composable security proofs including finite-size effects still fail due to Alice's homodyne detection.

B. Asymptotic secret key rate formula and Eve's maximal information
A lower bound on the asymptotic secret key rate in the case of reverse reconciliation for collective attacks is given by the Devetak-Winter rate [20] where I AB is the classical (Shannon) mutual information between Alice and Bob, χ EB is the Holevo quantity, the maximal quantum mutual information between Eve and Bob (the reference side of the information reconciliation), and β is the reconciliation efficiency.
The upper bound on the information extractable by Eve is given by the Holevo quantity [21] where S(ρ E ) is the von Neumann entropy of Eve's state, and S(ρ E|b ) is the von Neumann entropy of Eve's state conditioned on Bob's measurement. The Holevo information is obtained by allowing Eve access to the purification of the state shared between Alice and Bob ρ AB . The global state ρ ABE is pure and we can use the self-duality property of the von Neumann entropy to write S(ρ E ) = S(ρ AB ) and S(ρ E|b ) = S(ρ A|b ), where S(ρ A|b ) is Alice's mode conditioned on a measurement on Bob's mode.
For Gaussian states, χ EB may be calculated from the symplectic eigenvalues of the covariance matrices of ρ AB and ρ A|b in the equivalent entanglement-based (EB) version of a prepare-and-measure CV QKD protocol. Thus, Eve's information is given by i.e. the Holevo quantity is given in terms of von Neumann entropies which can be calculated using the symplectic eigenvalues v i of the covariance matrix of the state, i.e. Γ AB and Γ A|b , via the relation where N is the number of modes.

III. ASYMMETRIC PROTOCOL
The asymmetric minimum-leakage protocol from Ref. [13] is shown in Fig. 1. It is based on squeezed states modulated in a single direction with an overall Gaussian modulation and squeezing chosen so that the Holevo information χ EB is eliminated or minimized while the key rate is non-zero. Let us assume that the modulated quadrature is chosen to be the amplitude quadrature x. In a pure-loss channel, the requirement for complete elimination of χ EB is for the ensemble of squeezed states to have an overall amplitude quadrature noise variance of vacuum.
In the prepare-and-measure (PM) version of the protocol (the version Alice and Bob implement in practice), Alice prepares a Gaussian modulation of squeezed states of light to send to Bob. The PM scheme of the protocol is given in Fig. 1(a). Alice prepares x squeezed states with amplitude quadrature variance V sqz and applies modulation in the x quadrature, displacing each squeezed state according to a random Gaussian variable with variance V sig . The states are then sent to Bob through an asymmetric noisy channel with transmittance T x , T p and excess noise ξ x , ξ p in the x and p quadratures respectively. Bob performs homodyne measurements of the modulated x quadrature, but sometimes also measures the unmodulated p quadrature for estimating the properties of the channel in the p quadrature. Alice and Bob extract a secret key from the x quadrature data using a reversereconciliation procedure.
Security is analyzed using the equivalent EB version of the protocol, shown in Fig. 1b) which goes as follows. Alice prepares a two-mode squeezed vacuum state of variance µ, keeps one of the modes, and squeezes the x quadrature of the second mode with squeezing parameter r before sending it to Bob. Alice performs homodyne measurements in the x quadrature in order to project Bob's mode onto an ensemble of squeezed states with a Gaussian modulation. For instance, if Alice homodynes x then effectively she has sent a Gaussian modulation of x squeezed states in the x direction to Bob, equivalent to the PM version. When the variance of the overall ensemble is shot noise, information leakage is zero in a pure-loss channel.
As shown in Ref. [13], the PM condition which completely decouples the eavesdropper in a pure-loss channel is V sig + V sqz = 1, an overall x quadrature noise variance of vacuum. One can arrive at this condition by calculating Eve's maximal information via the covariance matrix shared between Alice and Bob in the EB version, and we refer you to Appendix A 1 and Ref. [13] for details. A similar relation minimizes the Holevo information under the assumption of a symmetric noisy channel but a little more squeezing is required because of the noise. By symmetric channel we mean T x = T p and ξ x = ξ p , which is a restricted eavesdropping attack since the input state to the channel is asymmetric. The dashed curve in Fig. 2 shows that Eve's information is minimized in a symmetric noisy channel for an appropriate choice of V sqz given that V sig = 0.5. In the next section we go beyond symmetric attacks.

A. General asymmetric attacks
In this section, we consider channels where the noise and loss may be asymmetric in each of the quadratures. Recall that during the protocol, Alice and Bob perform parameter estimation to derive the covariance matrix shared between them. Usually both quadratures are modulated allowing the full covariance matrix to be estimated, however, although only one quadrature is modulated, Bob can still estimate the variance of the unmodulated quadrature and use physicality of the quantum state to bound Eve's information, i.e., via the Heisenberg uncertainty principle [8].

Bounding Eve using the Heisenberg Uncertainty Principle
We use the EB version shown in Fig. 1 b) and bound Eve using the Heisenberg uncertainty principle. The EB parameters are related to the PM parameters such that where µ is the EPR variance and V = e −2r is the strength of the squeezing on the outgoing mode with squeezing parameter r. Equivalently, solving for µ and r, we have The covariance matrix shared between Alice and Bob after Eve's asymmetric attack is given by (see Appendix A 2 for details): where v B p and c p are unknown since the p quadrature is unmodulated. Fixing T x and ξ x , then given Bob's p quadrature variance v B p measured during an experiment (which he only does sometimes since he is mostly performing a homodyne measurement of the x quadrature), we bound the unknown correlation parameter c p by the physical requirement of the state given by Γ AB + iΩ ≥ 0, where Ω is the symplectic form Ω = n i=1 ω, ω = 0 1 −1 0 . To calculate a general upper bound on Eve's information we simulate an experiment and give Bob's variance v B p the value it would have if the channel were symmetric (i.e., in our simulated experiment v B p is calculated using T p = T x and ξ p = ξ x ), and then we maximize Eve's information χ EB by going over all physical covariance matrices. This is plotted in Fig. 2 (solid). Also plotted is for a symmetric channel (dashed), which has been done in [13]. Note that Alice prepares a Gaussian modulation of coherent states for no squeezing V sqz = 1. For coherent states, bounding Eve's information in this way is not very pessimistic, as can be seen in the plot since the gap between symmetric and upper bound is not significant. But as squeezing is increased, the gap between the symmetric channel and the general upper bound becomes very significant. To provide tighter bounds, it is better to measure all terms of the covariance matrix, thus, we need to do some estimation of the unmodulated quadrature.

Estimating the unmodulated quadrature
One must perform some estimation of the unmodulated quadrature in an experiment to obtain the full covariance matrix but it is difficult to estimate the unmodulated quadrature since it is antisqueezed. For illustration purposes, we assume that some estimation of the p quadrature is performed and that the excess noise is equal in both quadratures, ξ p = ξ x , and we again bound Eve's information using physicality of the covariance matrix and plot this in Fig. 2 (dotted). We do not have to simulate Bob's variance in this case because there is only one unknown parameter, T p . Compared to the general upper bound, assuming equal excess noise in both quadratures does much better. This shows that it is important to estimate the noise in the unmodulated quadrature, however, equal excess noise in both quadratures is not a realistic assumption because in an experiment the noise associated with the unmodulated quadrature is expected to be worse than the modulated quadrature. Therefore, one can expect Eve's information to lie between the symmetric channel and the upper bound. In summary, Eve's information is not necessarily minimized when considering stronger attacks beyond symmetric ones (and note that we have not yet considered finite-size effects). This motivates an exploration of symmetric protocols with the goal of minimizing Eve's information.

IV. HERALDING PROTOCOL
In this section, we symmetrize the asymmetric protocol of Ref. [13] in a heralding way to avoid worrying about asymmetric attacks in the security analysis. We present our heralding protocol in Fig. 3. In the PM version, Alice prepares two modes; two independent ensembles of squeezed states with a Gaussian modulation in a single direction, one mode modulated in the x direction and the other mode in the p direction. Note that here again we use V sqz for the squeezing variance and V sig for the modulation variance. These are combined on a beamsplitter and Alice homodynes one of the outputs (labelled A3) in either x or p at random while the other mode is sent to Bob (labelled B). The protocol heralds squeezed states modulated in a single quadrature, conditioned on Alice's homodyne measurement at A3. Bob independently and randomly homodynes in either x or p and, during classical post-processing, Alice and Bob sift their results, where they only keep the data for which they have used the same quadrature for the measurement.
In the EB version, Alice's ensembles are replaced with EPR states plus auxiliary squeezing on one of the modes. Mode B is squeezed in the x quadrature with squeezing parameter r and mode A3 is squeezed in the p direction by an equal amount. The EB version is equivalent to the PM version if Alice homodynes A1 in x and A2 in p.
Our heralding protocol consists of all Gaussian elements and is symmetric, so we can analyze security against Gaussian collective attacks in the asymptotic limit. For our heralding protocol we do know that in the asymptotic limit Gaussian attacks are optimal, and, unlike the asymmetric protocol, we no longer have to worry about asymmetric attacks since the protocol is symmetric. We still have the issue that we do not know Eve's optimal attack in the finite-size regime. Now we consider the EB version of the Heralding protocol. Before going through the channel, the mode that will go to Bob is symmetric and is given by After the channel, homodyne detection is performed on modes A3 and B independently and randomly in x and p. At the end Alice and Bob sift, and here, without loss of generality, we assume this sifting is in the x quadrature (shifted results in the p quadrature will behave similarly). After the quantum channel and Alice's heralding measurement (which we assume is homodyne A3 in x), the covariance matrix of Bob's mode is The full covariance matrix is given in Appendix B. For zero information leakage in a pure-loss channel we require that the variance of the x quadrature is equal to the shot noise. Solving for the prepare-and-measure parameters defined in Eq. (4), we find the following condition For example, if V sig = 0.3, then this condition implies that there are two solutions which eliminate the Holevo quantity: squeezing V sqz = 0.2821 or antisqueezing V sqz = 1.4179. We are interested in the squeezing solution, not antisqueezing (since mutual information I AB decreases with antisqueezing), and note that V sqz = 0 for infinite squeezing and V sqz = 1 for zero squeezing. Figure 4 shows Eve's information as a function of transmissivity T and squeezing variance V sqz for a) pure-loss and b) excess-noise ξ = 0.001. Eve's information is eliminated in a pure-loss channel when Eq. (9) is satisfied. For the noisy case Fig. 4 b) shows that when Eq. (9) is approximately satisfied and the Holevo information is minimized, the Holevo information actually decreases with loss. In other words, the Holevo information goes to zero as the transmissivity of the channel is decreases. That Eve's information is less the more loss there is may be counter-intuitive (since she steals more but gets less) but makes sense when one remembers that Alice and Bob's mutual information is also less for more loss.
In Fig. 5, we plot Eve's information as a function of signal modulation and squeezing in a pure-loss channel with fixed transmissivity T = 0.5. This shows that as squeezing is increased from V sqz = 1 (coherent states) to V sqz → 0 (infinite squeezing), the modulation variance V sig must also be increased in order to minimize information leakage.
In Fig. 6, we show the secret key rate of the heralding protocol as a function of distance for a thermal-loss channel with excess noise ξ = 0.05. Alice and Bob's mutual information is calculated assuming modes A1, A3 and B are all homodyned in x, and the parameters of the protocol are fixed such that Eq. (9) is satisfied and Eve's information is eliminated or approximately minimized at all distances. Alice and Bob's mutual information is calculated after Alice's heralding measurement of mode A3 in the x quadrature and is given by where V Bx is the variance of Bob's x quadrature, since this is the quadrature we assume he uses for the key, and V Bx|A1homx is the variance of Bob's x quadrature conditioned on Alice's homodyne measurement in x of mode A1. The heralding protocol optimizes for very large squeezing shown as the dashed line in Fig. 6. Infinite squeezing is V sqz →0 and the zero-leakage condition Eq. (9) means that V sig →0.5. Also shown is the key rate for the heralding protocol with 10dB of finite squeezing (dot-dashed). For comparison, we plot the optimized key rate for Gaussian-modulated squeezed state protocol and measured via homodyne detection [11] (red) and Gaussian-modulated coherent state protocol and heterodyne detection [6] (blue). The optimized squeezed state protocol with homodyne detection outperforms our heralding protocol. However, since our heralding protocol likewise is based on squeezed states and homodyne detection it can outperform the coherent state protocol with heterodyne detection.
We note here that rates for the heralding protocol are numerically identical as those for the asymmetric protocol against symmetric thermal-loss channel. Indeed, the rate for the heralding protocol for a pure-loss channel, infinite squeezing, and reconciliation efficiency β=1 reaches half of the fundamental repeaterless PLOB bound, which is given by [23]. This result for the asymmetric protocol was indeed mentioned in [13].

V. DISCUSSION AND CONCLUSION
In this paper, we have investigated CV QKD protocols designed to eliminate information leakage to Eve. We extended security analysis of the original asymmetric minimumleakage protocol by considering general asymmetric channels by demanding physicality of the quantum states. We also introduced a new protocol by symmetrizing in a heralding way.
The main results from this paper are the following: if the unmodulated quadrature can be properly estimated, the asymmetric protocol minimizes Eve's information and is secure against Gaussian attacks in the asymptotic regime. However, using the Heisenberg uncertainty principle for security analysis can give Eve a lot of information. Our heralding protocol minimizes Eve's information in a heralding way despite being symmetric with respect to the quadratures, but with added experimental complexity.
Another way to symmetrize the asymmetric minimumleakage protocol is by switching, i.e., randomly sending xsqueezed or p-squeezed states with an overall modulation of shot noise in each quadrature. However, the overall ensemble prepared by Alice will be non-Gaussian (specifically, it will not be a Gaussian thermal state as in the squeezed state protocol with homodyne detection from Ref. [11]). The non-Gaussianity complicates the security analysis, potentially giving more information to the eavesdropper, departing from our goal of minimum-leakage. We leave the investigation of minimum-leakage switching protocols for future work.
In conclusion, we have introduced a new CV QKD protocol which minimizes the amount of information leaked to an eavesdropper in a noisy channel and eliminates information leakage in a pure-loss channel, meaning that the classical part of the protocol will be computationally less complex, potentially leading to an overall increase in the secret key rate in situations of practical interest. Secret key rate of the heralding protocol as a function of distance for lossy thermal channel. The dashed and dot-dashed lines (black) show the heralding protocol with infinite and 10dB of finite squeezing respectively. The heralding protocol optimizes at very large squeezing, i.e., Vsqz→0, Vsig→0.5. For these plotted rates, the condition Eq. (9) is satisfied at all distances, hence, Eve's information is zero at all distances in a pure-loss channel and small in a noisy channel. Shown for comparison are optimized rates for the original squeezed state protocol with switching and homodyne detection [11] and no-switching coherent state protocol with heterodyne detection [6]. The excess noise is ξ = 0.05 for all protocols, the reconciliation efficiency is β=0.95 and we have considered optical fiber with loss of 0.2dB/km. The repeaterless PLOB bound [23] and is also plotted in the figure (gray). The covariance matrix for Alice's mode conditioned on Bob's homodyne measurement, which is assumed to be in the x quadrature, is Γ A|b = µ− T e −2r (µ 2 −1) T (e −2r µ+ξ)+1−T 0 0 µ . The Holevo quantity can be determined from the symplectic eigenvalues of these two matrices, Γ AB and Γ A|b . If r = ln √ µ then χ EB → 0 in a pure-loss channel.

Asymmetric attack
Now we write down the covariance matrix for general asymmetric channels. After the asymmetric channel, the covariance matrix shared by Alice and Bob is: where r is the squeezing parameter in the EB version. T x and ξ x are, respectively, the channel transmittance and excess noise, estimated by Alice and Bob in the x quadrature. However, since the p quadrature is not modulated, T p and ξ p are unknown.


Multi-Convolution Feature Extraction and Recurrent Neural Network Dependent Model for Short-Term Load Forecasting

Load forecasting is critical for power system operation and market planning. With the increased penetration of renewable energy and the massive consumption of electric energy, improving load forecasting accuracy has become a difficult task. Recently, it was demonstrated that deep learning models perform well for short-term load forecasting (STLF). However, prior research has demonstrated that the hybrid deep learning model outperforms the single model. We propose a hybrid neural network in this article that combines elements of a convolutional neural network (1D-CNN) and a long short memory network (LSTM) in novel ways. Multiple independent 1D-CNNs are used to extract load, calendar, and weather features from the proposed hybrid model, while LSTM is used to learn time patterns. This architecture is referred to as a CNN-LSTM network with multiple heads (MCNN-LSTM). To demonstrate the proposed hybrid deep learning model’s superior performance, the proposed method is applied to Ireland’s load data for single-step and multi-step load forecasting. In comparison to the widely used CNN-LSTM hybrid model, the proposed model improved single-step prediction by 16.73% and 24-step load prediction by 20.33%. Additionally, we use the Maine dataset to verify the proposed model’s generalizability.

I. INTRODUCTION
Electricity is a significant secondary energy source that has a sizable impact on both the national economy and daily life. With large-scale renewable energy connected to the power grid, we can expand the use of renewable resources, which has a number of benefits. However, there are additional complications associated with power distribution and dispatch. Short-term load forecasting (STLF) enables the generation of detailed forecasts for a time period that is critical for supply unit optimization, economy dispatch, and market transactions [1]. A detailed and reliable STLF enables the power system to operate safely and efficiently.
The associate editor coordinating the review of this manuscript and approving it for publication was Seifedine Kadry .
Due to the complexity of the factors affecting STLF, such as historical load curves, weather conditions, and calendar effects, staying on schedule is more difficult than ever. Numerous weather research experiments are currently being conducted to examine the four critical STLF variables: temperature, humidity, wind speed, and air pressures. Mukhopadhyay et al. [2] incorporated temperature and relative humidity into the load forecasting model. Friedrich and Afshari [3] included wind speed in their load forecasting model. Because the weather is influenced by both climate and geography, some of its forecast parameters are non-relational. Correlations between weather parameters and load data must be determined in order to determine which parameters (variables) to measure [4].The calendar effect was used to characterize the cyclical and seasonal variation in electricity consumption [5]. People's production and daily lifestyle changes can affect their energy consumption patterns, resulting in daily or weekly changes in the power load. Numerous calendar-dependent input features were incorporated into the load forecasting model to emphasize the importance of calendar variables via coding techniques.
Historically, a variety of load forecasting algorithms were employed to determine historical loads and load-related characteristics and produced accurate prediction results. Recently, deep learning models based on long short memory networks (LSTMs) and convolutional neural networks (CNNs) have been applied to anticipate wind energy generation, photovoltaic energy generation, and load forecasting. Their predicting abilities have been demonstrated to be superior to those of conventional shallow neural networks and machine learning techniques. The LSTM model accurately captures the time series' pattern information, whereas the CNN model extracts valuable features from the time series without requiring domain expertise. LSTMs are excellent at extracting temporal features, while CNNs are excellent at extracting spatial features [6]. Thus, by integrating the technological advantages of the two models, forecasting accuracy can be increased.
The periodicity of electric load and the effect of weather parameters on electric load are discussed in this paper. As input features, periodic calendar coding and weather parameters (temperature and humidity) are used. Different from other hybrid models of CNN and LSTM [7], [8], we propose a novel combined architecture of CNN and LSTM for STLF. We use three independent 1-dimensional convolution heads to extract historical load features, calendar features and weather features, and fuse the features into LSTM layer. This hybrid model is called multi-head CNN-LSTM(MCNN-LSTM). The independent convolutional head can concentrate on the input data, which can aid in feature extraction, and each independent convolutional head can finely tune the parameters to adapt to the feature extraction. At the same time, it can avoid the slow fitting speed caused by the increase of input feature dimension, thus reducing the training time. Currently, most of the methods used in deep learning focus operate on single-step load forecasting, while power market participants rely on multi-step forecasting. For multi-step load forecasting, we also utilize the proposed multi-input multi-output model. The main contributions of this paper are as follows: 1. We propose a novel MCNN-LSTM hybrid deep learning model for short-term load forecasting.
2. We compare the proposed model's performance to that of other machine learning models and deep learning models, demonstrating the model's validity.
3. We use polar coordinates coding to encode calendar variables, and analyze the impact of calendar coding on load forecasting at different time steps.
The rest of this paper is organized as follows. Section II describes related forecasting techniques. Section III details the basic framework of MCNN-LSTM hybrid deep learning model. Section IV gives the evaluation indicators of STLF model. Section V contains data processing and simulation analysis of experiments. Section VI presents the conclusions.

II. RELATED WORK
STLF methods can be divided into two categories: statistical methods and artificial intelligence (AI) methods.
The main statistical methods are linear regression (LR) [9], multiple linear regression (MLR) [10], auto regressive moving average (ARIMA) [11]. The goal of statistical methods is to measure changes in both current and past load series to establish a mathematical relationship. While time series correlation is considered, it is difficult to attain the requisite forecasting accuracy when working with non-stationary load series due to their inherent restrictions [1]. Because of the introduction of random and intermittent energy sources in the grid, the traditional statistical approach does not work for the complex load curve.
The use of machine learning models that overcame the shortcoming of statistical methods in STLF models were widely implemented for this class of problems, including support vector machine (SVM) [12], [13], light gradient boosting machine (LightGBM) [14] and artificial neural network(ANN) [15]- [17]. SVM's primary methodology is based on the principle of structural risk minimization. While SVM's time series performance is a strength, the accuracy with which the parameters are chosen has an effect on performance.
LightGBM is a promotion model for gradient boosted decision trees (GBDTs) that allows for equal gains in speed and accuracy. ANN makes it much easier to track the relationship between input and output variables due to the nonlinear processing. The ANN combined with a multi-layer perceptron (MLP) has become one of the most widely used load forecasting algorithms. However, the freedom of an ANN is constrained by its initial conditions and increases as the model becomes more complex, resulting in overfitting or underfitting [17].
As a new type of machine learning application, deep learning can solve the training problem of traditional neural network. In comparison to the shallow network, big data and unsupervised learning have garnered considerable interest. Deep learning, particularly convolutional neural networks (CNN), has aided in the study of STLF [18], [19] and long-term memory network(LSTM) [20], [21]. CNN's local connection and global sharing capabilities significantly reduce the model's training parameters and training time, with some organizations utilizing CNN for load prediction. Dong et al. [22] used CNN and K-means clustering to process large-scale power load data. Cai et al. [23] used gated CNN to forecast the short-term power consumption of commercial buildings, which has the best performance compared with gated recurrent neural network and seasonal ARIMAX. LSTM is a recurrent neural network, which is suitable for processing time series. Rahman and Zubair [24] presented a method of applying LSTM with data construction method for hourly electric load forecasting. VOLUME 9, 2021 Bedi and Toshniwal [25] proposed a deep learning framework based on LSTM to forecast electricity demand by learning history data dependencies. Sometimes a single model is not competent for all tasks, and deep learning is no exception. In order to take advantage of different models, some researchers have mixed different deep learning models into a hybrid network model [7], [14]. Alhussein et al. [26] proposed a hybrid CNN-LSTM model using CNN layer to extract features from the input data and LSTM layer for sequence learning to forecast household load. Kim et al. [27] proposed a recurrent inception convolution neural network (RICNN) that combines RNN and 1-dimensional CNN (1-D CNN) to forecast electricity load of three distribution industrial complexes in South Korea. The hybrid CNN and LSTM model has good performance in small-scale load forecasting of commercial buildings and individual household, while the STLF of large-scale regional level needs to be studied.
To capitalize on the CNN and LSTM hybrid model's advantages, many implementations of CNN and LSTM hybrid deep learning models have been developed in a variety of domains. Gao et al. [6] compared the forecasting performance of five different CNN-LSTM structures and proposed that the solar irradiance features be decomposed using complete ensemble empirical mode decomposition adaptive noise (CEEMDAN), and the prediction model of historical solar irradiance features obtained by multiple CNNs be fused into LSTM for optimal performance. Behnam Farsi et al. [28] introduced a parallel deep LSTM-CNN (PLCNet) model that relied on the parallel CNN layer and LSTM as the upper layer to extract the features of load data, and then connected the LSTM layer and the Dense layer to anticipate the final load data. Hourly load forecasting accuracy was 98.23% on the Malaysian dataset and 91.18% on the German dataset.The hybrid model of CNN and LSTM enables the incorporation of additional data features. Li et al. [29] introduced a cascaded CNN-LSTM model for PM2.5 prediction and demonstrated that the multivariate CNN-LSTM model is more accurate than the univariate CNN-LSTM model. Wang et al. [8] proposed the LSTM-Convolutional Network forecasting model for photovoltaic power generation. To begin, an LSTM layer is utilized to extract the temporal features affecting photovoltaic power, followed by a CNN layer to retrieve the spatial data. It is more accurate than a single CNN model or LSTM model. While our proposed model incorporates historical load and load-related characteristics and makes use of the CNN and LSTM hybrid model's multivariate feature processing capabilities, it also addresses the issue of training costs associated with the increased data dimension generated by multivariate features. It contains important references for tasks involving load forecasting with different input features.

A. 1-DIMENSIONAL CONVOLUTION NEURAL NETWORK
Convolutional neural network (CNN) was originally developed for image classification, and have been used in the field of computer vision with great success. Recently, CNNs are not only used in the field of natural language, but they are also extend to solving energy related problems, such as power quality issues [30], wind power prediction [31], solar irradiation prediction [32] and load prediction [23] etc. In comparison to a traditional full connected network, CNN has two main characteristics: local connection and weight sharing. The structure of CNN mainly includes convolution layer, pooling layer and full connection layer. Figure 1(a) shows the basic framework of one-dimensional CNN. Unlike a two-dimensional convolution image processing kernel, which moves in two dimensions from left to right and from top to bottom, the one-dimensional convolution kernel moves in just single dimension (time dimension). Time series can be defined as one-dimensional vector, and one-dimensional CNN is suitable for learning the characteristics of time series. In this study, CNN is applied for feature extraction of load forecasting. The operation example of one-dimensional convolution is shown in Figure 1(b). Convolution is the core concept of CNN, which is used to combine two sets of information into one. Convolution transforms the input data, and the convolution calculation formula is as follows: where w, n, x and b denote the weighting factors in kernels, number of kernels, input series and bias. The symbol ⊗ signifies the convolution operation. f is the activation function. The Rectified Linear Unit (ReLU) can effectively deal with the vanishing gradient problem and make network more trainable. It is adopted as the activation function in this paper. The formula is as follow: Pooling layer is mainly used to reduce the number of parameters. In this paper, maximum pooling is adopted.

B. LONG SHORT-TERM MEMORY NETWORK
In the traditional feedforward neural network, information flows from the input node to the hidden layer and then to the output layer in one direction, so the neural network cannot remember the input of different time series. Recursive network (RNN) can make use of internal state storage to realize memory function. Long short-term memory (LSTM) network is a special RNN, which can learn long-term dependence.
It was proposed by Hochreiter and schmidhub in 1997 [33], and has been improved and promoted by many people in the following works. It uses more complex internal structure cells instead of primitive low cell neurons to solve the problem of gradient explosion and gradient disappearance of primitive RNN. The basic unit of LSTM is shown in Figure 2. There are four important elements in the cell structure of LSTM: input gate, forgetting gate, output gate and cell state. Input gate, forgetting gate and output gate are used to update, maintain and delete the information contained in the status of control unit. The LSTM model updates the cell status and calculates the output as follows: where σ is sigmoid activation function. i t , f t , c t , o t are the output value of the input gate, forget gate, cell state and output gate. h t−1 is the hidden state of former  Figure 3 demonstrates the architecture of MCNN-LSTM.
CNN layer extracts the characteristics of historical load, time and weather factors respectively across three different positions of one-dimensional CNN. Using an independent one-dimensional CNN, which can obtain the characteristics of independent time series, and accurately identify the key features of the specific period (such as weekends and working day information) and weather changes that affect the load. The lower layer of MCNN-LSTM is LSTM. The time feature information related to power demand is extracted by CNN layer, and then processed by LSTM layer. LSTM is responsible for finding hidden time patterns from the extracted features. LSTM has the ability to remember the past events. Data passes through the unit neuron, which has two inputs: past historical data and current data. It can form internal memory that capture the actions through the whole sequence.

D. ARCHITECTURE OF THE PROPOSED MODEL
The proposed MCNN-LSTM hybrid model can adjust the parameters of each layer according to the composition of the network. The convolution layer can adjust the parameters of filter, kernel and stride, and the LSTM layer can adjust the parameters of hidden layer. Tuning the parameters will adjust the model's learning rate and learning effect of the model. Each model will differ. Dropout works to avoid over-fitting on the results. To improve continuity, the parameters of CNN layer and LSTM layer of MCNN-LSTM as shown in Table 1.

E. SIMULATION SOFTWARE AND HARDWARE
The proposed model is implemented in Python using backend libraries such as Keras and TensorFlow. This work is being conducted on a personal computer equipped with an Intel Core i7-4720HQ processor, a NVIDIA GeForce GTX 950m graphics card, and 16.0 GB of random access memory (RAM).

IV. PREDICTION EVALUATION
In order to evaluate the forecasting performance of the model proposed in this paper, the error metrics of five different viewpoints are adopted. They are MAE(Mean Absolute Error),

RMSE(Root Mean Square Error), MAPE(Mean Absolute
Percentage Error), R-squared and error as fellow: where y i is the true load data,ŷ i is the forecasting value, y i is the average of true load data, and N is the number of y i . Maine's electrical power and weather data are provided by ISO New England 3 . The data above is granular to the hour. The datasets from these two regions are divided into three components in this paper: training set, verification set, and testing set. The proportions of these three components are 6:2:2.

B. DATA PREPROCESSING AND FEATURE ENGINEERING
Outlier analysis and missing value filling are examples of data preprocessing. Due to the continuity of load data, the abnormal load value included in the data calculation and analysis process will have a negative impact on the load forecast result. A box plot is a graphical representation of digital data by its quartiles, it is a straightforward and effective method for visualizing outliers. Using the upper and lower tentacles as the distribution's boundaries, any data points that are higher or lower than the upper tentacles can be considered outliers.
In terms of missing value filling, our data has sparse missing values, which we fill using interpolation. The purpose of this study is to forecast future load at the large region level using historical load data, meteorological data, and calendar variables. The primary exogenous factors affecting STLF problems are weather conditions. Seasonal weather has an effect on electricity consumption. For example, the increased use of air conditioning and heating systems during the summer and winter seasons will affect the load curve's trend. We determine the correlation between load and four weather variables (temperature, humidity, air pressure, and wind speed) using the Pearson correlation coefficient method, and identify temperature and humidity as the most relevant input features. The load on the large level region accumulates the power system's large-scale energy consumption. In comparison to the stochastic microgrid, the periodicity of power load changes is more apparent [4]. STLF benefits from an appropriate mechanism for encoding calendar information in a way that accurately indicates the periodic change pattern. The most common methods of encoding a calendar in the traditional sense are natural encoding and one-hot encoding [11]. However, because these two coding methods do not take into account the periodicity of electricity behavior, they are ineffective for load forecasting. For instance,if natural coding is used, for example, 23:00 on the first day and 0:00 on the second day are adjacent, but their distance differs by 23. When one-hot encoding is used, the Euclidean distance between two distinct time points is always √ 2 and the distance between two identical time points is 0, making it difficult to distinguish the distance between the two time points.We use polar coordinates to encode calendar effect features in order to reflect the periodicity of temporal data. The polar coordinate coding method makes use of the sin and cos functions to identify calendar variables within a given period. Electricity consumption is influenced by people's production and lifestyle choices and exhibits varying trends over time. For instance, the load on weekdays and weekends is different, as is the load at various times of the day. As a result, we perform periodic coding for the hour of the day and the day of the week or year. The encoding process is by (14)(15)(16)(17)(18)(19) and

1) IRELAND SINGLE-STEP LOAD FORECASTING
The length of the proposed model series is crucial since it determines the amount of historical data that will be used in the training process. If the series is too brief, it is impossible to extract sufficient data to forecast the load. If the length is excessive, the presence of a significant amount of weakly correlated data increases the training cost and decreases the accuracy. We determine the length with the strongest connection by computing the load series' autocorrelation and partial autocorrelation coefficients. The autocorrelation coefficients of load series denote the correlation between a given load point and the load point at time t lagging, whereas the partial autocorrelation coefficients denote the correlation between the load point and the load series trailing over a specified time period. As illustrated in Figure 5, we use Python's statsmodels tool to plot the autocorrelation and partial autocorrelation of the Ireland load series. According to Figure 5, we design five models with 24-, 48-, 72-, 144-, and 168-step series lengths.The proposed model input set is historical load series and corresponding length of historical temperature (T), humidity (H) and calendar variables (p hsin , p hcos , p wsin , p wcos , p dsin , p dcos ). Table 2 compares the performance of MCNN-LSTM model with varying input set lengths.As the length of the input set rises, the training time increases as well, but the values of MAE,RMSE and MAPE firstly decrease and then increase. Although LSTM can solve the vanishing gradient problem of long series to a certain extent, too long series will contain more weak correlation information and affect the improvement of accuracy. When the first 72 steps were used on the input set of the Ireland dataset, the prediction accuracy was the highest. For Ireland's testing set, the results of all models' single-step load forecasting are calculated using the performance indicators MAE, RMSE, and MAPE. The evaluation indicators for the seven algorithms in Table 3 indicate that: (i) Deep learning models outperform traditional machine learning models in terms of prediction accuracy, while hybrid models outperform single models. (ii) Our proposed hybrid model's MAE, REMSE, and MAPE are the smallest and have the best predictive performance. (iii) When compared to the worst-precision SVM model, the proposed model's MAE is decreased by 42.83, RMSE is decreased by 42.58 MAPE is decreased by 1.35% and its R-squared is increased by 2.12%. (iv) Compared with the CNN-LSTM model, the proposed model's MAE decreases by 8.32, RMSE decreases 5.07, MAPE is decreased by 0.28%, and its R-squared is increased by 0.7%. Figure 6 illustrates error distributions using box plots. It can be seen that the proposed model's error is concentrated near the zero-valued baseline, indicating that the predicted load value is closer to the actual value. Figure 7 shows the simulation results of load forecasting curves of different models. All models predict values in a consistent manner relative to the actual values. To illustrate the predictive performance of various models, the details of the lower and peak load in Figure 7 have been partially   enlarged and are shown in Figure 8 (a) and (b). In comparison to other models, the proposed model's lower and upper peak loads are more similar to the actual value in terms of trend shape and degree of fit.
2) MAINE SINGLE-STEP LOAD FORECASTING Figure 9 depicts the autocorrelation coefficients and partial autocorrelation coefficients of Maine datasets. The prediction evaluation of different length load series in the Maine dataset is shown in Table 4, and the findings indicate that the proposed model performs best when a 48-step input set length is used. As a result of the variances in load curves, load data sets from different locations must select appropriate sequences to get the highest prediction performance.
Additionally, we forecast single-step load on Maine's testing set. Table 5 illustrates the evaluation indicators for various    also demonstrates that the error distribution of the proposed model is the smallest overall. Figure 10 illustrates the load forecasting simulation curves for various models. Figure 12 (a) and (b) are enlarged versions of the lower and peak load curves from Figure 11. The findings indicate that the proposed model is capable of accurately forecasting the electricity load and has superior fitting performance for lower and upper peak loads with a higher degree of randomness.

D. MULTI-STEP PREDICTION ANALYSIS
We compare the proposed MCNN-LSTM model to other models for 24-step load forecasting. For example, a sliding window multi-input multi-output forecasting method for MLP, CNN, LSTM, CNN-LSTM, and the proposed MCNN-LSTM models. The previous time-steps of data are used as input, and the predicted load for 24 time-steps is the output result. Because the SVM and LightGBM models do not support multiple outputs, multi-step load forecasting is accomplished by creating 24 models corresponding to 24 time steps.

1) IRELAND 24-STEP LOAD FORECASTING
We conduct 24-step load forecasting on Ireland's testing set, forecast the load for the next 24 hours. Table 6 summarizes the error evaluation metrics for the 24-hour ahead, including MAE, RMSE, and MAPE. In comparison to single-step load forecasting, the forecast errors of each model increase as the output dimensions increase. However, the proposed MCNN-LSTM model still has the best predictive performance, MAE is 67.31, RMSE is 105.12, MAPE is 2.10% and R-squared is 96.87%. The performance improvement of the CNN-LSTM hybrid model over the single CNN and single LSTM models is not immediately apparent. Compared with  To concretely demonstrate the forecast results, we evaluate different time points of each week. It can be observed from table 7 that the proposed model is smaller than other models in MAE, RMSE and MAPE, indicating that the proposed model has strong load forecasting capacity for weekdays and weekends. The simulation of a 24-hour ahead prediction curve for the Ireland testing set is shown in Figure 13. The findings indicate that deep learning-based methods such as CNN, LSTM, CNN-LSTM, and the proposed MCNN-LSTM are capable of accurately simulating the next 24-hour power consumption trend. The power consumption trends of MLP, SVM, and LightGBM, on the other hand, are poorly fitted. Additionally, the proposed MCNN-LSTM model can more accurately predict the 24-hour power consumption prediction's lower and upper peak loads.

2) MAINE 24-STEP LOAD FORECASTING
The weekly MAE, RMSE and MAPE show in Table 8, the proposed model has obvious advantages over other models. The overall evaluation index of Maine test set is shown in Table 9, and the prediction curves are shown in Figure 14. The results demonstrate that our proposed model provides the best 24-hour load forecasting performance and validates its generalizability.

E. COMPARISON OF TRAINING TIME
We compare the training times of four deep learning models with the best prediction performance, including CNN, LSTM, CNN-LSTM, and the proposed MCNN-LSTM. The CNN model is composed of two stacked one-dimensional convolutions with a total of 32 or 64 convolution kernels. The LSTM model is composed of two LSTMs stacked into a two-layer LSTM, with each hidden layer having a neural unit of 128. The CNN-LSTM hybrid model is constructed by stacking two-layer CNNs and two-layer LSTMs in series. The CNN layer contains between 32 and 64 convolution kernels, while the LSTM layer contains 128 hidden layer neurons. The MCNN-LSTM model's structural parameters are described in Section III D. To compensate for randomness, we run each model ten times over 100 epochs and employ early stopping to avoid overfitting. Table 10 summarizes the average ten-time training time for various deep learning models.
According to the results in Table 10, the CNN model requires the least amount of training time, while the LSTM model requires the most. The CNN-LSTM and proposed MCNN-LSTM models have a shorter training time than the LSTM model, owing to the use of the CNN layer to extract features and the convolution operation, which reduces the number of parameters and improves training efficiency. The training time for MCNN-LSTM is less than that for CNN-LSTM, because multiple convolution heads extract features independently, avoiding the issue of slow fitting speed caused by increased dimensionality of the input features.     Simultaneously, the Ireland test set has 5256 data points. Single-step load forecasting has a test time of 12.08s and an average hourly forecasting time of 0.0023s. The forecasting time for a 24-step load is 10.23s, and the average forecasting time every 24 hours is 0.047s.

F. THE IMPACT OF PERIODIC CODING
To demonstrate the effect of periodic coding, we compare the proposed MCNN-LSTM model to a model that does not include a convolution head for extracting calendar variables. The forecasting of single-step, six-step, twelve-step, and twenty-four-step loads is carried out using Ireland's testing set, and the error evaluation indicators are listed in Table 11.
As shown in Table 11, as the output dimension increases, the MAE, RMSE, and MAPE values increase as well. The proposed MCNN-LSTM model performs better than the MCNN-SLTM model without calendar variables. According to MAPE values, the performance of single-step load forecasting is improved by 1.39%, 6-step load forecasting is improved by 6.47%, 12-step load forecasting is improved by 10.23%, and 24-step load forecasting is improved by 20.45%. The experimental results indicate that periodic coding of calendar variables is more advantageous for forecasting loads with large time steps.

VI. CONCLUSION
We propose a multi-head CNN-LSTM (MCNN-SLTM) short-term load forecasting model in this article. Two datasets (the Ireland dataset and the Maine dataset) are used to evaluate the proposed MCNN-LSTM model's performance. On these two datasets, experiments with single-step and 24-step load forecasting are conducted. When compared to other machine learning and deep learning models, the proposed model achieves the highest accuracy for the evaluation indicators MAE, RMSE, and MAPE. The training time of four deep learning models is also recorded in this experiment. Although the proposed model requires significantly more training time than the CNN model, it is more accurate. In comparison to the CNN-LSTM model, the proposed model can reduce parameter variables and accelerate feature extraction speed through the use of multiple convolutional heads, while maintaining high accuracy and training speed. To investigate the effect of calendar encoding, we remove the proposed MCNN-LSTM model's convolution head about calendar variables. To investigate the effect of calendar encoding, we remove the proposed MCNN-LSTM model's convolution head about calendar variables. Comparative experiments indicate that the addition of calendar variables improves load forecasting performance slightly with short output steps but significantly with long output steps.
In general, the MCNN-LSTM short-term load forecasting model proposed in this paper is capable of producing acceptable results for both single-step and multi-step load forecasting. For future research, a variety of optimization algorithms could be considered to optimize the proposed model's parameters. Holidays are not treated differently in the proposed model, and necessary holiday elements can be introduced into future model upgrades.