Improving Relation Extraction with Knowledge-attention

While attention mechanisms have been proven to be effective in many NLP tasks, majority of them are data-driven. We propose a novel knowledge-attention encoder which incorporates prior knowledge from external lexical resources into deep neural networks for relation extraction task. Furthermore, we present three effective ways of integrating knowledge-attention with self-attention to maximize the utilization of both knowledge and data. The proposed relation extraction system is end-to-end and fully attention-based. Experiment results show that the proposed knowledge-attention mechanism has complementary strengths with self-attention, and our integrated models outperform existing CNN, RNN, and self-attention based models. State-of-the-art performance is achieved on TACRED, a complex and large-scale relation extraction dataset.

Introduction
Relation extraction aims to detect the semantic relationship between two entities in a sentence. For example, given the sentence: "James Dobson has resigned as chairman of Focus On The Family, which he founded thirty years ago.", the goal is to recognize the organization-founder relation held between "Focus On The Family" and "James Dobson". The various relations between entities extracted from large-scale unstructured texts can be used for ontology and knowledge base population (Chen et al., 2018a;Fossati et al., 2018), as well as facilitating downstream tasks that requires relational understanding of texts such as question answering (Yu et al., 2017) and dialogue systems (Young et al., 2018).
Traditional feature-based and kernel-based approaches require extensive feature engineering (Suchanek et al., 2006;Qian et al., 2008;Rink * Corresponding author. and Harabagiu, 2010). Deep neural networks such as Convolutional Neural Networks (CNNs) and Recurrent Neural Networks (RNNs) have the ability of exploring more complex semantics and extracting features automatically from raw texts for relation extraction tasks Vu et al., 2016;Lee et al., 2017). Recently, attention mechanisms have been introduced to deep neural networks to improve their performance (Zhou et al., 2016;Wang et al., 2016;. Especially, the Transformer proposed by Vaswani et al. (2017) is based solely on self-attention and has demonstrated better performance than traditional RNNs (Bilan and Roth, 2018;Verga et al., 2018). However, deep neural networks normally require sufficient labeled data to train their numerous model parameters. The scarcity or low quality of training data will limit the model's ability to recognize complex relations and also cause overfitting issue.
A recent study (Li and Mao, 2019) shows that incorporating prior knowledge from external lexical resources into deep neural network can reduce the reliance on training data and improve relation extraction performance. Motivated by this, we propose a novel knowledge-attention mechanism, which transforms texts from word semantic space into relational semantic space by attending to relation indicators that are useful in recognizing different relations. The relation indicators are automatically generated from lexical knowledge bases which represent keywords and cue phrases of different relation expressions. While the existing self-attention encoder learns internal semantic features by attending to the input texts themselves, the proposed knowledge-attention encoder captures the linguistic clues of different relations based on external knowledge. Since the two attention mechanisms complement each other, we integrate them into a single model to maximize the uti-lization of both knowledge and data, and achieve optimal performance for relation extraction.
In summary, the main contributions of the paper are: (1) We propose knowledge-attention encoder, a novel attention mechanism which incorporates prior knowledge from external lexical resources to effectively capture the informative linguistic clues for relation extraction. (2) To take the advantages of both knowledge-attention and self-attention, we propose three integration strategies: multi-channel attention, softmax interpolation, and knowledgeinformed self-attention. Our final models are fully attention-based and can be easily set up for end-toend training. (3) We present detailed analysis on knowledge-attention encoder. Results show that it has complementary strengths with self-attention encoder, and the integrated models achieve startof-the-art results for relation extraction.

Related Works
We focus here on deep neural networks for relation extraction since they have demonstrated better performance than traditional feature-based and kernel-based approaches.
Convolutional Neural Networks (CNNs) and Recurrent Neural Networks (RNNs) are the earliest and commonly used approaches for relation extraction. Zeng et al. (2014) showed that CNN with position embeddings is effective for relation extraction. Similarly, CNN with multiple filter sizes (Nguyen and Grishman, 2015), pairwise ranking loss function (dos Santos et al., 2015) and auxiliary embeddings (Lee et al., 2017) were proposed to improve performance. Zhang and Wang (2015) proposed bi-directional RNN with max pooling to model the sequential relations. Instead of modeling the whole sentence, performing RNN on sub-dependency trees (e.g. shortest dependency path between two entities) has demonstrated to be effective in capturing longdistance relation patterns Miwa and Bansal, 2016). Zhang et al. (2018) proposed graph convolution over dependency trees and achieved state-of-the-art results on TACRED dataset.
Recently, attention mechanisms have been widely applied to CNNs (Wang et al., 2016; and RNNs (Zhou et al., 2016;. The improved performance demonstrated the effectiveness of attention mechanisms in deep neural networks. Particu-larly, Vaswani et al. (2017) proposed a solely selfattention-based model called Transformer, which is more effective than RNNs in capturing longdistance features since it is able to draw global dependencies without regard to their distances in the sequences. Bilan and Roth (2018) first applied self-attention encoder to relation extraction task and achieved competitive results on TACRED dataset. Verga et al. (2018) used self-attention to encode long contexts spanning multiple sentences for biological relation extraction. However, more attention heads and layers are required for self-attention encoder to capture complex semantic and syntactic information since learning is solely based on training data. Hence, more high quality training data and computational power are needed. Our work utilizes the knowledge from external lexical resources to improve deep neural network's ability of capturing informative linguistic clues.
External knowledge has shown to be effective in neural networks for many NLP tasks. Existing works focus on utilizing external knowledge to improve embedding representations Liu et al., 2015;Sinoara et al., 2019), CNNs (Toutanova et al., 2015;Wang et al., 2017;Mao, 2019), andRNNs (Ahn et al., 2016;Chen et al., , 2018bShen et al., 2018). Our work is the first to incorporate knowledge into Transformer through a novel knowledge-attention mechanism to improve its performance on relation extraction task.

Knowledge-attention Encoder
We present the proposed knowledge-attention encoder in this section. Relation indicators are first generated from external lexical resources (Section 3.1); Then the input texts are transformed from word semantic space into relational semantic space by attending to the relation indicators using knowledge-attention mechanism (Section 3.2); Finally, position-aware attention is used to summarize the input sequence by taking both relation semantics and relative positions into consideration (Section 3.3).

Relation Indicators Generation
Relation indicators represent the keywords or cue phrases of various relation types, which are essential for knowledge-attention encoder to capture the linguistic clues of certain relation from texts.  We utilize two publicly available lexical resources including FrameNet 1 and Thesaurus.com 2 to find such lexical units. FrameNet is a large lexical knowledge base which categorizes English words and sentences into higher level semantic frames (Ruppenhofer et al., 2006). Each frame is a conceptual structure describing a type of event, object or relation. FrameNet contains over 1200 semantic frames, many of which represent various semantic relations. For each relation type in our relation extraction task, we first find all the relevant semantic frames by searching from FrameNet (refer Appendix for detailed semantic frames used). Then we extract all the lexical units involved in these frames, which are exactly the keywords or phrases that often used to express such relation. Thesaurus.com is the largest online thesaurus which has over 3 million synonyms and antonyms. It also has the flexibility to filter search results by relevance, POS tag, word length, and complexity. To broaden the coverage of relation indicators, we utilize the synonyms in Thesaurus.com to extend the lexical units extracted from FrameNet. To reduce noise, only the most relevant synonyms with the same POS tag are selected.
Relation indicators are generated based on the word embeddings and POS tags of lexical units. Formally, given a word in a lexical unit, we find its word embedding w i ∈ R dw and POS embedding t i ∈ R dt by looking up the word embedding matrix W wrd ∈ R dw×V wrd and POS embedding matrix W pos ∈ R dt×V pos respectively, where d w and d t are the dimensions of word and POS embeddings, V wrd is vocabulary size 3 and V pos is total number of POS tags. The corresponding relation indicator is formed by concatenating word embedding and POS embedding, If a lexical unit contains multiple words (i.e. phrase), the corresponding relation indicator is formed by averaging the embeddings of all words. Eventually, around 3000 relation indicators (including 2000 synonyms) are generated: K = {k 1 , k 2 , ..., k m }.

Knowledge-attention process
In a typical attention mechanism, a query (q) is compared with the keys (K) in a set of key-value pairs and the corresponding attention weights are calculated. The attention output is weighted sum of values (V ) using the attention weights. In our proposed knowledge-attention encoder, the queries are input texts and the key-value pairs are both relation indicators. The detailed process of knowledge-attention is shown in Figure 1 (left).
Formally, given text input x = {x 1 , x 2 , ..., x n }, the input embeddings Q = {q 1 , q 2 , ..., q n } are generated by concatenating each word's word embedding and POS embedding in the same way as relation indicator generation in Section 3.1. The hidden representations H = {h 1 , h 2 , ..., h n } are obtained by attending to the knowledge indicators K, as shown in Equation 1. The final knowledgeattention outputs are obtained by subtracting the hidden representations with the relation indicators mean, as shown in Equation 2.
where knwl indicates knowledge-attention process, m is the number of relation indicators, and d k is dimension of key/query vectors which is a scaling factor same as in Vaswani et al. (2017). The subtraction of relation indicators mean will result in small outputs for irrelevant words. More importantly, the resulted output will be close to the related relation indicators and further apart from other relation indicators in relational semantic space. Therefore, the proposed knowledgeattention mechanism is effective in capturing the linguistic clues of relations represented by relation indicators in the relational semantic space.

Multi-head knowledge-attention
Inspired by the multi-head attention in Transformer (Vaswani et al., 2017), we also have multi-head knowledge-attention which first linearly transforms Q, K and V h times, and then perform h knowledge-attentions simultaneously, as shown in Figure 1 (right).
Different from the Transformer encoder, we use the same linear transformation for Q and K in each head to keep the correspondence between queries and keys.
Besides, only one residual connection from input embeddings to outputs of position-wise feed forward networks is used. We also mask the outputs of padding tokens using zero vectors.
The multi-head structure in knowledgeattention allows the model to jointly attend inputs to different relational semantic subspaces with different contributions of relation indicators. This is beneficial in recognizing complex relations where various compositions of relation indicators are needed.

Position-aware Attention
It has been proven that the relative position information of each token with respective to the two target entities is beneficial for relation extraction task (Zeng et al., 2014). We modify the positionaware attention originally proposed by  to incorporate such relative position information and find the importance of each token to the final sentence representation.
Assume the relative position of token x i to target entity isp i . We apply position binning function (Equation 4) to make it easier for the model to distinguish long and short relative distances.
After getting the relative positions p s i and p o i to the two entities of interest (subject and object respectively), we map them to position embeddings base on a shared position embedding matrix W p . The two embeddings are concatenated to form the final position embedding for token Position-aware attention is performed on the outputs of knowledge-attention O ∈ R n×d k , taking the corresponding relative position embeddings P ∈ R n×dp into consideration: d a is attention dimension, and c ∈ R da is a context vector learned by the neural network.

Integrate Knowledge-attention with Self-attention
The self-attention encoder proposed by Vaswani et al. (2017) learns internal semantic features by modeling pair-wise interactions within the texts themselves, which is effective in capturing longdistance dependencies. Our proposed knowledgeattention encoder has complementary strengths of capturing the linguistic clues of relations precisely based on external knowledge. Therefore, it is beneficial to integrate the two models to maximize the utilization of both external knowledge and training data. In this section, we propose three integration approaches as shown in Figure 2, and each approach has its own advantages.

Multi-channel Attention
In this approach, self-attention and knowledgeattention are treated as two separate channels to model sentence from different perspectives. After applying position-aware attention, two feature vectors f 1 and f 2 are obtained from self-attention and knowledge-attention respectively. We apply another attention mechanism called multi-channel attention to integrate the feature vectors.
In multi-channel attention, feature vectors are first fed into a fully connected neural network to get their hidden representations h i . Then attention weights are calculated using a learnable context vector c, which reflects the importance of each feature vector to final relation classification. Finally, the feature vectors are integrated based on attention weights, as shown in Equation 6.
After obtaining the integrated feature vector r, we pass it to a softmax classifier to determine the relation class. The model is trained using stochastic gradient descent with momentum and learning rate decay to minimize the cross-entropy loss.
The main advantage of this approach is flexibility. Since the two channels process information independently, the input components are not necessary to be the same. Besides, we can add more features from other sources (e.g. subject and object categories) to multi-channel attention to make final decision based on all the information sources.

Softmax Interpolation
Similar as multi-channel attention, we also use two independent channels for self-attention and knowledge-attention in softmax interpolation. Instead of integrating the feature vectors, we make two independent predictions using two softmax classifiers based on the feature vectors from the two channels. The loss function is defined as total cross-entropy loss of the two classifiers. The final prediction is obtained using an interpolation function of the two softmax distributions: where p 1 , p 2 are the softmax distributions obtained form self-attention and knowledgeattention respectively, and β is the priority weight assigned to self-attention.
Since knowledge-attention focuses on capturing the keywords and cue phrases of relations, the precision will be higher than self-attention while the recall is lower. The proposed softmax interpolation approach is able to take the advantages of both attention mechanisms and balance the precision and recall by adjusting the priority weight β.

Knowledge-informed Self-attention
Since knowledge-attention and self-attention share similar structures, it is also possible to integrate them into a single channel. We propose knowledge-informed self-attention encoder which incorporates knowledge-attention into every self-attention head to jointly model the semantic relations based on both knowledge and data.
The structure of knowledge-informed selfattention is shown in Figure 3. Formally, given texts input matrix Q ∈ R n×d k and knowledge indicators K ∈ R m×d k . The output of each attention head is calculated as follows:  where knwl and self indicate knowledgeattention and self-attention respectively, and all the linear transformation weight matrices have the dimensionality of W ∈ R d k ×(d k /h) . Since each self-attention head is aided with prior knowledge in knowledge-attention, the knowledge-informed self-attention encoder is able to capture more lexical and semantic information than single attention encoder.

Baseline Models
To study the performance of our proposed models, the following baseline models are used for comparison: CNN-based models including: (1) CNN: the classical convolutional neural network for sentence classification (Kim, 2014). (2) CNN-PE: CNN with position embeddings dedicated for relation classification (Nguyen and Grishman, 2015). (3) GCN: a graph convolutional network over the pruned dependency trees of the sentence (Zhang et al., 2018). RNN-based models including: (1) LSTM: long short-term memory network to sequentially model the texts. Classification is based on the last hidden output. (2) PA-LSTM: Similar position-aware attention mechanism as our work is used to summarize the LSTM outputs . CNN-RNN hybrid model including contextualized GCN (C-GCN) where the input vectors are obtained using bi-directional LSTM network (Zhang et al., 2018). Self-attention-based model (Self-attn) which uses self-attention encoder to model the input sentence. Our implementation is based on Bilan and Roth (2018) where several modifications are made on the original Transformer encoder, including the use of relative positional encodings instead of absolute sinusoidal encodings, as well as other configurations such as residual connection, activation function and normalization.
For our model, we evaluate both the proposed knowledge-attention encoder (Knwl-attn) as well as the integrated models with self-attention including multi-channel attention (MCA), softmax interpolation (SI) and knowledge-informed selfattention (KISA).

Experiment Settings
We conduct our main experiments on TACRED, a large-scale relation extraction dataset introduced by . TACRED contains over 106k sentences with hand-annotated subject and object entities as well as the relations between them. It is a very complex relation extraction dataset with 41 relation types and a no relation class when no relation is hold between entities. The dataset is suited for real-word relation extraction since it is unbalanced with 79.5% no relation samples, and multiple relations between different entity pairs can be exist in one sentence. Besides, the samples are normally long sentences with an average of 36.2 words.
Since the dataset is already partitioned into train (68124 samples), dev (22631 samples) and test (15509 samples) sets, we tune model hyperparameters using dev set and evaluate model using test set. The evaluation metrics are microaveraged precision, recall and F 1 score. For fair comparison, we select the model with median F 1 score on dev set from 5 independent runs, same as . The same "entity mask" strategy is used which replaces subject (or object) entity with special NER -SUBJ (or NER -OBJ) tokens to avoid overfittting on specific entities and provide entity type information.
Besides TACRED, another dataset called SemEval2010-Task8 (Hendrickx et al., 2009) is used to evaluate the generalization ability of our proposed model. The dataset is significantly smaller and simpler than TACRED, which has 8000 training samples and 2717 testing samples. It contains 9 directed relations and 1 other relation (19 relation classes in total). We use the official macro-averaged F 1 score as evaluation metric.
We use one layer encoder with 6 attention heads for both knowledge-attention and self-attention since further increasing the number of layers and attention heads will degrade the performance. For softmax interpolation, we choose β = 0.8 to balance precision and recall. Word embeddings are fine-tuned based on pre-trained GloVe (Pennington et al., 2014) with dimensionality of 300. Dropout (Srivastava et al., 2014) is used during trianing to alleviate overfitting. Other model hyperparameters and training details are described in Appendix due to space limitations. Table 1 shows the results of baseline as well as our proposed models on TACRED dataset. It is observed that our proposed knowledge-attention encoder outperforms all CNN-based and RNNbased models by at least 1.3 F 1 . Meanwhile, it achieves comparable results with C-GCN and selfattention encoder, which are the current start-ofthe-art single-model systems.

Results on TACRED dataset
Comparing with self-attention encoder, it is observed that knowledge-attention encoder results in higher precision but lower recall. This is reasonable since knowledge-attention encoder focuses on capturing the significant linguistic clues of relations based on external knowledge, it will result in high precision for the predicted relations similar to rule-based systems. Self-attention encoder is able to capture more long-distance dependency features by learning from data, resulting in better recall. By integrating self-attention and knowledge-attention using the proposed approaches, a more balanced precision and recall can be obtained, suggesting the complementary effects of self-attention and knowledge-attention mechanisms. The integrated models improve performance by at least 0.9 F 1 score and achieve new state-of-the-art results among all the single endto-end models.
Comparing the three integrated models, softmax interpolation (SI) achieves the best performance. More interestingly, we found that the precision and recall can be controlled by adjusting the priority weight β. Figure 4 shows impact of β on precision, recall and F 1 score. As β increases, precision decreases and recall increases. Therefore, we can choose a small β for relation extraction system which requires high precision, and a large β for the system requiring better recall. F 1 score reaches the highest value when precision and re-  Table 1: Micro-averaged precision (P), recall (R) and F 1 score on TACRED dataset. †, ‡ and † † mark the results reported in , (Zhang et al., 2018) and (Bilan and Roth, 2018) respectively. * marks statistically significant improvements over Selfattn with p < 0.01 under one-tailed t-test. call are balanced (β = 0.8).
Knowledge-informed self-attention (KISA) has comparable performance with softmax interpolation, and without the need of hyper-parameter tuning since knowledge-attention and self-attention are integrated into a single channel. The performance gain over self-attention encoder is 1.2 F 1 with much improved precision, demonstrating the effectiveness of incorporating knowledgeattention into self-attention to jointly model the sentence based on both knowledge and data.
Performance gain is the lowest for multichannel attention (MCA). However, the model is more flexible in the way that features from other information sources can be easily added to the  model to further improve its performance. Table  2 shows the results of adding NER embeddings of each token to self-attention channel, and entity (subject and object) categorical embeddings to multi-channel attention as additional feature vectors. We use dimensionality of 30 and 60 for NER and entity categorical embeddings respectively, and the two embedding matrixes are learned by the neural network. Results show that adding NER and entity categorical information to MCA integrated model improves F 1 score by 0.2 and 0.5 respectively, and adding both improves precision significantly, resulting a new best F 1 score.

Results on SemEval2010-Task8 dataset
We use SemEval2010-Task8 dataset to evaluate the generalization ability of our proposed model. Experiments are conducted in two manners: mask or keep the entities of interest. Results in Table  3 show that the "entity mask" strategy degrades the performance, indicating that there exist strong correlations between entities of interest and relation classes in SemEval2010-Task8 dataset. Although the results of keeping the entities are better, the model tends to remember these entities instead of focusing on learning the linguistic clues of relations. This will result in bad generalization for sentences with unseen entities. Regardless of whether the entity mask is used, by incorporating knowledge-attention mechanism, our model improves the performance of selfattention by a statistically significant margin, especially the softmax interpolation integrated model. The results on SemEval2010-Task8 are consistent with that of TACRED, demonstrating the effectiveness and robustness of our proposed method.

Ablation study
To study the contributions of specific components of knowledge-attention encoder, we perform ablation experiments on the dev set of TACRED. The results of knowledge-attention encoder with and  without certain components are shown in Table 4. It is observed that: (1) The proposed multihead knowledge-attention structure outperforms single-head significantly. This demonstrates the effectiveness of jointly attending texts to different relational semantic subspaces in the multi-head structure.
(2) The synonyms improve the performance of knowledge-attention since they are able to broaden the coverage of relation indicators and form a robust relational semantic space. (3) The subtraction of relation indicators mean vector from attention hidden representations helps to suppress the activation of irrelevant words and results in a better representation for each word to capture the linguistic clues of relations. (4-5) The two masking strategies are helpful for our model: the output masking eliminates the effects of the padding tokens and the entity masking avoids entity overfitting while providing entity type information. (6) The relative position embedding term in position-aware attention contributes a significant amount of F 1 score. This shows that positional information is particularly important for relation extraction task.

Attention visualization
To verify the complementary effects of knowledge-attention encoder and self-attention   encoder, we compare the attention weights assigned to words from the two encoders. Table 5 presents the attention visualization results on sample sentences. For each sample sentence, attention weights from knowledge-attention encoder are visualized first, followed by self-attention encoder. It is observed that knowledge-attention encoder focuses more on the specific keywords or cue phrases of certain relations, such as "graduated", "executive director" and "founded"; while self-attention encoder attends to a wide range of words in the sentence and pays more attention to the surrounding words of target entities especially the words indicating the syntactic structure, such as "is", "in" and "of". Therefore, knowledgeattention encoder and self-attention encoder have complementary strengths that focus on different perspectives for relation extraction.

Error analysis
To investigate the limitations of our proposed model and provide insights for future research, we analyze the errors produced by the system on the test set of TACRED. For knowledge-attention encoder, 58% errors are false negative (FN) due to the limited ability in capturing long-distance dependencies and some unseen linguistic clues during training. For our integrated model 4 that takes the benefits of both self-attention and knowledgeattention, FN is reduced by 10%. However, false positive (FP) is not improved due to overfitting that leads to wrong predictions. Many errors are caused by multiple entities with different relations co-occurred in one sentence. Our model may mistake irrelevant entities as a relation pair. We also observed that many FP errors are due to the confusions between related relations such as "city of death"and "city of residence". More data or knowledge is needed to distinguish "death" and "residence". Besides, some errors are caused by imperfect annotations.

Conclusion and Future Work
We introduce knowledge-attention encoder which effectively incorporates prior knowledge from external lexical resources for relation extraction. The proposed knowledge-attention mechanism transforms texts from word space into relational semantic space and captures the informative linguistic clues of relations effectively. Furthermore, we show the complementary strengths of knowledgeattention and self-attention, and propose three different ways of integrating them to maximize the utilization of both knowledge and data. The proposed models are fully attention-based end-toend systems and achieve state-of-the-art results on TACRED dataset, outperforming existing CNN, RNN, and self-attention based models. In future work, besides lexical knowledge, we will incorporate conceptual knowledge from encyclopedic knowledge bases into knowledgeattention encoder to capture the high-level semantics of texts. We will also apply knowledgeattention in other tasks such as text classification, sentiment analysis and question answering.


extfieldsofstudy:
Engineering
Computer Science

text:
Perceptual Image Anomaly Detection

We present a novel method for image anomaly detection, where algorithms that use samples drawn from some distribution of “normal” data, aim to detect out-of-distribution (abnormal) samples. Our approach includes a combination of encoder and generator for mapping an image distribution to a predefined latent distribution and vice versa. It leverages Generative Adversarial Networks to learn these data distributions and uses perceptual loss for the detection of image abnormality. To accomplish this goal, we introduce a new similarity metric, which expresses the perceived similarity between images and is robust to changes in image contrast. Secondly, we introduce a novel approach for the selection of weights of a multi-objective loss function (image reconstruction and distribution mapping) in the absence of a validation dataset for hyperparameter tuning. After training, our model measures the abnormality of the input image as the perceptual dissimilarity between it and the closest generated image of the modeled data distribution. The proposed approach is extensively evaluated on several publicly available image benchmarks and achieves state-of-the-art performance.

Introduction
Anomaly detection is one of the most important problems in a range of real-world settings, including medical applications [1], cyber-intrusion detection [2], fraud detection [3], anomaly event detection in videos [4] and overgeneralization problems of neural networks [5]. Anomaly detection tasks generally involve the use of samples of a "normal" class, drawn from some distribution, to build a classifier that is able to detect "abnormal" samples, i.e. outliers with respect to the aforementioned distribution. Although anomaly detection is well-studied in a range of domains, image anomaly detection is still a challenge due to the complexity of distributions over images.
Generative Adversarial Networks (GANs) [6] present one of the new promising deep anomaly detection approaches. One network called the generator is trained to transform latent vectors, drawn from a latent distribution, to images in such a way that the second network, the discriminator, cannot distinguish between real images and generated ones. Thus after training, the generator performs a mapping of the latent distribution to the data distribution. This property has been used [7][8][9] to estimate the likelihood of abnormality for an input: if there is a vector in latent space, which after passing through the generator could reconstruct the input object, the object is normal, otherwise it is not. The difference between an input and its closest reconstruction (reconstruction error) is used as an anomaly score for this object.
Although there is a scope of methods that use GAN for anomaly detection, none of them were directly developed for anomaly detection on images. Usually, they apply the L1-norm or Mean Squared Error (MSE) between the pixels to compute a reconstruction error, which does not correspond to human understanding of the similarity between two images. Another problem of GAN-based approaches is how to find the latent vector that, after passing through the generator, recovers the input object. Previously, it was performed by a gradient descent optimization procedure [7,8], co-training the generator and the encoder that recovers the latent vector [9][10][11]. However, existing techniques are either time-consuming [7,8] or difficult to train [10,11], or consist of complex multi-step learning procedures [9]. Another problem is that the complete loss function consists of a sum of many components with weighting coefficients as hyper-parameters. The lack of a validation set (we do not have any anomaly examples during training), makes it difficult to choose these coefficients. In our work we propose solutions for each of these three problems:

Perceptual Image Anomaly Detection
1. We developed a new metric that measures the similarity between the perception of two images. Our metric, called relative-perceptual-L1 loss, is based on perceptual loss [13], but is more robust to noise and changes of contrast of images ( Figure 1). 2. We propose a new technique for training an encoder that predicts a latent vector jointly with the generator. We construct a loss function in such a way that the encoder predicts a vector belonging to the latent distribution, and that the image reconstructed from this vector by the generator is similar to the input. 3. We propose a way to choose the weighting coefficients in the complete loss functions for the encoder and the generator. We base our solution on the norm of the gradients (with respect to network parameters) of each loss function, to balance the contribution of all losses during the training process.
The proposed approach, called Perceptual Image Anomaly Detection (PIAD), allows us to improve performance on several well-known datasets. We experimented with MNIST, Fashion MNIST, COIL-100, CIFAR-10, LSUN and CelebA and made an extensive comparison with a wide range of anomaly detection approaches of different paradigms.

Related Work
Anomaly detection has been extensively studied in a wide range of domains [14]. However, anomaly detection on image data is still challenging. Classical approaches such as explicit modeling of latent space using KDE [15] or One-Class SVM [16] which learns a boundary around samples of a normal class, show poor quality when applied to image anomaly detection tasks. Due to the problem of the curse of dimensionality, these algorithms are weak in modeling complex high-dimensional distributions.
Deep autoencoders play an important role among anomaly detection methods [17][18][19]. Autoencoders that perform dimension reduction for normal samples learn some common factors inherent in normal data. Abnormal samples do not contain these factors and thus cannot be accurately reconstructed by autoencoders. However, image anomaly detection is still challenging for autoencoders, and usually they are applied only on simple abnormal samples, when the variability of normal images is low.
There are also "mixed" approaches that use autoencoders or other deep models for representation learning. GPND [20] leverages an adversarial autoencoder to create a low-dimensional representation and then uses a probabilistic interpretation of the latent space to obtain an anomaly score. The method described in [21] models a latent distribution obtained from a deep autoencoder using an auto-regressive network. In Deep SVDD [22], Ruff et al. show how to train a one-class classification objective together with deep feature representation.
GANs [6] created a new branch in the development of image anomaly detection. GAN-based approaches [7][8][9][10][11] differ in two parts: (i) how to find latent vectors that correspond to the input images, (ii) how to estimate abnormality based Figure 2: Comparison of four anomaly detection models. G denotes the generator, E the encoder, D * the discriminators, "rec. loss" reconstruction loss. p Z denotes the latent distribution, and z (white) its samples. In the same way, p X is the data distribution and x (white) are data samples.
on the input image and the reconstructed one. For the second problem, these methods use a linear combination of the L1-norm or the MSE between the input image and the reconstruction (reconstruction error), and the discriminator's prediction of the reality of the reconstructed image. For the first problem, approaches AnoGAN [7] and ADGAN [8] propose to use time-consuming gradient descent optimization of a latent vector. Other approaches train an encoder to predict a latent vector for each image. Figure 2 demonstrates the differences between the existing approaches. ALAD [11] and [10] train the encoder adversarially: the adversarial loss computed by the discriminator, which takes pairs (image, vector), forces the encoder to predict a latent vector that reconstructs the input image. However, discriminators of such models train with a cross-entropy loss function, which causes an unstable training process. The OCGAN model trains a denoising autoencoder. To improve the quality of mapping, authors added two discriminators: D X and D Z , and a classifier which searches for hard negative examples (bad generated images).

Perceptual Image Anomaly Detection
Conceptually the idea of PIAD follows the OCGAN. We apply the power of GANs two times, once for building a mapping from the latent space to the image space, and again to create an inverse mapping. A generator and an encoder are trained jointly to satisfy three conditions (see Figure 2d): 1. Generator G performs a mapping from latent distribution p Z to data distribution p X ; 2. Encoder E performs a mapping from p X to p Z ; 3. The image which generator G recovers from the latent vector that is predicted by encoder E must be close to the original image (reconstruction term): To accomplish conditions 1 and 2 we train the generator and the encoder with adversarial losses. Therefore, two discriminators D X and D Z are required. To evaluate the reconstruction term we propose to use our new relativeperceptual-L1 loss.
Ideologically, our approach differs from OCGAN. OCGAN is a denoising autoencoder with highly constrained latent space. On top of reconstruction loss, it uses adversarial loss to ensure that the decoder (the generator in our notation) can reproduce only normal examples. Our approach, however, is based on the power of adversarial loss for mapping two distributions. In practice, OCGAN differs in the classifier component, which helps to find weak places of latent space, which produce not "normal" images, but make the whole training process much complicated and multi-steps. Also, we do not add noise to image x before passing it through the encoder.
In order to train the encoder and generator to minimize a multi-objective loss function, we propose a new way of setting the weights of the loss functions that equalizes the contribution of each loss in the training process. Due to the fact that our approach relies on gradients of the parameters of the loss function, we called it gradient-normalizing weights policy.
After training the proposed model on samples of a normal class, we suggest to predict the abnormality of a new example x by evaluating the relative-perceptual-L1 loss between the input x and G(E(x)): (1) We consider the relative-perceptual-L1 loss in more detail in Section 3.1, the procedure for training models in Section 3.2 and the gradient-normalizing weights policy in Section 3.3.

Relative-perceptual-L1 Loss
Features obtained by a neural network, trained on a large dataset for the task of object recognition, can capture highlevel image content without binding to exact pixel locations [23]. In [13] Johnson et al. proposed content distance between two images, called perceptual loss: this metric computes the MSE between features taken at a deep level of a neural network that has been pre-trained on an object classification task.
Let f (x) be a feature map obtained from some deep layer of the network on image x, and C × H × W the shape of this feature map. Then the perceptual loss between image x and y is determined as: However, perceptual loss is very sensitive to changes in image contrast. Figure 1 shows three pairs of images: pairs 1b and 1c have lower contrast than 1a. Perceptual loss drops by 22% for images 1b compared to 1a, although for human supervision the pair 1b differs from the pair 1a very little. In this way, if we used perceptual loss for computing anomaly score, the model would tend to predict lower contrast images as less abnormal. Another problem is that perceptual loss applies the MSE over features, but the MSE penalizes the noise in the obtained feature values very heavily.
We tackled these problems and propose relative-perceptual-L1 loss, which is robust to contrast and noise. First of all, we noticed that features obtained at different filters can have a different scatter of values. As an example, Figure 3 (left) shows the standard deviations of filter responses of some deep layer of VGG-19 [24], computed over Imagenet [25]. We visualize the standard deviations since they indicate the overall value of the features, which are themselves distributed around zero. Standard deviations differ by a factor of 2-3, which means that the contributions per filter vary by a factor 2-3 as well. Therefore, as the first step of relative-perceptual-L1, we propose to normalize the obtained deep features by the mean and std of filter responses which are pre-calculated over the large dataset, like Imagenet. Secondly, we propose to use the L1-norm instead of the MSE, since the L1-norm is more robust to noise. Thirdly, to make the loss more resistant to contrast, we research how feature values behave under changes of contrast. Figure 3 (right) illustrates this behavior: during the reduction of image contrast, the feature value average decreases. In this way, the absolute error (the difference between features), which is used in perceptual loss, decreases as well for each pair of lower contrast images. Therefore, we propose not to use absolute error, but relative error, which measures the ratio of the absolute error of features to the average values of these features.

Perceptual Image Anomaly Detection
A PREPRINT Consequently, relative-perceptual-L1 is calculated as follows: where µ, σ are the pre-calculated mean and std of filter responses.

Training Objective
To train both discriminators we used the Wasserstein GAN with a Gradient Penalty objective (WGAN-GP) [26]. The training of a Wasserstein GAN is more stable than a classical GAN [6] (which was used in [9][10][11]), it prevents mode collapse, and does not require a careful searching schedule of generator/discriminator training. Thus, the discriminator D X learns by minimizing the following loss: where GP is Gradient Penalty Regularization [26] and λ is a weighting parameter. In the same way, D Z minimizes L disc (D Z ). Adversarial loss of the generator is Adversarial loss of the encoder L adv (E) is computed in the same way. Reconstruction loss is measured using the proposed relative-perceptual-L1 loss: Thus, the total objectives for the encoder and generator are as follows: where γ G and γ E are weighting parameters. The training process consists of alternating n dis steps of optimization of the discriminators and one step of optimization of the generator together with the encoder. Parameters γ G and γ E change every n param iterations following our gradient-normalizing weights policy. The full training procedure is summarized in Algorithm 1. (Steps "update gradient history" and "select weighting parameters" are explained in detail in the next Section).

Gradient-normalizing Weight Policy
Our objective function consists of the sum of multiple losses. To find weighting parameters for these losses, we cannot use cross-validation, because no anomaly examples are available to calculate anomaly detection quality. The work [9] chooses weights empirically based on reconstruction quality. However, it requires a person to manually select the coefficients for each experiment, and it is not objective and reproducible.
In order to choose weighting parameters automatically, we need to base our solution on measured values of an experiment. Letw = [w 1 , ..., w n ] be a vector of network parameters, l 1 (w) , l 2 (w) are losses calculated for this network, and L total (w) = l 1 (w) + γl 2 (w).
Depending on the nature of the loss functions l 1 and l 2 , the norms of ∂l1 ∂wi and ∂l2 ∂wi can differ by a factor of ten or even a hundred. Coefficient γ regulates the relative influence of the loss functions in the total gradient with respect to this parameter w i . To make the contribution of the loss functions equal, we can choose coefficient γ in the following way: However, due to using stochastic optimization, gradients are very noisy during training. To make this process more robust and stable, we propose to average the γ coefficients over all network parameters and over their previous values (history information). Our approach is summarized in Algorithm 2 and Algorithm 3.
Algorithm 1 Training procedure of PIAD. "Reset gradients" zeros all stored gradients of networks parameters. "Backpropagate loss" computes gradients of loss wrt network parameters and sums them to current stored gradients. "Update net" performs one step of gradient descent using stored gradients. Require: N, the total number of iterations. n dis , the number of iterations of training the discriminators. n weight , frequency of γ G and γ E change. 1: for iter = 0, ..., N do 2: for t = 0, ..., n dis do 3: Compute L disc (D X ), L disc (D Z ).

5:
Reset gradients; backpropagate L disc (D X ), L disc (D Z ); update D X , D Z .  Reset gradients. 16: Backpropagate L rec (G, E); multiply gradients of G by γ G , E by γ E . 17: Backpropagate L adv (G), L adv (E). In short: for each loss, we calculate the derivative (backpropagate loss) wrt each network weight w i . Then for each convolutional layer we compute the L2-norm of the derivative wrt the weight matrix and store it. This is done after every n weight iterations in training, and all previously calculated values are kept, thus creating a gradient history per loss, per layer. We calculate the L2-norm per layer (but not per each weight w i ) to reduce the size of the stored information. Computing the norm over all network parameters would lose too much information, since the last layers usually have more parameters, and hence information about gradients from the first layers would be lost. Firstly, the coefficient γ is calculated per layer: we perform exponential smoothing of the history values of each loss (to make values robust to noise), and then calculate the average ratio between the last N entries in the gradient history for loss 1 and the same for loss 2 . The final value for γ is computed as the average over the γ-s per layer.
Our approach simply generalizes to a loss function consisting of more than two contributions. It also leaves room for research on which norm to use and how to compute the final weights.

Experiments
We show the effectiveness of the proposed approach by evaluation on six publicly available datasets and compare our method with a diverse collection of state-of-the-art methods for out-of-distribution detection, including state-of-the-art GAN-based approaches.
Datasets. For evaluation we use the following well-known datasets (Table 1) Competing Methods. As shallow baselines we consider standard methods such as OC-SVM [16] and KDE [15]. We also test the performance of our approach against four state-of-the-art GAN-based methods: AnoGAN [7], ADGAN [8], OCGAN [9] and ALAD [11]. Finally, we report the performance of three deep learning approaches from different paradigms: Deep SVDD [22], GPND [20], and the Latent Space Autoregression approach [21] (results will be reported under the name LSA). All these methods have been briefly described in Section 2.
Evaluation Protocol. To test the methods on classification datasets, we use a one-vs-all evaluation scheme, which has recently been increasingly used in anomaly detection papers [8,9,11,[20][21][22]: to simulate out-of-distribution condition, one class of a dataset is considered as normal data while images of other classes are considered as abnormal. We evaluate results quantitatively using the area under the ROC curve (ROC AUC), which is a standard metric for this task.
Implementation Details. In all experiments we used pre-activation resnet blocks to build our generator, encoder, and discriminator; for computing relative-perceptual-L1 loss we used VGG-19 [24] network, pre-trained on Imagenet [25]. Other implementation details are presented in the supplementary material.

Results
MNIST and CIFAR-10. Following [8,9,11,21,22] we evaluated our approach on MNIST and CIFAR-10 using a train-test dataset split, where the training split contains part of the known class and the test split contains the unknown classes and the remainder of the known class. We run each experiment 3 times with different initializations and present the averaged ROC AUC in Table 2.
Since the MNIST dataset is easy, all methods work well. However, our approach allows to improve performance on several tricky digits (like 3 and 8) and outperforms all other approaches on average over the dataset. On the more diverse and complex CIFAR-10, the superiority of the proposed method is even more noticeable. This dataset contains 10 image classes with extremely high intra-class variation and the images are so small that even a human cannot always distinguish the kind of object on the image. Our approach based on the perceptual similarity of images can better capture class-specific information of an image, and hence, improves performance of anomaly detection in almost all experiments.
fMNIST and COIL-100. The work of GPND [20]    normal class. The remaining 20% of normal data and the same number of randomly selected anomaly instances are used for testing. We report the average performance of GPND and PIAD on the fMNIST and COIL-100 datasets in Table 3 (left), along with OCGAN since they came out the second best on the previous datasets, and they report fMNIST/COIL-100 results in the same evaluation scheme as well. For the COIL-100 dataset we randomly selected one class to be used as normal, and repeated this procedure 30 times (as it was done in [20]). The comparison shows that PIAD excels on both datasets.

LSUN.
We also compare PIAD with the ADGAN approach on the LSUN dataset, training a model on images of bedrooms and treating images of the conference room class as anomaly. We achieve a ROC AUC of 0.781 against 0.641 with ADGAN.
CelebA. In order to test our approach in conditions that are closer to a real-world use case, we experimented on the CelebA dataset, where we use attributes (Bald, Mustache, Bangs, Eyeglasses, Wearing Hat) to split the data into normal/anomaly cases. We train our model on 'normal' images where one of these attributes is not present and test against anomaly images, where that same attribute is present. Table 3  and makes bangs more transparent. However, our model failed to recognize the Bald anomaly. This may be a result of the complexity of the anomaly (see Figure 4c first image, where indeed the man is not completely bald) and also inexact annotation (on Figure 4c the first image is annotated as bald, but the second is not).

Ablation Study
We also performed an ablation study on the CIFAR-10 dataset to show the effectiveness of each proposed component of PIAD. We considered 5 scenarios: baseline, i.e. our model with MSE as reconstruction error during training and as anomaly score, with empirically chosen weighting parameters by human supervision of the quality of generated examples and reconstructions; + gr-norm w, the same, but with gradient-normalizing weights policy; + perc, where we further changed from MSE to perceptual loss; + perc-L1, where we added normalization on µ, σ in perceptual loss and used L1-norm over features instead of MSE, + rel-perc-L1, where we used the proposed loss of Section 3.1. We present the average ROC AUC over CIFAR-10 in Table 3 (right).
We note that our proposed gradient-normalizing weight policy shows the same result as carefully found weights through parameter selection by a human after running the model several times. Each further modification improved results as well. Figure 5 shows examples of images that were seen as the least and the most likely to be an anomaly, for different reconstruction losses. We note that only relative-perceptual-L1 loss is not prone to select monochrome images as the least anomalous, and furthermore, this loss selected classes that are closest to the car classes as less anomalous: truck, ship.

Conclusion
We introduced a deep anomaly detection approach, built directly for the image domain and exploiting knowledge of perceptual image similarity. For the latter, we proposed a new metric that is based on perceptual loss, but is more robust to noise and changes of contrast of images. As a part of our work we proposed an approach for selecting weights of a multi-objective loss function, which makes a contribution of all losses equal in the training process. We demonstrated the superiority of our approach against state-of-the-art GAN-based methods and deep approaches of other paradigms on a diverse collection of image benchmarks. In the future, we plan to perform a more extensive evaluation of our method on higher resolution image data, like medical images.

Implementation details
In all experiments images were rescaled to range [−1, 1]. We used the following schedule for training models: n dis = 2 rounds of discriminator training per one step of generator training (for experiments with LSUN and CelebA we raised n dis to 3). During discriminator optimization, the coefficient λ for the Gradient Penalty was set to 10 and we also added the regularizer E[D(x) 2 ] to the discriminator training objective with a small weight of 0.001, preventing the discriminator outputs to drift far away from zero (as it was done in [1]). Networks were trained using the Adam optimizer with betas of (0.5, 0.99) and learning rate 0.0005 for discriminators or 0.0002 for the generator (and the encoder).
We trained models for 20,000 iterations for the COIL-100 dataset, 50,000 iterations for MNIST and fMNIST, 100,000 for CIFAR-10, and 200,000 for LSUN and CelebA, with a batch size of 32.

Relative-perceptual-L1 Loss
Computation of the relative-perceptual-L1 loss between two images requires a deep convolutional network that has been pre-trained on an image classification task. Following [2], we used a 19-layer VGG [3] network, pre-trained on the Imagenet challenge [4]. We computed features using the relu 42 layer in the notation of [2] (the activation of the 2nd convolution of the 4th block). Normalizing coefficients µ and σ (means and standard deviations of filter responses) were calculated over a subset of Imagenet as well. For computing the loss on grayscale images, we simply duplicated the channel to get 3-channel images.

Gradient-normalizing Weight Policy
We used n weight = 100 frequency of saving gradient history and updating γ G and γ E . In our implementation, instead of computation L2-norm of gradients, we calculated standard deviation. However, since the mean of the derivatives is always close to zero, the standard deviation influences in the same way as L2-norm.

Network Architectures
For building the generator G, the discriminator D X , and the encoder E, we used pre-activation residual blocks [5] with two 3 × 3 convolutional layers with the same number of filters in both convolutions and Leaky ReLU activations with negative slope 0.2. A scheme of the pre-activation residual block is shown in Figure 1a. In the generator we used blocks that perform nearest-neighbor upsampling (BlockUp, see Figure 1b). In these blocks, if the convolutions change the number of filters, we additionally put a 1 × 1 convolution in the skip-connection path. For the discriminator D X and the encoder, we used average-pooling downsampling residual blocks (BlockDown, see Figure 1c). We also added Minibatch Standard Deviation layer [1] in the discriminator D X to improve capturing of variation of image data distribution. The discriminator D Z consists of three fully connected layers with Leaky ReLU activations.
The network architecture for COIL-100, MNIST and fMNIST is shown in Table 1, and further, deeper architectures are demonstrated for CIFAR10 in Table 2, for LSUN in Table 3, and for CelebA in Table 4.


extfieldsofstudy:
Computer Science
Physics

text:
DA-U-Net: Densely Connected Convolutional Networks and Decoder with Attention Gate for Retinal Vessel Segmentation

The segmentation of retinal vessels is greatly significant for doctors to diagnose the fundus diseases. However, existing methods have various problems in the segmentation of the retinal vessels, such as insufficient segmentation of retinal vessels, weak anti-noise interference ability. Aiming to the shortcomings of existed methods, this paper proposes an improved model based on the U-Net networks, which contains densely-connected convolutional network and a novel attention gate (AG) model, referred as Densely-Attention-U-Net (DA-U-Net), to automatically segment the retinal blood vessels. The method can alleviate the vanishing-gradient problem, strengthen feature propagation, substantially reduce the number of parameters, and automatically learn to focus on target structures without additional supervision. By verifying the method on the DRIVE datasets, the segmentation accuracy rate is 96.09%, higher than that of U-Net and R2U-Net.

Introduction
Medical image segmentation is a special computer vision area that is very important for many real-life applications. The fundus contains a large number of blood vessels, which is a deep microvascular system that can be directly observed in the human body without damage. It can provide doctors with a wealth of information about eye conditions and general system status. Ophthalmologists can detect early increases in the systemic vascular load caused by hypertension and diabetes, as well as early signs of vascular structural abnormalities such as retinal vein occlusion and retinal artery occlusion, the diseases caused by blood vessels and vascular systems can cause blindness. With the development of technology, it has been widely studied to explore an automatic method to segment retinal vessels, which can help doctors diagnose and analyze the patient's conditions and make early prevention and treatment of the fundus disease, so as to effectively avoid the visual loss caused by the degenerative disease. In many cases of biomedical applications, a small number of objects is to be found, on the other hand, only small datasets can be acquired, class imbalance is present, and very high recognition quality and robustness is required [1]. Convolutional neural networks (CNN) have already demonstrated their success in image classification, image segmentation and object detection. For almost any computer vision problems, CNN-based approaches outperform other techniques and in many cases even human experts in the corresponding field. In this work we intend to provide a new approach to the medical image segmentation tasks, which is based on well-known and highly-performing U-Net [2] convolutional neural network (CNN) of encoder-decoder style where initial convolutional feature maps are skip-connected [3] to upsampled layers from bottleneck layers. This skip-connection is crucial to segmentation tasks as the initial feature maps maintain low-level features. We refer to the neural network built as DA-U-Net. With the advent of convolutional neural networks (CNNs), near-radiologist level performance can be achieved in automated medical image analysis tasks. However, this approach leads to excessive and redundant use of computational resources and model parameters. For instance, similar low-level features are repeatedly extracted by all models within the cascade. To address this general problem, we propose a simple and yet effective solution, namely attention gates (AGs), and we replace the convolution layer with a Denseblock at the same time. CNN models with AGs can be trained from scratch in a standard way similar to the training of a FCN model, and AGs automatically learn to focus on target structures without additional supervision. The Denseblock can alleviate the vanishinggradient problem, strengthen feature propagation, encourage feature reuse, and substantially reduce the number of parameters. We demonstrate the implementation of AG and the Denseblock module in a standard U-Net architecture (DA-U-Net) and evaluate DA-U-Nets on DRIVE datasets, very high segmentation quality has been achieved.

Related Work
The retinal vascular network is the only deep blood vessel that carl be observed directly with noninvasive means in the human body. The changes of characteristics or morphology in the microvasculature can be leaded by any pathological changes of systemic and hematological characteristics. The segmentation of retinal blood vessels is the key step in retinal image processing and analysis, it has a good research value for the early prevention and diagnosis of systemic and hematological diseases. The characteristics of retinal images are complex, the automatic segmentation of retinal blood vessels is easy to be affected by external conditions and pathological changes, and in the retinal images, the difficulty of segmentation is increased by the reason that contrast ratio of tiny vessels and its background is low, therefore, to improve the segmentation accuracy is an important topic to research. Convolutional neural networks (CNNs) have become the dominant machine learning approach for visual object. We embrace this observation and introduce the Densely Convolutional Network (DenseNet), which connects each layer to every other layer in a feed-forward fashion. Whereas traditional convolutional networks with L layers have L connections-one between each layer and its subsequent layer-our network has L (L2+1) direct connections. For each layer, the feature-maps of all preceding layers are used as inputs, and its own feature-maps are used as inputs into all subsequent layers. Whilst following a simple connectivity rule, DenseNets naturally integrate the properties of identity mappings, deep supervision, and diversified depth. They allow feature reuse throughout the networks and can consequently learn more compact. DenseNets have several compelling advantages: they alleviate the vanishing-gradient problem, strengthen feature propagation, encourage feature reuse, and substantially reduce the number of parameters [4]. The Denseblock can be summarised as follows Figure 1:  Attention Gates: AGs are commonly used in natural image analysis, knowledge graphs, and language processing (NLP) for image captioning [5], machine translation [6], and classification [7,8] tasks. Initial work has explored attention-maps by interpreting gradient of output class scores with respect to the input image. Trainable attention, on the other hand, is enforced by design and categorised as hard-and soft-attention. Hard attention, iterative region proposal and cropping, is often non-differentiable and relies on reinforcement learning for parameter updates, which makes model training more difficult. Recursive hard-attention is used in [9] to detect anomalies in chest X-ray scans. Contrarily, soft attention is probabilistic and utilises standard back-propagation without need for Monte Carlo sampling. For instance, additive soft attention is used in sentence-to-sentence translation [10] and more recently applied to image classification [7,8]. Channel-wise attention is used to highlight important feature dimensions, which was the top-performer in the ILSVRC 2017 image classification challenge. Self-attention techniques have been proposed to remove the dependency on external gating information. For instance, non-local self attention is used in [11] to capture long range dependencies. The Attention Gate can be summarised as follows Figure 2:

Approach
In this experiment, we have 40 retinal vessels images, and we use 30 images to train,5 images to validation, and 5 images to test. Data augmentation is essential to teach the network the desired invariance and robustness properties, when only few training samples are available. In case of microscopical images we primarily need shift and rotation invariance as well as robustness to deformations and gray value variations. Especially random elastic deformations of the training samples seem to be the key concept to train a segmentation network with very few annotated images. Data Augmentation refers to the method of expanding the original data through a series of random transformations to increase the amount of data. The new data set generated by this method that cannot find any two identical images. This helps to suppress overfitting and thus improves The generalization ability of the deep learning model. This paper selects translation, rotation, shearing and scaling for data augmentation. The presented improved model, which we refer to as DA-U-Net, is depicted on Figure 4. It consists of basic blocks, and each of them follows the encoderdecoder architecture similar to U-Net [2]. We consider two kinds of basic blocks: Denseblock and Attention gate. They both feature connections (shown on the Figure 4), linking layers of the encoder and decoder, which are of very high importance.
In this paper, we replace the normal convolution layers as the Bottleneck layers in our network.
Although each layer only produces k output feature-maps, it typically has many more inputs. It has been noted in [12,13] that a 1×1 convolution can be introduced as bottleneck layer before each 3×3 convolution to reduce the number of input feature-maps, and thus to improve computational efficiency. We find this design especially effective for DenseNet and we refer to our network with such a bottleneck layer to the BN-ReLU-Conv (3×3), as shown in the Figure 3. And in the stage of the   Schematic of the AGs is shown in Figure 2. Feature selectivity in AGs is achieved by use of contextual information (gating) extracted in coarser scales.

Experiment
Experimental data provided by the DRIVE datasets, which is consisted of 40 color retinal images in total, in which 20 samples are used for training and remaining 20 samples are used for testing. The size of each original image is 565×584 pixels [14]. The data set is 900 sets of images generated after adopting the data augmentation method. In the training stage, we choose the Adam as the optimization, and the learning rate is 0.0001. The experimental platform is based on Windows 7 64-bit operating system, i7-4790 CPU, 16G memory, GTX 750, using a deep learning framework for TensorFlow. In order to further prove the effectiveness of the proposed method on retinal vascular segmentation, the comparison with the best found methods for the segmentation of retinal vessels in DRIVE datasets is presented in Table 1. Sensitivity (SE) indicates the percentage of correctly segmented blood vessel pixels as a percentage of true blood vessel pixels, and the specificity (SP) is the percentage of correctly segmented background pixels to the true background pixels. The accuracy (ACC) is the percentage of correctly segmented blood vessels and background pixels in the entire image.
Table1. Segmentation performance of retinal vessel on the DRIVE datasets.
The different prediction between the DA-U-Net and U-Net in the DRIVE can be made based on Figure 5.

Conclusion and Discussion
We present the model for the medical image segmentation based on Denseblock and Attention gate used in the well-konwn U-Net models. And the Segmented result by DA-U-Net is obviously better than the U-Net. We verified the feasibility of the proposed method on the DRIVE datasets, and the accuracy rates were 96.09%. By analyzing and comparing the segmented retinal blood vessel images, the proposed method is more advantageous than other methods. It can provide doctors with a wealth of information about eye conditions to help patients receive timely treatment. Still, our results fail todetect very thin vessels that span only 1 pixel. We expect that additional prior knowledge on the vessel structures such as connectivity may leverage the performance further. The use of artificial intelligence technology for medical image-assisted diagnosis is a rapidly developing field. With the rapid development of deep learning technology and the continuous opening of medical imaging data, artificial intelligence is gradually maturing in early stage disease screening and doctor-assisted diagnosis.