DeepViT: Towards Deeper Vision Transformer

Vision transformers (ViTs) have been successfully applied in image classification tasks recently. In this paper, we show that, unlike convolution neural networks (CNNs)that can be improved by stacking more convolutional layers, the performance of ViTs saturate fast when scaled to be deeper. More specifically, we empirically observe that such scaling difficulty is caused by the attention collapse issue: as the transformer goes deeper, the attention maps gradually become similar and even much the same after certain layers. In other words, the feature maps tend to be identical in the top layers of deep ViT models. This fact demonstrates that in deeper layers of ViTs, the self-attention mechanism fails to learn effective concepts for representation learning and hinders the model from getting expected performance gain. Based on above observation, we propose a simple yet effective method, named Re-attention, to re-generate the attention maps to increase their diversity at different layers with negligible computation and memory cost. The pro-posed method makes it feasible to train deeper ViT models with consistent performance improvements via minor modification to existing ViT models. Notably, when training a deep ViT model with 32 transformer blocks, the Top-1 classification accuracy can be improved by 1.6% on ImageNet. Code is publicly available at https://github.com/zhoudaquan/dvit_repo.

Introduction
Recent studies [7,37] have demonstrated that transformers [38] can be successfully applied to vision tasks [18] with competitive performance compared with convolutional neural networks (CNNs) [9,35]. Different from CNNs that aggregate global information by stacking multiple convolutions (e.g., 3 × 3) [9,10], vision transformers (ViTs) [7] take advantages of the self-attention (SA) mechanism [38] to capture spatial patterns and non-local dependencies. This allows ViTs to aggregate rich global information [7] Figure 1: Top-1 classification performance of vision transformers (ViTs) [7] on ImageNet with different network depth {12, 16, 24, 32}. Directly scaling the depth of ViT by stacking more transformer blocks cannot monotonically increase the performance. Instead, the model performance saturates when going deeper. In contrast, with the proposed Re-attention, our DeepViT model successfully achieves better performance when it goes deeper. without handcrafting layer-wise local feature extractions as CNNs and thus achieves better performance. For example, as shown in [37], a 12-block ViT model with 22M learnable parameters achieves better results than the ResNet-101 model which has more than 30 bottleneck convolutional blocks in ImageNet classification.
The recent progress of deep CNN models is largely driven by training very deep models with a large number of layers which is enabled by novel model architecture designs [9,41,31,20,47]. This is because a deeper CNN can learn richer and more complex representations for the input images and provide better performance on vision tasks [1,45,29]. Thus, how to effectively scale CNNs to be deeper is an important theme in recent deep learning fields, which stimulates the techniques like residual learning [9].
Considering the remarkable performance of shallow ViTs, a natural question arises: can we further improve performance of ViTs by making it deeper, just like CNNs?
Though it seems to be straightforward at the first glance, the answer may not be trivial since ViT is essentially different from CNNs in its heavy reliance on the self-attention mechanism. To settle the question, we investigate in detail the scalability of ViTs along depth in this work.
We start with a pilot study on ImageNet to investigate how the performance of ViT changes with increased model depth. In Fig. 1, we show the performance of ViTs [7] with different block numbers (green line), ranging from 12 to 32. As shown, as the number of transformer blocks increases, the model performance does not improve accordingly. To our surprise, the ViT model with 32 transformer blocks performs even worse than the one with 24 blocks. This means that directly stacking more transformer blocks as performed in CNNs [9] is inefficient at enhancing ViT models. We then dig into the cause of this phenomenon. We empirically observed that as the depth of ViTs increases, the attention maps, used for aggregating the features for each transformer block, tend to be overly similar after certain layers, which makes the representations stop evolving after certain layers. We name this specific issue as attention collapse. This indicates that as the ViT goes deeper, the self-attention mechanism becomes less effective for generating diverse attentions to capture rich representations.
To resolve the attention collapse issue and effectively scale the vision transformer to be deeper, We present a simple yet effective self-attention mechanism, named as Reattention. Our Re-attention takes advantage of the multihead self-attention(MHSA) structure and regenerates attention maps by exchanging the information from different attention heads in a learnable manner. Experiments show that, Without any extra augmentation and regularization policies, simply replacing the MHSA module in ViTs with Reattention allows us to train very deep vision transformers with even 32 transformer blocks with consistent improvements as shown in Fig. 1. In addition, we also provide ablation analysis to help better understand of the role of Reattention in scaling vision transformers.
To sum up, our contributions are as follows: • We deeply study the behaviour of vision transformers and observe that they cannot continuously benefit from stacking more layers as CNNs. We further identify the underlying reasons behind such a counter-intuitive phenomenon and conclude it as attention collapse for the first time.
• We present Re-attention, a simple yet effective attention mechanism that considers information exchange among different attention heads.
• To the best of our knowledge, we are the first to successfully train a 32-block ViT on ImageNet-1k from scratch with consistent performance improvement. We show that by replacing the self-attention module with our Re-attention, new state-of-the-art results can be achieved on the ImageNet-1k dataset without any pretraining on larger datasets.

Transformers for Vision Tasks
Transformers [38] are initially used for machine translation which replace the recurrence and convolutions entirely with self-attention mechanisms [28,14,48,11,39,17] and achieve outstanding performance. Later, transformers become the dominant models for various natural language processing (NLP) tasks [2,26,6,21]. Motivated by their success on the NLP tasks, recent researchers attempted to combine the self-attention mechanism into CNNs for computer vision tasks [40,3,4,32,24,50,49].. Those achievements also stimulate interests of the community in building purely transformer-based models (without convolutions and inductive bias) for vision tasks. The vision transformer (ViT) [7] is among the first attempt that uses the pure transformer architecture to achieve competitive performance with CNNs on the image classification task. However, due to the large model complexity, ViT needs to be pre-trained on largerscale datasets (e.g., JFT300M) for performing well on the ImageNet-1k dataset. To solve the data efficiency issue, DeiT [37] deploys knowledge distillation to train the model with a larger pre-trained teacher model. In this manner, vision transformer can perform well on ImageNet-1k without the need of pre-training on larger dataset. Differently, in this work, we target at a different problem with ViT, i.e., how to effectively scale ViT to be deeper. We propose a new design for the self-attention mechanism so that it can perform well on vision tasks without the need of extra data, teacher networks, and the domain specific inductive bias.

Depth Scaling of CNNs
Increasing the network depth of a CNN model is deemed to be an effective way to improve the model performance [30,33,34,10,15]. However, very deep CNNs are generally harder to train to perform significantly better than the shallow ones in the past [8,44]. How to effectively scale up the CNNs in depth was a long-standing and challenging problem [16]. The recent progress of CNNs largely benefits from novel architecture design strategies that make training deep CNNs more effective [9,35,13,12,36,52]. Transformer-alike models have modularized architectures and thus can be easily made deeper by repeating the basic transformer blocks or using larger embedding dimensions [2,19]. However, those straightforward scaling strategies only work well with larger datasets and stronger augmentation policies [51,46,43] to alleviate the brought training difficulties. In this paper, we observed that with the same dataset, the performance of vision transformers do saturate as the network depth rises. We rethink the self-attention mechanism and present a simple but effective approach to address the difficulties in scaling vision transformers.

Revisiting Vision Transformer
A vision transformer (ViT) model [37,7], as depicted in Fig. 2(a), is composed of three main components: a linear layer for patch embedding (i.e., mapping the highresolution input image to a low-resolution feature map), a stack of transformer blocks with multi-head self-attention and feed-forward layers for feature encoding, and a linear layer for classification score prediction. In this section, we first review its unique transformer blocks, in particular the self-attention mechanism, and then we provide studies on the collapse problem of self-attention.

Multi-Head Self-Attention
Transformers [38] were extensively used in natural language for encoding a sequence of input word tokens into a sequence of embeddings. To comply with such sequenceto-sequence learning structure when processing images, ViTs first divide an input image into multiple patches uniformly and encode each patch into a token embedding. Then, all these tokens, together with a class token, are fed into a stack of transformer blocks.
Each transformer block consists of a multi-head self-attention (MHSA) layer and a feed-forward multi-layer perceptron (MLP). The MHSA generates a trainable associate memory with a query (Q) and a pair of key (K)-value (V ) pairs to an output via linearly transforming the input. Mathematically, the output of a MHSA is calculated by: where √ d is a scaling factor based on the depth of the network. The output of the MHSA is then normalized and fed into the MLP to generate the input to the next block. In the above self-attention, Q and K are multiplied to generate the attention map, which represents the correlation between all the tokens within each layer. It is used to retrieve and combine the embeddings in the value V . In the following, we particularly analyze the role of the attention map in scaling the ViT. For convenience, we use A ∈ R H×T ×T to denote the attention map, with H being the number of SA heads and T the number of tokens. For the h-th SA head, the attention map is computed as with Q h and K h from the corresponding head. When the context is clear, we omit the subscript h.

Attention Collapse
Motivated by the success of deep CNNs [9,30,35,36], we conduct systematic study in the changes of the performance of ViTs as depth increases. Without loss of generality, we first fix the hidden dimension and the number of heads to 384 and 12 respectively 1 , following the common practice in [37]. Then we stack different number of transformer blocks (varying from 12 to 32) to build multiple ViT models corresponding to different depths. The overall performances for image classification are evaluated on Im-ageNet [18] and summarized in Fig. 1. As evidenced by the performance curve, we surprisingly find that the classification accuracy improves slowly and saturates fast as the model goes deeper. More specifically, we can observe that the improvement stops after employing 24 transformer blocks. This phenomenon demonstrates that existing ViTs have difficulty in gaining benefits from deeper architectures.
Such a problem is quite counter-intuitive and worth exploration, as similar issues (i.e., how to effectively train a deeper model) have also been observed for CNNs at its early development stage [9], but properly solved later [9,10]. By taking a deeper look into the transfromer architecture, we would like to highlight that the self-attention mechanism plays a key role in ViTs, which makes it significantly different from CNNs. Therefore, we start with investigating how the self-attention, or more concretely, the generated attention map A varies as the model goes deeper. Figure 3: (a) The similarity ratio of the generated self-attention maps across different layers. The visualization is based on ViT models with 32 blocks pre-trained on ImageNet. For visualization purpose, we plot the ratio of token-wise attention vectors with similarity in Eqn. (2) larger than the average similarity within nearest k transformer blocks. As can be seen, the similarity ratio is larger than 90% for blocks after the 17th one. (b) The ratio of similar blocks to the total number of blocks increases when the depth of the ViT model increases. (c) Similarity of attention maps from different heads within the same block. The similarity between different heads within the blocks is all lower than 30% and they present sufficient diversity.
To measure the evolution of the attention maps over layers, we compute the following cross-layer similarity between the attention maps from different layers: where M p,q is the cosine similarity matrix between the attention map of layers p and q. Each element M p,q h,t measures the similarity of attention for head h and token t. Consider one specific self-attention layer and its h-th head, A * h,:,t is a T -dimensional vector representing how much the input token t contributes to each of the T output tokens. M p,q h,t , thus, provides an appropriate measurement on how the contribution of one token varies from layer p to q. When M p,q h,t equals one, it means that token t plays exactly the same role for self-attention in layers p and q.
(2), we then train a ViT model with 32 transformer blocks on ImageNet-1k and investigate the above similarity among all the attention maps. As shown in Fig. 3(a), the ratio of similar attention maps in M after the 17th block is larger than 90% . This indicates that the learned attention maps afterwards are similar and the transformer block may degenerate to an MLP. As a result, further stacking such degenerated MHSA may introduce the model rank degeneration issue (i.e., the rank of the model parameter tensor from multiplying the layer-wise parameters together will decrease) and limits the model learning capacity. This is also validated by our analysis on the degeneration of learned features as shown below. Such observed attention collapse could be one of the reasons for the observed performance saturation of ViTs. To further validate the existence of this phenomenon for ViTs with different depths, we conduct the same experiments on ViTs with 12, 16, 24 and 32 transformer blocks respectively and calculate the number of blocks with similar attention maps. The re- sults shown in Fig. 3(b) clearly demonstrate the ratio of the number of similar attention map blocks to the total number of blocks increases when adding more transformer blocks.
To understand how the attention collapse may hurt the ViT model performance, we further study how it affects feature learning of the deeper layers. For a specific 32-block ViT model, we compare the final output features with the outputs of each intermediate transformer block by investigating their cosine similarity. The results in Fig. 4 demonstrate that the similarity is quite high and the learned features stop evolving after the 20th block. There is a close correlation between the increase of attention similarity and feature similarity. This observation indicates that attention collapse is responsible for the non-scalable issue of ViTs.

Re-attention for Deep ViT
As revealed above, one major obstacle in scaling up ViT to a deeper one is the attention collapse problem. In this sec- Figure 5: Impacts of embedding dimension on the similarity of generated self-attention map across layers. As can be seen, the number of similar attention maps decreases with increasing embedding dimension. However, the model size also increases rapidly.    tion, we present two solution approaches, one is to increase the hidden dimension for computing self-attention and the other one is a novel re-attention mechanism.

Self-Attention in Higher Dimension Space
One intuitive solution to conquer attention collapse is to increase the embedding dimension of each token. This will augment the representation capability of each token embedding to encode more information. As such, the resultant attention maps can be more diverse and the similarity between each block's attention map could be reduced. Without loss of generality, we verify this approach empirically by conducting a set of experiments based on ViT models with 12 blocks for quick experiments. Following previous transformer based works [38,7], four embedding dimensions are selected, ranging from 256 to 768. The detailed configurations and the results are shown in Tab. 1.
From Fig. 5 and Tab. 1, one can see that the number of blocks with similar attention maps is reduced and the attention collapse is alleviated by increasing the embedding dimension. Consequently, the model performance is also increased accordingly. This validates our core hypothesis-the attention collapse is the main bottleneck for scaling ViT. Despite its effectiveness, increasing the embedding di-mension also increases the computation cost significantly and the brought performance improvement tends to diminish. Besides, a larger model (with higher embedding dimension) typically needs more data for training, suffering the over-fitting risk and decreased efficiency.

Re-attention
It has been demonstrated in Sec. 3 that the similarity between attention maps across different transformer blocks is high, especially for deep layers. However, we find the similarity of attention maps from different heads of the same transformer block is quite small, as shown in Fig. 3(c). Clearly, different heads from the same self-attention layer focus on different aspects of the input tokens. Based on this observation, we propose to establish cross-head communication to re-generate the attention maps and train deep ViTs to perform better.
Concretely, we use the attention maps from the heads as basis and generate a new set of attention maps by dynamically aggregating them. To achieve this, we define a learnable transformation matrix Θ ∈ R H×H and then use it to mix the multi-head attention maps into re-generated new ones, before being multiplied with V . Specifically, the Reattention is implemented by: where transformation matrix Θ is multiplied to the selfattention map A along the head dimension. Here Norm is a normalization function used to reduced the layer-wise variance. Θ is end-to-end learnable.
Advantages: The advantages of the proposed Re-attention are two-fold. First of all, compared with other possible attention augmentation methods, such as randomly dropping some elements of the attention map or tuning SoftMax temperature, our Re-attention exploits the interactions among different attention heads to collect their complementary information and better improves the attention map diversity. This is also verified by our following experiments. Furthermore, our Re-attention is effective and easy to implement. It needs only a few lines of code and negligible computational overhead compared to the original self-attention. Thus it is much more efficient than the approach of increasing embedding dimension.

Experiments
In this section, we first conduct experiments to further demonstrate the attention collapse problem. Then, we give extensive ablation analysis to show the advantages of the proposed Re-attention. By incorporating Re-attention into the transformers, we design two modified version of vision transformers and name them as deep vision transform-ers (DeepViT). Finally, we compare the proposed DeepViT models against the latest state-of-the-arts (SOTA).

Experiment Details
To make a fair comparison, we first tuned a set of parameters for training the ViT base model and then use the same set of hyper-parameters for all the ablation experiments. Specifically, we use AdamW optimizer [23] and cosine learning rate decay policy with an initial learning rate of 0.0005. We use 8 Telsa-V100 GPUs and train the model for 300 epochs using Pytorch [25] library. The batch size is set to 256. We use 3 epochs for learning rate warm-up [22]. We also use some augmentation techniques such as mixup [46] and random augmentation [5] to boost the performance of baseline models following [47]. When comparing with other methods, we adopt the same set of hyper-parameters as used by the target models. We report results on the Im-ageNet dataset [18]. For all experiments, the image size is set to be 224×224. To study the scaling capability of current transformer blocks, we set the embedding dimension to 384 and the expansion ratio 3 for the MLP layers. We use 12 heads for all the models. More detailed configurations are shown in Tab. 2.

More Analysis on Attention Collapse
In this section, we show more analysis on the attention map similarity and study how the collapsed attention maps affect the model performance.
Attention reuse: As discussed above, when the model goes deeper, the attention maps of the deeper blocks become highly similar. This implies that adding more blocks on a deep ViT model may not improve the model performance. To further verify this claim, we design an experiment to reuse the attention maps computed at an early block of ViT to replace the ones after it. Specifically, we run experiments on the ViT models with 24 blocks and 32 blocks but share the Q and K values (and the resulted attention maps) of the last "unique" block to all the blocks afterwards. The "unique" block is defined as the block whose attention map's similarity ratio with adjacent layers is smaller than 90%. More implementation details can be found in the supplementary material. The results are shown in Tab. 3. Surprisingly, for a ViT model with 32 transformer blocks, when we use the same Q and K values for the last 15 blocks, the performance degradation is negligible. This implies the attention collapse problem indeed exists and reveals the inefficacy in adding more blocks when the model is deep. Visualization: To more intuitively understand the attention map collapse across layers, we visualize the learned attention maps from different blocks of the original ViT [7]. We take a 32-block ViT model as an example and pre-train it on ImageNet. The visualization of the attention maps with original MHSA and Re-attention are shown in Fig. 6. It can be observed that the original MHSA learns the local relationship among the adjacent patches in the shallow blocks and the attention maps tend to expand to cover more patches gradually. In the deep blocks, the MHSA learns nearly uniform global attention maps with high similarity. Differently, after implementing Re-attention, the attention maps at deep blocks keep the diversity and have small similarities from adjacent blocks.

Analysis on Re-attention
In this subsection, we present two straightforward modifications to the current self-attention mechanism as baselines. We then conduct a series of comparison experiments to show the advantages of our proposed Re-attention.
Re-attention v.s. Self-attention: We first evaluate the effectiveness of Re-attention by comparing to the pure ViT models using the same set of training hyper-parameters. We directly replace the self-attention module in ViT with Reattention and show the results in Tab. 4 with different number of transformer blocks. As can be seen, the vanilla ViT architecture suffers performance saturation when adding more transformer blocks. This phenomenon coincides with our observations that the number of blocks with similar attention maps increases with the depth as shown in Fig.  3(b). Interestingly, when replacing the self-attention with our proposed Re-attention, the number of similar blocks are all reduced to be zero and the performance rises consistently as the model depth increases. The performance gain is especially significant for deep ViT with 32 blocks. This might be explained by the fact that the 32 block ViT model has the largest number of blocks with similar attention maps and the improvements should be proportional to the number sim-  Our proposed re-attention mechanism. As shown, the original attention map is mixed via a learnable matrix Θ before multiplied with values.
ilar blocks in the model. These experiments demonstrate that the proposed Re-attention can indeed solve the attention collapse problem and thus enables training a very deep vision transformer without extra datasets or augmentation policies. Comparison to adding temperature in self-attention: The most intuitive way to mitigate the over-smoothing phenomenon is to sharpen the distribution of the elements in the attention map of MHSA. We could achieve this by as-signing a temperature τ to the Softmax layer of MHSA: As the attention collapse is observed to be severe on deep layers (as shown in Fig. 3), we design two sets of experiments on a ViT model with 32 transformer blocks: (a) linearly decaying the temperature τ in each block such that the attention map distribution is sharpened and (b) making the temperature learnable and optimized together with the model training. We first check the impact of the Soft-Max temperature on reducing the attention map similarity. As shown in Fig. 8(a), the number of similar blocks are still large. Correspondingly, the feature similarity among blocks are also large as shown in Fig. 8(b). Thus, adding a temperature to the SoftMax only reduces the attention map similarity by a small margin. The classification results on ImageNet are shown in Tab. 5. As shown, using a learnable temperature could improve the performance but the improvement is marginal. Comparison to dropping attentions: Another baseline we have attempted to differentiate the self-attention maps across layers is to use random dropout on the attention maps A. As the dropout will mask out different positions on the attention maps for different blocks, the similarity between attention maps could be reduced. The impacts on the attention maps and the output features of each block are shown in Fig. 8(a-b). It is observed that dropping attention does reduce the cross layer similarity of the attention maps. However, the similarity among features are not reduced by much. This is because the difference between attention maps comes from the zero positions in the generated mask. Those zero values do reduce the similarity between attention maps but not contribute to the features. Thus, the improvement is still not significant as shown in Tab. 5.
Advantages of Re-attention: Our proposed Re-attention brings more significant improvements over the temperaturetuning and the attention-dropping methods. This is because both adding temperature and dropping attention are regularizing the distribution of the originally over-smoothed selfattention maps, without explicitly encouraging them to be diverse. However, our proposed Re-attention mechanism uses different heads (whose attention maps are dissimilar) as basis and re-generate the attention maps via the transformation matrix Θ. This process incorporates the interhead information communication and the generated attention maps can encode richer information. It is worth noting that the original MHSA design can be thought as a special case of Re-attention with an identity transformation matrix. By making Θ learnable for each block, an optimized pattern could be learned end to end. As shown in Fig. 8(c), the learned transformation matrix assigns a diverse set of weights for each newly generated head. It clearly shows that the combination for each new heads takes different weights from the original heads in the re-attention process and thus reduces the similarity between their attention maps. As shown in Fig. 8(a), our proposed Re-attention achieves the lowest cross layer attention map similarity. Consequently, it also reduces the feature map similarity across layers as shown in Fig. 8(b).

Comparison with other SOTA models
With Re-attention, we design two ViT variants, i.e., DeepViT-S and DeepViT-L, based on the ViT with 16 and 32 transformer blocks respectively. For both models, we use Re-attention to replace the self-attention. To have a similar number of parameters with other ViT models, we adjust the embedding dimension accordingly. The hidden dimensions of DeepViT-S and DeepViT-L models are set as 396 and 408 respectively. More details on the model configuration are given in the supplementary material. Besides, motivated by [42], we add three CNN layers for extracting the token embeddings, using the same configurations as [42]. It is worth noting that we do not use the optimized training recipes and the repeated augmentation as [37] for training our models. The results are shown in Tab. 6. Clearly, our DeepViT model achieves higher accuracy with less parameters than the recent CNN and ViT based models. Notably, without any complicated architecture change as made by T2T-ViT [42] (adopting a deep-narrow architecture) or DeiT [37] (introducing token distillation), simply using the Re-attention makes our DeepViT-L outperforms them by 0.4 points with even smaller model size (55M vs. 64M & 86 M).

Conclusion
In this work, we found the attention collapse problem of vision transformers as they go deeper and propose a novel Re-attention mechanism to solve it with minimum amount of computation and memory overhead. With our proposed Re-attention, we are able to maintain an increasing performance when increasing the depth of ViTs. We hope our observations and methods could facilitate the development of vision transformers in future.

A. Experiment Implementation Details
Attention reuse: As shown in Fig. 3(b) and Tab. 3 in the main paper, the vision transformers with 24 blocks and 32 blocks have 11 and 15 blocks with similar attention maps, respectively. To verify the effectiveness of the attention maps from those blocks, we directly force those blocks to share the same attention map as the last 'unique' block as defined in Sec. 5.2. Specifically, we take the attention map of the last 'unique' block and denote it as A unique . For all the following blocks, the attention output is calculated by: where Θ is used to simulate the small variance between attention maps across layers since they are not identical. Norm is batch normalization used to adjust the variance across layers. As shown in Tab. 3, for a ViT with 32 blocks, forcing the top 15 blocks to share the same attention map causes negligible degradation on the classification accuracy on ImageNet. This proves that adding those blocks do not contribute to the accuracy improvement.
Training loss: We use the cross-entropy (CE) loss as the training loss for all experiments. To minimize the similarity of the attention maps across layers, we add the cosine similarity between layers into the loss function when training the model.
where Similarity(A l , A l+1 ) denotes the cosine similarity between layer l and l + 1 and A l denotes the attention map of layer l. B denotes the number of bottom blocks used for regularization and is a hyper-parameter. We set B to 4, 8 and 12 for training ViT models with 16, 24 and 32 blocks respectively.

B. DeepViT architecture design
As observed in Fig. 3(a), the attention maps of the transformer blocks become similar only at the top blocks. Thus, it is not necessary to apply re-attention to all blocks. To study the optimal number of blocks with re-attention, we conduct a set of experiments on a ViT model with 16 transformer blocks. For each experiment, we only apply reattention on the top K blocks where K ranges from 5 to 15. The rest of the blocks are using the original transformer block structure. We train each model on ImageNet with the same set of training hyper-parameters as those for baseline models as detailed in Sec. 5 in the main paper. The results are shown in Fig. 9.
It is observed that, as the number of re-attention blocks varies, the top-1 classification accuracy changes correspondingly. The highest accuracy appears at the position where the number of re-attention blocks is the same as the number of similar attention map blocks. Based on this observation, we define the architecture of DeepViT-S and DeepViT-L with 5 and 12 re-attention blocks respectively. Detailed configurations are shown in Tab. 7. Note that we adjust the embedding dimension to have a comparable size with other methods.

C. Impacts of hyper-parameters
In the main paper, all experiments are run with the same set of training hyper-parameters as the one used for reproducing ViT models. However, as shown in [37], an improved training recipe could improve the performance of ViT models significantly. In Tab. 8, we present the performance of DeepViT-S and DeepViT-L with the same set of training recipes as DeiT except that we do not use repeated augmentation. In Tab. 8, it is clearly shown that the performance of DeepViT could be further improved with optimized training hyper-parameters.

D. Similarity calculation
Cosine similarity between layers To measure the similarity between the attention maps, we define the similarity S p,q between the attention maps of two layers, p and q, as the ratio of the number of similar vector pairs to the total number of pairs between two attention maps: where τ is a hyper-parameter and used as a threshold for deciding similar vectors 2 .
Definition of similar blocks A block is counted as a similar block if the similarity between it's attention map and the adjacent block's attention map is larger than 80%. To measure the block similarity for a ViT model with B blocks, we take the ratio of the number of similar blocks to the total number of blocks as a measurement.


extfieldsofstudy:
Computer Science
Medicine

text:
Accurate pancreas segmentation using multi-level pyramidal pooling residual U-Net with adversarial mechanism

Background A novel multi-level pyramidal pooling residual U-Net with adversarial mechanism was proposed for organ segmentation from medical imaging, and was conducted on the challenging NIH Pancreas-CT dataset. Methods The 82 pancreatic contrast-enhanced abdominal CT volumes were split via four-fold cross validation to test the model performance. In order to achieve accurate segmentation, we firstly involved residual learning into an adversarial U-Net to achieve a better gradient information flow for improving segmentation performance. Then, we introduced a multi-level pyramidal pooling module (MLPP), where a novel pyramidal pooling was involved to gather contextual information for segmentation, then four groups of structures consisted of a different number of pyramidal pooling blocks were proposed to search for the structure with the optimal performance, and two types of pooling blocks were applied in the experimental section to further assess the robustness of MLPP for pancreas segmentation. For evaluation, Dice similarity coefficient (DSC) and recall were used as the metrics in this work. Results The proposed method preceded the baseline network 5.30% and 6.16% on metrics DSC and recall, and achieved competitive results compared with the-state-of-art methods. Conclusions Our algorithm showed great segmentation performance even on the particularly challenging pancreas dataset, this indicates that the proposed model is a satisfactory and promising segmentor.

Background
Organ segmentation from medical imaging is recognized as a difficult job, since the contours of organs tend to be indistinguishable from the background gray low-resolution images. Especially for organs with small volume and varied morphology, such as pancreas [1]. As deep learning thrives, convolutional neural networks (CNNs) show great potential on organ segmentation tasks and various methods based on CNNs have been raised for pancreas segmentation [2][3][4][5][6][7][8][9][10][11][12][13][14]. Zhang et al. incorporated multi-atlas registration and level-set for a pancreas segmentation framework, which mainly contained coarse, fine, and refine stages and utilized both the 3D and 2D information for segmentation [2]. Mo et al. proposed an iterative 3D feature enhancement network, which suppressed the non-target areas and increased the fine target details by leveraging the information in different layers, to segment pancreas from CT images [3]. Liu et al. raised an ensemble fully convolutional neural network for pancreas segmentation, which firstly generated candidate region and then combined output probability maps from five segmentation networks with different energy functions [4]. Roth et al. presented an automated holisticallynested convolutional network for pancreas localization Open Access *Correspondence: guosx@jlu.edu.cn 1 College of Electronic Science and Engineering, Jilin University, Changchun 130012, China Full list of author information is available at the end of the article and segmentation from 3D CT scans [5]. Zhou et al. put forward a fixed-point model for pancreas segmentation in abdominal CT scans through shrinking the input region by a predicted segmentation mask [6]. Cai et al. introduced a recurrent neural network to address the contextual learning and segmentation consistency problem and applied a novel Jaccard Loss to optimize training for improving pancreas segmentation performance [7]. Oktay et al. showed an attention gate model which is capable of automatically focusing on targets in medical imaging. The involvement of this attention gate into models conduces to suppressing irrelevant regions while highlighting useful features for a specific task. The U-Net model trained with the attention gate performs highly beneficial performance on 3D CT scans pancreas dataset [8]. These schemes indicate that the involvement of CNNs variants is effective for pancreas segmentation task.
Segmentation methods based on 3D networks are usually time-consuming and require more advanced server configuration. However, 2D segmentation models are not adept at capturing spatial information, thus tend to be limited in final segmentation performance. To solve this issue, we proposed a multi-level pyramidal pooling residual U-Net with adversarial mechanism for accurate pancreas segmentation in this paper. For this objective, we incorporated residual learning [15] into an adversarial U-Net to optimize the gradient information flow, thus improved its segmentation performance. Then, we put forward a multi-level pyramidal pooling module (MLPP) to collect more contextual information in search of a structure with better performance. Specifically, we introduced a novel pyramidal pooling into the adversarial U-Net with residual blocks as a substitute for the conventional pooling layer. And in order to obtain finer clues, we further introduced four groups of network structures with different numbers of pooling blocks for fusing features from different scales. To verify the effectiveness and stableness of MLPP in pancreas segmentation task, we applied pooling layers with different sizes to constitute two sets of MLPP and successively tested their segmentation performance. Overall, our proposed 2D segmentation model addresses the time-consuming and high configuration requirements of 3D networks. Then, the involvement of adversarial learning offsets the spatial information loss in simple 2D segmentation networks, depending on the ability of generative adversarial network to capture high-dimensional data distributions. And the introduction of multi-level pyramidal pooling module (MLPP) helps to collect more contextual information for segmentation. These steps contribute to compensate simple 2D segmentation networks for their spatial information loss.
The structure of the remaining paper was organized as follows. Firstly, we introduced the materials and methods used in this paper. Then, we set several groups of experiments and showed the specific results. Finally, we made a discussion and a conclusion for this work.

Data
We evaluated our proposed model on a public Pancreas-CT dataset [16,17], which is collected by the National Institutes of Health Clinical Center (NIH). It contains 82 contrast-enhanced abdominal CT volumes, where each scan is 512 × 512 × L and L represents slices number along the long axis. We resized the scans to [208,224] referring to the general scopes of label. Following the training protocol in [6,7,18], our experiments were carried on four-fold cross validation to randomly split these 82 patients. That is, three folds of patients were used as

Residual U-Net with adversarial mechanism
A typical generative adversarial network [19] is consisted of a discriminator and a generator. These two networks compete with each other in a minmax two-player game, as defined in Eq. (1). And in this formula, D and G respectively represent the discriminator and the generator. D(x) refers to the probability that the input x comes from the original dataset rather than the synthetic samples.
In the training process, the objective of the discriminator is to distinguish the real images from the synthetic samples produced by the generator, no matter how semblable they are. The generator aims to produce samples as realistic as possible to cheat the discriminator. During this process, the discriminator and the generator optimize their own network simultaneously until achieving Nash equilibrium [20]. A generative adversarial network is capable of capturing high-dimensional spatial information distributions through its specific competitive mechanism. Therefore, we introduced adversarial learning into conventional U-Net to obtain an adversarial U-Net to offset the spatial information loss in simple segmentation network, thus collecting much more useful information for pancreas segmentation.
Residual network proposed by He et al. [15] is to solve the problems of the increasing training error and degrading network performance with the network deepness added. In order to optimize the gradient information flow of our adversarial U-Net to further improve its segmentation performance, we incorporated residual learning into our proposed model.
Specifically, the U-Net proposed by Ronneberger et al. [21] was used as a basic framework in this work. Based on this segmentation network, we involved adversarial  [19] to further ensure the distributions of produced volumes resemble that of the ground truth images for a better segmentor, namely adversarial U-Net. In adversarial U-Net, a segmentation network (i.e., U-Net) is used as the generator in a generative adversarial network. Thus, the synthetic volumes produced by the generator refer to the obtained probability maps from the segmentation network. The adversarial network used here is consisted of five convolutional layers, and the kernel sizes are 7 × 7, 5 × 5, 4 × 4, 4 × 4, and 4 × 4 from the first to the fifth layer. The energy functions of the generator and discriminator in adversarial U-Net were respectively defined in Eqs. (2) and (3). N is the total amount of images. F n and R n respectively represent the fake samples from the segmentation network and the original images from the NIH Pancreas-CT dataset. D refers to the discriminator while G represents the generator (i.e., segmentor). Then, in order to further improve segmentation performance, we introduced residual learning into the adversarial U-Net, which contributes to a better gradient information flow within network. The specific structure of the involved residual block was shown in Fig. 1.

Multi-level pyramidal pooling module
Furthermore, we introduced a multi-level pyramidal pooling module (MLPP) to collect more useful details for segmentation. Specifically, inspired by [22], we involved a novel pyramidal pooling into (2) aforementioned segmentor (i.e., adversarial U-Net with residual blocks) for capturing contextual information in the model training process. Then, we set four groups of network structures with different numbers of pyramidal pooling blocks for fusing features from different scales, and selected the one with the optimal performance acted as the final version of our proposed algorithm. That is, in the first network, namely (i.e., B_ P1), we placed one pooling block after the fourth convolutional layer. In the second network (i.e., B_P2), we placed one block after the third and the fourth convolutional layer severally. In the third network (i.e., B_P3), we placed one block after the second, the third and the fourth convolutional layer severally. In the fourth network (i.e., B_P4), we placed one block after the first, the second, the third and the fourth convolutional layer severally. To assess the robustness of MLPP for pancreas segmentation task, we severally applied two kinds of updated pooling blocks based on the conventional pooling block displayed in Fig. 2a, to constitute MLPP and successively verified their segmentation performance. The structures of these two pyramidal pooling blocks (B1 and B2) were respectively shown in Fig. 2b, c.

Experimental environment
All experiments in this work were built on PyTorch [23] and implemented on an NVIDIA GeForce GTX 1080Ti graphics card with 11 GB memory. In the training phase, Adam was used as optimizer with initial learning rate 0.0001 and batch size was set to 1.

Evaluation metrics
To evaluate the segmentation performance of our proposed model, we used DSC and recall as evaluation metrics in this paper. DSC reflects the overlap between the output probability maps obtained from segmentation networks and their corresponding ground truths, as defined in Eq. (4). Let X represents the voxels in obtained probability maps while Y denotes the voxels in ground truths.  Recall measures the fractions of correctly predicted pixels in the total number of pancreas pixels, with its definition in Eq. (5) as below:

Residual U-Net with adversarial mechanism
In order to verify the effectiveness of residual learning, we firstly compared our proposed adversarial U-Net with residual blocks to a conventional segmentation network U-Net [21] and an adversarial U-Net. The segmentation results comparison was shown in Fig. 3. And the specific numerical values of these three models on metrics DSC and Recall were displayed in Table 1.

Multi-level pyramidal pooling module
In our proposed multi-level pyramidal pooling module (MLPP), we severally set one, two, three, and four pyramidal pooling blocks to obtain multi-scale feature representations, and picked out the one with optimal performance acted as the final network version. MLPP was severally consisted of two types of pooling structures: B1 and B2, as shown in Fig. 2. The specific numerical results were respectively displayed in Tables 2 and 3. Figure 4 showed the segmentation results from the optimal network ADR-U-Net with four pooling layers based on pyramidal pooling block B1 while Fig. 5 exhibited the segmentation results from the optimal ADR-U-Net with four pooling layers based on pyramidal pooling block B2.

Compared with the state-of-the-art methods
To evaluate the segmentation performance of our proposed model on pancreas segmentation task, we compared it with the state-of-the-art pancreas segmentation methods. The probabilistic bottom-up approach [18] based on multi-level deep convolutional networks achieves a DSC score of 71.8%, which is 11.23% lower than our proposed method. The holistically-nested convolutional neural network proposed by Roth et al. [5] obtains a mean DSC score of 81.27% and is inferior to our method 55% lower than our model. Also, our optimal DSC score in this work is 0.63% higher than the recurrent neural network architecture raised by Cai et al. [7] which addresses the contextual learning and segmentation consistency problem for pancreas segmentation, while our proposed method exceeds 0.66% on evaluation index DSC than the fixed-point model which involves a predicted segmentation mask to shrink the input region for a better performance when segmenting pancreas form abdominal CT scans in [6]. All these instances demonstrate that our proposed model is a powerful segmentor even on the tricky pancreas segmentation mission.

Residual U-Net with adversarial mechanism
In this work, we proposed a multi-level pyramidal pooling residual U-Net with adversarial mechanism to achieve accurate organ segmentation, and we conducted our proposed model on a particularly challenging dataset (i.e., NIH Pancreas-CT) to assess its segmentation performance. Specifically, we firstly involved a discriminator into a conventional U-Net to obtain an adversarial U-Net, and used this structure as the baseline architecture in our proposed segmentation model. Then, we introduced residual learning into the baseline segmentor to optimize gradient information flow for improving model performance. Figure 3 shows the segmentation results from U-Net, adversarial U-Net, and adversarial U-Net with residual blocks. It is obvious that the output probability volumes from the adversarial U-Net with residual blocks preferably resemble the ground truths in a myriad of subtle parts compared with U-Net and adversarial U-Net, which illustrates that the involvement of adversarial learning and residual blocks are all advantageous to improve pancreas segmentation performance.

Multi-level pyramidal pooling module
Furthermore, we introduced a multi-level pyramidal pooling module (MLPP) into the residual U-Net with adversarial mechanism to capture more contextual information for pancreas segmentation. Table 2 shows that B1_P4 precedes 1.12%, 0.65%, and 0.34% on DSC while 2.22%, 1.01%, and 0.98% on Recall than structures B1_P1, B1_P2, and B1_P3. This indicates that B1_P4 achieves the optimal segmentation performance compared to other three models as more contextual information are collected in B1_P4. The numerical values in Table 3 display the same trend as the Table 2, which further verify the above conclusion that B1_P4 performs the best on pancreas segmentation among these four sets of models. Thus, the structure B1_P4 is used as the final version of our segmentation model. Compared with the state-of-the-art methods for pancreas segmentation, our proposed model shows competitive results as displayed in experimental parts. There is still room, however, for our proposed segmentation network. As our model is 1.47%, and 2.43% lower than the methods proposed in [24,25]. Thus, future work will upgrade this existing model to quest a better segmentation performance on pancreas segmentation, and apply our model on different datasets to confront different kinds of organs segmentation tasks.

Conclusions
In this paper, we exploited a novel multi-level pyramidal pooling residual U-Net with adversarial mechanism for organ segmentation, and operated this proposed model on a particularly challenging NIH Pancreas-CT dataset. Experimental results show that our method achieves great improvements than the basic network and obtains satisfactory performance compared with the-state-of-art segmentation models even for the particularly challenging Pancreas-CT dataset. This also indicates that our proposed segmentation model is an effective and promising network for organ segmentation.


s2fieldsofstudy:
Computer Science
Engineering

text:
Power System Small-signal Stability Assessment Model Based on Residual Graph Convolutional Networks

Small-signal stability (SSA) is important to power system security. A data-driven approach is established for rapid prediction of the power system oscillation characteristics. The key of the approach is the Graph Convolution Networks (GCN) with residual mechanism, which works to aggregate features from high-dimension steady-state operation information and is denoted as ResGCN (RESidual GCN) in the paper. The residual mechanism helps to overcome the network degradation phenomenon. Both the oscillation frequency and damping ratio of multiple modes can be predicted by the proposed model. The performance of the proposed scheme as well as its adaptability to the power system topological changes is verified on the IEEE 39 Bus system.

Introduction
With the increase of renewable energy resources and HVDC converters in modern power systems, small-signal stability becomes more and more important for power system secure operation. Online small-signal stability assessment (SSA) is now an essential part in the security defense framework of the power dispatch center, which works to predict the power system oscillation mode and its damping characteristics quickly and accurately. Traditionally, SSA is solved by mathematical models based on physical principles, which can be divided into two categories. One is the eigen-analysis method based on linearization model [1][2]. Presently, the integration of a large number of converters poses severe challenges to this type of algorithms. Another type of approaches aim to extract the oscillation frequency and damping ratio from the oscillation curves obtained by the time-domain simulation with nonlinear models [3]. The drawback is that the results are subjected to the setting of perturbation types and locations. In the past decades, data-driven models and artificial intelligence technologies have developed rapidly, and the stability evaluation model based on sample learning has become a new research hotspot. However, most of the relevant researches focuse on the transient stability assessment problems. Studies on data-driven models for small-signal stability assessment are rare [4,5]. Among them, the dynamic characteristics of power system under faults are chosen as the inputs of the model. Ref. [4] adopted the improved XGBoost, whose inputs are manually selected generator characteristics including the 2 maximum and minimum acceleration of rotors, the maximum, minimum and average acceleration power and the difference between maximum and minimum angle speed. It aims at discriminating the damping level, i.e, unstable, weak and stable. In [5], the transient stability and SSA were combined into a multitask prediction model. The dynamics containing the voltage amplitude and phase angle at buses under large disturbance are fed in a Convolutional Neural Network (CNN) for transient stability assessment and a Long short-term memory model for system damping evaluation. However, the above method still depends on the transient information after a large disturbance. More recently, the research attention shifts to data-driven models for SSA with steady-state information, which is even more challenging since the inputs are high-dimensional in a large power grid. Note that the oscillation modes are not only affected by the generators and loads, but also closely related to the network topology. This leaves higher demand for the feature extraction of the model. In [6], Neural Networks with Random Weights is used to reconstruct the input features randomly, before an improved Random Bits Forest predicts the damping. Here the decision tree highly depends on the reconstruction, while it might be damaged due to the randomness of the feature learning based on NNRW. Graph Deep Learning [7-10] provides a new technical roadmap for high-dimensional feature aggregation considering topological information. Graph Convolutional Network (GCN) and Graph Attention Network have been applied for power system transient stability assessment in [9][10]. In this paper, a SSA model of power system based on the RESidual Graph Convolutional Network (ResGCN) is designed to predict the frequency and damping ratio of multiple oscillation modes in parallel, where feature aggregation is enhanced by GCN and the residual mechanism. The performance of the proposed model and its adaptability to topological changes are verified on the test system.

Structure of ResGCN
The overall structure of the ResGCN model is shown in figure 1. We utilize GCN to realize feature aggregation of the steady-state inputs and the residual connection (as orange lines) to overcome network degradation. Then several prediction modules composed by the Multilayer Perceptrons (MLPs) work in parallel to provide both the frequency and damping ratio of specific oscillation modes.

The Principle of GCN
When there are N buses and L transmission lines in the power system, we assume buses as the nodes V and lines as edges E. Then system structure can be represented by a graph ( , )  G V E , which is characterized by a node feature matrix X and an adjacency matrix A . The propagation principle of a GCN layer is expressed by Eq. (1):

Residual Mechanism
To alleviate the network degradation, the main idea of the residual mechanism is to concatenate the inputs and the outputs of the GCN layer, such that the input information can be preserved to the next layer [11]. Specifically, the residual mechanism is expressed by Eq. (2): where 1 l H represents nodes feature matrix after the residual mechanism, and || means the concatenation operation. In this paper, three GCN layers are constructed to form the feature aggregation module of ResGCN-SSA with the first two layers of GCN contain the residual mechanism.

The Proposed SSA Framework
The training and evaluation process of ResGCN-SSA is shown in figure 2, including data preparation, offline training and online application. iteratively by back-propagation of error to achieve the best performance. 3) Online application: During online application, real-time operation state of the power system is obtained from SCADA and the corresponding input quantities are fed into the trained model. According to the predicted damping ratio, power system operators can carry out necessary control measures to ensure the security of the power system.

Graphical Inputs 1) Node feature matrix:
The node feature matrix is derived from the arrangement of the voltage amplitude and phase at buses, the active and reactive power of generators, the active and reactive power of loads. The adjacency matrix A is relevant to the node admittance matrix, whose element ( , ) i j a is as follows: where ( , ) i j y , ( , ) max i j y denote the elements and the maximum element in the admittance matrix.

Generation of Labels
The frequency f and damping ratio  of each oscillation mode are selected as the labels. Their definitions are listed in Eqs. (4)- (6): where  represents the eigenvalue to determine the small-signal stability.

Data Preparation
The operation states with all transmission lines on service are denoted as the "Base" cases, while the "N-1" cases are generated by randomly switching off transmission line to test the topology learning of ResGCN. Besides, the loads are changed within 75% to 120% of the basic load level, while randomly adjust the generators for feasible power flow. we finally get 5,250 cases, which are divided into a training set (70%) and a test set (30%). The simulations are conducted on PSD-BPA for the labels. By comparisons of groups of parameters on the test set, the three-layer GCN in figure 1 is the best to fulfill the small-signal stability assessment.

Evaluation Metric
The average arctangent absolute error percentage (MAAPE) [12] is selected to measure the performance of the model. Eq. (7) measures the relative error between the two targets: where k i y refers to the labeled frequency or damping ratio of the k th mode of the i th sample, while k i y  refers to the predicted one. M, B is the number of the modes and samples.

Comparison of Model Precision and Its Adaptability to Topology
The proposed model are compared with two baseline models, where MLP is a shallow network and CNN is a deeper one. The results of different models are demonstrated in figure 3. MAAPEf and MAAPE- represent the predictive errors of the frequency and damping ratio.  Figure 3. Comparison with different models. The model proposed in this paper has better performance, whose MAAPE- and MAAPEf are 0.64% and 6.04% respectively. The CNN performs worse under "N-1" cases since it fails to capture the local changes of the operation topology. The high-dimensional inputs, however, lead to more difficulty in shallow learning based on MLP, whose MAAPE- and MAAPEf increase to 2.78% and 13.48%.

The Effectiveness of the Residual Mechanism
This subsection discusses the role of the residual mechanism in model performance. The test results concerning ResGCN and GCN are shown in figure 4. Here the MAAPE- of the samples predicted by ResGCN moves towards 0. Pay attention to "abnormal samples" with large MAAPE- higher than 15%. There are far more abnormal samples based on GCN than ResGCN, which means the residual mechanism can enhance the ability of the model to distinguish the topologies. As a result, the accuracy of ResGCN on MAAPE- is 1.74% higher than that of GCN.

Conclusion
In this paper, a ResGCN-based small-signal stability assessment model is proposed and verified on IEEE 39 Bus system. The conclusions are listed below: 1) Based on stacked GCN, the steady-state information of different nodes and their high-order neighbors can be fully and effectively aggregated. As a result, the adaptability of the model to topology changes becomes stronger. The experiments show that our method outperforms CNN and MLP under "N-1" cases. 2) The residual mechanism is designed for better performance. Compared with the conventional GCN model, the improved ResGCN can effectively improve the accuracy by reducing the risks of oversmoothing. In the future, we would like to seek solutions to improve the predictor(MLPs) for online application in larger-scale power systems.