extfieldsofstudy:
Computer Science
Engineering

text:
A study of CNN capacity applied to Left Venticle Segmentation in Cardiac MRI

CNN (Convolutional Neural Network) models have been successfully used for segmentation of the left ventricle (LV) in cardiac MRI (Magnetic Resonance Imaging), providing clinical measurements. In practice, two questions arise with deployment of CNNs: 1) when is it better to use a shallow model instead of a deeper one? 2) how the size of a dataset might change the network performance? We propose a framework to answer them, by experimenting with deep and shallow versions of three U-Net families, trained from scratch in six subsets varying from 100 to 10,000 images, different network sizes, learning rates and regularization values. 1620 models were evaluated using 5-fold cross-validation by loss and DICE. The results indicate that: sample size affects performance more than architecture or hyper-parameters; in small samples the performance is more sensitive to hyper-parameters than architecture; the performance difference between shallow and deeper networks is not the same across families.

Introduction
The use of deep learning has been growing and becoming very relevant for some types of medical image analysis. Deep learning includes CNN (Convolutional Neural Network) models, which have been successfully used for segmentation of the left ventricle (LV) in cardiac MRI (Magnetic Resonance Imaging) by Chen et al. [2020]. The myocardium segmentation (Figure 1) is the basis for other steps that provide measurements of ventricular function and myocardial viability (such as ejection fraction and ventricular mass), and is used for diagnosis of heart diseases, e.g. coronary artery disease [Aziz et al., 2013]. One of the most successful CNN architectures for medical image segmentation is the U-Net [Ronneberger et al., 2015]. It has been successfully applied in several medical imaging analyses such as pancreas detection [Oktay et al., 2018] and prostate segmentation [Ghavami et al., 2019]. Figure 1 shows an example of LV segmentation in cardiac MRI.
One of the limitations of deep learning approaches in many medical applications is the need for large labeled datasets, which are not always available. Creating labelled datasets requires clinical specialists for visual inspection and manual marking of images, which is costly and takes significant time. By considering this limitation, the U-Net architecture was designed for fast convergence and sufficiently accurate segmentation in relatively small datasets [Oktay et al., 2018]. U-Net is a CNN architecture with many possible variations such as different neural blocks, block width and network depth, so it is possible to balance the performance according to the images under study. Figure 1: Cardiac MRI images are acquired in two different axis: short-axis (a-c) and long-axis (d). In (a) we see a cardiac short-axis MRI of heart, centred in the heart, and cropped to frame the right and left ventricles; in (b) the left ventricle myocardium is highlighted, and (c) only shows the left ventricle myocardium segmentation mask. Images (a-c) are reproduced from the dataset analysed in [Nikolov et al., 2018] and (d) is reproduced from the Atlas of Human Cardiac Anatomy website [Spencer et al., 2013].
However, with the deployment of CNNs in medical practice, another two questions arise: 1) when is it better to use a small and shallow model instead of a larger and deeper one? 2) how the number of samples in a dataset might change the network performance? These questions motivate our work, which provides a framework of analysing the CNN capacity and dataset sizes for medical applications, and demonstrate its applicability to LV segmentation in cardiac MRI.

Our Contributions
In order to grasp how dataset size affects U-Net performance in segmentation, this paper describe a framework where we train 18 different U-Nets of different sizes (number of network parameters and depth) and families: VGG [Simonyan and Zisserman, 2015], ResNet [He et al., 2016] and EfficientNet . These U-Net variations were trained from scratch in six dataset sizes (suppose patients submitted to MRI study), and also with variations of hyper-parameters learning rate and L2 regularization. They were evaluated with 5-fold cross-validation by the loss and DICE index, and each fold was tested in a separate holdout set. We then analyse the variables and results in a factorial study using GAM (Generalized Additive Model), ANOVA (Analysis of Variance) and HSD (Honest Significant Difference) tests. In summary, our objectives in this paper are: • Observe the relationship between prediction error, dataset size and network size; • Identify hyper-parameter effects on performance by linear models, ANOVA and HSD; • Provide a framework to check when the choice for the network to be used in an application might change due to the number of images available in a training dataset.
This paper is organized as follows: Section 2 (Related work) reviews the base CNN architectures for image segmentation and related studies of neural network capacity; Section 3 (Materials and Methods) present the dataset preparation, the neural networks, hyper-parameters, linear models, ANOVA and HSD tests; Section 4 (Experiments) present the experiments, metrics, results and discussion; and Section 5 (Conclusion) summarizes our goals, the proposed framework, achievements and ideas for future investigations.

Related work
In this section we review related literature in the two main aspects of our framework: biomedical image segmentation (2.1), and how to evaluate neural network capacity (2.2).

Biomedical Segmentation with U-Net and its backbones
U-Net is an encoder-decoder CNN architecture for biomedical image segmentation, which encodes the input image in a sequence of four down-scaling layers (blocks), then up-scales the encoded tensors, concatenates them to the previous input and combines the results in the reversed order, forming a network that resembles an "U". It is possible to change its backbone by replacing the original convolution blocks for well-known blocks from other CNNs, e.g. residual blocks [Zhao et al., 2017], while keeping the U shape encoder decoder structure. The U-Net is trained by minimizing the cross-entropy between the class probabilities of each pixel in the output image and the segmentation mask given as class labels. The success of U-Net led to the proposal of backbone innovations, such as newer versions of the skip connections [Chaurasia andCulurciello, 2017, Zhou et al., 2018]. Other encoder-decoder architectures have also been proposed, some keeping the general U-net structure, and others with novel structures such as feature pyramids [Lin et al., 2017] and pooling pyramids [Zhao et al., 2017]. The standard U-Net backbone is based on VGG [Simonyan and Zisserman, 2015], but we extended our analysis by considering two more state-of-the-art backbones: ResNet [He et al., 2016] and EfficientNet .

Evaluating Neural Network Capacity
Given that there are several U-Net architectures available, making an appropriate choice is challenging and can impact the results of developing applications. These reflect in two critical aspects of medical studies planning: a) estimating the size of the sample needed for developing an effective solution (both in terms of images and patients) and b) selecting appropriate models, because different medical applications solve problems of different complexities, which thus require different network models of adequate capacity. However, there is no established method for calculating network capacity nor a consolidated understanding of how it relates to CNN depth and to the number of parameters [Bartlett et al., 2017, Neyshabur et al., 2019. Addressing these aspects in a clinical study is not trivial. While bigger networks might be prone to over-fit in small datasets, smaller networks might not have the required capacity for the challenge at hand. A further complication, at least for LV segmentation, is the fact that the relationship between U-Net, dataset sizes and generalization error is not well understood. Recent literature [Benkendorf andHawkins, 2020, Shahinfar et al., 2020] have been investigating sample size issues, so in our approach we extend this investigation to hyper-parameters and network architecture. Next we present our methodology, then experiments, results and conclusion.

Materials and Methods
In this section we present the dataset characteristics, preparation and splitting (Section 3.1, the U-Net CNN architecture and backbones based on VGG, ResNet and EfficientNet (Section 3.2), hyper-parameter optimization (Section 3.3), and the linear models and statistical tests used in the analysis (Section 3.4).

Dataset preparation
All of the datasets we created are sub-samples of the publicly available Left Ventricle Segmentation Challenge (LVSC) dataset [Nikolov et al., 2018]. The LVSC training set has cardiac cine-MRI pictures of 100 patients, with diverse number of slices (8-24), slice thickness (6-8mm), slice gaps (2-4mm), number of phases in the cardiac cycle (18-35), image sizes (138×192 to 512×512 pixels) and acquisition devices (GE, Phillips or Siemens MRI scanner systems with 1.5 or 3.0T). All images were loaded into float32 arrays, had the pixel values rescaled to [0,1], resized to 256×256 pixels with area and bi-cubic interpolation respectively for downscaling and up-scaling, and zero padding to maintain aspect ratio.
Due to anatomical and positional differences, patient id is a major source of variation, so we sub-sampled 100 images from each patient. In order to stratify samples in the whole cardiac volume, we selected an approximately equal number of images from each slice, regularly spaced throughout the cardiac cycle (images i × phases × slices/100 for i in [1,100]). This resulted in 10,000 images (100 patients × 100 images per patient). Then, an 80-20% holdout group split was performed for development (training + validation) and test. Afterwards, five smaller datasets were sub-sampled from the bigger dataset (10k), with 200, 500, 1000, 2500, and 5000 images. For each of those datasets, the development set was sub-sampled from the initial development set with 8000 images, while the test set was sub-sampled from the holdout test set with 2000 images. Both the holdout and the sub-sampling were performed by randomly selecting of patients, such that a given patient had either all of her/his images in one set only. In each execution the development set is again split in 5-folds of equal sizes for cross-validation: each fold is taken for validation using a model trained on the remaining four folds. Next we present the neural networks, hyper-parameters and linear models and statistical tests.

Neural Networks
U-Net [Ronneberger et al., 2015] implements the downscaling path similarly to the VGG network [Simonyan and Zisserman, 2015], using a sequence of 2D convolutions, ReLU activations and max-poolings. We also tested U-Net with modern backbones based on ResNet [He et al., 2016] and EfficientNet . ResNets divide the pathway in an input identity map path (a short-cut connection), and a residual path, that learns a residual function (which model the residuals from the inputs). By learning the residuals instead of the main signal, more convolution blocks can be stacked to build deeper neural networks without compromising their training. EfficientNets are based on mobile inverted bottleneck convolution blocks [Sandler et al., 2018] with squeeze-excitation optimization [Hu et al., 2018] and a loss function that balances accuracy and speed, by adjusting network depth, width and resolution hyper-parameters .
To study network size we selected small and medium-large sizes for each backbone: EfficientNets B0 and B5; ResNets 18 and 50, with approximately the same numbers of parameters to the chosen EfficientNets; and VGG 16 and 19, the two most common VGG sizes in literature. EfficientNet B0 has 5.3 million parameters and achieved 76.3% top-one accuracy in the ImageNet classification task, while B5 has 30 million parameters and achieved 83.3% accuracy in that same task. This is a bigger gain for scaling up than for other families of networks . Another aspect that impacts performance and convergence of neural networks is the choice of hyper-parameters, presented next.

Hyper-parameter optimization
Given that training strategies might differ regarding dataset and network size, we also chose to analyse some optimization parameters. However, as the time to compute all hyper-parameter space is combinatorial, only some values were selected. For each model and dataset combination, we tested different initial learning rates (10 −2 , 10 −3 and 10 −4 ) and different L2 regularization values (10 −2 , 10 −4 and 10 −6 ). The initial learning rate balances training speed and convergence stability. The regularization has a further implication in model capacity, as it balances accuracy and over-fitting by constricting network weights. In order to understand the effect of hyper-parameters in combination to dataset and network size, we analyse these variables with linear models, ANOVA and HSD tests of next section.

Linear Models, ANOVA and HSD
To understand the effect of CNN choices on the performance, we develop a linear regression model based on the dataset size, model architecture, model size, and hyper-parameters as input variables. Linear regression models are statistical approaches to analyse the dependence of a response variable Y on the linear combination of input variables X = x 1 , x 2 , ..., x n . These models follow the general formula Y = aX + b + , where a corresponds to weights for each variable x i , and b is a bias term (intercept), and is the error of the model. The parameters a and b are commonly adjusted by ordinary least squares regression (OLS). ANOVA for regression models tests the statistical significance of input variables (groups), with respect to the sums of squares (SS) of regressed variables and observed data. The HSD test was developed by Tukey for pairwise comparison of multiple group means, and is used as a post hoc adjustment [Tabachnick and Fidell, 2007].
Generalized linear models (GLM) extend linear models by applying a link function that transforms the predicted distribution, e.g. log(Y ) = aX + b →Ŷ = exp(aX + b) in the case of log link. Generalized additive models (GAM) further extends GLM by applying transformations in each variable, e.g. g(E[Y ]) = f 1 (x 1 ) + f 2 (x 2 ) + ... + f n (x n ) + b, and discovering the most appropriate functions f i for each input variable [Hastie and Tibshirani, 1990]. The linear models are evaluated by Akaike Information Criterion (AIC), according to the maximum likelihood of estimates and number of parameters of the model [Akaike, 1974]. We tested several linear models and settled our analysis with DICE ∼ ls + ls : F amily + log(Dataset) + log(lr) : ls : F amily + log(reg), which is presented in the next section along experiments, metrics, results and discussion.

Experiments
In this section we present the experimental configuration (4.1), evaluation metrics (4.2), results (4.3) and discussion (4.4). In Section 4.3 (Results) we present the experimental results, report the DICE index and BCE Loss of all models, compare their differences, and analyse the model's dependences on the chosen parameters by linear regression models, ANOVA and HSD tests. In Section 4.4 (Discussion) we comment our findings in relation to literature and place general considerations.

Experimental configuration
For each of the 324 different combinations of variables (model, dataset, initial learning rate, L2 regularization), the model was trained from scratch using 5-fold cross-validation. The development dataset was divided into 5 equally-sized subsets (folds) and each one is used for validation while the remaining four are used to fit the model. For each fold, every combination of parameters is trained for 50 epochs or less. We stopped training when there was no improvement in the validation loss after 5 epochs (early stopping). This results in 1620 training/evaluation runs, corresponding to 5 folds × 324 different conditions.
After all runs, we computed the metrics and fitted a linear regression model to the experiment parameters and results, then performed statistical ANOVA and HSD tests to assess significant differences. All experiments were performed in a Foxconn HPC M100-NHI with eight NVIDIA V100-SXM2-16GB GPUs, using the software we implemented with Python, PyDicom, Tensorflow, SegmentationModels, Slurm and StatsModels. Next we present the evaluation metrics, then results and discussion.

Evaluation metrics
To evaluate the quality of each model, we use two metrics: BCE (Binary Cross-Entropy) loss and DICE index [Dice, 1945]. These are computed on the validation set of each training run, and also separately on the held-out test set from the corresponding dataset. We then compare all configurations across datasets and models, considering the best hyper-parameters for each configuration. BCE and DICE formulas are given below: Where T P, F P, F N are true positives, false positives and false negatives, respectively, N is the number of pixels in the dataset, y are the binary labels for all pixels,ŷ are the corresponding predictions and · denotes rounding. BCE is a positive value which gives the amount of entropy mismatch in the predictions, the lower the better. DICE range from 0% (no correspondence) to 100% (images are perfectly equal) and is equivalent to F1-score. In the next section we present the results of the experiments, our analyses and discussion.

Results
All networks performed badly on the smallest dataset (due to insufficient training cases) and improved gradually when fed with more data. The network performed similarly well on the larger datasets ( Figure 2). However, the networks behaved very differently in the middle sizes (between 400 and 2000 samples), both in absolute performance and variability of the metrics. Loss function behaved similarly, showing a similar improvement related to increasing dataset size. Although, we opted to focus this report on DICE index because it is easier to interpret, has close relation to the task, and is common in literature.
After all runs, we fitted a linear regression model (Table 1) to the experiment parameters and results, then performed statistical ANOVA and HSD tests to assess significant differences (Table 2 and Figure 2). The variables F amily, ls, lr and reg correspond to the network architecture family (VGG, ResNet or EfficientNet), model version size (Long or Short, ls), initial learning rate (lr) and L2 regularization parameters (reg), respectively. The nesting of F amily within model version size (ls/F amily = ls + ls : F amily) in the regression formula is a way to access a rough effect of model size while not treating different architectures (long and short versions) as equivalents.
We also fitted a linear model with a similar formula and using a log link function instead of the log transformations on predictors: log(DICE) ∼ ls + ls : F amily + Dataset + lr : ls : F amily + reg, but it had a bigger AIC value of -582. According to the results, the dataset size had overall larger absolute coefficients (Table 1) and more significant effect (Tables 1 and 2) in DICE than architecture, initial learning rates or L2 regularization values in the linear model. HSD test was applied to the mean DICE and confidence intervals ( Figure 2). EfficientNets had consistently higher DICE scores up to 2000 images. ResNets and VGGs required more data to achieve a comparable level of performance, but slightly surpassed EfficientNets in the two largest datasets. In the next section we discuss these results in relation to other evidence in literature, and place general considerations.

Discussion
Apparently, smaller models (EfficientNet-B0, VGG16 and ResNet18) are not too small for LV segmentation tasks, and were able to achieve good levels of performance when given sufficient data. One could expect that different models would reach different plateaus. But we observed in all families that, as the training dataset size increases, the gain in DICE performance of deeper networks vanishes as the performance of shallower models levels up. Considering each network family separately, the network size effect is greater for the EfficientNets than for the ResNets and VGGs (Table  1 coefficients). As in the case of ImageNet classification, the gain of scaling up is bigger for the EfficientNets than in other families. We noticed that in all dataset sizes B5 performed significantly better than B0 (Figure 2), showing that it did not over-fit more easily with the increased number of parameters, even considering that B0 reached comparable performance with more data. These results contradict the common sense that bigger models would suffer more from over-fitting in smaller datasets, decreasing their performance, and that they would gain more performance as more data was made available [Martin and Mahoney, 2018]. Our findings with EfficientNet support the idea of Neyshabur et al. [2019] that bigger models does not necessarily over-fit, but this pattern does not hold for other families. Considering the ResNets, the deeper ResNet50 starts with a DICE performance below the shallow counterpart (ResNet18), but improved faster and caught up with 4000 samples. We observed that VGGs never differed significantly, probably because of their smaller differences in depth and number of parameters, and they also have high sensitivity to hyper-parameters. VGG16 had a particular tipping point from the worst results in the 800 images dataset to scoring near the top contenders on the next dataset with 2000 images.
A general consideration in the intermediate dataset sizes is that all models were relatively more sensitive to hyperparameters than to architecture, as their performances had higher variance than when more data was made available. This might affect sample size and model choice, as with 2000 images EfficientNet B5 performed significantly better than the ones from other families, but with 8000 images all models had no practical difference. Thus, in scenarios where bigger datasets are unavailable and acquisition is costly, it might be possible to use smaller data samples with deeper networks and careful optimization. When a bigger dataset size is available, neither model selection or optimization provides much performance gain, different approaches such as changing pre-processing or loss function might be required. However, as in the demonstrated case, any method should be tested in a range of training dataset sizes in order to guarantee its performance. In the next section we conclude this paper, summarizing our goals, experiments, results and future investigations.

Conclusion
In this work we described a framework to check when the choice for the network to be used in an application might change due to the number of images available in a training dataset. We demonstrated an application of this framework for LV segmentation in cardiac MRI images using the LSVC dataset. In our experiments we observed the relationship between prediction error, dataset size and network size; and also identified hyper-parameter effects on network performance, by modelling with linear model DICE ∼ ls + ls : F amily + log(Dataset) + log(lr) : ls : F amily + log(reg) and executing and ANOVA of its interaction terms. The results indicated that: sample size affected performance more than architecture or hyper-parameters; in small samples the performance was more sensitive to hyper-parameters than architecture; the performance difference between shallow and deeper networks was not the same across families.
Further investigation and extension of these experiments will focus on analysing the regularization effects' patterns regarding network size. Also, we plan to use similar strategies in Neural Architecture Search (NAS) and Automatic Deep Learning (AutoDL) to discover better segmentation networks for evolving and multi-centric medical imaging datasets.  Table 2: ANOVA table for linear model variables and their interaction. η 2 p is the effect size, measuring the proportion of SS explained by each term, while Residuals measure the unknown SS. The terms ls : F amily, log(Dataset) and log(lr) : ls : F amily are responsible for 80% of DICE index variance in the trained models. We observe that gross performance increment comes from larger datasets, while learning rates and network size give small but significant increments depending on the network architecture (F amily).


A Novel Optimized Asynchronous Federated Learning Framework

Federated Learning (FL) since proposed has been applied in many fields, such as credit assessment, medical, etc. Because of the difference in the network or computing resource, the clients may not update their gradients at the same time that may take a lot of time to wait or idle. That's why Asynchronous Federated Learning (AFL) method is needed. The main bottleneck in AFL is communication. How to find a balance between the model performance and the communication cost is a challenge in AFL. This paper proposed a novel AFL framework VAFL. And we verified the performance of the algorithm through sufficient experiments. The experiments show that VAFL can reduce the communication times about 51.02\% with 48.23\% average communication compression rate and allow the model to be converged faster. The code is available at \url{https://github.com/RobAI-Lab/VAFL}

I. INTRODUCTION
Machine learning and artificial intelligence need to be fed lots of data to get a well-performed model, which cased the demand of data that is stored in every device. For instance, smart city [1], medical [2], and credit assessment [3]. But, this data is sensitive. At the same time, the relationship between data is often isolated and fragmented. This limits its application for processing data.
In this context, federated learning was proposed. Federated learning is a privacy-preserving distributed machine learning technique. The data holders only need to provide a local model which was trained using their local data to interact with the central server. In this paradigm, only gradients need to be transmitted rather than the raw data, which can effectively reduce privacy risks. In federated learning, the client downloads the global model from the central server and trains the local data under the coordination of the central server. After a round of training is completed, the trained local model is uploaded to the central server for weighted aggregation to complete the update. After some iterations, the performance of the model is similar to the results of centralized machine learning. However, there are some challenges in federated learning [3] [4]. Traditional federated learning has obvious shortcomings in flexibility and scalability. These challenges are shown below: • Heterogeneous device: Since training data is stored on the client-side, heterogeneous device client used to have different data sizes, computing power, network, etc., and are affected by unstable communication conditions, the efficiency of traditional synchronous federated learning can be affected by the presence of dropped users. For example, when a few clients are disconnected due to network problems, other clients and server have to wait for them, causing the federated learning process slowly and spend lots of time idle. • Heterogeneous data: Since federated learning data are only stored locally, the inconsistency of this distribution among data in practical applications leads to significant degradation of model performance, such as feature distribution skew, label distribution skew, quantity skew [5], [6]. • Security risk: Even though federated learning does not involve the client raw data in the transmission, there is still a considerable security risk. For example, attackers may use GAN to learn potential information in the gradients [7]. In order to address these issues and challenges, FedAvg [8], FedProx [9], SkewScout [10], and other algorithms have been proposed one after another. Besides, asynchronous federated learning was proposed. Asynchronous federated learning mainly addresses the problem of how the central server parameters should be aggregated in the presence of any heterogeneous clients (heterogeneous data distribution, computing resources, network conditions, etc.). In this paper, we propose a new asynchronous federated learning optimization algorithm (VAFL) for the above challenges (mainly the first two). To address the first challenge, we consider the number of participating clients and provide an improved method for judging clients by checking obsolescence, introducing a function for evaluating the communication value of clients so that random selection is no longer performed. By selecting high-quality clients, the global model accuracy is improved. For the second challenge, we make a division between IID and Non-IID on the datasets, aiming to cope with practical federated learning scenarios, and explore applications in Non-IID environments.
In summary, our contributions are as follows: • We proposed a novel asynchronous federated learning algorithm VAFL, which can reduce the communication cost and the idle time and make the model converge faster. • We set up evaluation formulas for client selection to determine which clients upload the model. Experiment results show that this method accelerates the accuracy obtained at the beginning of training, and it also ensures that the communication compression rate can be improved to a certain extent. • We perform extensive experiments and validate the performance of VAFL.

II. RELATED WORK
In recent years, the Federated Learning framework proposed by Google has attracted great attention from academic scholars. The standard federated learning model needs to learn a single global statistical model from the stored data of a large number of remote devices. However, the standard model faces challenges such as high communication costs, system heterogeneity, statistical heterogeneity, and privacy issues [11].
In response to these challenges, McMahan [8]proposed a method for synchronous training of the Federated learning Average Algorithm (FedAvg), and recently Q Li [12] proposed MOON based on FedAvg. These algorithms are updated locally. The cost of communication is reduced; not only that, privacy protection is based on synchronization operations such as differential privacy [13]and secure aggregation [14] can solve the privacy problem to a certain extent.
However, many types of research on federated learning today are mainly about Non-IID data sets and data imbalances, and most of the researches are based on synchronization algorithms. However, federated learning based on synchronization algorithms cannot solve the problem of excessive communication costs. Issues such as differences in terminal performance due to different equipment terminals.
In response to the above challenges, many researchers have begun to try asynchronous algorithms for federated learning. For example, EAFLM proposed by Lu [15], this method can achieve the goal of reducing communication costs without searching for the optimal solution; SS Diwangkara [16]implemented it through an asynchronous aggregation algorithm The rapid convergence of each node and the reduction of the update frequency of the parameters can not only reduce the communication cost, but the method also achieves good results in the Non-IID data set. Although the above methods can reduce the communication cost better, they did not consider the serious delay caused by the lagging party. For this reason, McMahan B [8] proposed the use of multiple SGD updates and batch processing clients to alleviate the delay. Problem; Subsequently, Nishio T [17] suggested using the deadline method to solve the customer selection problem. Although this method can filter out some slow-response customers, it did not consider how to solve the model accuracy rate caused by the backwardness in training. Too low customers, for this reason, Chai [18] proposed a hierarchical-based federated learning system (TIFL), which can divide customers into different levels according to the user's response delay in the federated system. To select customers based on the system, the system can not only solve the problem of the user falling behind but also further alleviate the problem of Non-IID data. However, the TIFL system adaptively adjusts the client's participation based on the response delay of the client's machine, and it cannot solve some clients with high participation but low data quality.
Aiming at the excessively high communication cost in the federated learning training and the lagging customers, we propose an optimized federated learning asynchronous algorithm based on the communication value calculation of the gradient, the number of clients, and the accuracy of the model(VAFL). Different from the method of Chai [18], we use the client's gradient change, the accuracy rate on the test set, and the number of clients to evaluate the client's communication value, thereby determining whether the client can continue to participate in federated optimization. At the same time, considering the problem of low performance of client devices in reality, when we set up the experimental platform, we used a Raspberry Pi with poor performance to simulate client devices.

A. Communication Value Calculation
This section illustrates how VAFL is derived and worked. The communication value calculation is based on gradient, the number of clients, and model accuracy. The notations covered in this section are shown in Tab. I. The ith client's communication value The ith client's gradient ∇ k i Gradient at the kth round of training for the ith client Acc Accuracy of client models on the testset Acc i Accuracy of the ith client model on the testset n

Number of clients involved in federated learning
We refer to the work of Chen et al. [19], where the main idea of Chen et al.'s work is that not all communication back and forth between the server and the clients is equally important. Selecting only the important clients for communication reduces the communication cost and also maintains the convergence speed. Instead, our idea is to let the more communication-valuable clients communicate with the server, which we define as V , where the communication value of the ith client is denoted as V i .
How to determine the communication value of the client then becomes an important factor in the design of the algorithm. After following the prior work about the calculation of communication value, we decided to use the gradient of the clients' model as a variable in VAFL design. We use ∇ i to represent it. Considering that the model performance of each client reflects some extent the performance of the client and the combination of the datasets it has (in general, the more quality datasets it has, the better the performance of the model obtained by local training of the client will have better performance). For convenience, we choose an easy metric: the accuracy of the client model on the test set, as a variable for VAFL design. We define it as Acc, and The accuracy of the ith client model is Acc i . Recently, it has also been demonstrated that special client selection methods can optimize the federated learning process and improve the performance of the final model [20] [21]. Therefore, we choose the number of clients involved in federated learning as another variable in the design of VAFL. We define it as N ; Ultimately, we designed the evaluation formula for V i : Where ∇ k−1 i −∇ k i is the gradient difference of the model in the last two training processes of the ith client model. We use it to check whether the model is "old". This difference is smaller, indicating that the model is older, which means its V is smaller; n is the number of clients involved in federated learning, we divide it by 10 3 and plus 1 as the base of the power function with Acc i as the variable. Thus, as the number of clients involved in federated learning increases, the relative communication value of individual clients is further differentiated. The more valuable the client, the further the value will increase, while the less valuable the client, the further the value will decrease. To facilitate the design of VAFL.

B. An Asynchronous Federated Optimization Algorithm
This section introduce the design idea and algorithm flow of our asynchronous federated optimization algorithm based on client communication value.
We hope to compress the communication by measuring the communication value of each client participating in federated learning to decide whether they participate in federated optimization or not, so as to deal with the problem of communication bottleneck and unbalanced in asynchronous federated learning.
For the clients involved in federated learning, we calculate their communication value V by using Equation 1, and the server will be able to learn the V of each client. Add it to the asynchronous FedAvg only if the client's V satisfies the Formula 2.
The specific algorithm pseudo-code is shown in Algorithm 1

IV. EXPERIMENT SETTING
In this section, we will introduce how we built our experiment system and how VAFL is implemented.

A. Hardware Platform
VAFL are designed for varying edge devices. Following this principle, we used five Raspberry Pi devices and two laptops to build our experiment system for federated learning for edge computing, which is more realistic, compared to the simulated asynchronous environment. Among them, the laptop as the central server is configured with a 6core Intel(R) Core(TM) i7-9750H CPU running at 2.59GHz

Algorithm 1 V based Asynchronous FedAvg
N is the total number of clients , θ t is the parameters of current model, θ t i is the parameters of current model with index i, B is the batch size, E is the number of local epochs, S is a set of all clients model and η is the learning rate.
1: Server Update 2: Initialize θ 0 ; 3: for each round t = 1, 2, · · · do 4: for each i ∈ S do 5: // clients update model on local device and upload the V i to server; 26: return V to server with 8GB of memory, and the rest of the devices as edge clients, where the laptop is configured with a 4-core Intel(R) Core(TM) i5-9300H CPU running at 2.40GHz and 8GB of memory, one Raspberry Pi device is Raspberry Pi 4B, which is configured with a 4-core ARM Cortex-A72 running at 1.50GHz and 4GB of memory, and the remaining four Raspberry Pi devices are Raspberry Pi 4B with 8GB of memory.
All devices are connected to the same LAN with a network bandwidth of 2.4GHz and the speed for receiving and transmitting are 216 Mbps and 120 Mbps.

B. Framework
We use PySyft [22] as the basic framework to build our asynchronous federated learning experiment system. PySyft is a Python library for Federated Learning, Differential Privacy, and Encrypted Computation. We use the client class and server class provided by PySyft for connecting via WebSocket to facilitate the construction of our federated learning system. We use PyTorch as the deep learning backend which is supported by PySyft.
We run our client program on each client (a total of five Raspberry Pi devices and one laptop, which cannot be informed of each other's existence and ensures a certain degree of security and privacy between the clients). Each client is connected to the server, which can only be informed about the model of each client and the total number of clients. The server uses VAFL to update the model, and then returns the model obtained by the algorithm to the client, and the client uses the model to predict. The architecture of our experiment system is shown in Fig. 1.
We build our federated learning system using PySyft 0.2.4 with Python 3.7. We use ResNet [23] as the core part of model based on PyTorch 1.4.0, and its network structure is schematically shown in Fig. 2

C. Data
We constructed IID and Non-IID data distributed to each client using MNIST and conducted the experiments separately. For the IID case, the training set is equally distributed to all clients, each client contains samples with 10 labels. For the Non-IID dataset distribution, the training set varies in terms of labels and the number of samples with different labels. Some clients containing all labels and a large number of samples under each label, and some clients containing only a small number of labels and some samples under each label.
By constructing such a dataset, we hope to further explore the effect of whether the data is IID on the asynchronous federated learning system.

D. EAFLM
The main idea of EAFLM [15] is to ignore the "lazy" nodes in rounds and only communicate with the "hardworking" nodes. We were inspired by this and improved, so we choose Lu's algorithm for comparison. The difference between VAFL and EAFLM is that EAFLM selected the parameters and gradient of the model as the main factors for the design of the algorithm, whose algorithm is mainly as follows.
Where ∇ i θ k−1 is the k−1 th round gradient calculated by client i based on the k − 1 th round parameter. θ k−1 is the k − 1th round parameter of the server. β, ξ d and D are constant coefficients. α (0, 1) is the adjustable parameter in the parameter weights, which determines the decay rate. As α increases, the decay rate of the parameter weights increases. In the experiments of this paper, ξ d = 1 D , D = 1, α = 0.98.

V. EXPERIMENT AND ANALYSIS
In this section, we conduct four experiments, setting 3 or 7 clients with iid or non-iid data, to validate our proposed asynchronous federated optimization algorithm VAFL.

A. Experiment Metrics
We use the accuracy rate (Acc) and communication compression rate (CCR) as the evaluation metrics of the experiments. Where Acc is the highest Acc rate in multiple experiments and the communication compression rate (CCR) is calculated as: Where C t0 is the communication times before compression and C t1 is the communication times after compression.
The communication compression rate reflects how much the communication between the clients and the server is compressed, and the larger the compression rate, the higher the degree of compression. Although excessive compression rate reduces communication loss and saves time, it is usually accompanied by a decrease in model Acc.
Our experiments use two metrics to evaluate the effectiveness of VAFL, which is to achieve a certain communication compression rate while ensuring the loss of model Acc. The parameters for each of VAFL are shown in Tab. II.

B. Experiment
We conduct four experiments, setting 3 or 7 clients with iid or non-iid data.
• Experiment a: 3 clients with IID data. • Experiment b: 7 clients with data. • Experiment c: 3 clients with Non-IID data. • Experiment d: 7 clients with Non-IID data.
In the experiments of 3 client, we launch the client program to connect to the server on 3 Raspberry Pi devices, one of which has 4 GB of memory. And each client's training set have 20,000 samples. In experiments of 7 client, we launch the client program to connect to the server on 5 Raspberry Pi devices and 1 laptop (launched 2 processes). And each client's training set have 10,000 samples. The rest of the hardware configuration and software system have been mentioned in Section IV.
The neural network we use has been mentioned in Section IV also. We perform image classification experiments to verify whether our federated learning asynchronous training process can converge and the effectiveness of VAFL. We compare the three methods, ordinary asynchronous training, VAFL method, and EAFLM. The Acc of the federated model during the training period in four experiments are shown in Fig. 4.
We also compare the number of communications and the communication compression rate experienced by the three methods in training the model to achieve 94% Acc. The experimental results are shown in Tab. III.

C. VAFL Performance in Multiple Experiments
We also compared the performance of VAFL itself in different experiments. In the four experiments a, b, c, and d, the Acc of each client when executing VAFL are shown in Fig. 5. And the Acc when executing VAFL in different experiments is shown in the Fig. 6. The Communication times and the CCR are shown in the Tab. III. As the number of clients increases and the imbalance in the distribution of the dataset intensifies, the better VAFL performs.

D. Summary
Combining the results of the above experiments, it can be seen that compared with the basic asynchronous algorithm, VAFL enables the model to be trained faster, its Acc can be improved faster in a short period. Also, it can reduce the communication time of 51.02% and reach an average communication compression rate of 48.26%. Compared with EAFLM, VAFL reduces the number of communications  by about 3.23% and improves the compression rate of communications by an average of about 0.82%. In summary, VAFL is able to compress the communication to a large extent, reduce the communication burden between the client and the server, and improve model convergence speed compared to the common asynchronous training methods. Compared with EAFLM, VAFL is able to compress the communication to a better level and reduce the communication times during training period. And VAFL performs better as the number of clients increases and the imbalance in the distribution of the data set intensifies.

VI. CONCLUSION
In this paper. We proposed VAFL, a federated learning algorithm, focusing on the limitation and imbalance of the communication in asynchronous federated learning. The main idea of VAFL is that we expect to evaluate the communication values of each client to decide whether each one of them can participate in the optimization to compress communication values between server and clients. This study has designed a series of experiments aimed to identify the algorithm improving the efficiency of federated learning of edge devices compared with general algorithms and EAFLM, but also including some deficiencies. VAFL enables the model to be trained faster and can reduce the communication time of 51.02% and reach an average communication compression rate of 48.26%. But VAFL relies heavily on honest clients because the calculation of V i depends on the clients themselves. If the clients report V i incorrectly, this will exacerbate the unfairness between the data and clients.
Future work can introduce other reference factors to make the computed communication value more reflective of the client's real situation and make sure the value is trustworthy, such as local training for a certain amount of time before computing the communication value or setting up some mechanism to make the computation of the client communication value less frequent.
The approach proposed in this paper has potential applications in the fields of Internet of Things [24], robotics [25] [26] [27], and smart cities [1]. In addition, the idea of value assessment may be applied in the incentive mechanism of Federated learning [28] [29].


Bayesian Data Fusion of Imperfect Fission Yields for Augmented Evaluations

We demonstrate that Bayesian machine learning can be used to treat the vast amount of experimental fission data which are noisy, incomplete, discrepant, and correlated. As an example, the two-dimensional cumulative fission yields (CFY) of neutron-induced fission of U are evaluated with energy dependencies and uncertainty qualifications. For independent fission yields (IFY) with very few experimental data, the heterogeneous data fusion of CFY and IFY is employed to interpolate the energy dependence. This work shows that Bayesian data fusion can facilitate the further utilization of imperfect raw nuclear data.

Accurate modeling of nuclear fission remains an open problem although its significant applications are generally known [1]. In fact, for modern nuclear energy productions in a more compact and sustainable, cleaner and safer way, a supply of high quality data of multiple fission observables as a function of incident neutron energy is crucial [2]. In addition, more accurate fission yields are needed to address the reactor anti-neutrino anomaly [3,4]. In major nuclear data libraries [5][6][7][8], evaluated fission yields are only available for neutron incident energy at thermal energy, 0.5 and 14 MeV. Both experimental measurements and reliable modelings of nuclear fission are notoriously difficult. Nevertheless, there was a vast amount of raw experimental fission data [9] but these data are generally noisy, incomplete, and discrepant. Therefore it is desirable to exploit the further value of these imperfect fission data for novel nuclear applications as well as for constraints on fission theories.
The spreading fission product yields (FPY) are the most frequently studied observables, and FPY distributions evolve in time due to cascade decays of fragments. There are more data of cumulative fission yields (CFY) which are measured after post-fission β-decays but much fewer data of independent fission yields (IFY) which are identified promptly after fission [9]. Conventionally, semi-empirical models such as Brosa [10] and GEF [11] models with physics guidances are used for evaluation of fission data. However, semi-empirical models invoke many parameters, which undermines their applicabilities for new data and uncertainty quantifications. Advanced evaluation methods should take into account the correlations between different fission data [12]. In this context, the covariance matrix is widely used for uncertainty assessments [5,13,14].
Machine learning is now revolutionizing all scientific fields and is particularly powerful for treatments of complex big data. The belief behind data science is that data includes all correlations. There have been increasing applications of machine learning in nuclear physics (see a * Electronic address: peij@pku.edu.cn review [15]). In particular, machine learning can be used to extract and discover knowledge from existing nuclear data, which is referred to as data mining. For example, machine learning has been applied for inferences of nuclear structures [16][17][18][19][20][21] and nuclear reactions [22][23][24][25]. The application of machine learning to augment evaluations of nuclear data is very anticipated [26]. Previously we have applied the Bayesian neural network (BNN) for evaluation of incomplete fission mass yields [27] and discrepant charge yields [28]. The multi-layer BNN has been optimized for evaluation of fission yields with physical odd-even effects [28] and non-negative constraints [29]. There was also machine learning of fission yields using the mixture density network method [30]. Besides, the Gaussian process can also solve regression problems of nuclear data evaluations but is assumed with local correlations.
In this Letter, we aim to make the best utilization of vast raw experimental fission data by data fusion [31], which is a prevalent way to deal with imperfect data and includes all underlying correlations. The fused data is expected to be more informative than separated data sources. It is possible that some non-local and high-dimensional correlations could be weak but nonnegligible, in analogy to long-range and many-body interactions. The inference would be less precise when data in some energies is sparse, however, its correlations with other data in other energies can be beneficial to improve the inference. It is known that Bayesian machine learning is ideal for uncertainty quantification [32] which is key in fusion of imperfect data. Furthermore, the fusion of complex heterogeneous data can be naturally treated by Bayesian machine learning, which exhibits the powerful capabilities of machine learning by including correlations beyond the Gaussian process and the covariance matrix.
We employ BNN [33] for machine learning of distributions of fission yields. The posterior distribution p(ω|x, t) of BNN is based on a prior distribution p(ω) of network parameters ω and a likelihood function p(x, t|ω), Thus the resulting inference has a distribution and nat-arXiv:2111.14102v1 [nucl-th] 28 Nov 2021 urally provides the associated uncertainty. The data set is given as D = {x i , t i }, where x i is the input and t i is the output fission yields. The details of BNN has been described in previous works [27,32]. For the data fusion, it is crucial to taken into account the experimental uncertainties by the likelihood function, The cost function χ 2 (ω) reads: where f (x i , ω) denotes the network values. Note that the weights in the likelihood include a hyperparameter noise scale δ 2 i and the experimental uncertainty σ 2 i,expt . The hyperparameter δ 2 i is uniform for all data points and changes in the learning process until numerical convergence. It is reasonable to see that data points with large experimental uncertainties would have small weights in the data fusion. It is also interesting to study how the experimental uncertainties prorogate to the prediction uncertainties. The data fusion is in some sense similar to the conception of model averaging and model mixing [16] but doesn't suffer from the bias in model selections.
Firstly, we demonstrate the uncertainty prorogations by varying experimental uncertainties, or by deleting or adding data points deliberately to mimic incomplete and discrepant data. We use BNN to evaluate the fission mass distribution of 238 U as an illustrative example, with experimental data from Ref. [34]. The data set also includes the evaluated independent fission yields of 238 U with neutron incident energies at 0.5 and 14 MeV from JENDL [6]. Note that neutron-induced fission of 238 U doesn't occur at 0.5 MeV and the yield data of 0.5 MeV is a hypothetical evaluation. Here BNN adopts a single hidden layer with 10 neurons because the tests employ a small data set of 309 data points. Fig.1(a) displays the evaluation of fission yields of 238 U by including original experimental uncertainties. The BNN evaluations generally agree with experimental data except that the evaluation at mass A=100 is lower than experiments, since the JENDL evaluation is also lower. Fig.1(b) shows the inferred uncertainties by varying the experimental uncertainties in Fig.1(a) with a factor from 0.5 to 2.0. We see that uncertainties indeed increases with increasing experimental uncertainties but don't increase linearly. It is reasonable to see that the uncertainties are generally larger around two peaks. Fig.1(c) and Fig.1(d) show the predictions when the experimental data are deleted at the right or left peak, respectively. We clearly see that the resulted uncertainty of the absent data is larger than that with experimental data. Fig.1(e) and Fig.1(f) show the influences of discrepant data. In Fig.1(e), we add some extra data at the left peak that are not far from experiments with small uncertainties. We see the the evaluation moves toward the extra data with increased uncertainties. In Fig.1(f), we add some extra data that are far from experiments but with large uncertainties, However, we see the evaluation doesn't move significantly towards the extra data. Therefore the uncertainty prorogation is a comprehensive effect and would not significantly change due to a specific data.   Next we did the practical Bayesian data fusion of CFY of neutron-induced fission of 238 U from different experiments with different incident energies. The raw imperfect experimental fission yields are taken from the EXFOR library [9,35]. In BNN, input variables are given in terms of (Z, N , E), i.e., the atomic number Z, neutron number N of fragments and the neutron incident energy E. There are total 33 experiments with different energies of CFY of 238 U with about 1221 scattering data points. In addition, 2064 data points of evaluated CFY from JENDL at energies of 0.5 and 14 MeV are included as a learning constraint. Here BNN adopts a double-layer network with 20-20 neurons.
The yield-energy relations of some long-lived isotopes are of particular application interests for monitoring fission environments. For example, 135 Xe has large neutron absorption cross sections and is a "poison" for reactors [36]. Fig.2 shows the evaluated energy dependence of fission yields of 99 Mo, 135 Xe, 140 Ba and 147 Nd. In Fig.2, we see that the experimental data are indeed noisy, sparse and even discrepant. We see that data fusion can give reasonably the yield-energy relations and uncer-  tainty quantifications. Since JENDL evaluations are included, consequently BNN results are close to JENDL [6] and ENDF [5] [37]. The corresponding uncertainties become larger in this energy regime. In Fig.2, the inference uncertainties are larger than some experimental uncertainties but they are acceptable regarding the influences of discrepant data. The extrapolated yields at higher energies generally have increasing uncertainties. The BNN uncertainty includes two parts: the overall regression noise and the data-dependent posterior uncertainty [33], while they are not exclusive. In Fig.2, there is an overall noise scale of ∼0.6 which is related to the description capability of BNN and can be reduced by more complicated neural network or physics-informed neural network. Besides, Fig.1 and Fig.2 demonstrate that the experimental uncertainty, data sparsity and data discrepancy are reflected by the data-dependent uncertainty. Usually different experiments adopt different criterions for normalization of fission yields. The BNN evaluations can be further improved by careful calibrations of raw experimental data. Fig.3 displays the two-dimensional CFY distributions of neutron-induced fission of 238 U at energies of 2, 8 and 14 MeV from BNN data fusion. Recent experiments have made great progress in measurements of the isotopic mass yields using the inverse kinetic method [34], and it is still very difficult to obtain the complete distributions. The two-dimensional CFY distribution is very different from Gaussian distributions and it is difficult to be evaluated directly by semi-empirical models. In principle, our results include the yield-energy relations of all fragments. We can see that the highest yields are around Z=53∼54 and N =79∼81, which decrease gradually with increasing energies. It is known that symmetric fission would become more prominent as excitation energy increases [38,39]. The two peaks associated with asymmetric fission modes reduce with increasing energies, while the central distributions associated with the symmetric fission mode are increasing. Therefore, the energy dependence of fission modes can be reasonably described by BNN data fusion. Fig.3 (d, e, f) shows the corresponding uncertainties. There is an overall background noise scale ∼0.6 for all data. Obviously the uncertainty is energy dependent. The average uncertainties of Fig.3 (d, e, f) are 0.789, 0.805 and 0.793 for 1032 points, respectively. The larger uncertainty at 8 MeV is related to the data sparsity around this energy regime. The uncertainties of some FPY are as large as 1.4 where CFY have large values. The relative uncertainties around peaks are actually smaller. The one-dimensional charge yields can also be extracted from the two-dimensional CFY distributions. The precise and complete CFY are crucial to estimate the abnormal anti-neutrino spectrum at reactors [4] and our results will be useful for such studies.
Different from CFY, there are much fewer experimen- , in which CFY and IFY share the mass number A of fragments and the incident energy E, but their proton numbers are deliberately separated into two groups as Z I i and Z C i . Such a scheme is based on the fact that CFY mainly results from β-decays of IFY but they have very different distributions. The heterogeneous datasets are learned by the same network so that their multi-dimensional correlations are naturally included. The evaluated IFY from JENDL at 0.5 MeV and 14 MeV are used in the learning and our main purpose is to interpolate the energy dependence at other energies. Fig.4 displays the evaluation of IFY at 8 MeV with learning of pure IFY and heterogeneous CFY and IFY, respectively. The homogeneous fusion of IFY adopts a 16-16 network and the heterogeneous fusion of CFY and IFY adopts a 20-20 network. There are almost no experimental IFY data between 2 and 14 MeV. It is known that the energy dependence of PFY is nonlinear from Fig.2. Thus the evaluation of IFY at 8 MeV is not reliable by learning only IFY data. We see that there are almost no FPY from the symmetric fission between two peaks in Fig. 4(a), but there are considerable symmetric fission FPY in Fig. 4(b). This demonstrated that the heterogeneous data fusion of CFY and IFY can build the energy dependence information of CFY into IFY. The resulting IFY in Fig. 4(b) are consistent with the CFY in Fig.3(b). Fig. 4(c)(d) displays the uncertainties correspondingly. We see that the homogeneous fusion has a much smaller overall noise but some points have large uncertainties. The heterogeneous fusion has an overall background noise scale about 0.2 due to the influence of CFY. Indeed, it has been pointed out that cross-experiment correlations would result in increased final estimated uncertainties [14]. On the other hand, without the background noise, the data-dependent uncertainties of heterogeneous data fusion are generally smaller than that of the homogeneous fusion.
In summary, we have applied Bayesian machine learning for the evaluation of imperfect fission yields, in which the raw experimental data are generally noisy, incomplete, discrepant and correlated. The Bayesian data fusion can be used to augment the inference of energy dependence of imperfect fission data by including underlying correlations, which is a crucial need for next-generation nuclear energy productions. The twodimensional distributions of cumulative fission yields in terms of energy dependence are obtained. The evolution of fission modes is revealed by the smooth transition of two-dimensional fission yields. The yield-energy relations of key fragments are now given with uncertainty quantifications, in which energy dependencies of uncertainties are data dependent. We also applied the heterogeneous data fusion to interpolate the energy dependence of independent fission yields which have very few experimental data. We expect this method can also be applied to the heterogeneous data fusion of multiple fission observables and flexible physics constraints for a comprehensive understanding of nuclear fission.