Actor-Critic Instance Segmentation

Most approaches to visual scene analysis have emphasised parallel processing of the image elements. However, one area in which the sequential nature of vision is apparent, is that of segmenting multiple, potentially similar and partially occluded objects in a scene. In this work, we revisit the recurrent formulation of this challenging problem in the context of reinforcement learning. Motivated by the limitations of the global max-matching assignment of the ground-truth segments to the recurrent states, we develop an actor-critic approach in which the actor recurrently predicts one instance mask at a time and utilises the gradient from a concurrently trained critic network. We formulate the state, action, and the reward such as to let the critic model long-term effects of the current prediction and incorporate this information into the gradient signal. Furthermore, to enable effective exploration in the inherently high-dimensional action space of instance masks, we learn a compact representation using a conditional variational auto-encoder. We show that our actor-critic model consistently provides accuracy benefits over the recurrent baseline on standard instance segmentation benchmarks.

Introduction
Methods for instance segmentation have for the most part relied on the idea of parallel processing of the image elements and features within images [13]. However, previous work [31,32] suggests that instance segmentation can be formulated as a sequential visual task, akin to human vision, for which substantial evidence has revealed that many vision tasks beyond eye movements are solved sequentially [36]. While the segmentation accuracy of feed-forward pipelines hinges on a large number of object proposals, proposal-free recurrent models have a particular appeal for instance segmentation where the number of instances is unknown. Also, the temporal context can facilitate a certain order of prediction: segmenting "hard" instances can be improved by conditioning on the masks of "easy" instances segmented first (e.g., due to occlusions, ambiguities in spatial context etc.; [20]).
A pivotal question of a recurrent formulation for instance segmentation is the assignment of the ground-truth segments to timesteps, since the order in which they have to be predicted is unknown. Previously this was addressed using the Kuhn-Munkres algorithm [18], computing the maxmatching assignment. We provide some insight, however, that the final prediction ordering depends on the initial assignment. Furthermore, the loss for every timestep is not informative in terms of its effect on future predictions. Intuitively, considering the future loss for the predictions early on should improve the segmentation accuracy at the later timesteps. Although this can be achieved by unrolling the recurrent states for gradient backpropagation, such an approach quickly becomes infeasible for segmentation networks due to high memory demands.
In the past years, reinforcement learning (RL) has been showing promise in solving increasingly complex tasks [23,26,27]. However, relatively little work has explored applications of RL outside its conventional domain, which we attribute to two main factors: (1) computer vision problems often lack the notion of the environment, which provides the interacting agent with the reward feedback; (2) actions in the space of images are often prohibitively highdimensional, leading to tough computational challenges.
Here, we use an actor-critic (AC) model [5] to make progress regarding both technical issues of a recurrent approach to instance segmentation. We use exploration noise to reduce the influence of the initial assignments on the segmentation ordering. Furthermore, we design a reward function that accounts for the future reward in the objective function for every timestep. Our model does not use bounding boxes -often criticised due to their coarse representation of the objects' shape. Instead, we built on an encoder-decoder baseline that makes pixelwise predictions directly at the scale of the input image. To enable the use of RL for instance segmentation with its associated highdimensional output space, we propose to learn a compact action-space representation through the latent variables of a conditional variational auto-encoder [16], which we integrate into a recurrent prediction pipeline. To  c 2019 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.
Our experiments demonstrate that our actor-critic model improves the prediction quality over its baseline trained with the max-matching assignment loss, especially at the later timesteps, and performs well on standard instance segmentation benchmarks.

Related Work
Instance segmentation has received growing attention in the recent literature. One family of approaches focuses on learning explicit instance encodings [4,9,17,24,35], which are then clustered into individual instance masks using post-processing. Another common end-to-end approach is to first predict a bounding box for each instance using dynamic pooling and then to produce a mask of the dominant object within the box using a separate segmentation network [13,22]. These methods are currently best-practice, which can be attributed to the maturity of deep networkbased object detection pipelines. However, this strategy is ultimately limited by the detection performance, proposal set, and the need of additional processing to account for pixel-level context [2,8].
Making the predictions sequentially points at an alternative line of work. Romera-Paredes & Torr [32] used a convolutional LSTM [39] with a spatial softmax, which works well for isotropic object shapes and moderate scale variation. At each timestep, the recurrent model of Ren & Zemel [31] predicts a box location and scale for one instance. However, the extent of the available context for subsequent segmentations is limited by the box. Some benefits of the temporal and spatial context have been also reasserted on the task of object detection [7,21] and, much earlier, on image generation [12] and recognition [19]. In contrast to these works, our method obviates the need for the intermediate bounding box representation and predicts masks directly at the image resolution.
We cast the problem as a sequential decision process, as is studied by reinforcement learning (RL; [34]). Using the actor-critic framework [5], we define the actor as the model that sequentially produces instance masks, whereas the critic learns to provide a score characterising the actor's performance. Leveraging this score, the actor can be trained to improve the quality of its predictions. This is reminiscent of the more recent Generative Adversarial Networks (GANs; [11]), in which a generator relies on a discriminator to improve. In particular, our model is similar to Wasserstein GANs [1] in that the discriminator is trained on a regression-like loss, and to SeqGAN [40] in that the generator's predictions are sequential.
One obstacle is the action dimensionality, since the sampling complexity required for exploration grows exponentially with the size of the actions. A naive action representation for dense pixelwise predictions would lead to an action space of dimension in the order of O(2 H×W ) for im- Figure 1. Illustration of the max-matching assignment for instance segmentation. Consider an image with ground-truth instances a and b, and a recurrent model making predictions 1 and 2. In the constructed bipartite graph, each edge is assigned a weight corresponding to the IoU of the prediction with the connected ground truth. From the set of possible assignments, depicted by the orange and grey edges, max-matching finds the one that maximizes the sum of the IoUs. The loss is then computed independently for each timestep w.r.t. to this assignment to the ground truth.
ages with resolution H × W . This is significantly higher than the action spaces of standard problems studied by reinforcement learning (usually, between 1 and 20), or even its applications to natural language processing [3,30]. To address this, we suggest learning a compact representation using variational auto-encoders [16] to enable the crucial reduction of the problem from a high-dimensional discrete to a lower-dimensional continuous action space.

Motivation
As discussed above, we follow previous work in modelling instance segmentation as a sequential decision problem [31,32], yielding one instance per timestep t.
To motivate our work, we revisit the standard practice of using the Kuhn-Munkres algorithm [18] to assign the ground-truth instances to the predictions of a recurrent model. Let θ parametrise the model and U θ ∈ R n×n denote a matrix of elements u ij measuring the score of the i th prediction w.r.t. the j th ground truth (e.g., the IoU). The Kuhn-Munkres algorithm finds a permutation matrix as a solution to the max-matching problem where P is the set of n-dimensional permutation matrices, i.e. such that for all P ∈ P, we have j P ij = 1, i P ij = 1, P ij ∈ {0, 1}. Given a differentiable loss function l θ (i, j) (e.g., the binary cross-entropy), the model parameters θ are then updated to minimise ij,Pij =1 l θ (i, j). Consider a simple case of two ground-truth segments, a and b, illustrated in Fig. 1. Without loss of generality, assume that the initial (random) model parameters yield u 1a +u 2b < u 1b +u 2a , i.e. the sum of scores for segmenting instance a first and b second is lower than in the opposite order. This implies that max-matching will perform a gradient update step maximising the second sum, i.e. u 1b + u 2a , but not the first. As a consequence, for the updated parameters, the score for the ordering b → a is likely to dominate also at the later iterations of training. 1 Previous work [20,37] suggests that sequential models are not invariant to the order of predictions, including object segments (c.f . supplemental material). The implication from the example above is that sup u 1a + u 2b = sup u 1b + u 2a (the sup is w.r.t. θ). One conceivable remedy to alleviate the effect of the initial assignment is to introduce noise to the score matrix U (e.g., i. i. d. Gaussian), such that Eq. (1) becomes arg max P ∈P tr (U θ + )P . ( However, the noise in the loss function will not account for the inherent causality of the temporal context in recurrent models: perturbation of one prediction affects the consecutive ones.
In this work, we consider a more principled approach to encourage exploration of different orderings. We inject exploration noise at the level of individual predictions made by an actor network, while a jointly trained critic network keeps track of the long-term outcome of the early predictions. This allows to include in the gradient to the actor not only the immediate loss, but also the contribution of the current prediction to the future loss. We achieve this by reformulating the instance segmentation problem in the RL framework, which we briefly introduce next.

Notation and Definitions
In the following, we define the key concepts of Markov decision processes (MDPs) in the context of instance segmentation. For a more general introduction, we refer to [34].
We consider finite-horizon MDPs defined by the tuple (S, A, T, r), where the state space S, the action space A, the state transition T : S × A → S, and the reward r : S × A → R are defined as follows.
State. The state s t ∈ S of the recurrent system is a tuple of the input image (and its task-specific representations) and an aggregated mask, i.e. s t = (I, M t ). The mask M t simply accumulates previous instance predictions, which encourages the model to focus on yet unassigned pixels. Including I enables access to the original input at every timestep.
Action. To limit the dimensionality of the action space, we define the action a t ∈ A in terms of a compact mask representation. To achieve this, we pre-train a conditional variational auto-encoder (cVAE; [16]) to reproduce segmentation masks. As a result, the action a t ∈ A = R l is a continuous latent representation of a binary mask and has dimensionality l H · W , while the decoder D : R l → R H×W "expands" the latent code to a full-resolution mask.
State transition. As implied by the state and action definitions above, the state transition uses a pixelwise max of the previous mask and the decoded action, i.e. integrating the currently predicted instance mask into the previously accumulated predictions.
Reward. We design the reward function to measure the progress of the state transition towards optimising a certain segmentation criterion. The building block of the reward is the state potential [28], which we base on the max-matching assignment of the current predictions to the ground-truth segments, i.e.
where T i and S 1≤i≤t are the N ground-truth masks and t predicted masks; P(N ) is a collection of all permutations of the set {1, 2, ..., N }. F(·, ·) denotes a distance between the prediction and a ground-truth mask and can be chosen with regard to the performance metric used by the specific benchmark (e.g., IoU, Dice, etc.). We elaborate on these choices in the experimental section. The state potential in Eq. (4) allows us to define the reward as the difference between the potentials of subsequent states Note that since the (t + 1) st prediction might re-order the optimal assignment (computed with the Kuhn-Munkres algorithm), our definition of the reward is less restrictive w.r.t. the prediction order compared to previous work [31,32], which enforces a certain assignment to compute the gradient. Instead, our immediate reward allows to reason about the relative improvement of one set of predictions over another.

Overview
The core block of the actor model, shown in Fig. 2, is a conditional variational auto-encoder (cVAE; [16]). The encoder computes a compact vector of latent variables, encoding a full-resolution instance mask. The decoder recovers such a mask from the latent code. Using the transition function defined by Eq. (3), the latest prediction updates the state, and the procedure repeats until termination.
The actor relies on two types of context with complementary properties. As discussed above, the mask M t is a component of the state s t , which accumulates the masks produced in the previous steps. It provides permutationinvariant temporal context of high-resolution cues, encouraging the network to focus on yet unlabelled pixels. The Figure 2. Actor-critic model for instance segmentation. The model relies on two types of context: a spatial permutation-invariant state st accumulates the masks, whereas the hidden LSTM state ht models a temporal context sensitive to the prediction ordering. The State Pyramid propagates the high-res information at multiple scales to the decoder to compensate the loss of resolution at the bottleneck section.
hidden state of actor h t is implemented by the LSTM [14] at the bottleneck section and is unknown to the critic. In contrast to state s t , the representation of the hidden state is learned and can be sensitive to the prediction ordering due to the non-commutative updates of the LSTM state. The hidden state, therefore, contributes to the temporal context and is shown to be particularly helpful for counting in the ablation study.
We train our model in two stages as described next.

Pre-training
We pre-train the actor cVAE to reconstruct the mask of the target segment. The input to the network consists of the image and the binary mask of a randomly chosen groundtruth instance. To account for the loss of high-resolution information at the latent level, the decoder is conditioned on the input image and auxiliary channels of instance-relevant representations supplied at multiple scales, which we term State Pyramid. One channel contains the foreground prediction, while the other 8 channels encode the instance angle quantisation of [35], thereby binning the pixels of the object segment into quantised angles relative to the object's centroid. These features assist in instance detection and disambiguation, since a neighbourhood of orthogonal quantisation vectors indicates occlusion boundaries and object centroids. Following Ren & Zemel [31], we predict the angles with a pre-processing network [25] trained in a standalone fashion.
The auto-encoder uses the binary cross-entropy (BCE) as the reconstruction loss. For the latent representation, we use a Gaussian prior with zero mean and unit variance. The corresponding loss function is taken as the Kullback-Leibler divergence [16].

Training
During training, we learn a new encoder to sequentially predict segmentation masks. In addition to the image, the encoder also receives the auxiliary channels used during pre-training. In contrast, however, the encoder is encouraged to learn instance-sensitive features, since the decoder expects the latent code of ground-truth masks.
Algorithm 1 provides an outline of the training procedure. The actor is trained jointly with the critic from a buffer of experience accumulated in the episode execution step. In the policy evaluation, the critic is updated to minimise the error of approximating the expected reward, while in the policy iteration the actor receives the gradient from the critic to maximise the Q-value.
Episode execution. For an image with N instances, we define the episode as the sequence of N predictions. The algorithm randomly selects a mini-batch of images without replacement and provides it as inputs to the actor. Using the reparametrisation trick [16], the actor samples an action corresponding to the next prediction of the instance mask. The results of the predictions and corresponding rewards are saved in a buffer. At the end of each episode, the target Q-value can be computed for each timestep t as a sum of immediate rewards.
Policy evaluation. The critic network parametrised by φ, maintains an estimate of the Q-value defined as a function of the state and action guided by the policy µ: Note the finite sum in the expectation due to the finite number of instances in the image. The critic's loss L Critic,t for timestep t is defined as the squared L 2 -distance with a discounted sum of rewards where γ ∈ (0, 1) is a discount factor that controls the time horizon, i.e. the degree to which future rewards should be accounted for. The hyperparameter γ allows to trade off the time horizon for the difficulty of the reward approximation: as γ → 1, the time horizon extends to all states, but the critic has to approximate a distant future reward based only on the current state and action. We update the parameters of the critic to minimise Eq. (7) using the samples of the state-actions and rewards in the buffer, and set γ = 0.9 throughout our experiments.
Policy iteration. The actor samples an action a t ∈ A from a distribution provided by the current policy µ θ : S → A, parametrised by θ and observes a reward r t computed by Eq. (5). Given the initial state s 1 , the actor's goal is to find the policy maximising the expected total reward, , approximated by the critic. To achieve this, the state s t = (I, M t ) and the actor's mask prediction m t are passed to the critic, which produces the corresponding Q-value. The gradient maximising the Q-value is computed via backpropagation and returned to the actor for its parameter update.
We found that fixing the decoder during training led to faster convergence. Since the critic only approximates the true loss, its gradient is biased, which in practice can break the assumption we maintain during training -that an optimal mask can be reconstructed from a normally-distributed latent space. We fix the decoder and maintain the KLdivergence loss L KL while sampling new actions, thus encouraging exploration of the action space. In our ablation study, we verify that such exploration improves the segmentation quality. Note that we do not pre-define the layout of the actions, but only maintain the Gaussian prior.
To further improve the stability of the joint actor-critic training, we use a warm-up phase for the critic: episode execution and update of the critic happen without updating the actor for a number of epochs. This gives the critic the opportunity to adapt to the current action and state space of the actor. We could confirm in our experiments that pretraining the decoder was crucial; omitting this step resulted in near-zero rewards from which it proved to be difficult to train the critic even with the warm-up phase.
Termination. We connect the hidden state h t and the last layer preceding it (via a skip connection) to a single unit predicting "1" to continue prediction, and "0" to indicate the terminal state. Using the ground-truth number of instances, we train this unit with the BCE loss.
// Batch-update actor using critic Initialise states s 1 from buffer Inference. We recurrently run the actor network until the termination prediction. To obtain the masks, we discard the deviation part and only take the mean component of the action predicted by the encoder and pass it through the decoder. We do not use the critic network at inference time.
Implementation details. 2 We use a simple architecture similar to [32] for both the critic and the actor networks trained with Adam [15] until the training loss on validation data does not improve (c.f . supplemental material).

Discussion
In the actor-critic model the critic plays the role of modelling the subsequent rewards for states s i>t given state s t . Hence, if the critic's approximation is exact, the backpropagation through time (BPTT; [38]) until the first state is not needed: to train the actor, we need to compute the gradient w.r.t. the future rewards already predicted by the critic. The implication of this property is that memory-demanding networks, such as those for dense prediction, can be effectively trained with truncated BPTT and the critic, even in case of long sequences. Moreover, using the critic's approximation allows the reward be a non-differentiable, or even a discontinuous function tailored specifically to the task.

Experiments
In our experiments, we first quantitatively verify the importance of the different components in our model and investigate the sources of the accuracy benefits of the actorcritic over the baseline. Then, we use two standard datasets of natural images for the challenging task of instance segmentation, and compare to the state of the art.

Ablation study
We design a set of experiments to investigate the effect of various aspects of the model using the A1 benchmark of the Computer Vision Problems in Plant Phenotyping (CVPPP) dataset [33]. It comprises a collection of 128 images of plants taken from a top view with leaf annotation as groundtruth instance masks. We downsized the original 128 images in the training set by a factor of two and used a centre crop of size 224 × 224 for training. For the purpose of the ablation study, we randomly select 103 images from the CVPPP A1 benchmark for training and report the results on the remaining 25 images.
To compute the reward for our actor-critic model (Eq. 4), we use the Dice score computed as F(S, T ) = 2 i SiTi i Si+ i Ti . The dimensionality of the latent action space is fixed to 16. In the first part of the experiment, we look into how different terms in the loss influence the segmentation quality, measured in Symmetric Best Dice (SBD), and the absolute Difference in Counting (|DiC|). Specifically, we train five models: BL is an actor-only recurrent model trained with BPTT through all states. We use the BCE loss and Dicebased max-matching as a heuristic for assigning the ground truth to predictions, similar to [31,32]. BL-Trunc is similar to BL, but is trained with a truncated, one-step BPTT. We train our actor-critic model AC-Dice with the gradient from the critic approximating the Dice score. AC-Dice-NoKL is similar to the AC-Dice model, i.e. the actor is trained jointly with the critic, but we remove the KL-divergence term, which encourages exploration, from the loss of the actor. Lastly, we verify the benefit of the State Pyramid, the multi-res spatial information provided to the decoder, by comparing to a baseline without it (AC-Dice-NoSP).
The side-by-side comparison of these models summarised in Table 1 reveals that AC-Dice exhibits a superior accuracy compared to the baselines, both in terms of Dice and counting. Using the KL-divergence term in the loss improves the actor, which shows the value of action exploration in a consistent action space. We also observed that training AC-Dice-NoKL would sometimes diverge and require a restart with the critic warm-up. Furthermore, the State Pyramid aids the decoder, as removing it leads to a significant drop in mask quality. Surprisingly, BL-Trunc is only slightly worse than BL, which however has by far  . Dice score of our actor-critic model (AC-Dice) vs. our baseline with truncated BPTT (BL-Trunc) on CVPPP val, averaged for each timestep. We observe the advantage of our actorcritic model at later timesteps, which is an expected benefit of including the estimate of the expected reward in the loss at the earlier timesteps. Note that few images contain 20 instances, hence a large variance for this timestep. higher memory demands than both AC-Dice and BL-Trunc in the setting of long sequences and high resolutions.
To further investigate the accuracy gains of the actorcritic model, we report the average Dice score w.r.t. the corresponding timestep of the prediction in Fig. 3. The histogram confirms our intuition that incorporating the future reward into the loss function for every timestep, as modelled by the critic, should improve the segmentation quality at later stages of prediction: the Dice score of the actor-6 Model SBD ↑ |DiC| ↓ RIS [32] 66.6 1.1 MSU [33] 66.7 2.3 Nottingham [33] 68.3 3.8 IPK [29] 74.4 2.6 DLoss [9] 84.2 1.0 E2E [31] 84.9 0.8 Ours (AC-Dice) 79.1 1.12 Table 3. Segmentation quality of our actor-critic model on CVPPP test with Dice-based reward (AC-Dice) in terms of Symmetric Best Dice (SBD) and absolute Difference in Counting (|DiC|).
critic model tends to be tangibly higher especially at the later timesteps. Note that the contribution of this benefit to the average score across the dataset is moderated by not all images in the dataset having many instances.
In the next part of the experiment, we are interested in the reliance of the model on the recurrent state. Recall that our model maintains the mask accumulating the previous predictions as well as the hidden LSTM state. We alternately "block" either of the states by providing a zero tensor at every timestep. We consider only the first n predictions to compute the Dice score, where n is the number of groundtruth masks. We stop the iterations if no termination was predicted after 21 timesteps, since the largest number of instances in our validation set is 20. The results in Table 2 show that the LSTM plays an important role for counting (or, termination prediction), while having almost no effect on the mask quality. The networks have learned a sequential prediction strategy given only the binary mask of previously predicted pixels. Note that in contrast to the baseline models, actor-critic training reduced the dependence on the LSTM state for counting (AC-Dice), which suggests that the actor makes a better use of the state mask to make the next prediction.

Instance segmentation
We compare our method to other approaches on two standard instance segmentation benchmarks, each containing a rich variety of small segments as well as occlusions.
CVPPP dataset. For the CVPPP dataset used in our ablation study, this time we evaluate on the official 33 test images and train only our actor-critic model (AC-Dice) on the total 128 images in the training set.
The results on the test set in Table 3 show that our method is on par with the state of the art in terms of counting while maintaining competitive segmentation accuracy. From a qualitative analysis, see examples in Fig. 4a, we observe that the order of prediction follows a consistent, interpretable pattern: large leaves are segmented first, whereas small and occluded leaves are segmented later. This follows our intuition for an optimal processing sequence: "easy",  Table 4. Segmentation quality on KITTI. We evaluate our baseline with truncated BPTT (BL-Trunc) and the actor-critic with IoUbased reward (AC-IoU) in terms of mean weighted (MWCov) and unweighted (MUWCov) coverage, average false positive (AvgFP), and false negative (AvgFN) rates.
more salient instances should be predicted first to alleviate consecutive predictions of the "harder" ones. We also note, however, that the masks miss some fine details, such as the stalk of the leaves, which limits the benefits of the context for occluded instances. We believe this stems from the limited capacity of the critic network to approximate a rather complex reward function.
KITTI benchmark. We use the instance-level annotation of cars in the KITTI dataset [10] to test the scalability of our method to traffic scenes. We used the same data split as in previous work [31,35], which provides 3712 images for training, 144 images for validation, and 120 images for testing. While the validation and test sets have high-quality annotations [6,41], the ground-truth masks in the training set are largely (> 95%) coarse or incomplete [6]. Hence, good generalisation from the training data would indicate that the algorithm can cope well with inaccurate groundtruth annotation. The evaluation criteria for this dataset are: the mean weighted coverage loss (MWCov), the mean unweighted coverage loss (MUCov), the average false positive rate (AvgFP), and the average false negative rate (AvgFN). MUCov is the maximum IoU of the ground truth with a predicted mask, averaged over all ground-truth segments in the image. MWCov additionally weighs the IoUs by the area of the ground-truth mask. AvgFP is the fraction of mask predictions that do not overlap with the ground-truth segments. Conversely, AvgFN measures the fraction of the groundtruth segments that do not overlap with the predictions.
We use an IoU-based score function to compute the rewards, i.e.  The colourmap (middle) encodes the prediction order and ranges from blue (first prediction) to red (last prediction). Note how the prediction order follows a consistent pattern: large unoccluded segments tend to be segmented first, whereas small and occluded segments are usually predicted last.
benefits of our Actor-Critic model (AC-IoU) for structured prediction at higher resolutions, we also train and report results for a baseline, the actor-only model trained with one step BPTT (BL-Trunc). Considering the increased variability of the dataset compared to CVPPP, we used 64 latent dimensions for the action space.
The results on the test are shown in Table 4a. Given the relatively small size of the test set, we also report the results on the validation set in Table 4b, and use the available results from the equivalent evaluation of a state-of-the-art method [31] for reference.
The results indicate that our method scales well to larger resolutions and action spaces and shows competitive accuracy despite not using bounding box representations. Similar to our results on CVPPP, our model does not quite reach the accuracy of a recurrent model using bounding boxes [31] and a non-recurrent pipeline. We believe the segmentation accuracy is currently limited by the degree of the reward approximation by the critic and the representational power of the network architecture used by the actor model. As can be seen in some examples in Fig. 4b, without postprocessing the masks are not always well aligned with the object and occlusion boundaries. However, we note that the prediction order also follows a consistent, interpretable pattern: nearby instances are segmented first, while far-away instances are segmented last. Without hard-coding such constraints, the network appears to have learned a strategy that agrees with human intuition to segment larger, closeby objects first and exploits the resulting context to make predictions in the order of increasing difficulty.

Conclusions
In the current study, we formalised the task of instance segmentation in the framework of reinforcement learning. Our proposed actor-critic model utilises exploration noise to alleviate the initialisation bias on the prediction ordering. Considering the high dimensionality of pixel-level actions, we enabled exploration in the action space by learning a low-dimensional representation through a conditional variational auto-encoder. Furthermore, the critic approximates a reward signal that also accounts for future predictions at any given timestep. In our experiments, it attained competitive results on established instance segmentation benchmarks and showed improved segmentation performance at the later timesteps. Our model predicts instance masks directly at the full resolution of the input image and does not require intermediate bounding box predictions, which stands in contrast to proposal-based architectures [13] or models delivering only a preliminary representation for further post-processing, e.g. [9,35].
These encouraging results suggest that actor-critic models have potentially a wider application spectrum, since the critic network was able to learn a rather complex loss function to a fair degree of approximation. In future work, we aim to improve our baseline model of the actor network, which currently limits the attainable accuracy.

A. Ordering of Prediction
As we discussed in the main text, previous work suggests that the overall accuracy of a recurrent model is not invariant to the prediction order [20,37]. To confirm this experimentally for instance segmentation, we decouple the localisation and the segmentation aspect of a recurrent model in an oracle experiment. Reserving the location of the groundtruth segments as the oracle knowledge (e.g., with a bounding box) allows us to control the prediction order and study its role for the quality of the pixelwise prediction.
The setup for our oracle experiment is illustrated in Fig. 5. We supply the image and context patches (4 input channels) to a segmentation network in random order, while at the same time accumulating its predictions into the (global) context mask. We repeat this procedure 20 times for every image, with a different random order each. We use the CVPPP train/val split from our ablation study, and report the results on the 25 images in the validation set. The architecture of the network is the same as the segmentation module used by Ren and Zemel [31]. Figure 6a shows the statistic of the results (mean and standard deviation) for each of the 25 images. We observe that there is a tangible difference between the different segmentation orderings. The mean of the maximum and the minimum Dice across the images are 86.5 and 78.1 Dice, respectively, which corresponds to a potential gain of 8.4 mean Dice for the optimal prediction order. Notably, the deviation between the runs varies depending on the image. To investigate this further, we show two examples from the validation set with the highest (index #1, Fig. 6b) and the lowest (index #19, Fig. 6c) deviation. Example #1 appears to contain many more occlusions and more complex segment shapes (e.g., a single stalk) compared to example #19. This aligns well with our expectation: the benefit of the context should be more pronounced in more complex scenes.

B. Architecture
The architecture of the actor network and its parameters depending on the dataset used are shown in Tables 5 and 6. The critic has a similar architecture, but makes use of batch normalisation (BN) [46]. Additionally, one fully-connected layer (FC) from the last convolution is replaced by global maximum pooling. Further details are summarised in Table 7.
We use a vanilla LSTM [44] without peephole connections to represent the hidden state due to its established performance [45] and more economic use of memory compared to convolutional LSTMs [32,39].
The pre-processing network, which predicts the foreground and the angle quantisation of instances, is an FCN [25] with the same architecture as used by Ren and Zemel [31].

C. Training
To train our actor-critic model, we empirically found that gradually increasing the maximum sequence length leads to faster convergence than training on the complete length i (c) Example #19 Figure 6. Prediction of ground-truth segments in different order by using oracle bounding boxes centred on each of the target segments: (a) Mean Dice and standard deviation (error bars) of sequential mask prediction from 20 random permutations of oracle bounding boxes. The best and the worst prediction sequence averaged across the images yield 86.5 and 78.1 Dice, which implies the potential benefit of the optimal prediction ordering. (b) Example from CVPPP with the highest standard deviation, i.e. which can particularly benefit from the optimal ordering. (c) Example from CVPPP with the lowest standard deviation, i.e. where ordering of prediction has little effect on Dice. Note how the benefits of choosing an optimal ordering particularly occur in scenes with occlusions. from the start. Specifically, we train the actor to predict n remaining masks, where n starts with 1 and is gradually extended until the maximum number of instances in the dataset plus the terminations state (21 for CVPPP and 16 for KITTI) is reached. For example, with n = 1 the model learns to predict the last instance, i.e. the initial state s 1 input to the network is an accumulation of all but one ground-truth masks on a given image, which we choose randomly. We extend n by 5 once the segmentation accuracy on the validation set (either Dice or IoU) stops improving for several epochs.
We decrease the learning rate by a factor of 10 if we observe either no improvement or high variance of the results on the validation set. To trade-off the critic's gradient, we used a constant scalar β act = 10 −3 for the KL-divergence loss. The actor and critic networks use weight decay regularisation of 10 −5 and 10 −4 , respectively. We use Adam [15] for optimisation, as our experiments with SGD lead to considerably slower convergence. To manage training time, we downscale the original images with bilinear interpolation. The ground-truth masks are scaled down using nearest-neighbour interpolation. For CVPPP, we reduce the original resolution from 530 × 500 to 224 × 224, and for KITTI from the original 1242 × 375 to 768 × 256. We implemented our method in Lua Torch [43]. Both pre-training and training were performed on a single Nvidia Titan X GPU. The code is available at https://github.com/visinf/acis/.

D. Qualitative Examples
We provide qualitative segment-by-segment visualisations in Figs. 7 and 8 from the CVPPP and KITTI validation sets. Supporting our analysis in the main paper, the order of prediction exhibits a consistent pattern. Furthermore, we observe that our model copes well with inaccuracies of intermediate predictions -a common failure mode of recurrent networks [30].


extfieldsofstudy:
Physics
Computer Science

text:
Method of defining region of allowed configurations in multidimensional space of generalized coordinates using hypercube graph

The work is devoted to the development of a knowledge base on possible positions of an android arm mechanism with allowance for the position of the previously known forbidden zone. The knowledge base is used for intelligent control of the arm movement of an android robot that functions autonomously in an organized space without human-operator assistance. The specified knowledge base is determined by the region of allowed configurations in the multidimensional space of generalized coordinates. Based on the methods of geometrical modeling, a representation method of the allowable configuration region with the use of hypercube graph is proposed.

Introduction
A large number of studies are devoted to the problem of creating intelligent robots [1,2]. In solving this problem, computer virtual simulation of mechatronic object movements in organized environments is an important tool for analyzing and verifying the efficiency of the proposed algorithms for automatic control of manipulator mechanisms. One of the problems in developing intelligent control systems is to meet the requirements when collisions of the manipulator mechanism with the environment are absent [3][4][5][6]. These requirements are determined by the specified safe distance to the forbidden zones when a manipulator mechanism moves. This paper presents the results of studies on the determination of the set of allowed android arm configurations that meet the specified requirements. For this purpose, the varieties of points designated by the region Λ of the multidimensional space of generalized coordinates are determined, these points assigning configurations that satisfy the limit values of the generalised coordinates and the conditions of nonintersection with the forbidden zones. The robot control system should contain knowledge base about regions Λ for various forbidden zones that are known in advance. First it is necessary to create this base of knowledge that characterizes the property of maneuverability of the arm mechanism with allowance for forbidden zones, and then it may be used in synthesis of movements in the velocity vector in organized environment [7][8][9][10]. Defining the regions Λ is necessary for searching configurations that do not intersect forbidden zones. The creation of this knowledge base is quite a time-consuming task, as it is obtained as a result of processing a large array of points in a multidimensional space of generalised coordinates, taking into account the kinematic and geometric parameters of the investigated robot mechanism and the parameters of the shape and position of forbidden zones.

Problem Statement
In this paper a method for specifying the region Λ and on its basis the knowledge base of the robot using a graph of hypercubes is proposed. Figure1shows the kinematic diagram of the arm mechanism of the android robot AR-600E and the position of the forbidden zone P. The height of the base point A of the forbidden zone for the considered example in the coordinate system O o is equal to z op = 400 mm, the minimum safe offset of the manipulator base plate from the obstacle is equal to x op = 450 mm (Fig.  1).

Figure 1. Kinematic diagram of the android arm mechanism AR-600Е
The parameter z p = 100 mm determines the height of the forbidden zone (tool rack). Geometric parameters of the arm mechanism are specified in [7]. In the figure, the generalised coordinates are indicated by q i . The points O 1 -O 12 assign the coordinate origins used for specifying the kinematic chain model of the android arm mechanism. In order to determine the region of allowed configurations , taking into account the position of the forbidden zone P, in the work [7] a set of two-dimensional sections of the specified region in the four-dimensional generalised coordinate space is constructed. Particular positions of the arm mechanism links are investigated when q 1 = 0. The forbidden zone P, assigned by two horizontal and one profile planes of level Δ, Δ / and  (Fig. 2), defines in the space of generalized coordinates L q a certain region of the allowed configurations (RAC) Λ. A method for calculating the cross-section points of RAC to create a knowledge base about the past experience of motion synthesis in relation to the manipulator mechanism of a mobile robot was previously proposed in [8 -10]. In the work [7], a cross section representation method of the region Λ using a set of triangles is developed. However, the accuracy of the Λ region representation by this method requires further research in this field.
The operation of the manipulator mechanism takes place within a limited region  (four-dimensional space of generalised coordinates L q at q 1 = 0). Region  in a general case specified by hyperparallelepiped is that for any value of the vector q (q 2 , ..., q 5 ), specifying a point of the hypercube, the inequalities are satisfied: where min i q max i q are the lower and upper limits of generalised coordinates values. For this example, if q 1 = 0, the values of the parameter i satisfy the conditions 2 ≥ I ≥ 5. The values q i min and q i max for the mechanism of the android robot are respectively assumed to be equal to q i min (0 о , -25 о , -120 о , -120 о ) and q i max (120 о , 120 о , 120 о , 120 о ).

Theoretical Basis
The graph shown in figure 3 is used to assign the region Λ. Each node of this graph, which assigns hypercubes 1 1 at certain levels, allows specifying only such configurations of the android robot arm that satisfy the specified conditions of non-intersection with the forbidden zones. The advantages of the presented method of assigning the sets Λ compared with the known ones, are as follows: as a result of representation of the generalized coordinates space by elementary cells, it is possible to work not only with convex, but also with non-convex and composite geometric objects that define the region Λ; the method of studying the multidimensional space of generalized coordinates for the membership of arbitrary points in the regions Λ can significantly speed up the procedure for determining the conditions of non-intersection of the mechanism and the forbidden zones.
Let us note the main stages in the operation of this algorithm, in which the representation of the regions Λ of the multidimensional space L q is performed using a hypercube graph.
At the first step, the volume of the investigated multidimensional space L q is specified by a pdimensional hypercube. The side dimension of the hypercube (Fig. 4) is determined based on the ratio P p = 2q i max , where the parameter q i max is assigned in accordance with the specified limit values of the generalized coordinates q i max and q i min . The points Q p (φ 1k , φ 2k , ..., φ nk ) calculated according to a certain mesh step and belonging to a specified hypercube P p are checked for the condition of belonging to the set Λ. The values of the parameters q i , which specify points Q Λ , are found on the ratios q it = q it -1+q i maх , where q it are the values of the parameters q i at the current step t of the calculation, q it -1 is the value of the parameter q i at the previous step of the calculation, q i max is the maximum, originally adopted, increment of the parameters q i (Fig. 4). For the considered example q i max = 240•.
If one of the points Q Λ , represented in figure 4 by circles, turns out to belong to the set Λ (i.e. this point satisfies the conditions of non-intersection with forbidden zones), the space of the hypercube P p is divided into 2p p-dimensional hypercubes. The dimensions of new hypercubes P p /2 in this case are decreased, respectively, by a factor of two (Fig. 4)  The partition into hypercubes Р р /2 and examination of their points for membership to the set Λ occur as long as q max /2 > q min . Whereq min is specified initially. The parameter Δq min specifies the minimum mesh step in the study of points Q Λ , which is assumed to be Δq min = 20 o in this work.
Next, the hypercubes connection graph containing points Q p Λ is constructed (in general, the point O q , which defines coordinate origins of the space L q , may not belong to the region Λ). Note that if the points located on the boundaries of hypercubes belong to the set Λ, then the points of the boundary hypercubes are also investigated. The minimization of the computational time is achieved by excluding from the next calculation step those hypercubes describing the regions of the L q space in which the Q p  Λ points were not found at the current calculation step. In this context, this method of calculation allows us to study the region Λ at relatively small values of the parameter Δq min in less time.
The set of points Q p  Λ, which satisfies the conditions of non-intersection of configurations and forbidden zones, will determine the region Λ (Fig. 2). Defining the regions Λ is necessary for searching android arm configurations that do not intersect the forbidden zones. The found region Λ contains all possible values of the vector of generalized coordinates that meet the requirements of nonintersection of the arm mechanism with the forbidden zone.

Experimental Results
The set of sections of the allowed configurations region and their analytical assignment in the form of a hypercube graph is determined in advance based on the known forbidden zone position (for example, tool racks) and geometric and kinematic parameters of the arm mechanism. The region Λ defines the knowledge base and the model of the working environment. Based on the use of the Λ region, it is possible to create an algorithm to predict the development of events when the robot arm performs motor tasks. The algorithm for calculating the trajectory of the mechanism in the space of generalized coordinates using the construction of movements in the velocity vector and the region Λ is presented in [9]. The method of calculating the increments vector of generalized coordinates (the method of movement synthesis in the velocity vector) based on the use of matrices of partial gear ratios is described in [10]. The prediction of various situations when the manipulator moves is solved in paper [9] through the use of the mapping scale of the increment vector of the generalized coordinates, and the analysis of the trajectory position relative to the region Λ. Figure

Results Discussion
A more efficient way of finding the values of the generalized coordinate vector in the multidimensional space of generalized coordinates by traversing the nodes of a hypercube graph is proposed. Selection of nodes of this graph that define hypercubes 1 1 (Fig. 3) allows one to specify only such configurations of the android robot arm that do not intersect the forbidden zones.
The advantages of the presented method of assigning the regions Λ compared with the known ones, are as follows: as a result of description of the generalized coordinates space by elementary cells, it is possible to work not only with convex, but also with non-convex and separate regions Λ; the method of investigation of the multidimensional space of generalized velocities allows one to significantly speed up the procedure for determining the membership of an arbitrary point of the multidimensional space of generalized coordinates to the region Λ.
The region Λ for the android arm performing the movement in the specified velocity vector of OL is defined. It is proved that Λ regions can consist of separate subregions containing points Q  .


Burn image segmentation based on Mask Regions with Convolutional Neural Network deep learning framework: more accurate and more convenient

Background Burns are life-threatening with high morbidity and mortality. Reliable diagnosis supported by accurate burn area and depth assessment is critical to the success of the treatment decision and, in some cases, can save the patient’s life. Current techniques such as straight-ruler method, aseptic film trimming method, and digital camera photography method are not repeatable and comparable, which lead to a great difference in the judgment of burn wounds and impede the establishment of the same evaluation criteria. Hence, in order to semi-automate the burn diagnosis process, reduce the impact of human error, and improve the accuracy of burn diagnosis, we include the deep learning technology into the diagnosis of burns. Method This article proposes a novel method employing a state-of-the-art deep learning technique to segment the burn wounds in the images. We designed this deep learning segmentation framework based on the Mask Regions with Convolutional Neural Network (Mask R-CNN). For training our framework, we labeled 1150 pictures with the format of the Common Objects in Context (COCO) data set and trained our model on 1000 pictures. In the evaluation, we compared the different backbone networks in our framework. These backbone networks are Residual Network-101 with Atrous Convolution in Feature Pyramid Network (R101FA), Residual Network-101 with Atrous Convolution (R101A), and InceptionV2-Residual Network with Atrous Convolution (IV2RA). Finally, we used the Dice coefficient (DC) value to assess the model accuracy. Result The R101FA backbone network gains the highest accuracy 84.51% in 150 pictures. Moreover, we chose different burn depth pictures to evaluate these three backbone networks. The R101FA backbone network gains the best segmentation effect in superficial, superficial thickness, and deep partial thickness. The R101A backbone network gains the best segmentation effect in full-thickness burn. Conclusion This deep learning framework shows excellent segmentation in burn wound and extremely robust in different burn wound depths. Moreover, this framework just needs a suitable burn wound image when analyzing the burn wound. It is more convenient and more suitable when using in clinics compared with the traditional methods. And it also contributes more to the calculation of total body surface area (TBSA) burned.

Background
Burn injuries require immediate treatment by estimating the burn area and burn depth. Normally, this work is hard to solve by the general nurse or the doctor.

Current calculation methods
As shown in Fig. 1, the current assessment of a burn wound consists of three main methods [1][2][3][4]. The first method is to attach the sterile film to the wound of the patient, then draw the wound boundary with a marking pen, and finally calculate the area of the wound. This method is influenced by the subjective factors of the marking person and will bring about errors in some depths. The second method is to use the digital camera to calculate the wound area based on the principle of camera imaging. Then, this digital camera uses NIH ImageJ software to calculate the wound area. However, this software cannot automatically identify the edge of the wound, which makes it necessary to label the wound edge before calculating the wound area. The third method is to use the BurnCalc system [4]. The doctor uses this system to build 3D models of patients through specialized 3D scanning equipment, then draws the wound on the 3D model using a special drawing software, and finally calculates the wound area. This method needs to draw the artificial wound, which causes the final area calculation result to be highly inaccurate. In 2018, Cheah et al. designed a 3D application to calculate the burn area [5]. This application builds the 3D body model by the people's height and weight. Then, the doctors mark the burn area on the surface of the 3D model. After marking, the application can calculate the burn area automatically. However, this calculation relies on the precise burn area marked. Moreover, accurately marking the burn area is a time-consuming work which is not suitable for busy first-aid work.
We can conclude from above that the complicated and time-consuming part in these methods is to determine the wound region from the patient's skin, which is where many errors occur. To solve this problem, we used state-ofthe-art deep learning techniques to segment the wound region, and this method can be well applied to the calculation of the wound area.

Object detection
Analyzing object recognition and location in images is one of the most fundamental and challenging problems in computer vision. In the deep learning, Regions with Convolutional Neural Network (R-CNN) [6] is an object detection method in imagery analysis. This system consists of three parts. The first part generates category-independent region proposals by selective search [7]. The second part is a large convolutional neural network that extracts a fixed-length feature vector from each region. The last part is a set of class-specific linear support vector machines (SVM).
In order to realize faster training and evaluating, the Fast R-CNN [8] obtains feature vectors from the shared Fig. 1 Current mainstream diagnostic methods of burn wounds feature map. To avoid region proposals distortion, it adds a region of interest (RoI) pooling layer [8] on the basis of the R-CNN. Moreover, the author used the multi-task loss in the classification and regression to accelerate the system. In 2016, the Faster R-CNN [9] applied the region proposal network (RPN) [9] network to accelerate the speed of the generation of categoryindependent region proposals. With the powerful deep learning baseline, the object detection results can reach a high standard.

RPN
In the early object detection system, the generation of region proposals is exhaustive search. Then, in 2013, Uijlings et al. proposed a selective search to relieve the computational pressure [7]. Ren et al. then proposed an RPN which generates high-quality region proposals from the shared feature maps [9]. In generation, the RPN used a small sliding window on the shared feature map. Then, at each sliding-window location, the RPN generated multiple scale rectangle boxes on the original image, and these rectangle boxes are called anchors [9]. After that, the RPN network used a small convolutional neural network to predict the object score and box regression at each anchor. Then, the RPN sorted the score of each anchor and used the greedy non-maximum suppression to obtain region proposals.

Residual network (ResNet)
ResNet [10] was proposed in 2016 by Kaiming et al. In the convolutional neural network, the deeper the network is, the more features it can obtain. Although this feature of the convolutional neural network leads to a series of breakthroughs in image classification, it also brings a lot of new problems. For example, as the network depth deepens, the notorious problems of vanishing/exploding gradient occur. The deep residual learning network, which adds a reference on each layer to learn the residual function, can address the degradation problem properly. The main networks in the ResNet are the ResNet101 [10] and ResNet50 [10].

Feature pyramid network (FPN)
FPN [11] was proposed in 2017 by the Lin et al. This network can build high-level semantic feature maps at all scales. It contains three parts which are bottom-up, top-down, and lateral connections [11]. In general, ResNet is the backbone network of the FPN. The output feature map of each convolution layer is denoted as C 2 , C 3 , C 4 , and C 5 in the bottom-up part. The top-down part then upsamples the feature map by using a factor of 2, and the upsampled map is merged with the corresponding bottom-up map. Then a 3 × 3 convolution is appended on each merged map to generate the final feature map. The final outputs of feature maps are called P 2 , P 3 , P 4 , and P 5 .
In this article, we adopt the ResNet101 and Atrous [12] in FPN, and we called this backbone network as R101FA.

Mask R-CNN
In principle, Mask R-CNN is an intuitive extension of Faster R-CNN, yet building the mask branch properly is important for good results. Mask R-CNN is different from the Faster R-CNN in three points. The first point is that the Mask R-CNN adds a mask branch to predict the category of every pixel in the region proposals. The second point is that the use of the RoIAlign [13] to achieve the pixel-pixel alignment between network inputs and outputs. The third point is the definition of the mask loss. For each RoI associated with ground-truth class k, mask loss is only defined on the k-th mask (other mask outputs do not contribute to the loss).

Method
In this article, we used a novel method for employing the state-of-the-art deep learning framework Mask R-CNN. For a more refined result and a faster training speed, we modified the Mask R-CNN to adapt to our dataset. We changed the loss function of the class branch and adopted the Atrous Convolution [12] in the backbone network. In order to make a better segmentation result, we tried several mainstream backbone networks, and the R101FA demonstrated the best segmentation results. In order to obtain enough data, we used our smartphone to collect images of fresh burn wounds in the hospital every day. Then, we used our own software to annotate burn images and saved the marked content in Common Objects in Context (COCO) data set format. Figure 2 shows the software which we used to annotate. In order to ensure the accuracy of this framework, we annotated the burn images carefully under the guidance of the professional doctors and avoided mistaking confusing parts such as gauze and blood stains as wounds. With the help of doctors and nurses, we finally annotated 1000 burn images for training and another 150 for evaluating.

Network architecture
As shown in Fig. 3, our framework contains three parts. The first part is the backbone network to extract the feature maps. The second part is the RPN [9] network to generate the RoI [9]. Finally, we process the object detection and mask prediction from each RoI. Because there is only one category (here, we do not consider the depth of burn wound), we changed the loss function of the mask branch and classification branch to fit our data set. In the process of training, we collected almost all kinds of burn wound images to train our model, totaling 1000 after filtering. At the same time, in order to realize faster training speed and less evaluating time, we tried different backbone networks in our framework. Finally, we used the R101FA as the backbone network of our framework.
In this article, our backbone network is based on the R101FA. The ResNet101 is made up of 101 layers. We use C 1 , C 2 , C 3 , C 4 , and C 5 to define these output feature maps. As shown in Fig. 4, we obtain final feature maps P 2 , P 3 , P 4 , and P 5 .
Here, we use a 1 × 1 convolution kernel to get the first feature map P 5 by undergoing the output of the C 5 . Then, we upsample the P 5 to get the P * and produce C * by the 3 × 3 convolution kernel undergoing the C 5 , and the P 4 map is the result of merging C * with P * . After iterating all C, we can build P 2 , P 3 , P 4 , and P 5 .

Atrous
In the convolutional neural network, we employ atrous convolution [12] in ResNet. The traditional convolution  The kernel of atrous convolution is no longer a dense matrix, and it is shown by Fig. 5 that different rates represent different convolution kernels. Compared with traditional convolution kernels, employing a larger value of atrous rate enlarges the model's field-of-view, enabling object encoding at multiple scales. This structure is suited for our burn dataset which consists of varying burn depths and burn sizes. In this research, we set the rate at 2.

RPN in FPN
We adopt RPN in FPN to propose the candidate of region proposals. The detail is different from the original RPN network. The original RPN network just adopts one feature map, but in our network, we build several feature maps. In order to handle the images easier, we resized the images to 1024 × 1024 and filled the image with zero to prevent distortion. In order to contain all possible rectangular boxes, we defined five scales 32 × 32, 64 × 64, 128 × 128, 256 × 256, and 512 × 512. Every scale has three aspect ratios 0.5, 1, and 2. It was not necessary to define all scales on every feature map; we just defined one scale per feature map. Here, in order to correspond five scales, we added P 6 on the basis of P 5 and it is the output of the max-pooling after the P 5 . Hence, according to this idea, we can generate all possible rectangular boxes (anchor [9]) on the original image.
In the RPN network, we filter N numbers of RoIs by a small convolution network. This small network determines the object possibility of each anchor, and we call this possibility anchor score. We sorted all the anchors by this score and take the top N high score boxes as RoI. Moreover, in order to adjust the position of each anchor, the small network also predicts the regression offsets of each anchor. Therefore, in FPN, there are several feature maps and this small network is shared with all feature maps, and the detail is shown in Fig. 6.

RPN training
As shown in Fig. 6, the outputs of the RPN network are score and regression offsets of each anchor. Here, we define two loss functions to train the RPN network. The first is the score loss L rpnScore and the second is regression loss L rpnReg .
To calculate L rpnScore , we assign two kinds of labels which are the positive label and the negative label to each anchor. The anchor which has an intersection over union (IOU) overlap higher than 0.7 with any ground-truth bounding box is a positive label, and the anchor which has an IOU overlap lower than 0.3 with all ground-truth boxes is a negative label. Here, in order to ensure that all the ground-truth boxes correspond to at least one anchor, we will label the highest IOU anchor with each ground-truth box as a positive label. Therefore, we can get all the positive and negative anchors. We encode these anchors into a sequence of 0 and 1, Fig. 5 Atrous convolution. Atrous convolution with kernel size 3 × 3 and different rates. From left to right in the figure, the atrous rates are 1, 2, and 3. Standard convolution corresponds to atrous convolution with rate = 1. In this article, we set the rate as 2 and the sequence is the objective output in the RPN target judgment. As it is shown in Fig. 6, we apply the softmax function to the output of the RPN to get target possibility for all anchors. And then, we use the cross-entropy function to calculate the L rpnScore .
Then, we apply a linear function to the output of the RPN network and predict the regression parameters (t * ). We calculate the regression offsets (t) of each positive anchor. The regression offsets are the same as [8], and it contains four values (x, y, w, h). x and y are the offset ratios of the positive anchors' center point based on the associated ground-truth boxes center point. Then, w and h are the logarithmic values of the aspect ratio of positive anchors and the associated ground-truth boxes. Finally, we used the smooth L1 to calculate the L rpnReg which is shown in Eq. 1. Here, we stipulate that just the positive anchor will contribute the L rpnReg .
Here, i is the index of an anchor in the mini-batch and p Ã i is 1 if the anchor is positive; otherwise, p i * is 0. Here, t i and t Ã i are the four vectors representing the regression offset, and t i represents regression offset of a positive anchor based on the associated ground-truth box. And t Ã i represents the predicted regression offset. The regression loss function is shown in Eq. 2. The smooth L1 is defined in Eq. 3.
We used Eq. 4 to make a detailed explanation for regression offset.
After choosing the RoIs from the anchors, we map the RoIs on the feature map for subsequent operation of the framework. But in our framework, we have four feature maps. Unlike generating anchors, we do not make each RoI correspond to a feature map. Considering that P 2 contains all image features, we map all RoIs to P 2 . After mapping, the three parallel branches handle the mapping results.

Loss function
In our framework, our loss contains five aspects. The RPN network contains two losses. In the parallel branches, there are three losses. We define the three losses as L mCls , L mBReg , and L mMask . Therefore, our final loss is L =L rpnScore +L rpnReg +L mCls +L mBReg + L mMask .

Class loss
In Mask R-CNN, the author applied softmax on the output of the fully connected layer and used the cross-entropy function to calculate the class loss. This method was applied to solve the multi-class classification tasks. But in our task, our goal is simply to segment the burn wounds. Therefore, we used two classifiers to replace the multiple classifiers. We applied the sigmoid function on the output and used the cross-entropy function to calculate the loss. We used y to define the ground-truth of the N RoIs. The output of sigmoid is y * . Then, the L mCls is Eq. 5.
Bounding-box loss As mentioned above, the RPN network will predict the regression offset of each anchor. In the box branch, the input will be the coordinate of RoI. These coordinates are the result of the RoI that applied the regression offset of the RPN network. We then used the same way as L rpnReg to calculate the L mBReg.

Mask loss
In Mask R-CNN, the author applied a small FCN [14] network on the RoI. And in the mask branch, the author predicted the m × m mask. The mask is the output of the sigmoid function which is applied to each pixel. Then, the author calculated the mask loss according to the mask class to avoid the competition between classes and used the binary cross-entropy to define the loss. However, in this article, we just calculated the mask loss of the positive RoI and did not use the idea of competition between classes. Moreover, we defined the size of each ground-truth mask and predicted the mask as 28 × 28 to reduce memory consumption. Hence, the ground-truth RoI was scaled to 28 × 28 and padded with zero to avoid distortion. In the output of the mask branch, we will scale each RoI to the same size to calculate the mask loss.

Regularization loss
As mentioned above, we collected little data sets. And in order to prevent over-fitting of the model, we add the loss of regular term for entire loss function. We can see the details from the formula 6 This is the L2 regularization loss which represents the weight decay and aims to reduce the weight values to fit data well. In the formula, W i is the weight values of the i-th layer and N w i is the size of the W i . The λ is a hyper-parameter which is set as 0.0001 here.

Training detail
In order to obtain better training results, we did not randomly initialize weight in this framework. The initialization of the weight includes two parts. In the convolutional neural network, we used the pre-trained COCO model to initialize our backbone network. In the network head, we used the Gaussian distribution to initialize the weight values. Similar to the transfer learning, we fine-tune the convolutional neural network of our framework by collecting data.
Moreover, we tried several convolutional networks to extract the feature map from the original image. These backbone networks are Residual Network-101 with Atrous Convolution (R101A), Residual Network-101 with Atrous Convolution in Feature Pyramid Network (R101FA), and InceptionV2-Residual Network with Atrous Convolution (IV2RA). Through the experiment, we find the R101FA backbone has the best segmentation result. Before training, the images are resized to a 1024 width for a proper input in the network. Then, similar to the [10], the input data will undergo five convolution layers C 1 , C 2 , C 3 , C 4 , and C 5 which have strides of 2, 4, 8, 16, and 32 related to the input image.
After extracting a feature map, the RPN network handled the output of the backbone network. First, the RPN network generated N anchors (the anchor scale is for the original image) on the center of the sliding window. Then, we calculated the IOU value (per anchor) to judge if the anchor is positive or negative. As in [8,9], each image has N-sampled RoIs which have a ratio of 1:3 of positive to negatives. Then, we pooled each positive anchor to a fixed size. After that, we connected a fully connected network to extract a 2048 dimensions feature vector. This vector is used for the classifier and the box regressor. At the same time, the RoIs will undergo two convolution layers, then we predicted the image mask.

Burn area calculation
Burn area calculation is an important part of burn diagnosis. The framework as mentioned above is an auxiliary technique for calculating burn area and has great significance for fast, convenient, and accurate burn area calculation. As it is shown in Fig. 1, for example, the second method needs to manually mark the edge of the burn wound when calculating. Similar to the 3D application, it is not conducive for rapid treatment of patients. However, if we combine our segmentation framework with this software, we can get a more efficient and convenient area calculation tool. In a sense, we can apply our framework to the calculation of burn wound area.
In our plan, we intend to combine the 3D modeling and mesh parameterization technology with our segmented framework which calculates the burn wound area. The calculation mainly consists of three steps: Step-1: Building a 3D model of patient pictures through 3D reconstruction technology.
Step-2: Mapping the 3D model to the planar domain by mesh parameterization algorithm.
Step-3: Segmenting the burn regions by using our framework and calculate the total body surface area (TBSA) value.
Some 3D reconstruction technologies are already very mature, such as BodyTalk reconstruction system [15] and Kinect reconstruction system [16], which makes the 3D model building process easier. The mesh parameterization algorithm such as RiccFlow [17] and Authalic [18] make the second step easier to implement. Hence, our segmentation framework can achieve a faster, easier, and more accurate area calculation.

Burn characterization
There are four main depths of burn wounds: (i) superficial dermal burn, (ii) superficial partial thickness burn, (iii) deep partial thickness burn [19], and (iv) full-thickness burn. Figure 7 shows the four depths of burns across four images.
In the past, image processing techniques for total or partial segmentation often use evolving curvilinear boundaries because of their adaptive capacity and modeling of the internal structures of the images [19,20]. In this article, we come to the conclusion that the burn does not exhibit uniform boundaries. Moreover, various depths of wounds make the segmentation work harder. The traditional technologies no longer work in the burn segmentation if we want to segment all burn situations. Hence, we adopt the deep learning framework to realize the segmentation.

Segmentation results
In this paper, our framework mainly segments the burn wounds without classifying the depth of the wound. However, in order to show the stability and generalization ability of our work, we selected 150 images to evaluate our method. With the help of professional doctors, we combined these images from the different burn area size images and different burn depth images.

Segment different sizes
The first advantage is that our model can segment the different sizes of the burn wounds. Through the experiment, our model expressed high robustness of the different burn wound sizes. We chose the four depths of burn wounds. As it is shown in Fig. 8, our model performed fine segmentation in the %TBSA < 5% burn wound. Moreover, for the large burn area, our model also performed very well. This is also shown in Fig. 8.

Segment different depth
There are many reasons to cause a burn such as hydrothermal fluid, high-temperature gas, and flame. In addition, these reasons can lead to different burn wounds depths. Because of the variance of each depth of burn, it increases the difficulty of segmentation. However, in our model, we can segment the different burn wound depths successfully. Figure 9 shows the segmentation results of different depth burns.

Method comparison
We compared our method with traditional methods and modern methods.

Traditional methods
In the image segmentation, the traditional methods always use the edge features or spectral feature of the image to accomplish the segmentation. In this article, we tried the Watershed Algorithm [21] for our burn images. The Watershed Algorithm is based on the edge of the image and also labels the spectral feature on the image. Most of the time, this algorithm uses the color histogram to decide which color is to be the watershed. But in the burn images, there are many colors similar to the burn wound. This element made it difficult for the watershed algorithm to segment the burn wounds. We finally tried a different parameter to get better segmentation. As shown in Fig. 10, we can see that the Watershed Algorithm could not show good segmentation in a complex picture environment.

Modern methods
In recent years, the image segmentation method which is based on deep learning achieves excellent performance. In this article, therefore, we chose different architectures to be the backbone network in our framework. These backbone networks are IV2RA [22], R101A [10], and R101FA.
In training, for the best training effect, we set 20 epochs for IV2RA and R101A. But in our method, we set 16 epochs. Each epoch contained 1000 iterations.
First, we will show the loss reduction of the different backbone networks. As shown in Fig. 11, we see that our method can achieve a better loss reduction than two of the other backbone networks. Moreover, our method used fewer epochs than the other backbone networks.
During the evaluation, we chose 150 burn images to evaluate the different backbone networks. Here, because we only considered the wound segmentation effect, there was no need to use the mean average precision to evaluate the model. Hence, we chose the Dice coefficient (DC) [23] to evaluate the percentage of segmentation accuracy from the ground truth. The DC measures the concordance between two enclosed areas. The formula is as follows: In detail, the number of false positives is the FP value. The false positive represents the incorrectly segmented pixels. The FN is the number of false negatives. The false negative represents the target pixels that are not segmented. TP is the true positive. The true positive represents the correct segmentation pixels. Therefore, we calculate the DC value of the different backbone networks. The result is shown in Table 1. From the DC values, we discover that our backbone network R101FA has the highest accuracy. In other words, our model has a better result in burn image segmentation.
To more fully evaluate our model, we chose the different burn images. We chose a total of 120 pictures of different burn depths. There were 20 superficial burns, 50 superficial thickness burns, 40 deep partial thickness burns, and 10 full-thickness burn. Because of a lack of burn images, for the full-thickness burn, we were only able to analyze 10. Then, we calculated the DC values of the different backbone networks.
As shown in Table 2, our model showed better segmentation in the superficial, superficial thickness, and  deep partial thickness. Perhaps due to the lack of full-thickness burn images, the results for full-thickness burn in our model were slightly worse than the other models.
On the other hand, there were many patients with extensive burns in the clinical. Therefore, our method needed to demonstrate excellent results in this aspect. Therefore, we selected different burn area size images to evaluate different models. The result is shown in Table 3.
As shown in Table 3, the IV2RA has the highest average DC value for 5%< %TBSA < 20% wounds and the R101A has the highest average DC value for the %TBSA < 5% wounds. At the same time, our method has the highest average DC value for the %TBSA > 20% wounds and also has good results in two other sizes.
In order to ensure efficient hospital treatment, the prediction time for each picture must be short. We compared the prediction time of the different backbone networks. As shown in Table 1, our models needed only 0.37 s to predict an image which was the fastest prediction speed.

Discussion
Burn image segmentation is the first step in the intelligent diagnosis of burn wounds. The precise segmentation is important to the follow-up treatment. In this article, we propose a state-of-the-art segmentation framework to segment the burn images. Compared with the traditional method, this method greatly improves the accuracy of segmentation and contributes immensely to the burn clinic. However, there are still some problems to be solved in this framework. As is known to all, deep learning technology requires a large number of data to ensure the accuracy of the model. In this framework, due to the complexity of data collection and annotation, we only provided almost 1000 pictures to train this model. This makes the model show bad segmentation results in some burn images. In addition, our framework cannot classify the depth of burn wound. In general, the evaluation of wound depth information needs to be combined with the professional knowledge of the doctor, which makes the process of data annotation extremely complicated and difficult for non-professionals to   complete. Later, we will collect enough data sets to train the framework to improve the accuracy of the segmentation. Moreover, we will mark the burn wound depth information with the help of the professional, so as to classify the burn wound depth in this framework. And then we will apply our framework to calculate burn area.

Conclusions
This article proposed a new segmentation framework to segment the burn images based on deep learning technology. In the comparison experiment, we compared the feature extraction capability of different backbone networks. We found that the R101FA backbone network has the best result in accuracy and prediction speed. Finally, we achieved an average of 84.51% accuracy on 150 images. In practice, our method is more convenient than traditional methods and requires only a suitable RGB wound picture. It brings great benefits to the clinical treatment of the hospital.