Benchmarking Safety Monitors for Image Classifiers with Machine Learning

High-accurate machine learning (ML) image classifiers cannot guarantee that they will not fail at operation. Thus, their deployment in safety-critical applications such as autonomous vehicles is still an open issue. The use of fault tolerance mechanisms such as safety monitors is a promising direction to keep the system in a safe state despite errors of the ML classifier. As the prediction from the ML is the core information directly impacting safety, many works are focusing on monitoring the ML model itself. Checking the efficiency of such monitors in the context of safety-critical applications is thus a significant challenge. Therefore, this paper aims at establishing a baseline framework for benchmarking monitors for ML image classifiers. Furthermore, we propose a framework covering the entire pipeline, from data generation to evaluation. Our approach measures monitor performance with a broader set of metrics than usually proposed in the literature. Moreover, we benchmark three different monitor approaches in 79 benchmark datasets containing five categories of out-of-distribution data for image classifiers: class novelty, noise, anomalies, distributional shifts, and adversarial attacks. Our results indicate that these monitors are no more accurate than a random monitor. We also release the code of all experiments for reproducibility.

I. INTRODUCTION
Image classifiers based on machine learning (ML) are core components for many safety-critical autonomous applications, like autonomous vehicles. ML models make decisions based on trained past data. A common way to validate them is by analyzing the discrepancy between the predicted values and the ground truth (labels) in a testing activity. If the results are satisfactory, it goes to production. However, even modern ML techniques, such as deep learning, can be wrong in their predictions even with 100% confidence [1]. This situation may lead to hazardous situations like the whole behavior of the system may rely on the ML decision. Therefore, it is now urgent to deploy techniques to increase the confidence in such ML classifiers. Among techniques coming from the dependability community [2], fault tolerance is a technique that could be applied to deliver a correct service despite the occurrence of errors. Safety monitors (SM) keep the system in a safe state despite hazardous situations [3]. Potentially, these situations may include errors from the ML model. Many recent works focus on monitors dedicated to the ML model surveillance. They broadly fall in three types of monitors: observation of the inputs of the ML model [4], its outputs [5] or from intermediate layers in case of deep neural networks (DNN) [6], [7]. However, they are all based on the exploitation of the training data. Therefore, we refer to them as data-based monitors compared to safety monitors based on rules (or safety properties). Similar to uncertainties inherent to the use of ML, the confidence in such SM is an open issue.
Thus, it is essential to estimate how efficient such monitors are and if it is possible to include them in safety monitoring. This estimation should consider all potential situations leading to an error of the ML and should be based on metrics dedicated to measuring the monitor efficiency. Therefore, we focus on a framework for benchmarking such monitors, augmented with an additional primary mechanism to inhibit the decision in error detection. As presented in Figure 1, the complete system under test is composed of an SM that surveil the ML.
Our benchmark adapts and extends current metrics used in the ML community to estimate the SM detection performance at runtime, its impact on the system, and the overhead induced by the use of the monitor. Our main contributions are: work that proposes an initial framework that benchmarks SM for ML components from different perspectives. • A comprehensive benchmark experiment containing different data-based SM implementations, datasets, and results. Our experiments reveal the advantages and drawbacks of the main modern data-based SM approaches for image classifiers built with ML. We perform experiments on five challenging and important types of out-ofdistribution data for image classifiers. This work is organized as follows: in Section II, we provide background on ML classifiers, threats that may affect their results, and current safety monitoring approaches. In Section III, we present an overview of our framework, along with its main objectives. In Section IV we present our experiments, datasets, and results of the application of our framework to three ML monitors. Finally, in Section V, we present our conclusions.

II. BACKGROUND
This work focuses on monitoring ML image classifiers, especially when exposed to out-of-distribution data that can threaten their performance at runtime. In the following subsections, we give a brief explanation of these concepts.

A. Image classifiers with ML
An ML classifier is a software component that uses an ML algorithm for identifying, given an observation, in which class it belongs [8]. ML classifiers take an input, such as an image or a vector of numerical values, and outputs categorical values (classification) according to a category (class) previously seen during the training process. This component, called the ML model, is usually validated by analyzing the discrepancy between the classified values and the ground truth (labels). If the results are satisfactory, it goes to production. These classifiers are integrated into many perception pipelines of autonomous systems. The prediction of the ML is at the very heart of the safety-critical functions of such systems (e.g., collision avoidance, path planning, and so on).
B. Out-of-distribution data ML models tend to be biased to the training data [9], resulting in a natural inability of a model to generalize 100% of time even if all available data could be collected. Beyond this fundamental model generalization issue, there is another problem: data incompleteness. That is, rare conditions tend to be underrepresented since the training data represents a subset of all real-world possibilities [10]. It means that data represents the same target classes from training data, but that has different characteristics, different enough to threaten ML performance. Such data is known as out-of-the-distribution data (OOD). In this paper, we consider five types of OOD data that can threaten an ML model at runtime: • Novel classes: a scenario where new classes are introduced during runtime. For example, an ML model can be trained to identify dogs but fails to classify a new dog type not present in the training data [11].
• Adversarial inputs: a situation when ML fails with high confidence due to small input modifications [12]. • Distributional shifts: a condition that decreases the ML performance through time since the training dataset may differ from the real inputs. Such a situation is also known as a concept-drift [13]. Examples of distributional shift include a change in class attributes such as dimension, physical characteristics, contrast, brightness, and other pixel-related variations in the image. • Noise: a possibility to receive deteriorated images due to small failures in exteroceptive sensors or unexpected environmental interference. • Anomalies: a severe failure from exteroceptive sensors capable of severely corrupt images. However, contrary to noise, such failures corrupt data so that such images lose semantic value. It means that whatever the ML decision is, it can correspond to an inaccuracy. Examples of such images are black images, images with several shifted pixels, and so on. Next, we detail the current approaches for monitoring ML.

C. From SM to ML monitors
Safety monitoring is a well-known dependability technique already used in embedded systems such as robots or autonomous vehicles. This approach is generally based on a system model or the environment and on properties they should guarantee. We will use rule-based monitors for such approaches (in opposition to data-based monitors presented in the following sub-section). Some techniques for monitoring safety components, consider this component as a blackbox [3]. It is possible to apply this approach to systems, including ML algorithms. However, as they are considered black-box, the fact that it is an ML-based component has no real impact on the SM design. For example, safety rules can be implemented in an SM to verify if a vehicle can completely stop before reaching an obstacle [14] or to avoid a collision [15]. In both cases, external (exteroceptive) sensors, such as distance sensors, are observed along with internal (proprioceptive) sensors, such as speed, to evaluate if a safety property is violated. It means that it was possible in these situations to have a redundant mechanism to monitor the ML.
However, when ML is in charge of image classification, in most cases, no redundant classification mechanism could be developed to monitor the ML. Indeed, it is usually impossible to use external sensors and external software not based on ML to confirm or invalidate an ML prediction.
One possible direction is then to verify some assertions at the ML levels, for instance, with the use of model assertions [16]. This technique is an adaptation of the classical program assertions to monitor and improve ML models. The idea is to verify inputs/outputs that indicate when errors may be occurring in the system. For example, monitoring if an object flickers in and flickers out in the camera indicates a possible failure. Even applying these techniques during design and operation cannot guarantee that a DNN decision is safe. The reason is that, for some corner cases, ML outputs wrong decisions that cannot be verified by inspecting the code logic or the sensor values and lead to hazards [17].
We can deploy rule-based monitors when redundant observation sources are applied and safety properties are correctly expressed. However, monitoring an ML component for vision is particularly complex to find a redundant source of observation. Moreover, it is not apparent to express a safety property at the level of an ML prediction. For these reasons, a current approach is to develop monitors based on the ML training data: the data-based monitors.

D. Data-based monitors
This approach is usually based on data instead of using a system model or specifications provided by the developers. However, it means that we develop the monitor itself using the same training data as the ML. We propose to classify such monitors in three categories: monitors based on ML inputs, intermediate values, or outputs: 1) Monitoring DNN inputs: An example of this approach is to adversarially train an ML model to reconstruct a noisy image to an image similar to a previous known class [18]. It means that a new image arrives at runtime, and this ML model tries to reconstruct this image to a known one from the set of known classes previously seen in the training phase. Suppose there is a considerable difference between the reconstructed image and the actual image. In that case, this actual image is from an unknown class, indicating possible OOD data. A similar idea is tested in [19], in which the authors apply the Mahalanobis distance to improve the autoencoder reconstruction process in order to capture OOD samples better.
Another possible solution based on DNN inputs is to create a radius distance threshold calibrated during the training [4]. The idea is to perturb the DNN inputs, observe the correct answer, and determine how considerable the distance is regarding the DNN decisions. The advantage of this approach is that it does not need to inspect the internals of the DNN. The drawback is that it tends to be biased to the training data.
2) Monitoring the DNN intermediate values: An example of this approach is to monitor the neuron patterns observed from the layers of the DNN. Such an approach usually compares the recorded patterns of the DNN activation functions during the training with those during runtime. After the standard training process, a runtime monitor is created by feeding the training dataset to the network to store the neuron on-off activation patterns using binary decision diagrams (BDD) [6] or using a 2D projection [7]. For example, this projection is an abstraction box built by maximum and minimum activation function values during training. It inspects if the output of an activation function falls inside of this abstraction box during runtime after a new input pass through it. If not, it raises the alarm considering this input as a novel input.
Besides, uncertainty quantification methods can be applied to the intermediate outputs of a DNN [20], recognizing activation patterns that do not represent any data from the training dataset. An uncertainty-based supervisor accepts input as a typical sample if its uncertainty is lower than some threshold. Thus, a high threshold leads to many false negatives, and a low threshold leads to many false positives [21].
3) Monitoring the DNN outputs: For instance, an SM can monitor the values in the last layer of a DNN [5], verifying the decision's confidence level. However, it is unreliable since DNNs can output a wrong decision even with a high confidence level. For avoiding this problem, Liang et al. proposed enhancing the reliability of out-of-distribution image detection in neural networks (ODIN) [22] by applying techniques that decrease the DNN confidence values.
Uncertainty quantification methods can also be applied directly to DNN outputs. Some methods that can be used for this purpose are point-prediction networks, MC-dropout, and DNN ensembles. However, for evaluating such methods, it is necessary to apply metrics beyond the traditional combination of false/true positives/negatives, such as S-1 score [21].

E. Benchmarking SM for ML-based components
Despite several works for monitoring ML at runtime, just a few try to establish better benchmark methodologies. One of these works proposes a less biased evaluation of outof-distribution detectors, called OD-test [23]. Same as our work, the authors present a methodology that divides the data into three parts: training/validation, in-distribution testing set, and out-of-distribution testing set. Henrikson et al. [24] also proposed a framework for benchmarking DNN monitors, using six OOD datasets and seven metrics.
Nevertheless, our work differs from the two related works presented above in three different aspects. Firstly, we propose a methodology that covers the entire pipeline, from data generation to evaluation. Secondly, we apply metrics beyond the traditional accuracy and positive/negative rates, with additional statistical analysis in the results. Finally, we perform tests using five different types of OOD data across a greater amount of generated datasets. Next, we present our framework.

III. A BASELINE METHODOLOGY FOR BENCHMARKING SM FOR ML CLASSIFIERS
The proposed framework is an adaptation of the FARM [25] methodology. FARM is a methodology for fault injection for dependability validation. It uses the input domain of a target system as a set of faults F and a set of activations A that specifies the domain to test the target system. The output domain refers to a set of readouts R used for posterior evaluation with a set of metrics M . Our framework is divided into three modules, as illustrated in Figure 2.
After setting some initial parameters such as type of data generation, the amount of data to be tested, which ML models and SM to include, starts with the first module, called the data profile. This module generates benchmark datasets in the next module, called system under test (SUT).
The SUT is responsible for testing the ML and the SM performance, generating the results (readouts) at the end of the process. Readouts are inputs for the evaluation module.
The evaluation module is responsible for analyzing the readouts through several metrics for different components of the system. It provides the final analysis for each type of readout. Next, we explain each module and its components.

A. Data profile
This is the first module and it is composed of two items: • Activity: it contains in-distribution (ID) data, that is, instances from a distribution known by the ML. • Fault template: it contains rules for generating out-ofdistribution (OOD) data presented in Section II.B. The benchmark dataset receives ID and OOD data to test how the ML and the SM behave when exposed to expected and unexpected data. Even though all types of OOD can be part of a unique benchmark dataset, we generated datasets divided by each category of OOD data.
An essential premise is that the generated benchmark dataset is not applied to build or validate the SM or train the ML model. This premise guarantees unbiased experiment results and is more realistic since, in real scenarios, there are no guarantees about which type of data will arrive at runtime.

B. System under test
The system under test (SUT) module receives the benchmark dataset as input and outputs readouts for each component. To validate our framework, we chose ML models that classify images and one SM for each of the three categories mentioned in Section 2. Therefore, this module is composed of two main items: a third-party ML model already trained; and a third-party safety monitor containing a detection mechanism. We call the ML and the SM third-party components since they can be built aside from the framework.
This module simulates a stream of images randomly ordered. We apply datasets with random images since the image datasets available in the literature use such a setting.
The simulation works as a stream of images coming from the benchmark dataset set D that contains images X and labels y, arriving at one of each at a time. The ML model receives this image and makes aŷ classification. Next, the system triggers the SM, which checks a set of predefined properties. These properties can be the ML's classificationŷ with the associated confidence level, intermediate layers, the input X, or even a combination of two or more of these properties. In the case of this pseudo-code example, the SM inspects the ML model's internal properties (DNN's hidden layer) during the classification, as recently suggested in the literature [6], [7].
After the inspection, the SM detector mechanism raises a detection alarm or not (m). For this work, we consider a simple reaction strategy for the SM. If an alarm is raised, it invalidates the ML classification result; otherwise, it accepts the ML classification. Next, we observe whether such intervention produced a desirable outcome for the system or not. We call it overall detectionŝ. A desirable outcome means that canceling an ML output/agreeing with it was beneficial to the SUT. This result is independent of whether the SM-specific detection was correct or not in the OOD detection task. The criteria for considering these two dimensions of the detection (specific and overall) correct or not is discussed in the following subsection. This process continues until the end of the stream. Thus, the readouts are: the ground truth y, the ML classificationŷ, the SM specific detectionm, and its overall detectionŝ. They are divided into three different categories: 1) ML readouts: it contains classification (e.g., class number), confidence value from the last layer, and intermediate values from hidden layers. 2) SM readouts: it contains two types of detection: a) specific (e.g., it outputs 1 for OOD detection; 0 otherwise); b) overall (e.g., after OOD detection, it cancels the ML decision (1); 0 otherwise). 3) General readouts: it contains general outputs from all components above, such as processing time and memory. As can be noted, the SM readouts contain results in two dimensions: specific and overall. Considering two dimensions for detection is more realistic and complete than just analyzing the detection rate for the OOD data. The reason is that while the SM tries to detect OOD data, it also can avoid that the ML gives an answer that is different from the ground truth. The opposite is also true. As a result, the SM can incorrectly detect ID data as OOD data, hindering the correct decisions of ML. All possible situations are given in the next Subsection.

C. Evaluation
This module receives the readouts as inputs containing false positives/negatives and true positives/negatives regarding OOD data detection's specific task and the overall task of avoiding an unsafe outcome. This module evaluates two main aspects: 1) The SMs performance: the objective is to assess the SM results regarding the specific task of detecting OOD data. This evaluation's inputs are SM OOD detection (e.g., raises/does not raise the alarm).
2) The overall impact of the SM: the objective is to determine if the SM improves or worsens the overall SUT accuracy. We analyze the SUT using the ML alone (baseline) and ML with the SM. This evaluation's inputs are the overall decisions made by the SUT with ML alone and ML with SM, processing time, and memory. 3) Time and memory overhead induced by SMs: the objective is to investigate the memory and processing time efficiency of these methods during runtime. This module contains two major items: the oracle, and the metrics for ML and SM. The oracle determines positive and negative data and whether a test has passed or failed. Thus, the oracle considers a correct specific detection when the SM correctly detects OOD data. Besides, the oracle considers a correct overall detection for the SUT when the SM correctly avoids a wrong ML classification for ID or OOD data. The oracle takes into consideration the following scenarios to determine whether a readout is correct or incorrect: • ID data arrives in the stream -If the SM detects OOD data: it means a false positive for the specific task of OOD detection. * If the ML classification is equal to the ground truth: it means a false positive for the overall task of avoiding a failure since the SM intervened without necessity. * If the ML classification is different than the ground truth: it means a true positive for the overall task of avoiding a failure since it canceled the ML misclassification for ID data. -If the SM does not detect OOD data: it means a true negative for the specific task of OOD detection. * If the ML classification is equal to the ground truth: it means a true negative for the overall task of avoiding a failure since the ML gave a correct classification even though the SM did not detect OOD data. * If the ML classification is different than the ground truth: it means a false negative for the overall task of avoiding a failure since the ML misclassified the OOD data and the SM did not detect it, which could cancel a wrong ML output.
• OOD data arrives in the stream -If the SM detects OOD data: it means a true positive for the specific task of OOD detection. * If the ML classification is equal to the ground truth: it means a false positive for the overall task of avoiding a failure since the ML gave a correct classification but the SM wrongly intervened. * If the ML classification is different than the ground truth: it means a true positive for the overall task of avoiding a failure since the ML gave an incorrect classification and the SM correct canceled the ML decision. -If the SM does not detect OOD data: it means a false negative for the specific task of OOD detection. * If the ML classification is equal to the ground truth: it means a true negative for the overall task of avoiding a failure since the SM detection for OOD data also avoided an ML misclassification. * If the ML classification is different from the ground truth: it means a false negative for the overall task of avoiding a failure since the SM detection for OOD data also avoided an ML misclassification. The only exception for the above rules is for novelty class detection. When OOD data arrives in the stream, if the SM correctly detects OOD data, it will always be interpreted as true positive since the SM always correctly cancel the ML classification, independently of the ground truth. Conversely, if the SM does not detect OOD data, it will always be a false negative. If the SM lets the ML deal with a class that it was not trained in before, it can be regarded as unsafe. In order to evaluate these situations, we have selected in the literature a set of metrics that are pertinent for our study. Hence, we apply seven metrics: • Matthews correlation coefficient (MCC): it ranges from -1 if (ML or SM always wrong) passing through 0 (ML or SM is accurate as random) to 1 (ML or SM always right). This metric is more reliable than traditional metrics such as accuracy. It yields a high score only if the ML or SM can be assertive in all of the four confusion matrix categories [26]. In contrast, accuracy only considers the portion of the right answers. In a scenario in which the number of a class is 80%, the accuracy could yield a score of 80% for an SM that did not detect other classes. • False positive rate (FPR): also known as type-I error. It indicates how many false alerts the SM raises for the task to detect OOD data. High values means the SM indicates a problem wherein there are not in most cases. • False negative rate (FNR): also known as type-II error. It indicates how often the SM misses detecting OOD data. If this value is high, it indicates that the SM does not recognize the difference between the ID and OOD data. • Precision and recall (Pr and Re): the fraction of correct detection and the fraction of available OOD data. These metrics help to indicate how well the SM detected OOD data through the benchmark dataset. • Micro-F1: harmonic mean (global) for the prediction x recall. It helps to assess the quality of multi-label binary problems, which makes them suitable to be applied in the SUT evaluation. • Critical difference diagram: it is applied for all MCC results through all datasets. It shows how statistically different are the results between the SMs. Next, we detail each component's choice and its parameters for the three modules, and the benchmark results.

A. Data profile
We use three datasets as the activity of our framework: • GTSRB [27]: German traffic signs with 43 classes, with 39200 instances for training and 12600 for testing.
• BTSC [28]: Belgium traffic signs with 62 classes, divided into 7000 images for training and testing. • CIFAR-10 [29]: ten general classes (e.g., dog, car ...), with 50000 instances for training and 10000 for testing. Next, we apply a fault template for the five classes of OOD presented in section II-B: novelty, anomaly, distributional shift, noise, and adversarial inputs. For adversarial inputs generation, we apply the fast gradient signed method (FGSM) [30]. For the noise and distributional shift, we generate 19 different transformations with two types of intensity varying from 1 to 5 (e.g., snow (1) = image with a bit of snow; snow (5) = heavy snow) [31]. All transformations were applied over CIFAR-10 and GTSRB datasets. The benchmark datasets are composed of a transformed version of the ID datasets that we previously applied for training the ML algorithm and building the SM. Thus, we use 20% of the ID data for the benchmark dataset without transformation. The entire original dataset is transformed into a specific variation. For obvious reasons, this 20% of ID data was also not applied to the ML training.
For novelty class detection, the fault template applies one dataset as an ID dataset and another dataset with new classes as OOD data, resulting in three benchmark datasets: • GTSRB as ID data, and BTSC as OOD data: this combination tests whether the SM can distinguish new classes that have similar characteristics to the known ones. • GTSRB as ID data, and CIFAR-10 as OOD data: this combination tests whether the SM can distinguish new classes that are very different from the known ones. • CIFAR-10 as ID data, and GTSRB as OOD data: this combination tests the same as the aforementioned combination. However, since the ID data is different, the ML and the SM are built with different data. Hence, this permutation produces different outcomes. In total, we produced 79 benchmark datasets. More details can be found in the results section of our repository [32].

B. System under test
We simulate a scenario in which the SM has to detect OOD data by checking one RGB-colored image at a time in a randomly ordered stream of images. Once the SM makes a detection, it cancels the ML classification since this classification is potentially wrong. If nothing is detected, the ML classification is accepted. For the ML model, we use a LeNet [33] since it is a simple and traditional convolutional neural network (CNN) algorithm. It contains around 100,000 parameters and 128 neurons in the last hidden layer. For the monitors (noted as SM), we use one method of each of the three different strategies of data-based SM (presented in section II-D) for monitoring a DNN model: • DNN inputs: adversarially learned one-class classifier for novelty detection (ALOOC) [18]. This method learns how to reconstruct each class during the training phase. It receives an image during operation, tries to reconstruct this image to the known class, and analyzes the loss error resulting from this reconstruction. If this loss error surpasses a safe threshold, ALOOC flags it as OOD.
• DNN intermediate values: outside-of-the-box abstraction (OOB) [7]. This method projects a 2D-box region from the activation function values (RELU) from the hidden layers of the DNN during the training phase. At runtime, it receives an image and projects a point from it built from the outputs of the activation function from the same DNN layer used during the training. If this point falls outside the 2D box, OOB will flag this image as OOD. • DNN outputs: detector of out-of-distribution images in neural networks (ODIN) [22]. This method learns how to balance the confidence values outputted along with the DNN decision. It uses these values from the last layer of DNN during training and applies a method known as temperature scaling, which will scale the confidence values to a new threshold for considering an image as OOD. During the operation, ODIN verifies if the DNN confidence value over an incoming image respects the threshold. If not, ODIN will flag this image as OOD.

C. Parameter's evaluation and choice
Next, we present the chosen parameters for the SMs. To avoid biased results, all the initial parameters were tuned using only the ID dataset and are summarized in Table I. We explored the original parameters for each SM, and we also found/tested new values/methods not mentioned in their original papers (marked with *). For ALOOC, the authors did not investigate the influence of the optimizers in the original paper. Thus, we tested two different optimizers: ADAM [34] and RMSProp [35]. We also analyzed the best model for each class through different epochs. In general, all the models achieved a better convergence around 200 epochs. Since the threshold value for considering whether a class is considered OOD does not indicate in the original paper, we considered the average reconstruction loss for each class during the training.
For the OOB method, we test the parameter responsible for enlarging the size of boxes (τ ) with the same range of values proposed by the original paper. We applied the same number of clusters as suggested by the original paper (no clusters or three clusters). However, we also tested with a possible optimal number of clusters. The possible optimal number of clusters K for the K-means algorithm contained in the OOB method is chosen through the Elbow analysis [36]. Thus, the best K value was 5 and 17 for CIFAR-10 and GTSRB, respectively. Finally, we also tested three different approaches for the 2D-dimensionality reduction parameter: simple projection (proposed in the original paper), PCA [37], and ISOMAP [38]. ISOMAP is a popular nonlinear dimensionality reduction method, and PCA is also a popular method, but linear. With these two suggested methods, we can analyze if the choice of the dimensionality reduction methods influences the outcomes.
Regarding ODIN, we set the temperature (1000) and the magnitude parameters (0.0025 and 0.0014, for GTSRB and CIFAR-10, respectively) as suggested by the authors. However, we chose the confidence thresholds for determining OOD data by selecting the lower confidence value outputted from the method when exposed to the training data. In this case, 0.0237 and 0.1007, for GTSRB and CIFAR-10, respectively. We had to assume a confidence value threshold since it would not be possible to use ODIN as an SM without it.

1)
SMs performance: we use the positive/negative data and the evaluation metrics as mentioned in Section III.C. Since we measure the detection, there is no necessity to include the measurements for the ML. Best results are written in bold. Table II shows these results for novelty class: GTSRB as ID dataset and BTSC as OOD dataset; CIFAR-10 as ID and GTSRB as OOD; and GTSRB as ID and CIFAR-10 as OOD. The first benchmark dataset has the challenge of having ID and OOD data with a similar domain, while the other two have very different classes from each other. The MCC results indicate that the SM is sometimes slightly better than a random classifier (MCC ≈ 0). We observe that these methods' weakness is in the high amount of false positives, yielding a borderline MCC performance. According to the results, just ALOOC obtained a false-positive rate of around 50%. However, it had many false negatives, which can be considered worse depending on the scenario. In general, all SM had a poor performance due to the unreliable nature of DNN confidence values, and the high nonlinear nature of activation functions in ODIN and OOB, respectively.
Next, in Table III, we show the same analysis but using CIFAR-10 and GTSRB after being modified through an adversarial attack known as fast gradient sign method (FGSM).
According to the results, for CIFAR-10, the methods achieved negative values for MCC, which indicates performance worse than a random classifier. For GTSRB, the results are slightly better, but the MCC values can be considered as flawed as a random classifier. Despite the good values of micro-F1 for ODIN and ALOOC, in the CIFAR-10 and GTSRB datasets, respectively, the rate of false negatives was high. It means that the ML classifier did not give the correct prediction, and the SM did not detect the attack. Next, in Table IV, we show the results for CIFAR-10 and GTSRB datasets with different types of distributional shift.
Here, we see a surprising but negative outcome: OOB suffers from a lot of false positives. In contrast, ODIN suffers from many false negatives. It means that they always tend to miss a detection or to say that everything is OOD data. ALOOC got a fair MCC value for CIFAR-10 with intense fog and a good MCC value for GTSRB with heavy snow.  Another interesting result for ALOOC is that despite the low MCC performance on the other benchmark datasets, it got relatively good micro-F1 results. It highlights that this method tends to have a good amount of false negatives but rarely gives a wrong output when detection is signaled. Again, methods based on inspecting inputs seem to be more effective in detecting changes in the pixel values distribution.
Once we evaluate the methods in all datasets, we investigate how their results statistically differ from each other. The reason is to observe if there is an SM that is better than the others. We applied a Friedman test with Nemenyi posthoc with 95% the confidence level for all methods ranks through every dataset regarding the MCC metric. From that, we build a critical difference diagram illustrated in Figure 3. The result shows that ODIN achieved the best results in more benchmark datasets than the other two SM. However, there is no statistical difference between the SMs. It means that regarding the MCC results, there is no best SM method. Next, we evaluate how much the SM impacts the SUT when it works along with the ML classifier.
2) Overall impact of the SM: As previously explained, the objective is to evaluate how much the SM impacts the SUT when it works along with the ML classifier. Table V shows the overall SM's impact in the SUT when using GTSRB or CIFAR-10 regarding ID data. The results are expressed as MCC values. To evaluate the SM, we also use a percentage of relative change showing how much better/worse it performed compared to the baseline (ML alone). In the GTSRB dataset, the best values for the ALOOC method were obtained when using the ADAM optimizer. In contrast, the OOB method obtained the best results when using 3 clusters, ISOMAP, and an enlargement factor of 0.1. As mentioned earlier, for ODIN, the threshold was set to 0.0237.
For the CIFAR-10 dataset, the best optimizer for ALOOC was ADAM. For OOB, the best parameters were: 5 clusters, ISOMAP, and an enlargement factor of 0.35. For ODIN, we set the threshold to 0.1007.
As can be noted, all SM methods perform worse than the ML alone when exposed to ID data. This result is expected due to the generalization power of ML models. However, a reliable SM is also the one that can perform well when exposed to ID data, avoiding an ML misclassification or simply not raising a false alarm hindering a correct ML classification.
These results show the best SM's performance was obtained by the outside-of-the-box and ODIN, on GTSRB and CIFAR-10, respectively. Results indicate that the SM performed consistently better than a hypothetical random classifier (M CC = 0). However, they also show a significant gap between the performance of the ML alone and the SM for ID data, especially in the GTSRB that has four times more classes than CIFAR-10. This result is interesting since the tested monitors use information from the ML model and build their hypothesis over ID data.
3) Time and memory overhead induced by SMs: We show in Table VI an example of the performance for ALOOC, OOB, and ODIN, respectively, in the three benchmark datasets for novelty class detection. The values are expressed in seconds and represent the average time spent on the tasks of classification and monitoring on a single image. We used a computer with a Processor Intel(R) Core(TM) i7-9850H CPU @ 2.60GHz, and 32GB of memory.
Worth mentioning that we can have slight differences between the prediction time in the three methods since they use different deep learning libraries. ALOOC uses Keras, OOB uses Tensorflow, and ODIN uses PyTorch. Besides, the SUT time also contains the remaining time from other small processes involved in the experiment, so the summing between the prediction and monitoring time is lower than the SUT time. According to the results, ODIN is the fastest method. Except when using OOB with ISOMAP, all three methods needed no more than 0.07 seconds to spot OOD data. Such process time can be considered fast enough to be applied at runtime. Besides, they can be optimized, potentially reducing this time. However, the monitoring task is always slower than the prediction and can be responsible for 96% of the necessary time for a SUT to do the task. Since memory is also an important constraint, especially on embedded systems, we show in Table VII the memory efficiency of the methods when performing a novelty class detection over the GTSRB dataset. As can be seen, ALOOC needs a considerable amount of memory or disk space due to the necessity of developing one monitor for each class. For instance, even though ALOOC needs just 2.3MB per class, it is necessary almost 100MB to monitor all 43 classes contained in the GTSRB dataset. For OOB, it is also necessary to build one monitor for each class. However, the amount of memory needed is not huge since it uses just some stored arrays to make the boxes. For ODIN, it is not necessary to build one SM for each class. Moreover, since the algorithm needs to be applied during the training phase to collect the thresholds for the confidence values, it can use just a tiny amount of memory to do the inspection.

E. Threats to validity
In order to analyze and mitigate threats to the validity of the results, we present below a summary of arguments for external and internal validation.
• External validity: All tested SMs were already compared to other methods in their original papers. They obtained the best results during the comparison. Besides, we also chose three different approaches of monitoring (inputs, intermediate values, outputs) that may cover an acceptable range of SM approaches based on data. However, it is worth mentioning that this study is a first benchmark work. Thus, other approaches may be published with better performances in the future. • Internal validity: What could have to lead us to the wrong conclusions in our study? -Bad parameters choices for the SM: aiming at improving the SM performance, we explored additional parameter values than those explored in their original papers. For instance, we applied different dimensionality reduction methods for the outside-thebox. However, even though different dimensionality reduction methods can bring better results, they also introduce a high cost to time and memory performance. Regarding ALOOC and ODIN, we had to choose criteria to the threshold value for considering whether a class is considered out-of-distribution or not. This way, if one applies different criteria for these methods, they will result in different outcomes. However, such criteria had to be deducted since they are not contained in their original papers. Besides, ODIN used OOD data to calibrate their monitor, what can be considered unrealistic. Hence, in this work, all the parameters were chosen using just ID data. It makes the scenario more realistic and harder, which drastically decreases the SMs performance. -Choice of datasets: even though we chose datasets widely applied in the computer vision literature, the choice of the amount of ID data accessible for building the ML classifier and the SM can influence both. Furthermore, the amount of ID and OOD data in the benchmark datasets also can influence the results. However, we followed traditional ways to divide data (e.g., 80/20 for training and test). Besides, the authors of OOB [7], and ALOOC [18] test their methods considering the same dataset for ID and OOD data, which they test novelty detection by training the ML with 9 classes and omitting one, or training with 8 classes and omitting two, and so on.
However, we in this work we test novelty detection using an entirely new dataset as OOD data along with a part of ID data (ex: CIFAR-10 + GTSRB). Domain of validity: The setting CIFAR10-GTSRB constitutes two different datasets that do not have the same validity domain [39] (i.e., GTSRB is not only out-of-distribution but also out of the validity domain). Thus, the task of detecting OOD data should be less complicated in this scenario. However, the SM methods continued to be inefficient.
V. CONCLUSION AND RESEARCH DIRECTIONS In this work, we proposed a framework for benchmarking safety monitors for ML classifiers. We argue that there is a need for a better framework for benchmark SMs based on data by applying more metrics beyond accuracy and AUROC curves. Thus, we proposed a minimal set of measurements that should be considered when benchmarking such solutions.
Besides, we also showed that measuring these SMs is less straightforward than it seems. We presented two dimensions that need to be investigated: the SM's overall impact on the SUT; and the detection performance of the SMs. Our approach allowed insights into three categories of data-based monitors.
Our results indicate at least four general takeaways for the current solutions that are solely based on DNN's data: • The overall accuracy in the detection tends to be as bad as a random classifier due to the high amount of false positives or false negatives, depending on the scenario. • Such methods tend to negatively affect the system under test since they yield too many false positives interfering with the correct ML decision. • They need a considerable amount of memory or disk space since a monitor can consume up to 100MB. • They are not so fast to perform the detection and have a considerable overhead compared to the ML software. The tested SM were exposed just to novelty detection in their original papers. However, testing these SM to other four types of OOD data was important to show that: • SM based on DNN inputs: it has the advantage that it does not need to inspect the internals of the DNN. Thus, it can make the monitoring independent of the ML classifier. However, it has the drawback of being biased to the data used during the training. That is, it allows to look at the internal values that led the DNN to decide. However, the drawback is that the activation functions are insufficient to provide all information helpfully learned from the DNN. • SM based on DNN outputs: it has the advantage that it decreases and equilibrates the confidence values from the DNN. However, similarly to approaches based on DNN intermediate values, the drawback is that it relies heavily on the performance of the DNN. A limitation of this work is that we did not apply more and bigger DNN architectures. Such experiments could produce a more robust evaluation. Moreover, we benchmarked just over image datasets, not considering an entire system. Therefore, it is also essential to investigate the performance of the SMs inside a simulator to achieve a complete analysis. Finally, as future work, we also want to add SMs based on uncertainty quantification [20], [21] to our experiments.
The main conclusion is that using such data-based monitors does not provide sufficient confidence for their use in a safetycritical application. Thus, we want to explore how to combine data-based and model-based monitors.


Development of Unmanned Aerial Vehicle (UAV) Fixed-Wing for Monitoring, Mapping and Dropping applications on agricultural land

UAV technology is used in various fields, namely the military, to strengthen defence, surveillance, dropping logistics. In addition, UAV applications are also used in agriculture. In this study, a fixed-wing UAV will be developed as a monitoring, mapping and dropping process. Several things were carried out in the development process, namely design, manufacturing, installing electronic components, and test flights. The results showed that the vehicle could carry out the mission well. The monitoring process is carried out to review land security. Besides that, it can monitor if there is a fire in agricultural land. In the mapping process, photo data processing taken in the context of the mapping is carried out into an orthophoto map. Meanwhile, in the dropping stage, the aircraft managed to drop a payload weighing 0.5 kg with a distance of 20m from a predetermined centre point. The dropped payload is also equipped with a parachute for increased safety and reduced speed when the payload is dropped.

Introduction
An Unmanned Aerial Vehicle (UAV) is a generic aircraft designed to operate without a human pilot. In recent years, unmanned aerial vehicle technology has developed very rapidly. Applications of UAVs are used in the fields of industry, agriculture, military, and others. UAV can also be used as a monitoring and mapping technology. Based on research conducted by Nugroho et al. on the use of UAVs to monitor and map disaster areas, it resulted in 3 conclusions: settlements at risk of tsunamis, no evacuation routes, and no green zones around the coastline [1]. In some cases, UAV technology is also used for monitoring access to high-risk locations [2].
Unmanned Aircraft Vehicle (UAV), which has remote sensing instrumentation, can be utilized in disaster situations. When the UAV obtains photogrammetric data with appropriate image metadata, the UAV's capabilities for disaster management can be further implemented. Another application is for volcano monitoring and coastal monitoring using unmanned aerial vehicles [3]. Xi Wang also researched the use of UAVs to monitor hazardous environments, where experimental results show that the probability of UAVs completing missions in dangerous environments is higher than in operator mode within a specific time [4]. UAV technology is applied in various fields, namely the military, to strengthen the defence, as surveillance, dropping logistics. In addition, in the civil sector, UAVs are used in the agricultural sector, namely for monitoring, mapping and shipping goods [5]. Research on the application of drones was also carried out by Hakim et al., namely using drones to deliver goods for Covid-19 patients [6]. In this study, we will discuss the role of UAVs to support agricultural processes, namely for monitoring agricultural land, mapping the area, and shipping goods.

Method
This research on the application of UAV as monitoring, mapping, and dropping was conducted in Pracimantoro, Central Java, Indonesia. Furthermore, several stages were carried out to make the UAV, namely the design process, manufacturing, installation of electronic components, and flight tests, as shown in Figure 1. The next step is to determine the dropping area, where the location is 5 km from the initial location or the location of the Ground Control Station (GCS). The UAV will fly towards the dropping zone with an autopilot system; during the flight process towards the dropping, the camera will take photos and videos for monitoring and mapping purposes. After arriving at the waypoint, the UAV will drop the payload at a predetermined location, or what we know as dropping. The weight of the payload used is 0.5 kg. Figure 2 shows an illustration of the flight mission of a drone.

Manufacturing
Making a UAV aircraft consists of several stages: design, manufacture of aircraft parts, installation of electronic circuits, and assembling into an aircraft vehicle that can fly.

Design
At the design stage, simulations were also carried out to know the system analysis and modelling, operation data collection and observation data analysis. According to Pei-Hsiang Chung, the design process is used to 3 determine performance requirements, including stall speed, cruising altitude, maximum speed, absolute ceiling, and turning radius and speed [7]. The dimensions of the UAV aircraft are 2m long and 1m wide. The type of aircraft used is a twin-boom. Twin-boom aircraft have the advantage of having high efficiency; a double boom can distribute the load along the wingspan and strengthen the overall structure. Twin booms also have flexibility in setting the centre of gravity (CG) of the load so that the aircraft's performance will be equal to or without payload. The twin-boom design also has high stability due to the flat bottom airfoil design. Figure 3 shows the design of the aircraft used. The design of the aircraft is made aerodynamic so that it makes the aircraft more energy efficient. The twin-boom also has high stability due to its flat bottom airfoil design.

Manufacturing of aircraft parts
Making parts of the aircraft vehicle is done using a CNC machine to drive the vehicle precisely or close to the design drawing. The material used for the manufacture of the aircraft is rigid foam and coated with fibre and carbon fibre so that the aircraft is stronger and not easily destroyed when a crash occurs. The production methods we use are moulding and layup. You will get a sturdy and lightweight frame with the moulding method because a little resin is used, but it is enough to make the body sturdy. Figures 4 and 5 are the wings and fuselage that have been made using high-density rigid foam.

Electronic circuit installation
The vehicle is equipped with an autonomous system that can carry out missions outside of take-off and landing. The rides use batteries as a power source. The specifications for the electronic components of the UAV include a brushless electric motor and radio control system (transmitter and receiver) with a frequency of 433 Mhz, telemetry with a frequency of 433 MHz with a maximum power of 200 mW and a video transmitter with a frequency of 5.8 MHz with a maximum capacity of 1W. Figure 6 shows the process of assembling an electronic circuit with aircraft components.  Figure 7 is an aircraft vehicle that is ready for a flight test. The aircraft can carry out flying missions well. Furthermore, the aircraft can carry out the process of monitoring, mapping and dropping on agricultural land.  Figure 7. Aircraft rides

Monitoring
UAV applications for monitoring systems have been widely used globally. Several examples of the use of monitoring technology have been carried out by Shults, namely for monitoring warehouse development [8]. This technology is used to monitor the warehouse excavation work process. The monitoring process is carried out to review land security; besides, it can watch a fire in agricultural land. Figure 8 shows the results of monitoring from the aircraft vehicle.

.2. Mapping
The use of mapping techniques using unmanned aircraft has several advantages: mapping that is more accurate than using satellite imagery [9]. Mapping techniques are also very effective in terms of costs and time required. In general, the mapping technology concept consists of planning, installation of GCP (ground control point), data acquisition, data processing, reporting. Figure 9 shows the mapping results from the aircraft, and the mapping process is taken as far as 5 km.  Figure 9. The results of the mapping of aircraft rides

Dropping
In the dropping stage, the aircraft will drop a payload weighing 0.5 kg. In order to increase the safety of the dropped payload, the payload is parachuted to reduce the speed of the payload as it falls. The development of this research can also be used to send packages in quarantine areas for the Covid-19 infectious disease epidemic. This is done to ensure that monitoring activities and delivery of aid packages can be carried out quickly and effectively, which is expected to reduce the risk of disease transmission to other areas and aid workers. In addition, package delivery using UAV technology can be used to deliver fertilizer to agricultural land. Figure 10. The results of dropping aircraft rides

Conclusion
The research results on the development of unmanned aerial vehicle (UAV) fixed-wing for monitoring, mapping and dropping applications on agricultural land show that the vehicle has been able to carry out all missions well. The monitoring process is carried out to review land security; besides, it can monitor a fire in agricultural land. In the mapping process, photo data processing taken in the context of the mapping is carried out into an orthophoto map. Meanwhile, in the dropping stage, the aircraft managed to drop a payload weighing 0.5 kg at a distance of 20m from a predetermined centre point. The dropped payload is also equipped with a parachute for increased safety and reduced speed when the payload is dropped.


extfieldsofstudy:
Computer Science
Mathematics

text:
Sparse Multi-Decoder Recursive Projection Aggregation for Reed-Muller Codes

Reed-Muller (RM) codes are one of the oldest families of codes. Recently, a recursive projection aggregation (RPA) decoder has been proposed, which achieves a performance that is close to the maximum likelihood decoder for short-length RM codes. One of its main drawbacks, however, is the large amount of computations needed. In this paper, we devise a new algorithm to lower the computational budget while keeping a performance close to that of the RPA decoder. The proposed approach consists of multiple sparse RPAs that are generated by performing only a selection of projections in each sparsified decoder. In the end, a cyclic redundancy check (CRC) is used to decide between output codewords. Simulation results show that our proposed approach reduces the RPA decoder's computations up to $80\%$ with negligible performance loss.

I. INTRODUCTION
Reed-Muller (RM) codes were introduced by Muller [1] in 1954. Shortly after, Reed proposed a majority logic decoding algorithm [2] correcting errors up to half of its minimum distance. Since then, many approaches for decoding RM codes have been investigated. An overcomplete minimum weight parity check matrix is used in [3] to exploit the redundant code constraints; recursive list decoding in [4]- [6] achieves a performance close to maximum likelihood (ML) decoding with list size at-most 1024 for short block lengths; and the Sidel'nikov-Pershakov algorithm [7] decodes secondorder RM codes of length ≤ 1024 by exploiting derivatives of the received codeword and majority voting. Additionally, Sakkour's [8] variant of [7] simplifies the majority voting leading to achieving a smaller decoding error probability.
Recently, RM codes have received a great deal of attention. One reason for this surge of interest is their close connection to polar codes, a family of error-correcting codes that provably achieves capacity for any binary-input memoryless symmetric channel [9]. This connection was already mentioned in the seminal paper [9], and it was exploited to design a family of interpolating codes with improved performance at practical block lengths in [10]. Furthermore, it has been shown that, under ML decoding, RM codes achieve capacity on erasure channels [11]. Achieving similar results for a more general class of channels is a long-standing conjecture.
In general, ML decoding is not computationally efficient. This has spurred research on finding an efficient algorithm. D Many approaches taking advantage of different aspects of RM codes have been exploited recently. In [12], minimum-weight parity checks are employed taking advantage of the large automorphism group of RM codes. Berlekamp-Welch type algorithms on random errors and erasures are considered and analyzed in [13], [14]. Successive cancellation (SC) decoding [9] and SC list (SCL) decoding [15], initially proposed for polar codes, are also applicable to RM codes as they share a similar construction. An algorithm based on successive factor graph permutations is presented in [16]. For a thorough review on RM codes, we refer the reader to [17].
Most recently, a recursive projection aggregation (RPA) [18] decoder has been proposed. The RPA decoder uses projections of the codeword on its index space cosets in order to obtain valid RM codewords of smaller length. Then, the smaller codewords are recursively decoded and aggregated. RPA decoding performs close to ML decoding up-to length 1024 for second-order RM codes and also benefits from a parallel implementation. A recursive puncturing aggregation (RXA) decoder is presented in [19], and it is a modification of RPA decoding using puncturing instead of projections, built for high-rate codes.
In this paper, we focus on the RPA decoding algorithm. The near ML decoding performance benefits of the RPA decoder comes with the need for a large computational budget. We propose an optimized version of RPA decoding, namely a sparse RPA with multiple decoders (SRPA). The new method chooses a subset of recursions for each decoder at random. When combined, these smaller decoders can perform close to RPA decoding while requiring a significantly smaller computational budget.
We compare the proposed SRPA method with RPA decoding on code lengths ≤ 512. The results show that by using only two sparse decoders, one can achieve almost the same performance at 20% of the computational budget of RPA decoding. Also, the performance of SRPA is compared with SCL decoding while fixing the computational budget. The results indicate that for second-order RM codes, the performance of the proposed SRPA is more stable, staying relatively close to the performance of ML decoding as the block length increases.
The rest of the paper is organized as follows. In Section II, we give the basic definition of RM codes and the description of the RPA decoder. In Section III, we discuss the problem formulation and the proposed method. In Section IV, we present the simulation results. Finally, in Section V, we draw the main conclusions of the paper.

A. Reed-Muller Codes
There are many approaches to describing RM codes. Here we define them by using low-degree polynomials following the notations of [18].
as the polynomial ring of m variables. Since Z 2 = Z in F 2 , then the following set of monomials with size 2 m forms a basis for where [m] := {1, 2, . . . , m} and i∈∅ Z i := 1.

B. Recursive Projection Aggregation Decoder
For a binary-input memoryless channel W : {0, 1} → W the log-likelihood ratio (LLR) of the channel output is defined as Given L as a vector corresponding to the LLR values of the channel output, the RPA decoder obtains codewords of a shorter length by projecting L onto the cosets of the index subspaces. The shorter codewords are decoded recursively. Later, the results are aggregated via a voting procedure to estimate the original codeword.
Let E := F m 2 and let each codeword correspond to an mvariate polynomial of degree at-most r. Then, the coordinates of the codeword c ∈ RM(m, r) are indexed by the binary vectors z ∈ E, i.e., c = (c(z), z ∈ E). Definition 2 (Projection): Let B be an s-dimensional (s ≤ r) subspace of E and E/B be the set of all cosets of B in E; in other words, ∀T ∈ E/B , T = z + B for some z ∈ E. The projection of the codeword y on the cosets of B for the binary symmetric channel (BSC) is defined as where For general binary-input channels having vector L as LLR values of the noisy codeword, the definition above is modified into Note that, from Lemma 1 in [18], we have that if y ∈ RM(m, r), then y /B ∈ RM(m − s, r − s).
The RPA algorithm is obtained by setting s = 1. In this case, at level i in the recursion, each codeword will make 2 n−i − 1 branches. After obtaining the estimated codewords for each branch, the results are aggregated using the voting algorithm below.
Definition 3 (Aggregation/Voting Algorithm): Let L be the output LLRs,ŷ Bi be the decoded codeword corresponding to the cosets of the one dimensional subspaces B i = {0, z i }, z i ∈ F m 2 , andL be the final estimation. For each bit index z ∈ {0, 1} m , the estimation is computed aŝ where [z + B i ] is defined as the coset containing z. One way to look at (8) is that for a one dimensional is more likely to be 1, then their sign of LLR values should be different. On the other hand, if the sum is more likely to be 0, then their LLR values should also have the same sign. Fig. 1 shows an example of this procedure on RM(3, 2) and cosets of B 1 .
The algorithm starts from RM(m, r) and recursively decodes RM(m − 1, r − 1) until reaching RM(m − r + 1, 1). For first-order RM codes, the ML decoder has complexity O(n log n) and can be implemented using the Fast Hadamard Transform (FHT) [20]. According to Algorithm 3 in [18], the RPA decoding algorithm is iterative, in the sense that it repeats the procedure above at most m 2 times until it reaches a fixed point where |L(z) −L(z)| ≤ , ∀z ∈ F m 2 , for some small assigned constant .

III. SPARSE RECURSIVE PROJECTION AGGREGATION
Although the RPA decoding algorithm performs well on RM codes with respect to the block error rate (BLER), it requires significant computing power. Given an r-th order RM code of block length n, the algorithm runs in O(n r log n) time for a sequential implementation and in O(n 2 ) time for a parallel implementation using O(n r ) processors. That is because at the i-th level of recursion, the RPA decoder uses n2 −i − 1 recursive calls to vote for each sub-code. As we will see in Section III-A, having this amount of recursive calls is redundant for obtaining a reliable vote. Our goal is to achieve the same error probability to that of the RPA decoder using lower computational power. We will achieve this goal by reducing the number of recursive calls.

A. Random Selection of Recursions
In order to reduce the amount of computation required, several recursions are chosen at random to be kept and the rest are removed to generate a sparse decoder as defined below.
Definition 4: A sparse decoder D S (SRPA) that has at most t iterations and q recursions/subspaces per iteration, performs only a subset S = {s 1,1 , . . . , s t,q } of possible projections, where s i,j is the j-th chosen subspace in the i-th iteration that the codeword is projected on.
Definition 4 suggests that the aggregation formula in (8) for each z ∈ {0, 1} m is modified tô where S i = {s i,1 , s i,2 , . . . , s i,q } is the subset of S corresponding to the i-th iteration of SRPA decoding. The simulation results for RM(7, 2) in Fig. 2 show that removing up to half of the recursions causes a negligible loss in performance, which confirms the high redundancy of the RPA decoding algorithm. However, if only 1/4 of the projections are kept, the error probability starts to degrade. The reason for this degradation is that each projection estimates certain codewords more reliably and by removing a large fraction of recursions, it is more likely to miss the suitable projections.

B. Choice of Multiple Decoders
The loss of performance that is caused by the pruning method in SRPA can be compensated by using multiple SRPA decoders. The idea is that the low-complexity decoders work well for a subset of errors. Choosing multiple low-complexity decoders at random will improve the error probability of SRPA while having lower complexity than RPA decoding.
Let us select k sparse decoders D S1 , D S2 , . . . , D S k at random. We call this set of k SRPA decoders the k-SRPA decoder. The input LLR L is individually decoded by each of the k SRPA decoders resulting in estimationsL 1 ,L 2 , . . . ,L k . The final estimation is then chosen via one of the following methods: 1) Most likely codeword: The most likely codeword is chosen by computing argmax i L i , L , where ·, · denotes the scalar product.
2) CRC verification: The final codeword is chosen by first checking the CRC condition and then choosing the most likely codeword among the codewords satisfying the CRC condition.
Note that lower levels of recursions can only use the first approach in this setting, as projections do not allow the concatenation of CRC at these levels.

IV. SIMULATION RESULTS
The proposed SRPA is tested on RM codes of up to third order, with block lengths 128, 256, and 512. The performance of the proposed method is compared with the original RPA decoding algorithm. In the implementation of SRPA, the 3-bit CRC (CRC-3-GSM: x 3 + x + 1) is used to select the final codeword.
Note that each projection and aggregation has complexity Θ(n), and the decoding of first-order RM codes has complexity Θ(n log n) since it is equivalent to performing an FHT. These operations are exactly the same in both RPA decoding and SRPA decoding. The difference lies in the number of operations performed, and we elaborate on this difference for RM codes of second and third order below.
In order to decode RM(m, r), we denote by q i , t i and d i the number of projections, iterations, and decoders used by SRPA for the sub-code RM(m − i, r − i). Then, in order to decode RM(m, 2), the total number of projections, aggregations, and FHTs is d 0 t 0 q 0 . Furthermore, in order to decode RM(m, 3), we need d 0 t 0 q 0 + d 0 t 0 q 0 d 1 t 1 q 1 projections, d 0 t 0 q 0 + d 0 t 0 q 0 d 1 t 1 q 1 aggregations, and d 0 t 0 q 0 d 1 t 1 q 1 FHTs. Recall that the complexity of the FHT has an extra log n factor with respect to projections and aggregation. Note also thatin our experimental setting -d 0 t 0 q 0 is much smaller than d 0 t 0 q 0 d 1 t 1 q 1 . Thus, in order to compare the computational budget of RPA and SRPA, it suffices to count the number of FHTs executed by the two algorithms.
In the implementations in this paper, we use two randomlyselected decoders. For second-order RM codes, each of the decoders keep 1 8 of the recursions chosen uniformly at random. For second-order RM codes, having at most 8 SRPAs, we can achieve a performance better than RPA decoding or even approach the ML bound.
Thus, SRPA roughly leads to a 4× speed-up with respect to the original RPA decoding. For third order RM codes, 1 8 of the first level recursions are chosen. When the recursions reach RM(m−1, 2), four decoders are used instead of two to avoid an increase in the error rate. This leads to a factor of 1 8 × 1 2 reduction in the number of FHTs. Thus, SRPA roughly leads to an 8× speed-up. Fig. 3 shows the BLER of different decoders as a function of the SNR (E b /N 0 ). It is visible that the 2-SRPA decoder has close performance to that of RPA decoder. In addition, by adding more decoders to the 2-SRPA decoder for second-order RM codes, one can gain considerable improvement in the error probability and approach the performance of ML decoding. Table I provides a detailed comparison of the number of FHTs used by 2-SRPA and RPA. We report (i) the average number of FHTs used for RPA decoding at different E b /N 0 levels where the early stopping condition is the same as [18], (ii) the number of FHTs used by RPA when there is no early stopping (see the column Full Rounds), and (iii) the number of FHTs required for SRPA decoding. As SRPA decoding does not necessarily converge to a fixed point, it runs for a fixed number of iterations and does not require a comparison at different E b /N 0 levels. The last column of Table I provides the savings in the number of FHTs guaranteed by 2-SRPA when compared to the RPA decoder with no early stopping (Full Rounds). It can be seen that the proposed SRPA decoder can save up to 87% of the FHT operations with respect to RPA decoding. When comparing with the average number of FHT operations in RPA decoding at different E b /N 0 levels, the 2-SRPA decoder leads to a saving between 50% and 79% of RPA's computational budget, while having little to no performance loss. Fig. 4 compares the performance of 2-SRPA decoding with SCL decoding. Recall that, given an RM code of order r, decoding each of its first-order RM sub-codes takes 2 −(r−1) n log(2 −(r−1) n) operations. Furthermore, SCL decoding has complexity Ln log n, where L is the list size. Thus, in our comparison, we assume that the decoding complexity of SCL with list size L is roughly the same as SRPA using 2 r−1 L FHTs. For second-order RM codes, the list size for each code is chosen such that either the computational budget of SCL is roughly the same as that of SRPA, or SCL reaches the ML decoding bound with less computation. It is visible that the performance of SCL decoding deteriorates quickly as  (8,2) and RM(9, 2), SRPA yields a performance improvement while requiring the same computational budget; for third-order RM codes, SCL performs better than RPA methods. the block length increases: for N = 128 and N = 256, SCL slightly improves upon SPRA using the same computational budget; However, for N = 512, SRPA provides a significant improvement. This fact does not hold for third-order RM codes. As seen in plots for RM (7,3) and RM (8,3), the SCL decoder with list size 32 and 1024, respectively, outperforms RPA methods, while requiring a much smaller computational budget.

V. CONCLUSIONS
RPA decoding incorporates the algebraic structure of RM codes by using projections. By sparsifying the decoder on each iteration, this paper suggests that there exist more efficient ways to utilize these projections to lower the computational power required. Future work includes determining whether there exists a strategy for choosing the recursions that minimizes the error probability.


A Fabric-Based Integrated Sensor Glove System Recognizing Hand Gesture

 The research on wearable glove sensor system has been increasing over recent years because of the need for portability and comfort. This study presents a fabric-based integrated sensor glove system with five sensing zones. Five sensors are knitted by silver-plated nylon yarn and embedded into glove directly using intarsia technology. Various parameters including sensor elasticity, sizes of embedded sensor as well as glove are discussed, respectively. Further, selected or chosen integrated sensor glove is manufactured and tested for recognizing gestures. Results show that elasticity affects effective sensing range of knitted sensors, size has significant influence on sensors’ sensitivity, and appropriate glove size helps avoiding nonlinear sensing phenomenon. Finally, the glove system, by extracting feature data, can distinguish Chinese number gestures very well and has also the potential to recognize more hand gestures in the future.

Introduction
Sensing of human hand signal plays an important role in rehabilitation medicine, biomechanics, tele-operation, and human-computer interaction, like robot arm or virtual reality. Consider an example specifi cally-mature sensors that track hand gesture may provide opportunity for a person to communicate with deaf-mutes as well as understand them.
Until now, numerous hand motion sensor systems have been put forward and investigated. Early researches are focused on vision-based methods [1,2], such as camera and refl ective markers on glove, and infrared-based methods. Nevertheless, these methods are limited by hand gestures signifi cantly. For instance, it is hard to recognize one fi nger when it is shielded by another [3]. Moreover, multiple devices fi lming or photographing hand motions are not suitable in some cases.
Moreover, most studies used the method of integrating fl exible or hard sensors with off-the-shelf gloves. Hard sensors used are common transducers that mainly consist of acceleration sensor, angle transducer, inertial measurement unit (IMUs), magnetic sensors [4], etc. However, rigid components need to be small enough so that they are easily wearable, which may lead to diffi culty during integrating [5]. Flexible sensors consist of familiar components that include soft bending sensors [6,7], fi ber optical sensors (FOS) [8][9][10], and various self-assembly stretch sensors. When the factors such as simplicity, portability, and comfort are considered, fl exible glove sensor system is becoming a key system for realizing hand signal recognition.
Except for commercial component, different materials are selected to generate wearable sensor such as nanomaterial [11][12][13] (carbon nanotube, silver nanowires, graphene, etc.), PEDOT:PSS [14], nylon-based silver yarns or other conductive yarns [15,16], etc. However, conductive coating layer may face the problem of generating irreversible cracks during stretch and cohesive materials may have impact on original elasticity and extensibility of fabric. Compared with coating or pasting, fl exible knitted sensors can be embedded into glove directly through the whole-garment knitting process. Regarding wholegarment technology, some underwear or pants [15,17] have had a shot and demonstrated that embedded knitted sensor of these clothing can cling to skin perfectly and monitor human body postures with good sensitivity and stability.
In a previous research [18], an integrated glove with one knitted sensor embedded into it was manufactured for index fi nger and results exhibited that the sensor was able to refl ect fi nger bending condition. In this paper, impact of fabric parameters on sensor properties were investigated further and an integrated glove with fi ve applicable strain sensors was fabricated using a fl at knitted machine. The sensor glove was connected with selfmade fi ve-channel read-out electronics to construct a tracking system. Eventually, results show the system can distinguish numerical hand gestures from 1 to 9 well.

Figuration and mechanism
It is known that lengthwise loops of weft plain knitted fabrics are called wales and crosswise loops are called courses. For fabric-based sensors knitted by conductive yarns, stretching along the lengthwise and crosswise direction generates change in effective contact area between adjacent loops, and as a result, equivalent resistance of the sensor varies. Generally, the extensibility of knitted fabric along the lengthwise direction is larger than that of crosswise direction. For knitted sensors located on chest band or T-shirt, their electrical-mechanical performance is studied based on the crosswise direction, along which fabrics deform. In this study, considering that fi nger bending leads to skin stretch along the lengthwise direction, properties of knitted sensor was tested all along the fabric lengthwise direction.
In the previous study [19], conductive weft plain-knitted fabrics made of silver-plated nylon yarn and elastic nylon/spandex yarns were stretched along the lengthwise direction within certain strain range and found that the sensor resistance increased with good linearity and sensitivity. Hence, this structure is adopted to design new sensors of the glove system here. Figure 1a shows the construction of knitted sensor, where conductive area (i.e., sensor area) was knitted by nylon/ spandex yarns and silver-plated nylon yarn, other area was made of nylon/spandex yarns and elastic polyester yarns, which is non-conductive. Owing to addition of spandex yarns, both the two areas have elasticity.
By designing knitting procedure on computer, an integrated glove with fi ve effective sensors can be obtained from a SHIMA Seiki SWG 061N-15G computer fl at knitting machine directly. For thumb, the embedded sensor cover one interphalangeal joint (IP), but for other four fi ngers, each sensor covers the proximal interphalangeal (PIP), and the distal interphalangeal joint (DIP), as Figure 1b-c shows. Due to glove knitting sequence and design limitation, intarsia technology cannot be used to knit sensors covering all IP, PIP, and MCP joint consequently. Therefore, IP/DIP and PIP are selected and studied here. Being off from knitted machine, two ends of silver yarns of every sensor are connected with read-out electronics so as to obtain resistance value. In addition, knitted elastic glove is washed and ironed to insure stabilization before wearing

Selection of elastic materials
Nylon/spandex yarns that include 20/75 (nylon 8.3 tex, spandex 2.2 tex), 40/75 (nylon 8.3 tex, spandex 4.4 tex), and 70/30 (nylon 3.3 tex, spandex 7.8 tex) are chosen as plating yarn in whole fabric and in addition, silver-plated nylon yarn (44 dtex/12 fx8, resistivity of each filament is 15 Ω/cm) and elastic polyester yarn (33.3 tex) are used as ground yarn of sensor areas and other areas, respectively. Three kinds of knitted sensors are fabricated and elasticity is measured, and Table 1 shows those values. Fixed elongation test method is selected to obtain sensor elasticity, in which the fixed elongation is set as 20%, clamping distance is 100 mm, stretching speed is 100 mm/min, the pre-tensioning force is 1N, the stagnant time is 60 s, the recovery time is 180 s, and number of stretch cycle is 5 times. Three sensors are stretched along the lengthwise direction in one cycle and repeated cycles, and their resistance-strain values are recorded, and Figure 2 depicts these values.
It can be seen from Table 1 and Figure 2 that elasticity of knitted sensor is affected by content of spandex in used yarns, and elasticity obviously influences the electrical-mechanical performance of knitted sensors. Experiments results show that all sensors exhibit increased resistance with good linearity and sensitivity during the first elongation phase, but the corresponding strain range also called as effective working range will be larger when the sensor has better elasticity. Beyond that, it is known that the sensor with better elasticity shows better repeatability during repeated stretch process. In brief, effective working range of sensor S3 can meet the requirement of finger dorsum skin strain (up to 35%), and it has the best repeatability against fabric plastic deformation. Hence, sensor S3 is selected to be used in this glove system.

Size of sensors
To produce an integrated glove with five effective sensors, size parameter of every sensor needs to be considered. Size parameters of embedded sensor include number of rows and wales. In the previous investigation [19], experiment results show that knitted sensor with less wales exhibits better sensitivity but have only smaller effective sensing range (where resistance exhibits growth linearly). On the contrary, knitted and testing, washing and ironing parameters are controlled equally for different glove samples as described later. When experimenter bends fingers, elastic sensors clinging to skin tightly will be stretched and exhibit resistance variation.

Materials and size
In the investigation [19], the strain range where resistance exhibits linear response of elastic knitted sensor is limited (0~10%), however, stretch of finger skin in lengthwise direction can be more than 35% when people do some simple gestures in daily life. As is reported, elastic deformation process of knitted sensors has effect on their electrical-mechanical performance [20]. As the elastic deformation process depends on fabric elasticity and for the sake of obtaining applicable embedded sensor with targeted linear working range in this study, three nylon/spandex yarns of various content of spandex are used to fabricate flexible sensors with different elasticity. Resistance response was recorded by the VICTORY 4105A ground resistance tester (GRT) and YG (B) 026ET fabric mechanical analysis machine (FMA, supplied by Wenzhou Darong) was used to stretch knitted sensor.
As the glove and sensors needs integration, sizes have become important parameters during integration, which includes size of glove (Gs) and size of embedded sensors (ESs). Glove sensor system being too loose or snug for a person may affect sensing process. Here, three sizes of glove are knitted and performance of one single sensor on index finger is compared to determine appropriate Gs. It's worth mentioning that size of finger being monitored and fabric shrinking percentage of post processing should be considered to obtain appropriate Gs. Secondly, ESs also play an important role in affecting sensor performance. In this part, electrical-mechanical performances of different sizes of glove and sensors were recorded by GRT and FMA.

Monitoring finger movement
Preparative gloves are worn by experimenter and connected with a self-made electronic read-out system where five resistance channels are set to obtain resistance signals of one hand, as shown in Figure 1c. Each channel records resistance information of one embedded sensor. In the read-out software used, five resistance response curves were displayed simultaneously. Except for pursuing better linearity and sensitivity, ESs also need to meet the requirements of covering DIP and PIP joints and to save the used materials as much as possible. It is concluded that knitted sensor containing less wales but more rows should be selected based on the fact that it will be able to cover targeted fi nger joints. Considering length of each fi nger is different, various sizes of sensor are determined for 5 wales but with 45, 60, 60, 45, 35 rows, respectively, from thumb to little fi nger.

Size of glove
In addition, three sizes of expected glove are fabricated and the index fi nger sensors' performances are investigated simply.
sensors with more wales show larger sensing range but lower sensitivity. Here, four sensors containing 5 wales are made but under four different rows 10, 20, 30, and 40, respectively and the yarn material is same as used in sensor S3. Photos and longitudinal electrical-mechanical curves of these sensors are shown in Figure 3.
It can be observed from Figure 3 obviously that all four sensors show good linearity over their effective sensing range. Nevertheless, knitted sensor containing 40 rows exhibits better sensitivity, while the sensor having 5 wales and 10 rows exhibits lower sensitivity. But the effective sensing range of four sensors are similar, which is about 0~50%.  http://www.autexrj.com/ exhibits linear increase and decrease when the fi nger bends and returns once in a sequence, and so the resistance signal only generates one summit in one motion cycle. However, both of K 2 and K 3 exhibit two summits in one cycle, while K3 is more remarkable. Two summits appearing in one cycle means resistance response of two phases (bending and recovering) is non-linear, which may be due to the small size of these gloves. After wearing tight gloves, knitted sensor has been pre-stretched before monitoring fi ngers motion, which leads to nonlinear resistance (changing) phenomenon later. In a word, K 1 is selected as a size reference for fabricating following glove system.

Monitoring fi nger movement
Herein, combining chosen elastic materials, size parameters of embedded sensors and glove, a whole fabricating process is determined and an integrated sensing glove is obtained, as shown in Figure 5. Five pairs of electric wires drawn from embedded sensors are connected to the central processing unit part (CPU); in turn CPU gets and sends resistance signals to related computer software. Experimenter performs numbered gestures from one to nine, each gesture is done for about 7 s, and resistance responses are exhibited as shown in Figure 6. It can be seen that each embedded sensor refl ects corresponding fi nger motion successfully. Due to difference of joint angle in various gestures, resistance signal presents diverse characteristics. For instance, all fi ve sensors showed higher resistance value when the hand performed the number 9 First of all, default Gs in design system is chosen and a glove with random size is obtained. After washing and ironing, a shaped glove is generated and its Sp of post processing can be fi gured up. Hand sizes (Hs) of appointed experimenter are also measured, and expected size of sensor gloves (EGs) can be calculated by the formula (1) and (2): where K is the value refl ecting glove's size (larger value demonstrates looser glove), DGs is designed size of sensor gloves, including length and circumference of fi ngers and palms. For a glove to be elastic and to cling to fi nger tightly, K must be a negative number. Here, K is selected to be 0, −0.1, −0.2, and three corresponding sensor gloves are obtained and named K 1 , K 2 and K 3 respectively, which is shown by Figure 4a. Embedded sensors' specifi cation of three gloves is completely identical.
For the appointed experimenter, it's obvious the K 1 is loosest and K3 is tightest among those three sensor gloves. Sensor embedded on the snug gloves gets broadened after being worn, as Figure 4b shows. After wearing the three gloves, the experimenter followed fi nger motion trace, as Figure 4b shows, to observe and compare detected resistance signal of the index fi nger sensor. It can been seen from Figure 4d that K1 than number 5. In summary, the sensor glove with fi ve effective embedded sensors can monitor and recognize simple gestures well now.
However, just obtaining fi ve channels resistance signal is not enough to recognize hand gestures further. To distinguish various number gestures, a mass of raw digits of each gesture are recorded to extract feature data.
In this part, number gesture '5' is selected as a reference position, where fi ve fi ngers are in "straighten" position. Experimenter changes hand from gesture '5' to other number gestures and repeats for 150 times for respective gestures. For example, Figure 7 demonstrates four cycles of fi ve-channels resistance changing from gesture '5' to gesture '6', where the arrow points to resistance of sensor on ring fi nger. When the experimenter performs gesture '5', original resistance (OR) is obtained. When the experimenter performs gesture '6', ambition resistance (AR) is got. Here, the ratio of OR to RR is chosen as a key characteristic value to distinguish different gesture, which is called feature C later. Figure 8 depicts a boxplot of ratios, that is, the value C extracted from 150 cycles for each gesture, where T, I, M, R, L represents the sensors on thumb, index fi nger, middle fi nger, ring fi nger and little fi nger, respectively. The red short line in the middle of the boxplot is the median of the data, which demonstrates average level; the upper and lower line refl ects quartiles of the data, and hence, the box contains 50% of the data. The width of the box refl ect degree of data fl uctuations; the upper and lower short line corresponds to the maximum and minimum value of the data, while red dots represent the abnormal value.
According to Figure 8, the feature C of some gestures shows larger box width, which means there is larger data fl uctuation.
For example, C of the sensor on ring fi nger of gesture '6' varies from 0.3 to 1, which is also affected by diffi culty in repeating accurate and identical gestures completely. In general, from Figure 8, contrast between feature data of different numerical gestures is obvious from Figure 8. So, we can know that the manufactured sensor gloves do have the potential to distinguish various hand gestures with advantages of portability and fl exibility. Further, hand gesture recognition will be investigated using machine learning in future studies and accurate recognition rate will be obtained.

Conclusion
In summary, this work investigates an integrated sensor glove system with fi ve embedded sensors, which is knitted by silverplated nylon yarns and nylon/spandex yarn directly using whole-garment technology. To obtain an applicable sensor   glove system, effect of elastic materials, size of embed sensors, and glove on sensing performance are discussed respectively.
First, elasticity of sensors of various content of spandex is investigated. It is demonstrated that conductive knitted sensor with better elasticity exhibits larger sensing range, better repeatability but lower sensitivity. As a result, 70/30 nylon/ spandex yarn is chosen as elastic plating yarn in whole glove. Second, sensors with various courses but same wales are designed and it is discovered that knitted sensor containing less wales but with more courses has better sensitivity. Considering embedded sensor needs to cover DIP and PIP fi nger joints, sensor with more courses and less wales is designed in later gloves. In addition, according to size of experimenter hand and fabric shrinkage during post-processing, three sizes of gloves are manufactured and results show that glove with largest size can play the role of best sensor. Finally, taking above factors into consideration, an integrated sensor glove is obtained and experiment results demonstrate that the glove system can distinguish nine number gestures well through generating various feature data. For future work, we have planned to build recognizing model by machine learning and obtain accurate recognition rate.