Statistically Profiling Biases in Natural Language Reasoning Datasets and Models

Recent work has indicated that many natural language understanding and reasoning datasets contain statistical cues that may be taken advantaged of by NLP models whose capability may thus be grossly overestimated. To discover the potential weakness in the models, some human-designed stress tests have been proposed but they are expensive to create and do not generalize to arbitrary models. We propose a light-weight and general statistical profiling framework, ICQ (I-See-Cue), which automatically identifies possible biases in any multiple-choice NLU datasets without the need to create any additional test cases, and further evaluates through blackbox testing the extent to which models may exploit these biases.

Introduction
Deep neural models have shown to be effective in a large variety of natural language inference (NLI) tasks [Bowman et al., 2015;Wang et al., 2018;Mostafazadeh et al., 2016;Roemmele et al., 2011;Zellers et al., 2018]. Many of these tasks are discriminative by nature, such as predicting a class label or an outcome given a textual context, as shown in the following example: Example 1. Natural language inference in SNLI dataset, with ground truth bolded.
Premise: A swimmer playing in the surf watches a low flying airplane headed inland.
Hypothesis: Someone is swimming in the sea.
The number of candidate labels may vary. Humans solve such questions by reasoning the logical connections between the premise and the hypothesis, but previous work [Naik et al., 2018;Schuster et al., 2019] has found that some NLP models can solve the questions fairly well by looking only at the hypothesis (or "conclusion" in some work) in the datasets. It is widely speculated that this is because in many datasets, the hypotheses are manually crafted and may contain artifacts that would be predictive of the correct answer. Such "hypothesis-only" tests can identify problematic questions in the dataset if the question can be answered correctly without the premise. While such a method to evaluate the quality of a dataset is theoretically sound, it i) usually relies on training a heavy-weight model such as Bert, which is costly to evaluate, ii) does not provide explanation why the question is a culprit, and iii) cannot be used to evaluate a model since a model that can make a correct prediction using only the hypothesis is not necessarily a bad model: it is just not given the complete data.
Inspired by black-box testings in software engineering, CheckList [Jurafsky et al., 2020] assesses the weakness of models without the need to know the details of the model. It does so by providing additional stress test cases according to predefined linguistic features. Unfortunately, to ensure the correctness of these additional cases, the templates must be carefully crafted with substantial restrictions, thus limiting the testing space and complicating the implementation. Furthermore, with CheckList, you only get to know what the model is incapable of doing but won't know what the model has learned from the data. (1) not a cue (2) (3) Figure 1: Example of a cue.
In this paper, our view is that the existing test sets for these tasks are not sufficiently exploited. Why do we go the extra mile to generate new test cases which are potentially incorrect, when we can test the models using existing test sets but from different perspectives? With this objective in mind, we propose ICQ ("I-see-cue"), an open-source evaluation framework 1 for evaluating both the dataset and the corresponding models. In this framework, one test dataset can be seen as multiple test sets from different perspectives.
Previous studies [Gururangan et al., 2018;Sanchez et al., 2018;Poliak et al., 2018] showed that statistical bias on linguistic features (e.g., sentiment, repetitive words and even shallow n-grams) which have statistical correlation with specific labels in benchmark datasets can be predictive of the correct answer. We call such biases artificial spurious cues when they appear both in the training and test datasets with a similar distribution over the prediction values. We illustrate this in Figure 1. Once these cues are neutralized from the test data, previously successful models may degrade substantially on it, suggesting that the model has taken advantage of the biased feature and is hence not as robust as assumed against with such cues.
In summary, this paper makes the following contributions: • we provide a light-weight but effective method to uncover the statistical biases and cues in NL reasoning datasets; • we propose two simple tests to quantitatively and visually assess whether a given model has taken advantage of a spurious cue when making predictions; • we evaluate the statisical bias issues comprehensively on 10 popular NLR datasets and 4 models and not only reaffirm some of the findings from previous work but also discover some new perspectives in these datasets and models; • we created an online demonstration system to showcase the results and invite users to evaluate their own datasets and models.

Preliminary Definition
We define an instance x of a natural language reasoning (NLR) task dataset X as where p is the context against which to do the reasoning (p corresponds to "premise" in Example 1); h is the hypothesis given the context p; l ∈ L is the label that depicts the type of relation between p and h. The size of the relation set L varies with tasks. There is another type of natural language reasoning tasks which are also in the form of multiple-choice questions, but their choices are a fixed set of labels, as shown below.  Ending 2: He is happy now.
We can transform the this case into two separate problem instances, still in the same form as in Eq.
(1), u 1 = (context, ending1, f alse) and u 2 = (context, ending2, true), where L = true, f alse. The ICQ framework is illustrated in Figure 2. In data filtering phase, it extracts from the dataset those problem instances that contain a given linguistic feature f . In cue discovery phase, it identifies the possible cues in this dataset. Finally, in model probing phase, it does two tests: "accuracy test" and "distribution test". Next we will discuss these phases in more details.

Linguistic Features
In this work, we consider the following linguistic features: Word (unigram tokens in the input sentences), Typos, NER (named entity recognition), Tense (temporal order of events), Negation, Sentiment, and Overlap (words occurring both in the premise and the hypothesis). The above list is by no means exhaustive, but just a starting point for users who can come up with additional features that are relevant to their task or domain.

Word
For a dataset X, we collect a set of all words V that ever exist in X. A word feature is defined as the existence of a word w ∈ V either in the premise or the hypothesis. Because V is generally very large, in practice, we may narrow it down to words that sufficiently popular in X. That is, we may remove words that seldom appear in X.

Sentiment
For each data instance x, we can compute its sentiment value as: where polar(w) is the sentiment polarity (-1, 0, or 1) of w determined by a look-up from a pretrained sentiment lexicon 2 .
We say x has a positive/negative/neutral sentiment feature if S(x) = 1, -1 or 0, respectively.

Tense
We say that an instance x has past, present or future tense feature if x carries one of these tenses, respectively, by the POS tag of the root verb in p or h.

Negation
Previous work has observed that negative words ("no", "not" or "never") may be indicative of a certain label in NLI tasks for some models. The existence of a negation feature in x is decided by dependency parsing 3 .

Overlap
In many models, substantial word-overlap between the premise and the hypothesis sentences causes incorrect inference, even if they are unrelated [McCoy et al., 2019]. We define that an overlap feature exists in x if there's at least one word (except for stop words) that occurs both in p and h.

NER
We define the NER feature as the existence of either PER, ORG, LOC, TIME, CARDINAL entity in x. We use the NLTK ner toolkit for this purpose.

Typos
We say an instance x has typo feature if there exists at least one typo in x. We use a pretrained spelling model 4 to detect all typos in a sentence. We don't distinguish the types of mispellings here. As we mentioned in Example 2, multiple choice questions are each split into two data instances with opposite labels (T or F), and the premises in these two instances are identical. Therefore, it is not useful to detect features within the premises alone. Consequently, for MCQ type of datasets, all the above features except for Overlap are only applied on the hypotheses.

Discovering Cues in Dataset
Once we have defined the linguistic features, we can build a data filter for each feature values. A filter takes a dataset and returns a subset of instances associated with that feature value. For example, there is a filter for the word "like"; there is a filter for "PER" entity; and there is filter for "negative" sentiment, etc.
For each feature f , we apply its filter to both the training data and test data of X, denoted as R and S in Figure 2, resulting in R f and S f . Only those features that appear both in the training and test data are qualified as possible cues for a dataset. Let y i be the number of instances with label l i in the filtered dataset, then we can compute the bias of the label distribution for a filtered set as the mean squared error (MSE): whereŷ i is the mean of y i . The larger mse(Y ), the more "pointed" the label distribution and more biased. Furthermore, if the filtered training set and the filtered test set are biased similarly, the Jensen-Shannon Divergence [Lin, 1991] between them is small: where M = 1 2 (R f + S f ). Finally, we define a cueness score as which represents how much a dataset X is biased against a feature f .

Probing Bias in Model
Suppose we already know that a dataset X is infected with a cue f from previous test in Section 3.2. However, just because the dataset is infected with a cue doesn't mean the model trained from this dataset necessarily exploits that cue.
Here we propose a simple method to probe any model instance trained from the given biased dataset 5 to see if it actually takes advantage of that cue f and by how much. We can do that through two simple tests: accuracy test and distribution test. In accuracy test, we simply assess the prediction accuracies of the model M on the filtered test set and on the remaining test data, and call them acc(S f ) and acc(S nf ), respectively. The accuracy test says that if the difference between these two accuracies, i.e, ∆ = acc(S f ) − acc(S nf ) is greater 0, then the model is considered to be biased and to have exploited this cue. The value of ∆ measures the extent of the bias.
Distribution test is a visual test. We first create a "stress data set" S f by "flattening" the label distribution in S f . We achieve that by replicating random instances from all labels except the most popular label in the filtered test set and adding them back into the set. The repetition augmentation procedure stops when the feature distribution based on each label is balanced. This way we have effectively removed the bias in the filtered test set and presumably posed a challenge to the model. Next we apply the same model on the stress test set to get prediction results. We compare the label distribution of the prediction results on the stress test set with the label distribution of the filtered training data. The idea is, if the filtered training data contains a cue, its label distribution will be skewed toward a particular label. If the model exploits this cue, it will prefer to predict that label as much as possible, even amplifying the skewness of the distribution, despite that the input test set has been neutralized already. We hope to witness such an amplification in the output distribution to capture the weakness in the model.
The above two tests are related but not equivalent and their outcome complement and reinforce each other.
We first show the experimental setup and then give the results on cue discovery as well as model probing along with some analysis.
The whole framework has been implemented into an online demo at http://anonymized.for.blind.review.

Setup
We evaluate this framework on 10 popular NLR datasets and 4 well-known models, namely FASTText (FT), ESIM (ES), BERT (BT) and RoBERTA (RB) on these datasets. All these datasets except for SWAG and RECLOR are collected through crowdsourcing. SWAG is generated from an LSTMbased language model. Specifications of the datasets are listed in Table 1

Hypothesis-only Tests
As a comparison to our framework, we first show the hypothesis-only test results on our 4 models and 10 datasets. In this test, we apply the models trained on full training data (with both premise and hypothesis) of the 10 datasets, and test their accuracies on the hypothesis-only test data (by stripping the premises from the questions in the test set). Table  2 shows the results, compared with the original accuracies using the full test data.
We further plot the four models hypothesis-only accuracies against voting by majority results in Table 2 in Figure 3. This figure depicts the "weakness" of the datasets to these models. The higher the bars, the weaker the dataset. We can see that SWAG, SNLI and MNLI are generally easier, whereas ARCT ADV is a hard task (the deviation results approach to zero).
Next, we plot the differences between the model accuracies on the full test data and the accuracies on the hypothesis-only  data in Figure 4. This experiment evaluates the robustness of the models. We see that the bars for FastText or ESIM are very short, while the bars are much longer for BERT and RoBERTA. This shows that from the hypothesis-test point of view, BERT and RoBERTA are more robust against the artifacts in the datasets, because the duo do not perform as well when given only the ending of the questions. These preliminary results serve as the basis for our findings next.

Cues in Datasets
We first filter the training and test data for each dataset using all the features defined in this paper. Left half of Table  3 shows the top 5 cues we have discovered as well as their cueness score for each of the 10 datasets. ARCT adv is an adversarial dataset and is known to be well balanced on purpose.
As a result, we only managed to find one cue which is OVER-LAP and its cueness score is very low. This is not surprising, because OVERLAP is the only "second-order" feature in our list of linguistic features that is concerned with tokens in both the premise and hypothesis, and likely escaped from the data manipulation by the creator. In most of the datasets, the top 5 cues discovered are word features, but besides OVERLAP, we do see NEGATION and TYPO showing up in the lists. In fact, SENTIMENT and NER features would have shown up if we expanded the list to top 10. It is also interesting to see several features previously reported to be biased by other works, such as "not" and NEGATION in ARCT, "no" in MNLI and SNLI, and "like" in ROCStory. Especially in MNLI, all the five cues discovered are related to negatively toned words, suggesting significant human artifacts in this datasets.
In the results, we also identify that some of the word cues are indicative of certain syntactic/semantic/sentiment patterns in the questions. For example, "because" in SNLI indicates a causal-effect structure; "like" in ROCStory indicates positive sentiment; "probably" and "may" in RACE indicate uncertainty, etc.
If we sum up the top 5 cueness scores for each dataset, we find that overall SNLI, RACE, ROCStory and MNLI carry more cueness than others, while ARCT adv has the smallest cueness. This result is generally consistent with our findings in the hypothesis-only test (see Figure 3) earlier, though now we can pinpoint what causes the weaknesses in these four datasets. The only exception is SWAG, which didn't emerge as a very weak dataset in the experiment here, but was found very biased in the hypothesis-only test. The reason is SWAG is the biggest dataset among the ten and we discovered more cues in it than the other 9 combined. Therefore, the top 5 cues are insufficient to represent the full scale of its weakness, which is showed by the hypothesis test as it gives the full picture.

Biases in Models
For each feature and a dataset, we train four models on its original training set, and test the models on its full test set, feature-filtered test set, and neutral test set. We first show the result of "accuracy tests" in Table 3. If ∆ is positive for a model on a feature, it means that the model exploits the existence of this feature. Conversely, the model exploits the non-existence of the this feature. A model is more robust against biases in the data, if ∆ is close to zero. Therefore, the bottom of Table 3 shows that, across all 10 datasets, by the sum of the absolute values of ∆, RoBERTA < BERT < ESIM  < FastText. This again is consistent with our hypothesis-only test earlier and the community's common perception of these popular models. However, if we delve into individual datasets and features, situation can be a bit murkier. For example, it seems that FastText tends to pick up more individual word cues than semantic cues, but more complex models such as BERT and RoBERTA appear to be more sensitive to structural features such as NEGATION and SENTI-MENT, which are actually classes of words. This can be well explained by the way FastText is designed to model words more than syntactic or semantic structures.
The fact that FastText is strongly negatively correlated with TYPO is also interesting. We speculate that FastText might have been trained with a more orthodoxical vocabulary and thus less tolerant to typos in text.
We further invesigate the models using the "distribution test". We show three insteresting findings in Figure 5. We observe that all models on cue "no" in MNLI achieve positive ∆ in Table 3, and fastText in particular. Consistent with the "Ac- curacy test", we find the predicting label distribution skewness is amplified in Figure 5a for fastText and ESIM -with "no" in sight, they prefer to predict "Contradiction" even more than the ground truth in training data. On the contrary, BERT and RoBERTA are only moderate in following the training data.
If cue "no" is very good at tricking the models, cue "above" is not as successful. Figure 5b shows that the distribution of predicting result for ESIM in ARCT is completely opposite to the training data. This explains while ∆ = −8.43 in Table  3 and demonstrates that models may not take advantage of the cue even though it is right there in the data. Similarly, the "flatness" in BERT and RoBERTA can also explain their low ∆ values in Table 3.
The example of cue "threw" presents an outlier for BERT, because the distribution test result is inconsistent between the accuracy test: The accuracy deviation is very high for BERT, but its prediction distribution is flat. We haven't seen a lot of such contradictory cases so far. But when it happens, as it is here, we give BERT the benefit of the doubt that it might not have exploited the cue "threw".

Related Work
Our work is related to and, to some extent, comprises of elements in three research directions: spurious features analysis, bias calculation and dataset filtering.
Spurious features analysis has been increasingly studied recently.
Much work [Sharma et al., 2018;Srinivasan et al., 2018;Zellers et al., 2018] has observed that some NLP models can surprisingly get good results on natural language understanding questions in MCQ form without even looking at the stems of the questions. Such tests are called "hypothesis-only" tests in some works. Further, some research [Sanchez et al., 2018] discovered that these models suffer from an insensitivity to certain small but semantically significant alterations in the hypotheses, leading to speculations that the hypothesis-only performance is due to simple statistical correlations between words in the hypothesis and the labels. Spurious features can be classified into lexicalized and unlexicalized [Bowman et al., 2015]: lexicalized features mainly contain indicators of n-gram tokens and cross-ngram tokens, while unlexicalized features involve word overlap, sentence length and BLUE score between the premise and the hypothesis. Naik et al. [2018] refined the lexicalized classification to Negation, Numerical Reasoning, Spelling Error. McCoy et al. [2019] refined the word overlap features to Lexical overlap, Subsequence and Constituent which also considers the syntactical structure overlap. Sanchez et al. [2018] provided unseen tokens an extra lexicalized feature.
Bias calculation is concerned with methods to quantify the severity of the cues. Some work [Clark et al., 2019;He et al., 2019;Yaghoobzadeh et al., 2019] attempted to encode the cue feature implicitly by hypothesis-only training or by extracting features associated with a certain label from the embeddings. Other methods compute the bias by statistical metrics. For example, Yu et al. [2020] used the probability of seeing a word conditioned on a specific label to rank the words by their biasness. LMI [Schuster et al., 2019] was also used to evaluate cues and re-weight in some models. However, these works did not give the reason to use these metrics, one way or the other. Separately, Ribeiro et al. [2020] gave a test data augmentation method, without assessing the degree of bias in those datasets.
Dataset filtering is one way of achieving higher quality in datasets by reducing artifacts. In fact, datasets such as SWAG and RECLOR evaluated in this paper were produced using variants of this filter approach which iteratively perturb the data instances until a target model can no longer fit the resulting dataset well. Some methods [Yaghoobzadeh et al., 2019], instead of preprocessing the data by removing biases, leave out samples with biases in the middle of training according to decision made between epoch to epoch. Bras et al. [2020] investigated model-based reduction of dataset cues and designed an algorithm using iterative training. Any model can be used in this framework. Although such an approach is more general and more efficient than human annotating, it heavily depends on the models. Unfortunately, different models may catch different cues. Thus, such methods may not be complete.

Conclusion
We develop a light-weight framework that evaluates the potential biases and cues in NLR multiple choice datasets and further shed light on the exploration of models at least from the perspective of the statistical cues. We experimented on a large range of datasets covering different tasks and conclude that the new evaluation framework is effective in discovering bias problems in both the datasets and some popular models.


extfieldsofstudy:
Computer Science
Engineering

text:
Safe multi-agent deep reinforcement learning for joint bidding and maintenance scheduling of generation units

This paper proposes a safe reinforcement learning algorithm for generation bidding decisions and unit maintenance scheduling in a competitive electricity market environment. In this problem, each unit aims to find a bidding strategy that maximizes its revenue while concurrently retaining its reliability by scheduling preventive maintenance. The maintenance scheduling provides some safety constraints which should be satisfied at all times. Satisfying the critical safety and reliability constraints while the generation units have an incomplete information of each others' bidding strategy is a challenging problem. Bi-level optimization and reinforcement learning are state of the art approaches for solving this type of problems. However, neither bi-level optimization nor reinforcement learning can handle the challenges of incomplete information and critical safety constraints. To tackle these challenges, we propose the safe deep deterministic policy gradient reinforcement learning algorithm which is based on a combination of reinforcement learning and a predicted safety filter. The case study demonstrates that the proposed approach can achieve a higher profit compared to other state of the art methods while concurrently satisfying the system safety constraints.

Introduction
Finding an optimal bidding strategy and maintenance scheduling of generation units in an electricity market would lead to a higher profit and an improved reliability of the entire system. The electricity market creates competition among the units since the profit of each unit depends on the market clearing price which is determined by the system operator (ISO), while the market clearing price is impacted by the bidding strategies of all units [1]. This problem can be categorized as a multi-agent bi-level decision-making problem [2]. The units, which are referred to as agents, are strategic and price-takers. In the first level, the agents decide about their strategies individually. In the second level, the ISO clears the market and obtains the power generation amount for each unit such that the demand of the system can be satisfied in all periods of time [3,4].
In this competition, the main aim of the unit is to maximize its profit by choosing the best bidding strategy. In addition to this goal, increasing the reliability and safety is another important objective that should be considered by the unit [5]. In order to retain the reliability, the units need to perform preventive maintenance actions that take a defined period of time enabling, thereby, to prolong the lifetime of the units. However, performing maintenance imposes maintenance costs on the units. In this case, the units need to obtain their optimal maintenance scheduling which makes a trade-off between increasing reliability and decreasing the maintenance cost of the system [6,7]. Hence, in summary, the goal of the unit in the electricity market can be considered as finding the optimal bidding strategies and maintenance schedule which maximizes the system's profits as well as reliability and minimizes the maintenance cost.
Nash equilibrium (NE) is one typical solution to the competition problem where no unit can increase its profit by changing its strategies while the other units are not changing their strategies [8]. Many papers address the optimal bidding of the electricity market using the game theory approach [9,10,1,11,12]. In these research studies, increasing reliability and maintenance schedules are, typically, not considered in the objective functions of the units. In fact, finding the NE of this problem is a challenging problem since the (1) Chair of Intelligent Maintenance Systems, ETH Zurich, Switzerland. Corresponding author, email address:ofink@ethz.ch.
Date: December 21, 2021. units are neither aware of the bidding of others nor of the market-clearing price of the system. In addition, another challenge is that the electricity demand should be satisfied at all time even when some units are out of operation and are performing maintenance. This imposes some constraints on the maintenance scheduling of units and the NE of the game should satisfy these critical constraints. Hence, the units face the incomplete information game associated with some uncertainties and constraints [2]. This is a challenging problem and needs some coordination among the units [13,14] Besides the works that address the bidding strategies of units, many papers address the distributed maintenance scheduling problem [15,16]. These papers proposed optimization approaches in which the units decide about their maintenance scheduling. To handle the challenges of electricity demand satisfaction, an algorithm based on the coordination and negotiation between ISO and units is proposed in [17,18]. These papers assume that units send their marginal cost as the bidding price to the ISO. Therefore, obtaining the optimal bidding strategies is not their goal.
All of the above works assume that the units have complete information of the market-clearing price as well as the bidding of their competitors. Thereby, they can solve the optimization problem and reach their NE. It is worth noting that this is a limiting assumption in the real market system. Reinforcement learning (RL) is a promising tool to handle challenges of incomplete information and system uncertainty [19]. In the case of an electricity market, this incomplete information and uncertainty are due to the unknown information about the actions of other units as well as the market clearing price.
Q-learning is one of the typical RL algorithms which applies look-up tables to find the optimal strategy in each state of the system [20] . However, since a bidding strategy in electricity markets is a continuous action, this method has a high computational time [21]. Deep Q-learning can tackle this challenge by using neural networks as function approximators for obtaining the strategies of the units [22,23]. Some research studies address the bidding strategies using deep RL algorithm [2,24,25]. However, they do not consider maintenance scheduling in addition to the bidding strategies.
As mentioned above, applying maintenance scheduling as one of the aims of the generation units imposes some critical constraints on the system such as electricity demand satisfaction and allocating the time for performing preventive maintenance. These constraints related to system safety and reliability should be satisfied at all time periods. Unfortunately, deep RL cannot ensure that all the system constraints can be satisfied during the training of the units.
One way to solve this issue is to combine a predicted safety filter with a RL algorithm [26]. In the proposed approach, a safety filter is responsible for satisfying system safety constraints and finding strategies which are as close as possible to the RL strategies while also satisfying all the system constraints. In other words, if RL strategies violate some constraints of the system, the predicted safety filter adjusts the decisions of the units to ensure that the constraints are satisfied.
In this paper, we propose to define the joint bidding and maintenance scheduling of generation units in the electricity market system as a safe deep RL problem. The proposed approach is able to satisfy the safety constraints and also handle the challenges of incomplete information due to the unknown bidding strategies of other units. In this approach, the units do not need to know the model and the bidding strategies of other units. They can update their strategies by getting feedback and reward from the electricity market system in each time step of the learning algorithm. In other words, at each time step, the units obtain their strategies and send them to the ISO. Then, they obtain their profit based on the information that they get from ISO, and update their policy. The main contributions of this paper are as follows: 1) We propose to formulate the maintenance scheduling and electricity bidding process of multiple generation units as a bi-level optimization problem. In the first level, the units aim to obtain the strategies which maximize their reward function while satisfying the system safety constraints related to the maintenance scheduling of agents. In the second level, the ISO clears the market and obtains the electricity price and power generation of each unit such that the total cost of the system is minimized and all the network constraints are satisfied.
2) We develop a multi-agent deep RL using the deterministic policy gradient (DDPG) algorithm to obtain the maintenance scheduling and bidding price such that the reward function of the agents is maximized without considering the safety constraints. This algorithm can handle the high-dimensional continuous action and state space. Moreover, it can tackle the non-stationary and uncertainties of the environment where each unit is not aware of the decisions of other units.
3) We design a predicted safety filter to guarantee that all the safety constraints are satisfied. This filter ensures that during the training as well as testing of the RL algorithm the units can perform maintenance for some specified time intervals. Moreover, this filter provides the conditions which ensure that the load of the system can be satisfied at all times by limiting the number of units that can perform maintenance simultaneously.
4) The proposed methodology can be applied online to the real system since the safety filter makes sure that even during the training of the learning algorithm all the safety and demand constraints can be satisfied. In other words, the agent does not need to learn its strategies offline and we do not need to build a virtual electrical market environment which is associated with many uncertainties.
The rest of the paper is organized as follows. The preliminaries on RL methodology and algorithm are introduced in Section 2. The units' and ISO's objective functions are formulated in Section 3. The solution method based on a combination of RL and predicted safety filter is proposed in Section 4. Simulation results of the case study are presented in Section 5. Concluding remarks are made in Section 6.

Preliminary on RL algorithm
The aim of RL is to maximize the total discounted reward function where the environment can be modeled as a Markov decision process (MDP) [19]. An action-value function is defined as follows: where s(t) ∈ S and a(t) ∈ A are the state and the action at time t. The policy is the probability of choosing action a given state s. The reward and discount factor are expressed by r and γ where γ ∈ [0, 1). The expected value of the reward over policy π is given by E π [ · ]. The aim of the RL algorithm is to find the policy which maximizes the action-value function as follows: (1) The conventional RL methods such as Q-learning can solve the action-value function (2) using a look-up table. In this case, a table is established to store the action-value function values for all possible state-action pairs. One typical way to update a table is to implement the temporal difference method [27] in which the value-action function is updated as follows: where l is the iteration index.
However, in cases where either the state or action space is continuous, Q-learning faces the "curse of dimensionality" problem. Deep RL is one way to tackle this challenge. In this method, instead of a look-up table, a neural network is trained to estimate the action-value function. This network can create a continuous mapping from the state and action pairs to the action-value function.

Problem formulation
In this section, we formulate the optimization problem of generation units and ISO for the electrical market with a uniform price. In the proposed model, the generation units decide about their maintenance scheduling u i and bidding strategy k i while the ISO obtains the amount of power generation for each unit g i and the electrical market price λ i through the following optimization problems.

Maintenance model of generation units
In this part, inspired by [18], we model the unit maintenance cost and its corresponding constraints. In the proposed model we assume that unit i is charged the maintenance cost c i at each time while performing maintenance and seeks to find the strategy that minimizes the total maintenance cost. In addition, we assume that during the decision horizon T unit i needs to perform maintenance at least for the total duration of H i . This preventive maintenance activity ensures that the generation unit can retain the system reliability. We assume that H i is determined by the safety regulation for each unit which is set based on the generation unit's data and model. Based on these preventive maintenance activities, we impose the following constraint: In addition, there is another constraint which represents the condition that if the maintenance of each unit starts at time t, it should be completed, which takes D i periods of time. We formulate this constraint as follows: It is worth to mention that these two constraints, (3.1) and (3.1), imply that for the whole decision horizon of time t ∈ T , unit i can perform maintenance several times with an interruption between them, provided that each maintenance action takes D i periods of time and the total period of maintenance is larger than the required duration H i of preventive maintenance of generation unit i.
In the following, we propose the optimization model of generation units using the above discussion.

Optimization model of generation units
In this section, we propose the optimization problem in which the goal of the units is to obtain their optimal bidding and maintenance scheduling strategies which maximize their revenue and also satisfy the maintenance constraints (Equations (3.1) and (3.1)). In this optimization problem formulation for clarity reasons and without loss of generality, we assume that each producer has only one generation unit. In addition, we assume that the start up and shut down costs are zero and each unit has a linear marginal cost. The optimization problem is formulated as follows: subject to : where the first term in the objective function λ(t)g i (t) expresses the revenue that unit i obtains by producing and selling the power. The second term λ i,m g i (t) and the third term c i u i (t) are the marginal and the maintenance costs of unit i. Constraint A 1 expresses the limit of the strategic bidding variable. In fact, each unit chooses a bidding strategy k i (t) and offers k i (t)λ i,m where k i (t) = 1 means that unit i offers its actual marginal cost and behaves competitively, and 1 < k i (t) ≤ k max i means that unit i behaves strategically. Constraints A 2 and A 3 are the maintenance constraints which have been explained in Section 3.1. Constraint A 4 denotes that the maintenance decision is a binary variable where u i (t) = 1 indicates that the unit performs maintenance at time t.
In problem (3.2), the price of the market λ(t) and the power generation of units g i (t) are obtained through the ISO optimization problem which is described in the following section.

ISO optimization model
In this section, we derive the optimization formulation of ISO where the maintenance decision u i and bidding strategy k i of the units are obtained through Eq.(3.2). ISO optimization aims to minimize the cost of the market while satisfying the load demand of the system. To achieve this aim, we formulate the problem as follows: subject to : where the objective function is to minimize the total costs of the units. The clearing price of the system λ is the dual multiplier of Constraint C 1 . Constraint C 1 expresses the demand-supply balance constraint. Constraint C 2 denotes the maximum and minimum power that unit i can produce. In fact, this constraint implies that when the unit is performing maintenance, its production output is zero. Constraints C 3 and C 4 express the ramp up and ramp down constraints of unit i.
In the following section, we explain our proposed methodology to solve problems (3.2) and (3.3).

Solution methodology: safe multi-agent deep RL
As we can see from the problem formulation in Section 3, we face a bi-level optimization problem. In this problem, in the upper level, the units aim to maximize their reward by obtaining their optimal bidding strategies and maintenance schedule. Also, the objective function of problem (3.2) depends on the price λ(t) of the market which can be obtained through To solve these challenges, we propose a algorithm which is a combination of RL and a safety filter. The proposed algorithm comprises two parts. In the first part, the units learn the strategies of other units using RL algorithm. In the second part, all the critical safety constraints (A 2 , A 3 ) are satisfied using the safety filter. In other words, the RL algorithms is able to find the strategies of the units without considering the safety constraints. Then, the units send their maintenance decisions to the safety filter and the filter obtains the decisions which are close to the decisions of the units while satisfying the safety constraints as well. In the next step, the bidding and maintenance decisions are sent to the ISO and it obtains the price and power generation of the units by clearing the market. Based on this information, the units update their strategies in order to gain more profit. The schematic of this approach is shown in Figure 1.
The formulation of the RL algorithm and the predicted safety filter are explained in the following sections.

Deep RL using DDPG algorithm
As mentioned above, the RL algorithm aims to obtain the strategies of the agents which consist of maintenance scheduling and bidding algorithm without considering the safety constraints A 2 -A 3 of (3.2). Since the bidding strategy is a continuous action, we apply deep RL using deep deterministic policy gradient (DDPG) algorithm in which the two neural networks are used to obtain on the one hand the value function and on the other hand the policy for each of the units. To implement this algorithm, we need to define the state, the action, and the reward of each agent. Let us define the state of the system at time t as follows: which consists of the electrical price and the demand of the system at the previous time step. This state can be provided by ISO and describes the system environment. The action of unit i can be defined as follows: where u i (t) = {0, 1} and 1 ≤ k i (t) ≤ k max i are the maintenance and bidding strategies of generating unit i.
Let us consider the objective function of (3.2), the reward function of unit i at time t can be formulated as: Equation (4.1) implies that the reward of unit i is the price that it obtains by selling its power to the market subtracting its generation and maintenance cost.
In the following, we explain the details of the DDPG algorithm and how the neural network weights are updated. As mentioned before, this algorithm comprises two neural networks which are estimating the action-value function and the policy of the units. They are also known as actor-critic networks. It should be mentioned that since we have N units and each unit decides about its own strategy individually, we have an individual actor-critic network for each unit. In other words, we have N actor-critic networks in which each unit updates the weights of the corresponding network using the DDPG algorithm. In this algorithm, each agent first obtains its action a i (t) using the actor-network. In the next step, it executes the action and gets the reward r i (t) and the new state s(t + 1). Then, it stores the tuple (s(t), s(t + 1), a i (t), r i (t)) and uses a mini-batch (s j (t), s j (t + 1), a j i (t), r j i (t)) to update the actor-critic network weights. In the following, we explain the details of the DDPG algorithm.
Critic network: In the DDPG algorithm, for each unit, two critic networks are considered to estimate the action-value functions which are called target and behaviour critic networks. The inputs of the networks are the state s(t) and the action of the units a i (t) which are defined as (4.1) and (4.1), respectively. The output is the associated action-value function Q i (s(t), a i (t)). Let us consider θ ′ i and θ i as the weights of the target and the behaviour critic networks of unit i, respectively. The loss function for updating the weights of the behaviour critic networks is defined as follows: where the target network Q i,target is calculated as follows: which in fact is the summation of the current reward r i (t) and the discounted value of the maximum actionvalue at the next time step t + 1 which is obtained by the target network.
During training and updating of the weights of the networks, the target network is updated at a slower speed than the behaviour critic network. This condition helps to stabilize the learning process. The weights of the behaviour network for unit i are updated as follows: where η i is the learning rate of the behavior network.
The weights of the target critic network for unit i are updated as follows: where τ i is the learning rate of the target network of unit i.
Actor network: The actor network of unit i is designed to obtain the optimal policy by considering the estimated action-value function, which is defined as: The input of the actor network is the current state s(t) and the output is the action a i (t) which maximizes Q i (s(t), a i (t)). Let us define the loss function of the actor network as follows: where θ µ i is the weight of actor network and µ i (s j (t), θ µ i ) is the policy generated by the network. In fact, the goal of unit i is to find the policy which maximizes Q i (s(t), a i (t)). Hence, by minimizing the loss function as (4.1) we can obtain the policy that maximizes Q i (s(t), a i (t)).
The weights of the actor network are updated as follows: where η µ i is the learning rate of the actor network of unit i. In the same way as for critic network, for the sake of stability, there is also a target actor network whose weights are updated as: θ The schematic of the DDPG algorithm for unit i is depicted in Figure 2.

Predicted safety filter model
As mentioned above, we aim to formulate the safety filter which obtains a maintenance decision close to the units' original decision while making sure that the updated decision guarantees that all the safety constraints of problem (3.2) are satisfied for the predicted horizon time as well. To achieve this aim, at each instant t, the units send their decisions u i (t) to the predicted safety filter. Then, the predicted safety filter obtains maintenance decisions u f,i (t) which satisfy constraints A 2 , A 3 for all periods of t ′ = {t, · · · , T }. Moreover, one additional aim is to ensure that while several units perform maintenance simultaneously and are not able to produce any power in that period of time, the load at the network level can still be satisfied. To achieve this goal, we restrict the number of units that can perform maintenance simultaneously. By considering all of these restrictions, we propose to formulate the safety filter optimization as follows: where x f,i (0) = 0. The objective function implies that if the maintenance decisions of the units u i (t) can satisfy all the constraints, the output of the safety filter is the same as the units' decisions. Constraints S 1 and S 2 are equivalent to constraint A 2 of problem (3.2). In fact, x f,i (t ′ ) measures the duration that the unit performs maintenance which at the end of the decision horizon T should be longer than H i . Constraint S 3 ensures that maximum M units can perform maintenance simultaneously, in which choosing the appropriate value of M guarantees that at each time instant there is a sufficient number of units in the power grid system that can satisfy the electrical demand of the system. Constraints S 4 and S 5 are the same as A 3 and A 4 which imply that the units need D i time for performing a maintenance action and the decision of the predicted safety filter is a binary variable.
Due to the nonlinear constraint S 1 , Problem (4.2) is a nonlinear programming problem. Using big-M method, we convert (4.2) to a mixed integer linear programming (MILP) problem. Let us consider Z i (t ′ ) = u f,i (t ′ )x f,i (t ′ ) and add the following constraints to (4.2) as an equivalent of the nonlinear term: where M is a large positive constant [28].
Then, using (4.2), we have the following MILP optimization:

Learning algorithm for electricity market bidding and maintenance scheduling
In this section, we summarize the proposed formulation of safe RL in Algorithm 1. This algorithm shows how all units can find their maintenance scheduling and electricity market bidding.

Case study evaluation results
In this section, we evaluate the performance of the proposed algorithm on IEEE 30-bus system with 6 generation units [29]. We consider a linear marginal cost for each generation and neglect the ramp up and down of the generating units. The maximum bidding k max i is considered as being 2 (Constraint A 1 in 3.2). Moreover, to increase the safety of the units, we impose the constraint that each unit should perform maintenance for at least 1 day during 100 days (Constraint A 2 in (3.2)). In addition, the repair duration time of a single maintenance action for all of generating units D i is considered to be one day. It means that once they decide to perform maintenance, it should be taken at least one day (Constraint A 3 in (3.2)). We Algorithm 1 Safe RL algorithm for bidding and maintenance scheduling Initialize the system state s(t) using (4.1) 5: for t ∈ T do 7: Obtain the action from the actor network a i (t) = max(min(µ i (s(t), θ µ i ), k max i ), 1), ∀i ∈ N 8: Obtain the maintenance scheduling which satisfies the safety filter u f,i (t) using (4.2) 9: Obtain the market clearing price λ(t) and units' power generation g i (t), ∀i ∈ N , using (3.3)

19:
Update the target actor networks' weights θ ′ µ i , ∀i ∈ N , using (4.1) 20: end for 21: end for  .2)). This limitation makes sure that the load of the system can be satisfied at all periods of time. The other parameters of the generating units are shown in Table 1. The parameters of the network are obtained from [29].
In this case study to implement Algorithm 1, we consider 100 episodes in which each episode consists of 30 days. We set two hidden layers for the actor-critic networks of each generating unit. In each layer, we use the Rectified Linear Unit (ReLU ) activation function. The initial value for the weights of neural networks is chosen randomly between 1 and 3. The number of samples N s = 100 is chosen for the DDPG algorithm. In the following, we evaluate the performance of the safe RL Algorithm 1.

Generation unit reward
We obtain the average reward over each episode for all units (r i (episode) =  training stage. This figure shows that the reward is increasing during training which means that the units can increase their profit by changing their bidding and maintenance strategies. The algorithm converges after about 40 episodes.
The bidding strategies of units k i (t), ∀i ∈ N , are shown in Figure 4. This figure shows that the bidding of units converges after about 40 episodes of the algorithm and that the units learn how to change their bidding to improve their profit.

Maintenance scheduling
The maintenance scheduling of generation units between day 700 until 900 is shown in Figure 6 (we only show a limited time interval to visualize the frequency of maintenance scheduling more clearly). This figure shows that at each time step, the maximum of two generating units can perform maintenance simultaneously. Moreover, each generation performs maintenance at least once during 100 days. This increases the system reliability and safety.
The average maintenance cost per unit for each episode and during the learning algorithm is demonstrated in Figure 6. According to this figure, the maintenance cost is decreasing during the learning algorithm which means that units learn how to find the sub-optimal maintenance scheduling and do not perform maintenance frequently.

Effect of the safety filter
As mentioned in Section 4, the RL algorithm obtains the units' strategies without considering the safety constraints. The predicted safety filter is then responsible to satisfy the safety constraints. In the case that we do not consider the safety filter, there is not any guarantee that the units perform maintenance before they fail. For-example, during the interval between iteration 700 and 900, the units do not perform maintenance which decreases the system safety. In addition, during training, when units are not aware of the other units' maintenance decision, did not learn their best strategies, and explore different strategies to learn the behaviour of system, using only RL without the safety filter might result in the case that most of the units may schedule their maintenance simultaneously. In this case, the demand of the system would not be able to be satisfied. Figure 7 shows the time interval where most of the units perform maintenance simultaneously and there is not any guarantee for satisfying the load of the system.

Performance comparison of different RL algorithms
In this section, we compare the results of Algorithm 1 with the Q-learning algorithm. In Q-learning, the RL algorithm applies a look-up table to update its policy. To obtain the look-up table, the states and actions of the system should be discretized. Hence, we consider discrete points for price and load as the state and bidding strategy as the action. Figure 8 shows the average profit of the safe DDPG algorithm and the safe Q-learning algorithm. As we can see, the DDPG algorithm converges in fewer episodes than the Q-learning. Moreover, the DDPG algorithm can achieve a higher profit than the Q-learning algorithm. These results can be explained with the fact that in this problem, the state and the actions are continuous. Hence, using a look-up table which approximates the continuous space with some discrete values decreases the performance and accuracy of the algorithm.

Conclusions
In this paper, we address the problem of joint bidding and maintenance schedule in the electricity market environment. We propose the safe deep RL algorithm to solve the problem. In the first step of this algorithm, each unit obtains its strategies using deep RL methodology without considering the safety constraints. Then in the second step, the safety filter modifies the units' decisions to ensure that all the system constraints can be satisfied. The proposed algorithm can handle the challenges of uncertainty and safety constraints. The results on the case study show that during the training, the profit of the units is increasing, while the generation units also perform preventive maintenance. In addition, the results demonstrate that the proposed methodology can achieve a higher profit compared to the Q-learning algorithm. As future work, one can consider some other sources of uncertainties in the electricity market such as renewable energies. In addition, the proposed algorithm can be applied to a larger system with more generation units by considering additional constraints such as contingency of the network.