Gaussian Process Meta Few-shot Classifier Learning via Linear Discriminant Laplace Approximation

The meta learning few-shot classification is an emerging problem in machine learning that received enormous attention recently, where the goal is to learn a model that can quickly adapt to a new task with only a few labeled data. We consider the Bayesian Gaussian process (GP) approach, in which we meta-learn the GP prior, and the adaptation to a new task is carried out by the GP predictive model from the posterior inference. We adopt the Laplace posterior approximation, but to circumvent the iterative gradient steps for finding the MAP solution, we introduce a novel linear discriminant analysis (LDA) plugin as a surrogate for the MAP solution. In essence, the MAP solution is approximated by the LDA estimate, but to take the GP prior into account, we adopt the prior-norm adjustment to estimate LDA's shared variance parameters, which ensures that the adjusted estimate is consistent with the GP prior. This enables closed-form differentiable GP posteriors and predictive distributions, thus allowing fast meta training. We demonstrate considerable improvement over the previous approaches.

Introduction
Few-shot classification [7,36,14,15] is the task of predicting class labels of data instances that have novel unseen class semantics, potentially from a novel domain, where the learner is given only a few labeled data from the domain. It receives significant attention recently in machine learning, not only due to the practical reason that annotating a large amount of data for training deep models is prohibitively expensive, but also the constant endeavor in AI to build human-like intelligence where the human is extremely good at recognizing new categories from a few examples.
In order to build a model that can generalize well to a novel task with only a few samples, meta learning [27,3] forms a training stage that is similar to the test scenario. More specifically, during the training stage, the learner sees many tasks (or episodes) where each task consists of the support and query sets: the learner adapts the model to the current task using a few labeled data in the support set, and the performance of the adapted model is measured on the query set, which is used as a learning signal to update the learner. This is in nature a learning-to-learn paradigm [32,28,34,8], and it often leads to more promising results in certain scenarios than simple supervised feature (transfer) learning. Although recently there were strong baselines introduced for the latter with some feature transformations [35,5], in this paper we focus on the meta learning paradigm.
As meta few-shot learning essentially aims to generalize well from only a few observations about a new task domain, it is important to learn prior information that is shared across different tasks. In this sense, the Bayesian approach [29,31] is attractive in that we can express the prior belief effectively, and easily adapt our belief to a new task based on the given evidence, in a principled manner. In Bayesian meta learning [11,41,10,16,39], the adaptation to a new task corresponds to posterior predictive distribution inference, and meta learning amounts to learning a good prior distribution from many training episodes.
To enable efficient Bayesian meta learning, the posterior predictive inference needs to be fast and succinct (e.g., closed form). To this end, we consider the Gaussian process (GP) model with the linear deep kernel [38] that allows parametric treatment of GP via the weight-space view [24]. Although there was similar attempt recently [22], they resort to regression-based likelihood model for the classification problem to derive closed-form inference, and such an ad hoc strategy can potentially lead to performance degradation. Instead, we propose a novel Laplace approximation for the GP posterior with a linear discriminant plugin, which avoids iterative gradient steps to find the maximum-a-posterior (MAP) adaptation solution, and allows a closed-form predictive distribution that can be used in stochastic gradient meta training efficiently. Hence, it is computationally more attractive than gradient-based adaptation approaches [8,11,16] by construction, and more amenable to train than neural net approximations of the predictive distribution (i.e., amortization) such as [10].
We show the improved performance of our GP approach over the regression-based previous work [22] and other state-of-the-arts on several benchmark datasets in both within-and cross-domain few-shot learning problems.

Problem Setup and Background
We provide the formal training/test setup for the meta few-shot classification problem (Sec. 2.1). We then briefly review the recent GP-based few-shot learning algorithm (GPDKT) [22] in Sec. 2.2 due to its close relation to our proposed approach.

Meta few-shot learning framework
The (C-way, k-shot) episodic meta few-shot classification problem can be formally defined as follows: • Training stage (repeated for T times/episodes): 1. Sample training data (S, Q) for this episode: support set S = {(x, y)} and query set Q = {(x, y)}, where S consists of C · k samples (k samples for each of the C classes), and Q contains C · k q samples (k q samples per class). We denote by y ∈ {1, . . . , C} the class labels in (S, Q), however, the semantic meaning of the classes is different from episode to episode. 2. With (S, Q), we train a meta learner F (S) → h where the output of F is a C-way classifier, h : X → {1, . . . , C}. The training objective is typically defined on the query set, e.g., the prediction error of h on Q.
• Test stage: 1. The k/k q -shot test data (S * , Q * ) are sampled, but the query set Q * is not revealed.
For the k-shot support set S * = {(x, y)}, we apply our learned F to S * to obtain the classifier h * = F (S * ). Again, the semantic meaning of the test class labels are different from those in the training stage. The performance of h * is measured on the test query set Q * .
For instance, in the popular ProtoNet [28], the meta learner learns the parameters θ of the feature extractor φ θ (x) (e.g., convolutional networks), and the meta learner's output h = F (S) is the nearest centroid classifer where the centroids are the class-wise means in S in the feature space. Note that h(x) admits a closed form (softmax), and the meta training updates θ by stochastic gradient descent with the loss, E (x,y)∼Q [CrossEnt(y, h(x))]. Another example is the GP meta learning framework that essentially considers h = F (S) as a GP posterior predictive model, that is, where f is a GP function, h(x) is defined as a probabilistic classifier p(y|x, S), and p(f |S) ∝ p(f ) · (x,y)∈S p(y|f (x)). Meta training of F amounts to learning the GP prior distribution p(f ) (i.e., GP mean/covariance functions). The recent GPDKT [22] is one incarnation of this GP framework.

Brief review of GPDKT
GPDKT assumes the GP regression model (its usage to classification will be described shortly), where the GP covariance function k θ is defined as the deep kernel [38]: where φ θ (x) is the feature extractor (comparable to that in ProtoNet) andk(·, ·) is a conventional kernel function (e.g., Gaussian RBF, linear, or cosine similarity). We abuse the notation to denote by θ all the parameters of the deep kernel, including those from the outer kernelk. They pose the meta training as the marginal likelihood maximization on both support and query sets: Due to the regression model, the marginal likelihood admits a closed form, and one can easily optimize (5) by stochastic gradient ascent.
To extend the GP model to the classification problem, instead of adopting a softmax-type likelihood p(y|f (x)), they rather stick to the GP regression model. This is mainly for the closed-form posterior and marginal data likelihood. In the binary classification problem, they assign real-valued y = ±1.0 as target response values for positive/negative classes, respectively, during training. At the test time, they threshold the real-valued outputs to get the discrete class labels. For the multi-class C-way problem with C > 2, they turn it into C binary classification problems by one-vs-rest conversion. Then during training, they maximize the sum of the marginal log-likelihood scores over the C binary problems, while at test time the one with the largest predictive mean E[y|x, S] over the C problems is taken as the predicted class. Although this workaround allows fast adaptation and training with the closed-form solutions from GP regression, the ad hoc treatment of the discrete class labels may degrade the prediction accuracy.

Our Approach
In this section we describe our Laplace approximation GP posterior formulation for the task adaptation, where we introduce the novel linear discriminant plug-in to circumvent the iterative optimization for the MAP solution and enable the closed-form formulas. Our formalism admits the softmax classification likelihood model, more sensible than the regression-based treatment of the classification problem.
We adopt the weight-space view of the Gaussian process model [24] using the linear-type deep kernel, and consider the softmax likelihood model with We let W = [w 1 , . . . , w C ] and B = [b 1 , . . . , b C ] be the weight-space random variables for the GP functions. Note that in (7) the prior (scalar) parameters β, β b are shared over all C functions, which is reasonable considering that the semantic meaning of classes changes from episode to episode. And it is easy to see that the i.i.d. priors on (w j , b j ) makes {f j (·)} C j=1 i.i.d. GPs with a zero mean and the covariance function, This can be interpreted as adopting a linear outer kernelk(z, z ′ ) = z ⊤ z ′ in the deep kernel (4) with some scaling and biasing. Although our formulation excludes more complex nonlinear outer kernels (e.g., RBF or polynomial), it was shown that the linear or cosine-similarity outer kernel empirically performed the best among other choices [22]. Note that the latter cosine-similarity kernel is obtained by unit-norm feature transformation (φ(x) → φ(x) ||φ(x)|| ). Given the support set S = {(x, y)}, the GP posterior distribution of f j (x) at some arbitrary input , and this is determined by the posterior p(W, B|S), where (up to constant) The posterior p(W, B|S) is used to build the task(S)-adapted classifier p(y|x, S), the GP predictive distribution derived from (1). And the meta training in our model amounts to optimizing the classification (cross-entropy) loss of the adapted classifier on the query set with respect to the GP prior parameters (i.e., β, β b , and the parameters θ of the feature extractor network φ), averaged over all training episodes. That is, our meta training loss/optimization can be written as: where p(y|x, S) = p(W, B|S)p(y|x, W, B)dW dB, and the expectation is taken over (S, Q) samples from training episodes/tasks. Considering the dependency of the loss on these prior parameters as per (10), it is crucial to have a succinct (e.g., closed-form) expression for p(W, B|S), as well as the predictive distribution p(y|x, S). However, since p(W, B|S) does not admit a closed form due to the non-closed-form normalizer (i.e., the log-sum-exp of (24) over {w j , b j } j ), we adopt the Laplace approximation that essentially approximates (24) by the second-order Taylor at around the MAP estimate {w * j , b * j } j , i.e., the maximum of (24).

Laplace approximation via LDA plugin with prior-norm adjustment
Specifically we follow the diagonal covariance Laplace approximation with diagonalized Hessian of (24), which leads to the factorized posterior p(W, B|S) = C j=1 p(w j , b j |S). The approximate posterior can be derived as j , and all operations are element-wise. However, obtaining the MAP estimate {w * j , b * j } j , i.e., the maximum of (24), although the objective is concave, usually requires several steps of gradient ascent, which can hinder efficient meta training. Recall that our meta training amounts to minimizing the loss of the task-adapted classifier p(y|x, S) on a query set with respect to the feature extractor φ θ (·) and the GP prior parameters β, β b , and we prefer to have succinct (closed-form-like) expression for p(y|x, S) in terms of θ, β, β b . The iterative dependency of p(y|x, S) on φ, β, β b , resulting in a similar strategy as MAML [8], would make the meta training computationally expensive. To this end, we propose a novel linear discriminant analysis (LDA) plugin technique as a surrogate of the MAP estimate.
LDA-Plugin. We preform the LDA on the support set S, which is equivalent to fit a mixture of Gaussians with equi-covariances by maximum likelihood [4]. More specifically, we consider the Gaussian mixture model (with some abuse of notation, e.g., p(x) instead of p(φ(x))), p(x, y) = p(y)p(x|y) = π y N (φ(x); µ y , σ 2 I), where we assume the shared spherical covariance matrix across different classes. The maximum likelihood (ML) estimate on S can be derived as: where S j = {(x, y) ∈ S : y = j}, n j = |S j |, n = |S|, and d = dim(φ(x)). Then our idea is to use this ML-estimated Gaussian mixture to induce the class predictive model p(y|x) = p(x, y)/p(x), and match it with our GP likelihood p(y|F (x)) in (6) to obtain {w j , b j } j , which serves as a surrogate of the MAP estimate {w * j , b * j } j . More specifically, the class predictive from the Gaussian mixture is: . (15) We match it with the GP likelihood model p(y|F (x)) from (6), that is, which establishes the following correspondence: where α is a constant (to be estimated). We aim to plug the LDA estimates (14) in (17), to obtain the MAP surrogate. However, there are two issues in this strategy: First, the ML estimate σ 2 * can raise a numerical issue in the few-shot learning since the number of samples is too small 1 , although π * and µ * incur no such issue. Secondly, it is only the ML estimate with data S, and we have not taken into account the prior on {w j , b j } j . To address both issues simultaneously, we propose a prior-norm adjustment strategy, which also leads to a sensible estimate for σ 2 .
Prior-norm adjustment. We will find σ 2 that makes the surrogate w j in (17) become consistent with our prior p(w j ) = N (0, β 2 I). Since w j sampled from the prior can be written as w j = [βǫ j1 , . . . , βǫ jd ] ⊤ with ǫ j1 , . . . , ǫ jd iid ∼ N (0, 1), we have: where the approximation to the expectation gets more accurate as d increases due to the law of large numbers. The equation (18) implies that any w j that conforms to the prior has the norm approximately equal to β √ d. Hence we enforce this to the surrogate w j in (17) to determine σ 2 . To consider all j = 1 · · · C, we establish a simple mean-square equation, (1/C) C j=1 ||µ * j /σ 2 || 2 = β 2 d, and the solution leads to the prior-norm adjusted MAP surrogate (denoted by w * j ) as follows: Determining α. We adjust b j to take into account its prior, and from (17) this amounts to finding α properly. We directly optimize the log-posterior (24) with respect to α. Denotingb j = log π * j − ||µ * j || 2 /(2σ 2 * ) (i.e., b j =b j + α), we solve = 0, and have a MAP surrogate (denoted by b * j ) as: Algorithm 1 GPLDLA meta training and meta test.
Summary. We have derived the Laplace approximated posterior p(w j , b j |S) in (11)(12) with the MAP surrogate (w * j , b * j ) from (19) and (20). From this GP posterior, we derive the predictive distribution p(y|x, S) that is used in our meta training (10) as well as meta test. We adopt the Monte Carlo estimate with M (reparametrized) samples from the posterior: where w where ǫ

Related Work
Few-shot/meta learning [3,27] has received enormous attention recently with the surge of deep learning, and it now has a large body of literature [36,14,15]. The approaches in few-shot learning can broadly fall into two folds: feature transfer and the other meta learning. The former uses the entire training data to pretrain the feature extractor network, which is then adapted to a new task by finetuning the network or training the output heads with the few-shot test data [35,5]. On the other hand, the meta learning approaches [28,34,8] follow the learning-to-learn paradigm [32], where the meta learner is trained by the empirical risk minimization principle.
In the Bayesian meta learning [11,41,10,16,39], the prior on the underlying model parameters typically serves as the meta learner, and the adaptation to a new task corresponds to inference of the posterior predictive distribution. In this way the meta learning amounts to learning a good prior distribution from many training episodes. For the efficient meta training, the posterior predictive inference, i.e., adaptation procedure, needs to be fast and succinct (e.g., in closed forms). Some previous approaches used neural net approximation of the posterior predictive distribution (i.e., amortization) [10,39], while others are based on gradient updates [8,11,16]. The main focus of meta few-shot learning lies on how to learn the meaningful prior model that can be quickly and accurately adaptable to novel tasks with only a limited amount of evidence.
Another recent Bayesian meta learner closely related to ours is MetaQDA [42], where they consider a mixture-of-Gaussians (MoG) classifier possibly with non-equal covariances, thus representing quadratic decision boundaries. With the Normalized-Inverse-Wishart prior on the MoG parameters, the posterior admits a closed-form expression by conjugacy. One of the key differences from our approach is that the MetaQDA deals with the joint MoG modeling p(x, y), while we focus on the discriminative p(y|x). It is known that the discriminative model has lower asymptotic error and is more data efficient without requiring marginal input distribution modeling [20]. But this comes at the cost of the non-closed-form posterior, and we had to resort to Laplace approximation with the prior-norm adjusted MAP estimates. Despite superb performance, there are several shortcomings of MetaQDA: it involves a large number of Wishart prior parameters to be trained, O(Cd 2 ) for C-way classification and d-dimensional features. On the other hand, ours has only two extra scalar parameters β, β b . Moreover, MetaQDA's performance is rarely known when the backbone feature extractor network φ(x) is jointly trained. They rather fix the features and only learn the prior QDA model. Its performance is highly reliant on the underlying feature extractor used.

Experiments
In this section we test our GPLDLA on several popular benchmark tasks/datasets in meta few-shot classification. We demonstrate the performance improvement over the state-of-the-arts, especially highlighting more accurate prediction than the previous GP few-shot model, GPDKT [22].
Implementation details. For fair comparison with existing approaches, we use the same feature extractor backbone network architectures φ θ (x) (e.g., convolutional networks or ResNets [13]) as competing models such as ProtoNet [28], Baseline++ [5], SimpleShot [35], and GPDKT [22]. For all experiments we use normalized features (φ(x) → φ(x) ||φ(x)|| ), which corresponds to the cosinesimilarity outer kernel with the original feature in our deep kernel GP covariance function (8). As the GP prior parameters β, β b , the only extra parameters, are constrained to be positive, we represent them as exponential forms and perform gradient descent in the exponent space. The number of Monte Carlo samples is fixed as M = 10 for all experiments. The details of the optimization hyperparaemters are described in Appendix.

Within-domain classification
The results on the CUB, miniImageNet, and tieredImageNet datasets are summarized in Table 1, Table 2, and Table 3, respectively. To have fair comparison with existing approaches, we test our model on the four-layer convolutional network (known as Conv-4) used in [28,34] and ResNet-10 as the backbone networks for the CUB dataset. For the miniImageNet and tieredImageNet, we use the Conv-4 and ResNet-18. We compare our GPLDLA with several state-of-the-arts, including MAML [8], ProtoNet [28], MatchingNet [34], and RelationNet [30]. We also consider the simple feature transfer, as well as strong baselines such as Baseline++ [5] and SimpleShot [35]. Among others, the (hierarchical) Bayesian approaches including VERSA [10], LLAMA [11], and Meta-Mixture [16], are also compared. However, we exclude methods that use more complex backbones or more sophisticated learning schedules [40,2,21,23], and those that require a large number of extra parameters to be trained [42].
Our approach achieves the best performance on most of the setups. On the CUB dataset, GPLDLA attains the highest accuracies for three cases out of four. On the miniImageNet, GPLDLA exhibits significantly higher performance than competing methods when the simpler backbone (Conv-4) is used, while being the second best and comparable to SimpleShot with the ResNet18 backbone 2 . And our GPLDLA outperforms GPDKT with all different kernels in most of the cases. GPLDLA also performs the best on tieredImageNet.   Table 4 where we use the Conv-4 backbone for both cases. GPLDLA exhibits the best performance for most cases outperforming GPDKT, except for one case. Our GPLDLA also performs comparably well with recent approaches with the ResNet-18 backbone on the miniImageNet→CUB task as shown in Table 5.

Calibration errors
Considering the practical use of the machine learning algorithms, it is important to align the model's prediction accuracy and its prediction confidence. For instance, when model's prediction is wrong, it would be problematic if the confidence of prediction is high. In this section we evaluate this alignment measure for our approach. Specifically we employ the expected calibration error (ECE) [12] as the measure of misalignment. The ECE can be computed by the following procedure: the model's prediction confidence scores on the test cases are sorted and partitioned into H bins (e.g., H = 20), and for each bin we compute the difference between prediction accuracy (on the test examples that belong to the bin) and the confidence score of the bin. The ECE is the weighted average of these differences over the bins with the weights proportional to the numbers of bin samples. Hence the smaller the better.
Following [22], we sample 3000 tasks from the test set on the CUB dataset, and calibrate the temperature parameter by minimizing the negative log-likelihood score, and use another 3000 tasks from the test data to evaluate the ECE loss. The ECE losses averaged over five random runs are summarized in Table 6. On the 1-shot case, our GPLDLA attains the lowest calibration error, while being slightly worse than ProtoNet and GPDKT on 5-shot.

Ablation study
To verify the impact of the proposed approximation strategy of LDA plugin and prior-norm adjustment, we conduct ablation study in this section. We compare three models: i) Laplace approximation that finds the MAP solution without any approximation (neither LDA plugin nor prior-norm adjustment), ii) approximate MAP estimate by LDA plugin alone without prior-norm adjustment, and iii)  both LDA plugin and prior-norm adjustment (hence our GPLDLA). For Laplace approximation we take 5 Newton steps to find the MAP solution, where the cascaded update operations are differentiable for meta learning of the prior parameters. For the LDA plugin alone, we need to estimate the class-conditional variance parameter σ 2 , and we adopt the median distance heuristics: σ set as the median of all pairwise feature distances (in the support set), a common practice for determining feature scales in kernel machines [9]. Table 7 summarizes the test performance of the three models. Overall our GPLDLA and Laplace approximation perform equally well whereas GPLDLA slightly outperforms Laplace approximation most of the time. This may be attributed to the ProtoNet-like effect of the LDA plugin estimator, which serves as an additional regularizer for few-shot learning. We also see that LDA plugin alone significantly underperforms GPLDLA, implying that prior-norm adjustment is more effective than heuristic median-distance rule in determining the class conditional feature scale. However, the differences are less pronounced for larger backbones (ResNets) in which the importance of the sophisticated features is dominant.

Running times
Next we measure the wall clock running times for competing Bayesian meta few-shot methods. The per-episode inference time with the Conv-4 backbone on tieredImageNet is reported in Table 8. It shows that GPLDLA is the fastest thanks to the efficient closed-form inference steps. GPDKT [22] has computational overhead of solving C binarized problems separately where C is the number of Table 8: Per-episode inference time (milliseconds) with the Conv-4 backbone on tieredImageNet. We exclude the feature computation times and only measure the inference time, that is, time for computing p(y|x, S) for (x, y) ∈ Q.

Conclusion
We proposed a novel GP meta learning algorithm for few-shot classification. We adopt the Laplace posterior approximation but circumvent iterative gradient steps for finding the MAP solution by our novel LDA plugin with prior-norm adjustment. This enables closed-form differentiable GP posteriors and predictive distributions, thus allowing fast meta training. We empirically verified that our approach attained considerable improvement over previous approaches in both standard benchmark datasets and cross-domain adaptation scenarios. We would like to conclude the paper by discussing some limitations of the proposed approach. In the current study we have not carried out rigorous theoretical analysis on the impact of the cascade of approximations that we used. Our strategy looks reasonable, and works well on empirical data, but further theoretical study needs to be carried out as future work.


Active Learning of Abstract Plan Feasibility

Long horizon sequential manipulation tasks are effectively addressed hierarchically: at a high level of abstraction the planner searches over abstract action sequences, and when a plan is found, lower level motion plans are generated. Such a strategy hinges on the ability to reliably predict that a feasible low level plan will be found which satisfies the abstract plan. However, computing Abstract Plan Feasibility (APF) is difficult because the outcome of a plan depends on real-world phenomena that are difficult to model, such as noise in estimation and execution. In this work, we present an active learning approach to efficiently acquire an APF predictor through task-independent, curious exploration on a robot. The robot identifies plans whose outcomes would be informative about APF, executes those plans, and learns from their successes or failures. Critically, we leverage an infeasible subsequence property to prune candidate plans in the active learning strategy, allowing our system to learn from less data. We evaluate our strategy in simulation and on a real Franka Emika Panda robot with integrated perception, experimentation, planning, and execution. In a stacking domain where objects have non-uniform mass distributions, we show that our system permits real robot learning of an APF model in four hundred self-supervised interactions, and that our learned model can be used effectively in multiple downstream tasks.

I. INTRODUCTION
Long horizon sequential manipulation tasks still pose a challenging problem for robotic systems. Tasks such as assembly depend on using many objects with varying physical properties. Finding a plan to achieve a task in these domains consists of reasoning over large spaces that include discrete action plans, as well as low level continuous motion plans.
These problems can be effectively addressed hierarchically: at the highest level of abstraction the system searches over plausible abstract action sequences, and at the lower level it plans for detailed concrete motion plans and object interactions. The complexity of the search space for the concrete planner is greatly reduced when constrained by the abstract action sequence. Further computational efficiencies can be gained if we lazily [17] postpone concrete planning until we have a complete abstract plan that is likely to succeed, avoiding the need to query the concrete planner multiple times. A version of this approach is used in skeleton-based task and motion planning systems [25,16,22].
The success of this lazy strategy hinges on our ability to predict whether an abstract action sequence will be feasible to execute. In this work, we call an abstract action sequence feasible if both the concrete planner returns a solution and this solution is reliably executed in the real world with the intended outcome. If the abstract planner does not take into account errors in execution due to phenomena unmodeled by the concrete planner, the robot may end up attempting a plan that fails during execution.
Fortunately, even an approximately correct estimator of abstract plan feasibility (APF) can offer huge computational advantages during planning. In some cases it may be possible to approximate APF via coarse-grained simulation. However, this strategy still requires a coarse dynamics model, which may not capture complex phenomena needed to accurately predict feasibility in the real world.
Instead, we explore a strategy in which we learn a model that predicts APF by exploring the space of real plan executions without a specific planning problem or task at hand -a form of curious exploration [27]. Data efficiency is a primary concern in enabling real robot learning of feasibility models. Here, a training instance is the execution trace of an abstract plan, labeled by success or failure. Labeling each such plan is very expensive as it involves finding and executing a concrete motion plan, potentially taking several minutes on a real robot. Furthermore, due to the combinatorial input space of abstract action plans, randomly executing actions is unlikely to elicit interesting behavior.
To address the data efficiency problem, we observe that in the process of training the APF model, some observations may be more valuable than others. Active learning is a technique for identifying unlabeled instances that are most informative in learning a target concept. The technical challenge is how to (C) After observing the true plan feasibility, φ, the robot uses this new labeled data to update its APF model, represented as an ensemble of neural networks. Evaluation Phase (right) Once an APF model has been learned, the robot can use it to perform various tasks, such as building the tower with the longest overhang from a given set of blocks. find plans of interest -an active learning approach requires both a way to generate candidate plans, and a way to score how informative a candidate plan might be given the current model.
To determine how informative a plan is with respect to the learned APF model, we adopt an information-theoretic active learning approach [26,19]. To generate candidate plans, we exploit an important property of abstract action sequences: for an action sequence (a 1 , . . . , a n ), if any prefix (a 1 , . . . , a i ) is infeasible, then any longer prefix (a 1 , . . . , a j ) for i < j ≤ n is also infeasible. This infeasible subsequence property gives us leverage during data acquisition. A complex plan instance may contain many elements that are highly informative for model learning, but will never be experienced because early elements in the plan will fail with high probability.
We apply this active learning strategy to the concrete problem of stacking blocks with a real robot, where the blocks are each unique and have non-uniform mass distributions. The robot autonomously designs, plans, and executes experiments to learn a feasibility model using a Franka Emika Panda robot arm ( Figure 1). The robot is also capable of resetting the world state after each experiment, enabling continuous autonomous experimentation. The learned feasibility predictor is later used to build towers with previously unseen blocks that satisfy several different objective functions, including the tallest possible tower or the tower with the longest overhang. This sample-efficient autonomous learning process relieves engineers from supervising data collection, resetting the experimental environment, and having to specify accurate dynamics models for planning. This results in a highly flexible and robust system for planning and executing complex action sequences in the real world.
In summary, our contributions are: • A method to learn an Abstract Plan Feasibility model by synthesizing hypothetical plans; • A data acquisition approach which leverages the infeasible subsequence property when sampling potential plans; • A robotic system which conducts autonomous selfsupervised learning via integrated perception, experimen-tation, planning, and execution.

II. PROBLEM FORMULATION
Our objective is to learn a model that predicts the success of an abstract action sequence when it is executed by the robot. That is, to learn the parameters Θ that predict where φ ∈ {0, 1} is the success of the sequence of abstract actions, a = (a 1 , . . . , a n ). Furthermore, we wish to learn Θ using as few labeled action sequences (a, φ) as possible.
Note that in general, APF may need to consider the initial state in which a plan is to be executed, however in this work we make the assumption that the first abstract action in a sequence will be feasible, and does not depend on the initial state. The method applies regardless of whether this assumption is made.
To learn this APF model, our system operates in two phases as illustrated in Figure 2. In the experimentation phase, the robot curiously explores the space of possible plans, learning the parameters of the APF model; in the evaluation phase, the robot is given specific goals to achieve, and uses the learned APF model to efficiently plan over abstract action sequences.
Both phases depend on the ability to construct and execute concrete plans on a real robot given an abstract action sequence. We assume that a system is available that can integrate perception, planning, and control to execute an abstract action sequence and observe whether or not it was successful. In Section IV we describe our implementation of this system in detail.

A. Experimentation phase
During the experimentation phase, the robot performs active learning to efficiently design, plan, and execute abstract action sequences that are informative about APF. This phase operates in a loop. At each iteration, the robot first generates candidate action sequences that may be informative according to some sampling strategy. Each action sequence is then scored using the current model according to some acquisition function, f : A → R. The highest scoring plans are executed on the robot to obtain feasibility labels which are then used to update the model for future iterations of the active learning loop. The effectiveness of active learning in reducing data complexity depends critically upon the choice of acquisition function and sampling strategy -or which action sequences are considered as possible experiments. These choices are discussed in more detail in Section III.

B. Evaluation phase
After the APF model has been learned, it can be used to achieve multiple objectives. We assume that the robot is given an objective function V which maps action sequences to real values and its goal is to find the abstract action sequence, a ∈ A with the highest expected value, where the expectation takes into account the likelihood that an abstract action sequence is feasible when executed on the real robot. To maximize this objective, we use a Monte Carlo planner which randomly samples action sequences and selects the one with the maximum expected value.

III. ACTIVE LEARNING OF ABSTRACT PLAN FEASIBILITY
Collecting data on real robot platforms is both time and cost-intensive. To minimize the amount of data needed to learn the APF model, we take an information theoretic approach to active learning [19]. Concretely, we maintain a distribution over model parameters that are consistent with the data we have observed so far. Our objective is to select new data that minimize the entropy over this distribution as quickly as possible. Efficient active learning requires: (1) a model class that captures uncertainty in model parameters, (2) a way to score unlabeled plans based on how informative they may be, and (3) a method of generating potentially informative plans. We discuss each of these in turn.

A. Abstract plan feasibility model
Our APF model, Pr(φ | a; Θ), aims to capture the uncertainty in the underlying stochastic process of predicting the feasibility of abstract action sequences. This uncertainty can be attributed to phenomena such as the robot's motor capabilities, errors in perception, or unmodeled behaviors of the planning process, and is referred to as aleatoric uncertainty. Our goal is to learn parameters Θ such that this uncertainty is adequately captured and our model can be leveraged, along with a low level planner, to achieve a goal.
We take a Bayesian approach to learning the model parameters, and maintain a distribution over the parameter space, Pr(Θ). This distribution aims to capture the uncertainty we have regarding the accuracy of our predictions, referred to as epistemic uncertainty. In general, for complex model classes such as neural network classifiers, an explicit representation of Pr(Θ | D) for training data D is difficult to construct or update with new data. We therefore follow the strategy of Beluch et al. [3] and represent this uncertainty with an ensemble of N models, (θ 1 , . . . , θ N ), where θ i ∈ R d . Initial parameters are drawn independently at random and are updated to incorporate new data via gradient descent.
The design of the models in the ensemble is selected to match the underlying structure of the prediction problem. In a naive implementation, we can directly estimate feasibility from the entire sequence of actions. We will refer to this approach as a complete model, denoted by Θ comp . However, we observe that the infeasible sub-sequence property provides a strong constraint on the set of feasible plans: sequence (a 1 , . . . , a n ) is only feasible if all of its prefixes are also feasible. As such, the model can instead learn the probability a specific action is feasible given all previous actions were feasible. We will refer to a model that considers this property as Θ ss . Under this model, the feasibility of a plan, a, is: where Φ i represents whether action a i is feasible given a 1:i−1 were feasible, and we use the subscript 1 : n to refer to a sequence of variables. In our method we assume initial actions are feasible, meaning Pr( The Θ comp model only considers full action sequences, and therefore entire plans correspond to a single label once they are executed. On the other hand, Θ ss requires labels after each action is executed. In both of these models, the length of plans for which we require predictions varies. Therefore, we use graph neural networks (GNNs), which make predictions based on aggregations of local properties and relations among the input entities, and exploit parameter tying to model global properties of plans of arbitrary size using a fixed-dimensional parameterization Θ. For a detailed description of GNNs, see the overview by [39]. More details about the specific architecture we use can be found in Section IV-C.

B. Entropy reduction
Following [26,6], we guide our active learning by picking a sequence of data D that maximally reduces the entropy of Pr(Θ | D). The general problem of designing a sequence of experiments to minimize entropy -or equivalently, maximize information gain -is a difficult sequential decision-making problem. Fortunately, due to sub-modularity of the objective, a myopic approach that considers only the next experiment to conduct can be shown to be a good approximation to the optimal experimentation strategy [9].
Given a model distribution Pr(Θ | D) that depends on the data we have seen so far, D, we choose the action sequence a ∈ A that reduces the entropy of the posterior distribution as much as possible: While the robot can select the plan, a, to experiment with, it cannot select the outcome φ, so to compute this quantity, we have to take an expectation over the outcome, using our current model distribution.
Estimating the entropy over a high-dimensional parameter space is expensive, so we follow the approach of Houlsby et al. [19] to reformulate the objective in (3) as: allowing the computation of entropies to take place in the lower-dimensional label space, Φ. This is known as Bayesian Active Learning by Disagreement, or BALD. The BALD objective invites an appealing interpretation: maximizing the first term encourages selecting an a that our model is overall uncertain about, and minimizing the second term encourages selecting an a for which the individual models in (θ 1 , . . . , θ N ) can make confident predictions about the outcome φ. If we think of the overall uncertainty as a combination of epistemic and aleatoric uncertainty, then this objective seeks an experiment with high overall uncertainty and low aleatoric uncertainty, which therefore has high epistemic uncertainty. Intuitively, if the various possible models in Pr(Θ | D) are individually confident but about differing outcomes, then observing the φ value corresponding to a is likely to prove some of those outcomes incorrect.
Using an ensemble of equally weighted parameter vectors (θ 1 , . . . , θ N ) to represent Pr(Θ | D) allows us to compute a global feasibility prediction, as well as find the experiment that maximizes the estimated BALD objective in the form:

C. Sampling Strategies
Now that we have established an informational score for experiments, we consider several sampling strategies for optimizing over A, the set of plans up to a fixed length L. Maximizing the BALD objective over the entire set A is difficult because we need to consider all discrete plans up to length L, as well as all possible assignments to each continuous abstract action parameter.
Complete One strategy we consider is uniformly sampling complete plans from A and scoring the samples. We call this the complete strategy.
Unfortunately, to achieve a consistent sampling density, the number of required samples scales exponentially with the length of the plan. Additionally, this strategy might generate most of its samples in the infeasible part of the space.
Greedy Another strategy requiring fewer samples is a greedy approach, in which we select the next action a n which maximizes the BALD objective, given that we have already optimistically constructed a 1:n−1 . This strategy does not take into consideration that if a plan fails early, we do not get to learn from the full plan execution.
We can leverage additional structure afforded to us by the infeasible subsequence property when we do active learning using the Θ ss model class. In the following, we consider two possible strategies.
Sequential When generating a potentially informative plan for the Θ ss model, we can take into account the probability a specific action will be attempted (i.e., that the plan was successful up until that action). This allows us to find plans whose informative outcomes have a high probability of being observed. The resulting objective is: where the probability is calculated as in Equation 2. This equation computes the expected information gain for executing a plan, taking into account the probability that plan execution fails at any given step as predicted by the learned APF model. We refer to this as the sequential strategy.
We implement the sequential approach naively by sampling and scoring entire plans. Like complete, this method requires exponentially more samples for longer plans, however, in the domain we considered, a sampling approach was acceptable as we had a relatively short plan horizon. In the future work we hope to extend this strategy to a search-based method which prunes candidate plans by their predicted feasibility.
Incremental We also consider a strategy where we only consider plans (a 1 , . . . , a n ) for which we have already observed the prefix to be feasible. In other words, the prefix (a 1 , . . . , a n−1 ) together with result φ 1:n−1 = 1 are in the current data set D. We call this the incremental strategy. argmax a1:n∈A 1 (a1:n−1,φ=1)∈D BALD(a 1:n ; Θ ss ) Although this strategy finds more feasible plans than complete, it also requires the robot to reuse plan prefixes, so we do not gather novel observations when repeating a plan prefix. Constraining our experiments to build on previously feasible plans may be overly restrictive.

IV. IMPLEMENTATION
We have implemented this framework for a class of problems in which the robot manipulates objects to construct towers. All of our experiments use the 7-DOF Panda robot from Franka Emika, in simulation and in the real world.

A. Domain
The world consists of the robot and a set of objects, O, with which it can interact. In this work, we consider cuboids with non-uniform mass distributions ( Figure 3). Each object, o ∈ O, is described by a tuple, (d, c, m), where d ∈ R 3 are the dimensions, c ∈ R 3 is the offset of the center of mass from the center of geometry, and m ∈ R is the object's mass. The cuboids for this manipulation task were constructed from laser cut plywood. A 25mm diameter lead ball is mounted inside some of the objects to significantly alter the mass distribution. Right: Unique ArUco markers are applied to each face of each object, for object identification and localization. Visible in the foreground is one of the two external cameras mounted around the workspace.
During the experimentation phase we use a set of 10 blocks, and all evaluations are performed with a different set of 10 blocks. The block parameters from each set are sampled from the same uniform distribution over the dimensions of the objects and the locations of the center of mass within the objects.
The abstract actions are to place objects onto a stack; they are specified by a = (o, r) where r ∈ SE(3) is the relative pose of o with respect to the object placed in the previous action (or the table if this is the first object placement).
An abstract plan is feasible if the detailed planning and execution system can find and execute robot commands (i.e., grasp poses and motion plans) such that the objects are placed on top of one another and the resulting tower is stable. Note that, for this property to hold, each prefix of the plan must also be feasible -that is, each subtower is stable. The learned APF model is applied to three different objectives in the evaluation phase: 1) Tallest Tower: The objective is to construct the tallest possible tower. 2) Longest Overhang: The objective is to construct the tower with the maximum distance from the center of geometry of the bottom block to the furthest vertical side of the top block. 3) Maximum Unsupported Area: The objective is to construct the tower where each block has as much area possible unsupported by the block below it. See Appendix C for a discussion on the generalizability of our method outside of the towers domain.

B. Perception, planning, and execution
Perception For the system to robustly pick up blocks and recover from unstable towers falling in unpredictable configurations, we require a perception system that can identify and localize objects at arbitrary positions. Although more advanced perception systems might be needed for arbitrary objects, vision is not the immediate focus of this work, so we pattern our objects with ArUco markers to simplify perception. To indicate identity and avoid orientation ambiguity, each object has a unique ArUco marker on each face.
Two RealSense D435 depth cameras mounted statically on a frame observe the workspace and allow for localizing the objects with minimal occlusions. If an object is not visible, it is assumed to be at its home position behind the arm. Due to the resolution of the cameras and size constraints of the tags on the blocks, we found that the pose estimates from the static cameras can have up to 1 centimeter of error. This level of error is acceptable, as rough pose estimates are refined with a third RealSense D435 camera mounted on the robot wrist. As the arm moves to a pre-grasp pose computed from the noisy object pose estimate, the wrist-mounted camera collects images closer to the the object to be grasped, allowing for a refined pose estimate and more precise grasp.
Planning In this domain, executing an abstract action requires a multi-step task and motion plan. For example, to place an object on a tower with an arbitrary relative pose to the block below it, regrasping may be necessary. When a tower falls over, the robot will also need to move fallen blocks out of the way to build the next tower.
To handle these scenarios, we turn to a large body of work in task and motion planning. We use PDDLStream [13], which integrates PDDL task-level planning with lower level motion planning. The PDDL domain describes actions including picking and placing objects and moving the arm through free space, while a Bidirectional RRT [23] in joint configuration space performs motion planning and collision checking with a surrogate world model implemented in PyBullet [7]. In this work we make the simplifying assumption that there are dedicated positions for the base of the tower, regrasping, and storage for each of the objects. These constraints are specified to the planner to reduce planning time by limiting the search space of possible action parameters.
Execution The motion plans generated by PDDLStream are executed using joint-space controllers on the robot. When constructing a tower in the experimentation phase, after each block placement the wrist camera is used to check for tower stability. If a tower was unstable, then the last manipulated block will not be near its expected pose in front of the gripper.
To improve data collection efficiency, we parallelize execution and planning. As the robot executes a motion plan to assemble or disassemble a tower, the planner produces plans to move each individual block in that tower under the assumption that all actions will be successful. This parallelism is interrupted if a tower falls over prematurely, prompting a replan to clear the fallen blocks.
If the state of the world is such that a robot is unable to find a plan to proceed with experimentation or execute an existing plan, human intervention may be required. We have provisioned for several of these cases, including blocks falling off the table, or too close to one another for the planner to find feasible grasp candidates. Once such issues are manually resolved (e.g., by putting the block back on the table), the robot can update its estimate of block poses and resume planning.

C. Learning
As discussed in Section III-A, the distribution over APF model parameters is represented by an ensemble of networks. In our implementation, the ensemble is made up of GNNs with domain-specific connectivity. The input to the GNN is an abstract action sequence, a. Each action, or block placement a i , is passed into a separate node in the graph, and each node (corresponding to a block placement in the tower) is connected to nodes above it in the tower. This mirrors the analytical computation of tower stability, in that each subtower's combined center of mass must be within the contact patch of the block below it. We refer to this network architecture as a TGN, or Towers Graph Network. Other network architectures which are invariant to task plan length are a Fully Connected Graph Network (FCGN) and a LSTM model. The FCGN uses the same node and edge networks as our TGN model, but has edges between each node. The LSTM model passes each block's vector representation through the network in order starting from the top of the tower, and most closely matches our TGN connectivity. A visualization of the connectivity of each architecture is given in Figure 4.
In our experiments, 10 networks are used in the ensemble. Each individual network is randomly initialized and trained using the binary cross-entropy loss function with early stopping according to the loss on a validation set, which is also collected actively.
Before active experimentation, each model in the ensemble is initialized by training on the same dataset (shuffled differently for each) of 40 randomly generated towers. For the complete strategy the towers are of size 2 − 5, and for the sequential, incremental, and greedy strategies they are of size 2. During the experimentation phase, at each iteration the top 10 most informative towers are chosen and labeled by attempting to build each with the robot. 20% of the collected data is added to a validation set and the remainder is added to the training set. We perform data augmentation by rotating each collected tower 90, 180, and, 270 degrees about the axis normal to the table surface.

V. EVALUATION
In the following sections we evaluate the utility of various sampling strategies (Section V-A) and model architectures (Section V-B) for increasing data efficiency and generalization. In simulation, we show that these choices greatly influence the robot's accuracy and performance on multiple downstream tasks. In addition, Section V-C gives the performance of our incremental sampling strategy on a real robot. We show that not only can the robot learn an APF model from real data and use it to perform downstream tasks, but also that learning on the real robot allows us to be robust to noise that would be difficult to model in simulation.
When evaluating task performance, we randomly select 5 blocks from a set of novel blocks and execute the best tower (that uses all 5 blocks) found by the Monte Carlo planner, given the task objective and our Learned APF model. The reward received from executing this tower is used to calculate normalized regret, which is the difference between this received reward and the largest reward of a stable tower considered by the planner (found using an Analytical model). If a tower is unstable, we assign a reward of zero.
A. Impact of Sampling Strategy Figure 5 shows the task performance of models trained using the four sampling strategies described in Section III-C. Each strategy has results aggregated from 4 independent training runs and 50 task evaluations per run all performed in a simulated environment. Different sets of blocks are used between training and evaluation.
Our APF model performs best on the Tallest Tower task, successfully minimizing regret after constructing only 200 towers with the sequential method. The incremental method also performs well, successfully minimizing regret with minimal variance across runs. The complete method on average performs well, but has very high variance for the more challenging tasks, Longest Overhang and Maximum Unsupported Area. Note that the shaded region represents the quartile distribution -a region that extends to 1.0 means that more than a quarter of the trials were unstable. This highlights the importance of considering the infeasible sub-sequence property when sampling and scoring plans in the action space. Finally, the naive greedy strategy performs the worst, and is only able to achieve decent performance on the Tallest Tower task after seeing roughly 800 training towers, likely due to the fact that it is not considering the feasibility of subtowers when searching the actions space, just greedy single-block placements.
The Maximum Unsupported Area and Longest Overhang tasks are more challenging for the robot because they require deep understanding of the tower stability decision boundary, while the Tallest Tower task only requires a rough understanding of how to build stable towers with high confidence. These results show that in spite of the difficulty of the first two tasks, the active learner is able to improve its understanding of the decision boundary well enough to perform tasks with very low regret and low variance.
In Appendix B we give additional results which compare against baselines that do not leverage active learning.

B. Model Architecture and Generalization
We compare our TGN to the other network architectures discussed in Section IV-C and shown in  evaluation, we report accuracy on a held-out test set of towers built with a novel set of blocks. The test set consists of half feasible and half infeasible towers, and 1000 towers for each tower size. Our models were trained in simulation on towers consisting of up to 5 blocks, but our results give model accuracy for towers ranging from 2 to 7 blocks, shown in Figure 6.
In the towers domain, it is necessary to consider the joint centers of mass for groups of blocks above support blocks. While the LSTM architecture could remember the previous blocks as it iterates through the tower, in practice we find that it is outperformed by the graph network architectures. We believe this is because the connectivity of the graph networks allows them to precisely compare adjacent blocks in addition to aggregating information about multiple blocks. The weakness of the LSTM is more pronounced as the number of blocks in a tower increases.
Our TGN architecture is structured to be biased towards our particular domain, so it is able to improve its predictions much faster than the other architectures. This enables good planning time performance as seen in Section V-A with similar longterm performance to the FCGN architecture.

C. Real Robot Experiments
Finally, we give results for executing the entire active learning pipeline on a real Panda robot (see Section IV for details of the real robot setup). In total, the robot built 400 towers while training over a period of 55 hours.
For the experimentation phase, we used a fixed set of 10 training blocks. The TGN ensemble is initialized with 40 random 2-block towers labeled in simulation with added relativepose noise. We generated candidate experiments using the incremental strategy described in Section III-C, and produced stability labels for the constructed towers by observing the outcome with the cameras.
During the evaluation phase, we test the robot's ability to use its learned APF model to perform all tasks described in Section IV-A with a separate set of 10 held-out evaluation blocks. We compare the learned APF model to two baselines. First, we compare to a hand-engineered model of plan feasibility that calculates whether a candidate tower is feasible in a noiseless world. We also compare to a noisy simulator feasibility model, which predicts a tower is feasible only if  the candidate tower is also stable to 10 normally distributed perturbations, with a standard deviation of 5mm, for each block placement.
For each task and model we select 5 blocks at random from the evaluation set and plan to maximize the given objective. If a constructed tower is unstable, it gets zero reward. The robot constructs 10 towers for each task and feasibility model. In Table I, we report average normalized regret, and the number of total trials that resulted in a stable tower. Figure 7 includes  images of the robot performing the Longest Overhang task using the different models for APF. From these results, it can be seen that the Analytical model can build towers with high reward when the tower is stable, but the towers it chooses to build are rarely stable across all three tasks. However, the Simulation and Learned APF models can still build towers with large overhang while considering the effects of noisy action execution on a real robot. Our Learned model performs competitively with the Simulation model, and in aggregate leads to similar stability across all tasks (27 versus 24 stable towers out of 30). However, note that the Simulation model presents higher variability in tower stability across tasks. This is because the model makes assumptions about the type of noise distribution (Gaussian only in the plane normal to the table) and its parameters (mean and variance), which may be more suitable for certain tasks. The Learned model, on the other hand, may capture other complex real-world phenomena that significantly contribute to plan feasibility in a task-agnostic setting.

VI. RELATED WORK
Hierarchical Planning It is a common approach to longhorizon planning problems; decomposing the solution into high level reasoning over abstract actions and lower level reasoning over concrete actions [25,31,34]. Singh and Kelly [33] performed early work on considering the feasibility of high level plans to improve efficiency when planning in high dimensional spaces. Recent works have proposed methods that predict feasibility of an action as a way to reduce the number of calls to expensive solvers and enable more efficient planning [8,36]. Our work builds upon this literature by presenting a method to learn feasibility actively when detailed physical models are not available.
Learning for Task and Motion Planning In this work we leverage a task and motion planning framework to plan for concrete actions, then execute those actions to determine plan feasibility. Others have explored learning the feasibility of actions, specifically an action's preconditions and effects. [35] uses a Gaussian process to learn these action parameters, with a specialized acquisition function to guide learning. Our work differs in that our acquisition strategy considers observations of entire plans, which means our active learning is nonmyopic. In addition, for the stacking domain we found graph neural networks to be a more applicable class of function approximators in which plan length can vary. A similar approach is taken in [21], a precursor to [35].
Active Learning Active learning [26] is a well-established learning paradigm that aims to minimize the number of samples needed to learn the target concept. Recently, Gal et al. [12] have extended BALD [19] to complex model domains of deep neural networks using MC-dropout [11]. However, in this work, we follow the approach of Beluch et al. [3] and use an ensemble of deep networks for active learning.
Ideas from active learning have been used for efficient learning in model-based reinforcement learning tasks [28,32]. Pathak et al. [28] explore the environment by taking actions that maximize disagreement between an ensemble of forward models. Instead of predicting continuous states, our work learns a model that predicts feasibility of abstract plans where the focus is on performing long-horizon tasks.
Learning Dynamics Many recent works have focused on learning predictive dynamics models in scenes with varying numbers of objects. In such scenarios, it has been shown that explicitly representing objects and their relations in the model can lead to more efficient learning and generalization [1,2,5,38]. As such, graph networks are becoming a more common modeling choice in these domains [22,30]. Specifically, in a stacking domain, Hamrick et al. [18] show that a graph neural network can predict stability properties of towers when given access to a large training set.
Stacking Domain Tasks that involve stacking objects in a Blocks World have a long history in artificial intelligence. Early works developed methods to compute the stability of block placements and construct a target configuration [4,37]. More recently, computer vision researchers have developed scene understanding algorithms that take into account known geometries and stability properties of objects within the scene [15,20,29]. Furrer et al. [10] developed a system that can build stacks out of stones using detailed models of the objects.
Recent work has shown the ability to predict tower stability using deep learning techniques [14,18,24]. However, typically these works have used passively collected datasets which include orders of magnitude more samples than required in this work -making them infeasible to actively collect on a real robot. An active learning approach allows the robot to explore efficiently, and learn a feasibility model under the real-world noise distribution. In addition, vision-based systems would not be effective when the state contains non-visual properties, such as the center of mass in our stacking domain [14,24].

VII. CONCLUSION
We have presented a system which leverages informationtheoretic active learning to acquire an Abstract Plan Feasibility model, and shown that incorporating plan feasibility into the active learning strategy can dramatically improve sample efficiency. We deployed our system on a real Franka Emika Panda robot arm in a block-stacking domain, enabling the robot to learn a useful APF model with only 400 experiments.
In future work, we are eager to apply this approach to other domains. We believe that this self-supervised method of curious exploration is an exciting direction, as it may someday allow the millions of robots sitting powered-off in laboratories around the world to make effective use of their downtime.

A. Real Robot Evaluations
In this section we provide additional results for the experiments described in Section V-C. Figure 8 shows the towers built when performing the Tallest Tower task with each of the analytical, simulation, and learned models. 10 towers were built for each task using 5 blocks. Towers that have fallen over or do not have 5 blocks were infeasible plans. Figures 9  and 10 show the evaluation towers for the Longest Overhang and Maximum Unsupported Area tasks, respectively.

B. Supervised vs Active Learning Comparisons
In this section we compare the active learning methods described in this paper to supervised learning baselines which do not perform active data collection to train the APF model. Figure 11 compares our methods which use the Θ ss model class to a random-ss strategy. random-ss randomly samples action sequences and labels each subsequence. The results show that our active methods which leverage the infeasible subsequence property, sequential and incremental, outperform random-ss. random-ss only outperforms greedy, showing that a random strategy is able to find more interesting training towers than a myopic approach which simply tries to maximize the BALD objective for a single block placement. Figure 12 compares our method, complete, which uses the Θ comp model class, to a random-comp strategy. random-comp randomly samples action sequences to train on and only labels the full sequences. As expected, training on sequences which maximize the BALD objective is more effective than randomly sampling action sequences.
The data used to train the random methods is not actively collected, but we show how training on increasingly larger datasets compares to the actively collected dataset.

C. Method Generalizability
Here, we motivate a more general class of problems for which our system applies and clarify which components of the system are specific to our chosen domain. Our method most benefits domains where the feasibility of an action depends strongly on the preceding action sequence (e.g., adding a fifth block to a tower that already has four blocks). Domains that include construction tasks, like ours, will commonly benefit from non-myopic information gathering and a feasibility predictor that incorporates previous actions into its predictions. To apply our method to a new domain (e.g., consider a packing problem where many objects need to be placed in a larger container), the overall system/methodology would remain unchanged. However, one would need to adapt the following domain-specific components: 1) Abstract action definitions. Parameterize an abstract action that is appropriate for the domain and connect it to concrete actions (e.g., object locations within the container). 2) Experimental infrastructure. Additional capabilities to autonomously plan and execute abstract actions in the physical world (e.g., motion and grasp planning infrastructure). 3) Feasibility detector. The notion of APF will depend on the desired outcome of an abstract action. For a new action, we require a method to autonomously acquire the feasibility label during execution (e.g., whether the gripper will collide when placing the object or if an object will be damaged). 4) Additional inductive bias (optional). Additional structure to the learner can further increase data efficiency, as shown by our TGN method. However, this is not required as we show in Figure 4 that a general purpose graph network can be used (e.g., a graph network that has connectivity between all objects).