extfieldsofstudy:
Computer Science
Mathematics

text:
Cross-Lingual Word Embedding Refinement by \ell_{1} Norm Optimisation

Cross-Lingual Word Embeddings (CLWEs) encode words from two or more languages in a shared high-dimensional space in which vectors representing words with similar meaning (regardless of language) are closely located. Existing methods for building high-quality CLWEs learn mappings that minimise the ℓ2 norm loss function. However, this optimisation objective has been demonstrated to be sensitive to outliers. Based on the more robust Manhattan norm (aka. ℓ1 norm) goodness-of-fit criterion, this paper proposes a simple post-processing step to improve CLWEs. An advantage of this approach is that it is fully agnostic to the training process of the original CLWEs and can therefore be applied widely. Extensive experiments are performed involving ten diverse languages and embeddings trained on different corpora. Evaluation results based on bilingual lexicon induction and cross-lingual transfer for natural language inference tasks show that the ℓ1 refinement substantially outperforms four state-of-the-art baselines in both supervised and unsupervised settings. It is therefore recommended that this strategy be adopted as a standard for CLWE methods.

Introduction
Cross-Lingual Word Embedding (CLWE) techniques have recently received significant attention as an effective means to support Natural Language Processing applications for low-resource languages, e.g., machine translation (Artetxe et al., 2018b) and transfer learning (Peng et al., 2021).
The most successful CLWE models are the socalled projection-based methods, which learn mappings between monolingual word vectors with very little, or even zero, cross-lingual supervision Artetxe et al., 2018a;Glavaš et al., 2019). Mainstream projection-based CLWE models typically identify orthogonal mappings by * Chenghua Lin is the corresponding author. minimising the topological dissimilarity between source and target embeddings based on 2 loss (aka. Frobenius loss or squared error) (Glavaš et al., 2019;. This learning strategy has two advantages. First, adding the orthogonality constraint to the mapping function has been demonstrated to significantly enhance the quality of CLWEs (Xing et al., 2015). Second, the existence of a closed-form solution to the 2 optima (Schönemann, 1966) greatly simplifies the computation required (Artetxe et al., 2016;Smith et al., 2017).
Despite its popularity, work in various application domains has noted that 2 loss is not robust to noise and outliers. It is widely known in computer vision that 2 -loss-based solutions can severely exaggerate noise, leading to inaccurate estimates (Aanaes et al., 2002;De La Torre and Black, 2003). In data mining, Principal Component Analysis (PCA) using 2 loss has been shown to be sensitive to the presence of outliers in the input data, degrading the quality of the feature space produced (Kwak, 2008). Previous studies have demonstrated that the processes used to construct monolingual and cross-lingual embeddings may introduce noise (e.g. via reconstruction error (Allen and Hospedales, 2019) and structural variance ), making the presence of outliers more likely. Empirical analysis of CLWEs also demonstrates that more distant word pairs (which are more likely to be outliers) have more influence on the behaviour of 2 loss than closer pairs. This raises the question of the appropriateness of 2 loss functions for CLWEs.
Compared to the conventional 2 loss, 1 loss (aka. Manhattan distance) has been mathematically demonstrated to be less affected by outliers (Rousseeuw and Leroy, 1987) and empirically proven useful in computer vision and data mining (Aanaes et al., 2002;De La Torre and Black, 2003;Kwak, 2008). Motivated by this insight, our paper proposes a simple yet effective post-processing technique to improve the quality of CLWEs: adjust the alignment of any cross-lingual vector space to minimise the 1 loss without violating the orthogonality constraint. Specifically, given existing CLWEs, we bidirectionally retrieve bilingual vectors and optimise their Manhattan distance using a numerical solver. The approach can be applied to any CLWEs, making the post-hoc refinement technique generic and applicable to a wide range of scenarios. We believe this to be the first application of 1 loss to the CLWE problem.
To demonstrate the effectiveness of our method, we select four state-of-the-art baselines and conduct comprehensive evaluations in both supervised and unsupervised settings. Our experiments involve ten languages from diverse branches/families and embeddings trained on corpora of different domains. In addition to the standard Bilingual Lexicon Induction (BLI) benchmark, we also investigate a downstream task, namely cross-lingual transfer for Natural Language Inference (NLI). In all setups tested, our algorithm significantly improves the performance of strong baselines. Finally, we provide an intuitive visualisation illustrating why 1 loss is more robust than it 2 counterpart when refining CLWEs (see Fig. 1). Our code is available at https://github.com/Pzoom522/ L1-Refinement.
Our contribution is three-fold: (1) we propose a robust refinement technique based on the 1 norm training objective, which can effectively enhance CLWEs; (2) our approach is generic and can be directly coupled with both supervised and unsupervised CLWE models; (3) our 1 refinement algorithm achieves state-of-the-art performance for both BLI and cross-lingual transfer for NLI tasks.
2 Related Work CLWE methods. One approach to generating CLWEs is to train shared semantic representations using multilingual texts aligned at sentence or document level (Vulić and Korhonen, 2016;Upadhyay et al., 2016). Although this research direction has been well studied, the parallel setup requirement for model training is expensive, and hence impractical for low-resource languages.
Recent years have seen an increase in interest in projection-based methods, which train CLWEs by finding mappings between pretrained word vectors of different languages (Mikolov et al., 2013;Peng et al., 2020). Since the input embeddings can be generated independently using monolingual corpora only, projection-based methods reduce the supervision required for training and offer a viable solution for low-resource scenarios. Xing et al. (2015) showed that the precision of the learned CLWEs can be improved by constraining the mapping function to be orthogonal, which is formalised as the so-called 2 Orthogonal Procrustes Analysis (OPA): where M is the CLWE mapping, O denotes the orthogonal manifold (aka. the Stiefel manifold (Chu and Trendafilov, 2001)), and A and B are matrices composed using vectors from source and target embedding spaces. While Xing et al. (2015) exploited an approximate and relatively slow gradient-based solver, more recent approaches such as Artetxe et al. (2016) andSmith et al. (2017) introduced an exact closed-form solution for Eq. (1). Originally proposed by Schönemann (1966), it utilises Singular Value Decomposition (SVD): where M denotes the 2 -optimal mapping matrix. The efficiency and effectiveness of Eq. (2) have led to its application within many other approaches, e.g., Ruder et al. (2018),  and Glavaš et al. (2019). In particular, PROC-B (Glavaš et al., 2019), a supervised CLWE framework that simply applies multiple iterations of 2 OPA, has been demonstrated to produce very competitive performance on various benchmark tasks including BLI as well as cross-lingual transfer for NLI and information retrieval.
While the aforementioned approaches still require some weak supervision (i.e., seed dictionaries), there have also been some successful attempts to train CLWEs in a completely unsupervised fashion. For instance,  proposed a system called MUSE, which bootstraps CLWEs without any bilingual signal through adversarial learning. VECMAP (Artetxe et al., 2018a) applied a self-learning strategy to iteratively compute the optimal mapping and then retrieve bilingual dictionary. Comparing MUSE and VECMAP, the latter tends to be more robust as its similarity-matrixbased heuristic initialisation is more stable in most cases (Glavaš et al., 2019;. Very recently, some studies bootstrapped unsupervised CLWEs by jointly training word embeddings on concatenated corpora of different languages and achieved good performance (Wang et al., 2020).
The 2 refinement algorithm. CLWE models often apply 2 refinement, a post-processing step shown to improve the quality of the initial alignment (see  for survey). Given existing CLWEs {X LA , X LB } for languages L A and L B , bidirectionally one can use approaches such as the classic nearest-neighbour algorithm, the inverted softmax (Smith et al., 2017) and the crossdomain similarity local scaling (CSLS)  to retrieve two bilingual dictionaries D LA →LB and D LB →LA . Note that word pairs in D LA →LB ∩ D LB →LA are highly reliable, as they form "mutual translations". Next, one can compose bilingual embedding matrices A and B by aligning word vectors (rows) using the above word pairs. Finally, a new orthogonal mapping is learned to fit A and B based on least-square regressions, i.e., perform 2 OPA described in Eq. (1).
Early applications of 2 refinement applied a single iteration, e.g. (Vulić and Korhonen, 2016). Due to the wide adoption of the closed-form 2 OPA solution (cf. Eq. (2)), recent methods perform multiple iterations. The iterative 2 refinement strategy is an important component of approaches that bootstrap from small or null training lexicons (Artetxe et al., 2018a). However, a single step of refinement is often sufficient to create suitable CLWEs Glavaš et al., 2019).

Methodology
A common characteristic of CLWE methods that apply the orthogonality constraint is that they optimise using 2 loss (see § 2). However, outliers have disproportionate influence in 2 since the penalty increases quadratically and this can be particularly problematic with noisy data since the solution can "shift" towards them (Rousseeuw and Leroy, 1987). The noise and outliers present in real-world word embeddings may affect the performance of 2 -lossbased CLWEs.
The 1 norm cost function is more robust than 2 loss as it is less affected by outliers (Rousseeuw and Leroy, 1987). Therefore, we propose a refinement algorithm for improving the quality of CLWEs based on 1 loss. This novel method, which we refer to as 1 refinement, is generic and can be applied post-hoc to improve the output of existing CLWE models. To our knowledge, the use of alternatives to 2 -loss-based optimisation has never been explored by the CLWE community.
To begin with, analogous to 2 OPA (cf. Eq. (1)), 1 OPA can be formally defined and rewritten as where tr(·) returns the matrix trace, sgn(·) is the signum function, and ∈ O denotes that M is subject to the orthogonal constraint. Compared to 2 OPA which has a closed-form solution, solving Eq. (3) is much more challenging due to the discontinuity of sgn(·). This issue can be addressed by replacing sgn(·) with tanh(α(·)), a smoothing function parameterised by α, such that Larger values for α lead to closer approximations to sgn(·) but reduce the smoothing effect. This approach has been used in many applications, such as the activation function of long short-term memory networks (Hochreiter and Schmidhuber, 1997). However, in practice, we find that Eq. (4) remains unsolvable in our case with standard gradient-based frameworks for two reasons. First, α has to be sufficiently large in order to achieve a good approximation of sgn(·). Otherwise, relatively small residuals will be down-weighted during fitting and the objective will become biased towards outliers, just similar to 2 loss. However, satisfying this requirement (i.e., large α) will lead to the activation function tanh(α(·)) becoming easily saturated, resulting in an optimisation process that becomes trapped during the early stages. In other words, the optimisation can only reach an unsatisfactory local optimum. Second, the orthogonality constraint (i.e., M ∈ O) also makes the optimisation more problematic for these methods.
We address these challenges by adopting the approaches proposed by Trendafilov (2003). This method explicitly encourages the solver to only explore the desired manifold O thereby reducing the 1 solver's search space and difficulty of the optimisation problem. We begin by calculating the gradient ∇ w.r.t. the objective in Eq. (4) through matrix differentiation: where Z=α(AM−B) and is the Hadamard product. Next, to find the steepest descent direction while ensuring that any M produced is orthogonal, Here I is an identity matrix with the shape of M. With Eq. (6) defining the optimisation flow, our 1 loss minimisation problem reduces to an integration problem, as where M 0 is a proper initial solution of Eq. (3) (e.g., 2 -optimal mapping obtained via Eq. (2)). Empirically, unlike the aforementioned standard gradient-based methods, by following the established policy of Eq. (6), the optimisation process of Eq. (7) will not violate the orthogonality restriction or get trapped during early stages. However, this 1 OPA solver requires extremely small step size to generate reliable solutions (Trendafilov, 2003), making it computationally expensive 2 . Therefore, it is impractical to perform 1 refinement in an iterative fashion like 2 refinement without significant computational resources.
Previous work has demonstrated that applying the 1 -loss-based algorithms from a good initial state can speed up the optimisation. For instance, Kwak (2008) found that feature spaces created by 2 PCA were severely affected by noise. Replacing the cost function with 1 loss significantly reduced this problem, but required expensive linear programming. To reduce the convergence time, Brooks and Jot (2013) exploited the first principal component from the 2 solution as an initial guess. Similarly, when reconstructing corrupted pixel matrices, 2 -loss-based results are far from satisfactory; using 1 norm estimators can improve the quality, but are too slow to handle large-scale datasets (Aanaes et al., 2002). However, taking the 2 optima as the starting point allowed less biased reconstructions to be learned in an acceptable time (De La Torre and Black, 2003).
Inspired by these works, we make use of 1 refinement to carry out post-hoc enhancement of existing CLWEs. Our full pipeline is described in : perform integration to solve Eq. (7) for M , with initial value M0 ← I, until stopping criteria are met Algorithm 1 (see § 4.3 for implemented configurations). In common with 2 refinement (cf. § 2), steps 1-4 bootstrap a synthetic dictionary D and compose bilingual word vector matrices A and B which have reliable row-wise correspondence. Taking them as the starting state, in step 5 an identity matrix naturally serves as our initial solution M 0 . During the execution of Eq. (7), we record 1 loss per iteration and see if either of the following two stopping criteria have been satisfied: (1) the updated 1 loss exceeds that of the previous iteration; (2) on-the-fly M has non-negligibly departed from the orthogonal manifold, which can be indicated by the maximum value of the disparity matrix as where is a sufficiently small threshold. The resulting M can be used to adjust the word vectors of L A and output refined CLWEs. A significant advantage of our algorithm is its generality: it is fully independent of the method used for creating the original CLWEs and can therefore be used to enhance a wide range of models, both in supervised and unsupervised settings.

Datasets
In order to demonstrate the generality of our proposed method, we conduct experiments using two groups of monolingual word embeddings trained on very different corpora: Wiki-Embs : embeddings developed using Wikipedia dumps for a range of ten diverse languages: two Germanic (English|EN, German|DE), two Slavic (Croatian|HR, Russian|RU), three Romance (French|FR, Italian|IT, Spanish|ES) and three non-Indo-European (Finnish|FI from the Uralic family, Turkish|TR from the Turkic family and Chinese|ZH from the Sino-Tibetan family). News-Embs are considered to be more challenging for building good quality CLWEs due to the heterogeneous nature of the data, while a considerable portion of the multilingual training corpora for Wiki-Embs are roughly parallel. Following previous studies Artetxe et al., 2018a;Zhou et al., 2019;Glavaš et al., 2019), only the first 200K vocabulary entries are preserved. In the original implementations, MUSE, PROC-B and JA were only trained on Wiki-Embs while VECMAP additionally used News-Embs. Although all baselines reported performance for BLI, they used various versions of evaluation sets, hence previous results are not directly comparable with the ones reposted here. More concretely, the testsets for MUSE/JA and VECMAP are two different batches of EN-centric dictionaries, while the testset for PROC-B also supports non-EN translations.

Implementation Details of Algorithm 1
The CSLS scheme with a neighbourhood size of 10 (CSLS-10) is adopted to build synthetic dictionaries via the input CLWEs. A variable-coefficient ordinary differential equation (VODE) solver 3 was implemented for the system described in Eq. (7). Suggested by Trendafilov (2003), we set the maximum order at 15, the smoothness coefficient α in Eq. (5) at 1e8, the threshold in Eq. (8) at 1e-5, and performed the integration with a fixed time interval of 1e-6. An early-stopping design was adopted to ensure computation completed in a reasonable time: in addition to the two default stopping criteria in § 3, integration is terminated if dt reaches 5e-3 (dt is the differentiation term in Eq. (7)).
In terms of the tolerance of the VODE solver, we set the absolute tolerance at 1e-7 and the relative tolerance at 1e-5, following the established approach of Kulikov (2013). These tolerance settings show good generality empirically and were used for all tested language pairs, datasets, and models in our experiments.

Results
We evaluate the effectiveness of the proposed 1 refinement technique on two benchmarks: Bilingual Lexicon Induction (BLI), the de facto standard for measuring the quality of CLWEs, and a downstream natural language inference task based on cross-lingual transfer. In addition to comparison against state-of-the-art CLWE models, we also report the performance of the single-iteration 2 refinement method which follows steps 1-4 of Algorithm 1 then minimises 2 loss in the final step.
To reduce randomness, we executed each model in each setup three times and the average accuracy (ACC, aka. precision at rank 1) is reported. Following Glavaš et al. (2019), by comparing scores achieved before and after 1 refinement, statistical significance is indicated via the p-value of two-tailed t-tests with Bonferroni correction (Dror et al., 2018) (note that p-values are not recorded for Tab. 2b given the small number of runs).
To put these improvements in context, Heyman et al. (2019) reported an improvement of 0.4% for VECMAP on same dataset and language pairs.
Our method tends to work better on the more distant language pairs. For instance, for the distant pairs EN-{RU, ZH}, the increments achieved by MUSE-1 are 1.6% and 1.3%, respectively; whereas for the close pairs EN-{DE, ES, FR} the average gain is a maximum of 0.9%. A similar trend can be observed for JA-MUSE-1 and VECMAP-1 . (As the VECMAP algorithm always collapses for EN-ZH, no result is reported for this language pair).
Another set of experiments were conducted to evaluate the robustness of our algorithm following the main setup of Artetxe et al. (2018a), who tested four language pairs based on the more homogeneous News-Embs. Tab. 1b shows that JA-MUSE-1 and VECMAP-1 consistently improves the original VECMAP with an average gain of 1.2% and 1.0% (p<0.01). Obtaining such substantial improvements over the state-of-the-art is nontrivial. For example, even a very recent weakly supervised method by  is inferior to VECMAP by 1.0% average ACC. On the other hand, MUSE fails to produce any analysable result as it always collapses on the more challenging News-Embs. Improvement with 1 refinement is also larger when language pairs are more distant, e.g., for VECMAP-1 the ACC gain on EN-FI is 1.8%, more than double of the gain (0.7%) on the close pairs EN-{DE, IT} (cf. Tab. 1a and above).
We also conduct an ablation study by reporting the performance of 2 refinement scheme ({MUSE, JAMUSE, VECMAP}-2 ). This observation is in accordance with that of , who reported that after performing 2 refinement in the first loop, applying further iterations only produces marginal precision gain, if any.
Overall, the 1 refinement consistently and significantly improve the CLWEs produced by base algorithms, regardless of the embeddings and setups used, thereby demonstrating the effectiveness and robustness of the proposed algorithm.

Refining supervised baselines.
To test the generalisability of our method, we also applied it on state-of-the-art supervised CLWE models: PROC-B (Glavaš et al., 2019) and JA-RCSLS (Wang et al., 2020). Following the setup of Glavaš et al. (2019), we learn mappings using Wiki-Embs and 1K training splits of their dataset.
Their evaluation code retrieves bilingual word pairs using the classic nearest-neighbour algorithm and outputs the Mean Reciprocal Rank (MRR). As shown in Tab. 2a, both JA-RCSLS-1 and PROC-B-1 outperform the baseline algorithms for all  Table 3: MRR (%) of BLI for non-EN language pairs. Rows marked with ␃ are from the supplementary of Glavaš et al. (2019). MUSE yielded one unsuccessful run for TR-IT, and we only record the average of the two successful scores with *.  language pairs (with the exception of EN-IT where the score of PROC-B is unchanged) with an average improvement of 0.9% and 0.5%, respectively (p<0.01). JA-RCSLS-1 and PROC-B-1 were also tested using News-Embs with results shown in Tab. 2b 5 .
1 refinement achieves an impressive improvement for both close (EN-{DE, IT}) and distant (EN-FI) language pairs: average gain of 1.9% and 3.9% respectively and over 5% for EN-DE (PROC-B-1 ) in particular. The 2 refinement does not benefit the supervised baseline, similar to the lack of improvement observed in the unsupervised setups.
Comparison of unsupervised and supervised settings. This part provides a comparison of the effectiveness of 1 refinement in unsupervised and supervised scenarios. Unlike previous experiments where only alignments involving English were investigated, these tests focus on non-EN setups. Glavaš  Results shown in Tab. 3 demonstrate that the main baselines (MUSE, JA-MUSE, VECMAP, JA-RCSLS, and PROC-B) outperform these other models by a large margin. For all these main baselines, post applying 1 refinement improves the mapping quality for all language pairs (p < 0.01), with average improvements of 1.7%, 1.4%, 1.8%, 1.1%, and 0.8%, respectively. Consistent with findings in the previous experiments, 2 refinement does not enhance performance. Improvement with 1 refinement is higher when language pairs are more distant, e.g., for all inter-language-family pairs such as FI-HR and TR-IT, even the minimum improvement of MUSE-1 over MUSE is 2.3%.
Comparing unsupervised and supervised approaches, it can be observed that MUSE, JA-MUSE and VECMAP achieve higher overall gain with 1 refinement than JA-RCSLS and PROC-B, where JA-MUSE-1 and VECMAP-1 give the best overall performance. One possible explanation to this phenomenon is that there is only a single source of possible noise in unsupervised models (i.e. the embedding topology) but for supervised methods noise can also be introduced via the seed lexicons. Consequently unsupervised approaches drive more benefit from 1 refinement, which reduces the influence of topological outliers in CLWEs.
Topological behaviours of 1 and 2 refinements. To validate our assumption that 2 refinement is more sensitive to outliers while its 1 counterpart is more robust, we analyse how each refinement strategy changes the distance between bilingual word vector pairs in the synthetic dictionary D (cf. Algorithm 1) constructed from trained CLWE models. Specifically, for each word vector pair we subtract its post-refinement distance from the original distance (i.e., without applying additional 1 or 2 refinement step). Fig. 1 shows visualisation examples for three algorithms and language pairs, where each bar represents one word pair. It can be observed that 1 refinement effectively reduces the distance for most word pairs, regardless of their original distance (i.e., indicated by bars with negative values in the figures). The conventional 2 refinement strategy, in contrast, exhibits very different behaviour and tends to be overly influenced by word pairs with large distance (i.e. by outliers). The reason for this is that the 2 -norm penalty increases quadratically, causing the solution to put much more weight on optimising distant word pairs (i.e., word pairs on the right end of the X-axis show sharp distance decrements). This observation is in line with Rousseeuw and Leroy (1987) and explains why 1 loss performs substantially stronger than 2 loss in the refinement.
Case study. After aligning EN-RU embeddings with unsupervised MUSE, we measured the distance between vectors corresponding to the groundtruth dictionary of  (cf. Fig. 1a). We then detected large outliers by finding vector pairs whose distance falls above Q3 + 1.5 · (Q3 − Q1), where Q1 and Q3 respectively denote the lower and upper quartile based on the popular Inter-Quartile Range (Hoaglin et al., 1986). We found that many of the outliers correspond to polysemous entries, such as {state (2× noun meanings and 1× verb meaning), состояние (only means status)}, {type (2× nominal meanings and 1× verb meaning), тип (only means kind)}, and {film (5× noun meanings), фильм (only means movie)}. We then  re-perform 2 -based mapping after removing these vector pairs, observing that the accuracy jumps to 45.9% (cf. the original 2 -norm alignment it is 43.8% and after 1 refinement it is 45.6%, cf. Tab. 1). This indicates that although all baselines already make use of preprocessing steps including vector normalization, outlier issues still exist and harms the 2 norm CLWEs. However, they can be alleviated by the proposed 1 refinement technique.

Natural Language Inference
Finally, we experimented with a downstream NLI task in which the aim is to determine whether a "hypothesis" is true (entailment), false (contradiction) or undetermined (neutral), given a "premise". Higher ACC indicates better encoding of semantics in the tested embeddings. The CLWEs used are those trained with Wiki-Embs for BLI. For MUSE, JA-MUSE and VECMAP, we also obtain CLWEs for EN-TR pair with the same configuration. Following Glavaš et al. (2019), we first train the Enhanced Sequential Inference Model (Chen et al., 2017) based on the large-scale English MultiNLI corpus  using vectors of language L A (EN) from an aligned bilingual embedding space (e.g., EN-DE). Next, we replace the L A vectors with the vectors of language L B (e.g., DE), and directly test the trained model on the language L B portion of the XNLI corpus .
Results in Tab. 4 show that the CLWEs refined by our algorithm yield the highest ACC for all language pairs in both supervised and unsupervised settings. The 2 refinement, on the contrary, is not beneficial overall. Improvements in cross-lingual transfer for NLI exhibit similar trends to those in the BLI experiments, i.e. greater performance gain for unsupervised methods and more distant language pairs, consistent with previous observations (Glavaš et al., 2019). For instance, MUSE-1 JA-MUSE-1 and VECMAP-1 outperform their baselines by at least 2% in ACC on average (p < 0.01), whereas the improvements of JA-RSCLS-1 and PROC-B-1 over their corresponding base methods are 2% and 2.1% respectively (p < 0.01). For both unsupervised and supervised methods, 1 refinement demonstrates stronger effect for more distant language pairs, e.g., MUSE-1 surpasses MUSE by 1.2% for EN-FR, whereas a more impressive 2.7% gain is achieved for EN-TR.
In summary, in addition to improving BLI performance, our 1 refinement method also produces a significant improvement for a downsteam task (NLI), demonstrating its effectiveness in improving the CLWE quality.

Conclusion and Future Work
This paper proposes a generic post-processing technique to enhance CLWE performance based on optimising 1 loss. This algorithm is motivated by successful applications in other research fields (e.g. computer vision and data mining) which exploit the 1 norm cost function since it has been shown to be more robust to noisy data than the commonly-adopted 2 loss. The approach was evaluated using ten diverse languages and word embeddings from different domains on the popular BLI benchmark, as well as a downstream task of cross-lingual transfer for NLI. Results demonstrated that our algorithm can significantly improve the quality of CLWEs in both supervised and unsupervised setups. It is therefore recommended that this straightforward technique be applied to improve performance of CLWEs.
The convergence speed of the optimiser prevented us from performing 1 loss optimisation over multiple iterations. Future work will focus on improving the efficiency of our 1 OPA solver, as well as exploring the application of other robust loss functions within CLWE training strategies.

Ethics Statement
This work provides an effective post-hoc method to improve CLWEs, advancing the state-of-the-art in both supervised and unsupervised settings. Our comprehensive empirical studies demonstrate that the proposed algorithm can facilitate researches in machine translation, cross-lingual transfer learning, etc, which have deep societal impact of bridging cultural gaps across the world.
Besides, this paper introduces and solves an optimisation problem based on an under-explored robust cost function, namely 1 loss. We believe it could be of interest for the wider community as outlier is a long-standing issue in many artificial intelligence applications.
One caveat with our method, as is the case for all word-embedding-based systems, is that various biases may exist in vector spaces. We suggest this problem should always be looked at critically. In addition, our implemented solver can be computationally expensive, leading to increased electricity consumption and the associated negative environmental repercussions.


s2fieldsofstudy:
Computer Science
Environmental Science

text:
Fast and Robust Registration of Partially Overlapping Point Clouds

Real-time registration of partially overlapping point clouds has emerging applications in cooperative perception for autonomous vehicles and multi-agent SLAM. The relative translation between point clouds in these applications is higher than in traditional SLAM and odometry applications, which challenges the identification of correspondences and a successful registration. In this paper, we propose a novel registration method for partially overlapping point clouds where correspondences are learned using an efficient point-wise feature encoder, and refined using a graph-based attention network. This attention network exploits geometrical relationships between key points to improve the matching in point clouds with low overlap. At inference time, the relative pose transformation is obtained by robustly fitting the correspondences through sample consensus. The evaluation is performed on the KITTI dataset and a novel synthetic dataset including low-overlapping point clouds with displacements of up to 30m. The proposed method achieves on-par performance with state-of-the-art methods on the KITTI dataset, and outperforms existing methods for low overlapping point clouds. Additionally, the proposed method achieves significantly faster inference times, as low as 410ms, between 5 and 35 times faster than competing methods. Our code and dataset are available at https://github.com/eduardohenriquearnold/fastreg.

I. INTRODUCTION
P OINT cloud registration is the problem of estimating the rigid relative pose transformation that aligns a pair of point clouds into the same coordinate system. This is a key problem in many downstream applications including 3D scene reconstruction [1], localisation [2] and SLAM [3]. Recent applications such as Augmented Reality (AR) [4], cooperative (multi-agent) perception for autonomous vehicles [5] and multi-agent SLAM [6] introduce new challenges to this problem. Specifically, these applications require registration methods that are robust to point clouds with low overlap, e.g. when sensors are far apart, and capable of operating in realtime.
Existing registration methods are often designed and evaluated assuming a significant overlap between the input point clouds. This assumption is valid for applications such as  SLAM [3] and lidar odometry [7], where pairs of point clouds are obtained sequentially in adjacent time steps by a single vehicle navigating in a driving environment. On the other hand, applications such as cooperative perception [5] and multi-agent SLAM [6] require registering point clouds obtained simultaneously from a pair of sensors on two different vehicles that are potentially far apart, and thus, may have low field-of-view overlap, e.g. Figure 1. As the relative translation between the sensors increases, the number of identifiable correspondences decreases, which poses challenges in registering the point clouds accurately. The majority of existing point cloud registration methods cannot guarantee real-time execution. Traditional local registration methods such as Iterative Closest Point (ICP) [8] solve the problem iteratively assuming an initial relative pose. However, the iterative nature of such methods renders them unfeasible for real-time applications, particularly considering large scale point clouds. These methods are also prone to nonoptimal solutions when the initial pose estimate is poor, which may be addressed with global-optimisation variants [9], [10] at the cost of higher computational complexity. Another category of methods identify correspondences between point clouds using a distance metric between hand-engineered features [11] or learned point-wise features [12]. These correspondences are often contaminated by a large number outliers and must be filtered using RANSAC [13], [14] or learned models [15], which further increases the registration execution time. Furthermore, state-of-the-art learning-based models [12], [15] require computationally demanding 3D convolutions and generate numerous putative correspondences, introducing a bottleneck on the RANSAC loop and rendering real-time execution unfeasible.
To mitigate the aforementioned limitations, we propose a novel point cloud registration method capable of operating in real-time and robust to low-overlapping point clouds. The proposed method identifies correspondences between the source and target point clouds by learning point-wise features. A novel encoder hierarchically subsamples the point clouds to reduce the number of key points and improve the run-time performance. The resulting features are refined using selfand cross-attention based on a graph neural network. The attention network leverages geometrical relationships between key points and their features across to improve the correspondence accuracy, particularly in regions of low overlap. The relative pose parameters are obtained by fitting the learned correspondences using Random Sample Consensus (RANSAC) to robustly reject outliers. During inference, the RANSAC fitting is done efficiently considering a small number of correspondences, which allows end-to-end inference times below 410ms. The model is trained and evaluated separately on the KITTI odometry dataset and a novel Cooperative Driving Dataset (CODD). The relative translation between sensors in CODD ranges up to 30m, introducing challenging pairs of point clouds with low overlap, which we hope will create a new research benchmark. Our contributions are summarised as: • A computationally efficient point-wise feature encoder that allows identifying correspondences between point clouds; • A graph neural network that provides self-and crossattention between point clouds and improves the quality of correspondences; • A novel registration method for point clouds that is robust to partially-overlapping point clouds and capable of operating in real-time; • A new synthetic lidar dataset containing low overlapping point clouds in a wide range of driving scenarios;

II. RELATED WORKS
This section reviews existing point cloud registration methods in the literature and highlights how the method proposed in this paper differs from these existing works. We divide existing methods in the literature into two categories: traditional registration methods and learning-based methods.

A. Traditional Registration Methods
Iterative Closest Point (ICP) [8] is a local registration method that assumes an initial relative pose and iteratively computes the transformation parameters that minimise the distance between each point in the source point cloud and its closest neighbour in the target point cloud. This method is highly sensitive to the initial pose estimate, and converges to non-optimal local-minima results when the initial pose estimate is poor. To mitigate this, [9], [10] estimate global optimum solutions for ICP considering branch-and-bound search over the transformation space. However, such global methods have significantly higher execution times, which prevents their usage in real-time applications.
Other traditional approaches use handcrafted features to find correspondences between the point clouds. Fast Point Feature Histogram (FPFH) [11] encodes the local geometry of 3D points using multi-dimensional feature vectors. But the correspondences obtained by comparing FPFH features are often contaminated by a large number of outliers, which prevents accurate registration. For this reason, RANSAC methods are used to filter out the outlier correspondences. More recently, TEASER [16] reformulates the registration problem using a truncated least-squares cost, which results in improved registration accuracy compared to RANSAC when considering a high number of outliers.

B. Learning-based Methods
One group of learning-based registration methods focus on learning accurate correspondences between the point clouds. Generally, these methods learn a mapping from the original Euclidean space to a latent feature space and optimise the mapping such that corresponding points have a small distance in the latent space. Deng et al. [17] uses a PointNet [18] model to learn point-wise features and trains the model using an N -tuple loss. VCR-Net [19] learns point-wise feature vectors using multi-layer-perceptrons (MLPs) to extract local features, which are refined using global attention and used to identify correspondences between point clouds. In contrast, [12] uses sparse fully convolutional networks to obtain voxelwise features and trains the model using variations of triplet loss with hard negative mining. The resulting correspondences are often contaminated with outliers and need to be pruned using RANSAC [13] or further learning-based filtering [15] methods before estimating the pose transformation parameters.
Another group of methods solve the problem end-to-end by learning the relative pose transformation directly. For example, PCRNet [20] uses a PointNet [18] model to encode a global feature vector for both source and target point clouds and directly regress the transformation parameters. DeepVCP [21] uses PointNet++ [22] to create point-wise features, then selects top K salient points using a learned weighting network to generate a deep feature embedding. The embeddings are fed to a 3D CNN to obtain soft-correspondences which are finally used to compute the transformation parameters in closed form.
The proposed method differs from previous works in the following ways. First, our method considers a novel and computationally efficient point-wise feature encoder based on Set Abstraction (SA) and Feature Propagation (FP) layers [22]. While previous works [21] have used PointNet++ feature encoders, we distinguish our encoder by adopting an architecture that hierarchically subsamples points at each layer, resulting in improved computational performance. Secondly, we improve the quality of the correspondences using a novel graph-based attention network that allows to efficiently combine self-and cross-information across point clouds. Differently from the feature-based attention in [19], the proposed graph attention leverages both spatial and feature dimensions of local neighbourhoods to refine point-wise features. Finally, we extend our analysis beyond existing datasets and evaluate our model performance in challenging low overlapping point clouds using a novel dataset where the translation between sensor poses vary uniformly up to 30 meters.

III. PROBLEM FORMULATION
Given two input point clouds P X ⊂ R 3 and P Y ⊂ R 3 , the registration problem is to estimate the rigid relative pose transformation that aligns P X into the coordinate system of P Y . This transformation is parametrised by a rotation matrix R ∈ SO(3) and a translation vector t ∈ R 3 . The problem can be solved by identifying pairs of correspondences between P X and P Y . Given a set of correspondences, . . , N are correspondence pairs, the transformation parameters are obtained by the minimisation of the least-squares error: This error is a form of the Orthogonal Procrustes problem [23] and admits the closed-form solution described below. First, the centroids are computed as and the covariance matrix, denoted by H, is obtained using Finally, the rotation matrix and translation vector R, t that minimise Eq. 1 are computed in closed-form as considering the Singular Value Decomposition (SVD) H = U SV T . The next section proposes a novel method to efficiently obtain correspondences between pairs of point clouds.

IV. PROPOSED METHOD
This section presents a novel method for robust point cloud registration using learned correspondences targetting efficient, real-time inference. Figure  (D) The relative transformation parameters R, t are robustly estimated using a RANSAC formulation of the problem defined in Section III. The aforementioned components and the training process are described in the following subsections.

A. Point-wise Feature Encoder
The encoder is a core component of the pipeline, as it computes point-wise features that will be used to identify correspondences. We propose a novel and computationally efficient encoder network based on Set Abstraction (SA) and Feature Propagation (FP) layers [22]. The proposed encoder architecture, including the hyper-parameters of each layer, is depicted in Figure 3. The encoder outputs subset of sampled coordinates (key points) from the source and target point clouds, denoted by X and Y , and their respective feature vectors, f X and f Y . The input to the encoder consists of 3D point coordinates and corresponding features, e.g. lidar return intensity. Note that the input features are optional, but in this work they consist of a single scalar per point representing the lidar intensity. The source and target point clouds are fed to the encoder independently.
The first four encoder layers are SA layers. A SA layer consists of four operations: 1) n coordinates are sampled from the previous layer using Farthest Point Sampling (FPS) [24]. is computed using an aggregation function (max-pooling) over the MLP output of the points in the respective neighbourhoods. Each SA layer hierarchically subsamples and aggregates information from the previous layer with progressively larger receptive volumes, which is a fundamental step in reducing the computational cost of our pipeline. At the same time, it is also important not to discard valuable information, i.e. prioritising that points from one layer are within the neighbourhood of sampled points in the next layer. This trade-off is achieved by tuning the layers' hyper-parameters, namely n, r, L, such that the sampled points' neighbourhood include most points from the previous layer. These hyper-parameters were optimised for large outdoor driving environments and would require finetuning for indoor scenarios.
The last encoder layer is an FP layer. It propagates high level information from SA4 to the points in the previous layer (SA3) as illustrated in Figure 3. This is achieved by interpolating the feature vectors in SA3 layer using the features from the three nearest-neighbour coordinates in SA4. The final features are obtained fusing the original SA3 features with the interpolated SA4 features using a shared MLP, represented by a list L of intermediate nodes. More details about the interpolation can be found in [22].

B. Graph-based Attention
The feature vectors obtained with the encoder network represent local point cloud information. However, these features are agnostic to the global context of the point cloud. For example, if a point cloud contains multiple objects, e.g. trees, it would be difficult to distinguish between individual trees. Another problem, most critical for low overlapping point clouds, is that regions of overlap generally have different point densities in each point cloud, challenging the correct identification of correspondences since a point's features change with the density of points in its neighbourhood. To mitigate both problems, we propose a graph-based attention module which transforms points' feature vectors considering the wider point cloud context (self-attention) and the context of both source and target point-clouds (cross-attention). Self-attention increases the distinctiveness of key points by attending to their surrounding context. The cross-attention layer learns to refine point-wise features by attending to the most similar features across point clouds. Both layers, illustrated in Figure  4, increase the likelihood of finding accurate correspondences, even in cases of low overlap.
The self-attention layer introduces attention between points within the same point cloud. A graph connecting the points (nodes) is created for each point cloud using the k-Nearest-Neighbours of the points' spatial coordinates. Let f i be a feature vector from either f X or f Y . The self-attention layer computes a residual term for f i using the Crystal Graph In the selfattention layer, the k-NN distance is the Euclidean distance between points' spatial coordinates. In the cross-Attention graph, the k-NN distance is the dot product between points' feature vectors.
Convolution Operation [25]: where N (i) indicates the k neighbour nodes of i, z i,j = [f i , f j ] is the aggregated features of nodes i, j. σ(·) indicates the sigmoid function, and the Softplus function is defined as Softplus(z) = log(1 + e z ). The attention matrices W f , W s are parameters to be learned and the operation represents element-wise multiplication. We adopt the number of nearest neighbours k = 32, which provides a good trade-off between accuracy and computational efficiency. This process is performed independently with shared parameters for both source and target point clouds.
Following the self-attention layer, the cross-attention layer allows interaction between the source and target point cloud features. This layer creates a bi-partite graph between source and target points. Each source point is connected to the k-Nearest-Neighbours nodes in the target point cloud, where the distance metric is the dot product between the feature vectors of the respective points. This layer uses the same residual update rule from Eq. 5, considering the different underlying graph and independent attention matrices W f , W s . The graph-attention network output is given by the updated feature vectors from source and target point clouds, denoted byf X andf Y , respectively.

C. Identifying Correspondences
The correspondences between the point clouds can be obtained by comparing the point-wise features of the source and target key points, denoted respectively asf X ,f Y ∈ R N ×D . These feature vectors are normalised to unity D-dimensional vectors using the Euclidean norm. A matching probability map indicating the probability of correspondences between X and Y is computed as where T is a temperature hyper-parameter and the Softmax function is applied row-wise. Each element φ ij represents the probability that the i-th key point in X matches the j-th key point in Y . The Softmax function scales the coefficients of each row φ i , ensuring a probability distribution over the points in Y . The temperature parameter, denoted by T , controls the entropy of distribution across points in Y . In the limit, when T → 0 + , the coefficients become the one-hot encoding of the point in Y with the highest similarity (i.e. dot product). Finally, each key point x i ∈ X is matched to the key point in Y with highest correspondence probability, resulting in the set of correspondence pairs {(x i , y arg max j φij ), i = 1, . . . , N }.
The ordered set of correspondences in Y is denoted asŶ = {y arg max j φ1j , . . . , y arg max j φ N j }.

D. Estimating Transformation Parameters
The previous step computes a correspondence for every point in X. In practice, only a fraction of points in X will have correspondences in Y , particularly in the case of partially overlapping point clouds. Sensor noise and varying point densities can also lead to encoding errors and erroneous correspondences. To mitigate the effect of correspondence outliers, a common practice is to use sample consensus algorithms such as RANSAC [13], [11]. A general version of this algorithm applied to this problem consists of three steps: 1) Create a hypothesis: Sample a minimal set of three correspondences from the set of correspondences and compute the transformation parameters using Eq. 4. 2) Score the hypothesis based on consensus: Compute the number of inlier correspondences, where a corre- where κ is the inlier threshold, R, t are the hypothesis parameters computed in the first step.
3) Repeat the previous steps for L times and select the hypothesis with the highest number of inliers. The number of tested hypotheses, denoted by H, offers a tradeoff between computational performance and robustness to outliers. H can also be derived to achieve a desired confidence of the selected hypothesis [13], i.e. sampling an outlier-free set of correspondences. While previous works used RANSAC with a large set of putative correspondences [11], this may be unfeasible for real-time systems. In this work, we achieve negligible RANSAC computational cost by learning a small set of correspondences (N = 512), which allows to reduce the number of hypotheses being tested.

E. Training Process
The training process consists of optimising the encoder and attention networks to find accurate correspondences where they exist. We maximise the matching probabilities of groundtruth correspondences and minimise the matching probability of non-corresponding points. This is achieved by directly minimising the following loss function: where the binary variable δ i indicates whether the source point x i has a correspondence in Y andĵ represents the index of the corresponding point in Y . Additionally, N c = N i=1 δ i is the number of ground-truth correspondences and λ is a hyper-parameter scaling the contributions of incorrect matches into the loss function. A point in X is considered to have a correspondence in Y if, under the ground-truth transformation, it is within a distance smaller or equal to 1.6 meters from a point in Y . This inlier distance is arbitrary and was chosen based on the smallest encoder radius. Data augmentation is employed by applying random rotation transformations to both input point clouds and adjusting the ground-truth rotation matrix accordingly. The optimisation details are described in Section V-C.

V. PERFORMANCE EVALUATION
In this section, we first describe the datasets and the evaluation metrics, followed by the implementation details. We then compare the performance of the proposed method with traditional baselines, including ICP [8], FPFH RANSAC [11] and TEASER [16]; and two state-of-the-art learningbased methods: FCGF [12] and DGR [15]. We also provide an ablation study identifying the impact of the proposed attention network into the registration performance.

A. Dataset
The KITTI Odometry dataset [26] is traditionally used to evaluate point cloud registration methods in outdoor environments. We follow the evaluation protocol of recent methods [27], [12], [15], [28], which adopt sequences 0 to 5 for training, 6 to 8 for validation and 9 to 10 for testing. In each sequence, the samples are created by selecting pairs of points clouds obtained sequentially by a single vehicle such that the translation between the poses is less than 10m. The groundtruth pose is provided by GPS and refined using ICP to reduce misalignment.
The distribution of poses in the KITTI dataset is limited to the trajectory of a single vehicle as it navigates the environment. In practice, registration methods must be resilient to point clouds with arbitrary relative pose, where the overlap between point clouds may vary significantly across samples. To this end, we introduce Cooperative Driving Dataset (CODD) [29], an open-source synthetic dataset containing lidar point clouds collected simultaneously from multiple vehicles. This dataset is created using CARLA [30] and features a diverse range of driving environments, including rural areas, suburbs, and dense urban centres. The dataset consists of 108 sequences, which are split into three independent subsets for training, validation and testing, as detailed in Table I. The samples are created by selecting all pair-wise combinations of point clouds obtained from vehicles driving simultaneously within a vicinity considering a maximum distance of 30m. Figure 5 presents the cumulative density plots of the relative distance (translation vector norm), rotation angle and overlap ratio of the pairs of point clouds in each dataset. The overlap ratio measures the overlap between point clouds as the percentage of points in the source point cloud that, when aligned, are within a distance smaller than γ to any point in the target point cloud. Our CODD dataset has a significantly broader distribution of relative distance, rotation angles and overlap ratio between the point cloud pairs, which provides representative scenarios for cooperative perception and multiagent SLAM.

B. Evaluation Metrics
Following previous studies [12], [15], the registration performance is evaluated in terms of the translation and rotation errors given by where R g , t g denotes the ground-truth rotation matrix and translation vector, respectively. These metrics are reported considering their mean value over all dataset samples, denoted as Mean Translation Error (MTE) and Mean Rotation Error (MRE), respectively. We also consider the recall rate, measured as the ratio of successful registrations to the total number of samples, where the success criteria is TE < 0.6m and RE < 5deg following [15]. The runtime performance is evaluated as the average inference time for the registration of a pair of point clouds disregarding the data loading time.

C. Implementation Details
Our proposed method is implemented using PyTorch [31], PyTorch Geometric [32], the CUDA implementation of SA and FP layers from [22] and the Open3D [33] Procrustes RANSAC implementation. The model is trained independently for each dataset using the Adam [34] optimiser with a learning rate of 0.1, = 10 −4 , β 1 = 0.9, β 2 = 0.999 and a batch size of 6 (pairs of point clouds). The model is trained for twenty epochs and the learning rate is reduced in half after every five epochs. For evaluation, the model with the lowest validation loss is selected. The temperature hyper-parameter, described in Section IV-C, is set to T = 10 −2 , and the loss scaling hyper-parameter is set to λ = 10. During inference, the RANSAC inlier threshold, κ, is set to 0.5m, and the maximum number of RANSAC iterations, denoted by H, is computed to achieve 0.999 confidence in the selected hypothesis within a limit of 10 5 iterations. The point clouds from both datasets are downsampled using voxel sizes of 0.3m following previous methods [12], [15]. The overlap ratio distance threshold, γ, is set to 0.3m, following the down-sampling voxel size. The experiments are carried out on a Xeon ES-1630 CPU and Quadro M4000 GPU with 8 GB of memory. For fair comparison, all baselines are also evaluated on the same hardware. The official implementation and pre-trained models (30cm voxel) are used for the evaluation of [12], [15] in the KITTI dataset; likewise, the official TEASER [16] implementation is adopted; and the Open3D [33] implementation of FPFH, RANSAC and ICP is used for the evaluation of the respective methods.

D. Performance on the KITTI dataset
The evaluation results, presented in Table II, show that our proposed method achieves competitive registration errors compared to other methods at a significantly lower inference time -more than five times faster than the fastest baseline. While the proposed method has a marginal increase in mean translation error compared to baseline methods, it achieves on-par recall rate relative to baseline methods. The mean translation and rotation errors of our method can be further reduced using ICP for refinement (Ours + ICP), at the cost of increased inference time.

E. Performance on the CODD dataset
We aim to evaluate the performance of the proposed method and baselines on challenging low-overlapping pairs of point clouds. For a fair comparison, learning-based methods [12], [15] are also trained on the CODD dataset. Table III presents the results on the CODD test set, aggregated by the overlap ratio between point clouds in four progressively larger intervals -the last interval contains all samples. Traditional methods are not resilient to low overlapping points clouds, as the registration error increases significantly when considering lower overlap ratios, as shown in the first three rows of Table III. In contrast, the learning-based baselines are reasonably robust to low overlapping point clouds and achieve high recall rates on all intervals. However, the latter methods demand substantial running times due to their complex encoders and the filtering of a high number of putative correspondences. In contrast, our proposed method achieves similar or better recall rates to the learning-based baselines with more than 35 times faster inference times. This is achieved by our efficient encoder design which outputs a small number of correspondences, which in turn reduces the RANSAC inference time. Our efficient encoder strategy comes at the cost of a slight increase of the MTE and MRE metrics, as compared to DGR [15].
To mitigate this, we apply ICP refinement to our model's output (Ours + ICP), which allows achieving similar MTE and MRE for highly overlapping point clouds and outperforming all baselines on low-overlapping point clouds. Although the ICP refinement comes with an additional computational cost, we still achieve a nine-fold speed-up compared to learningbased baselines. Qualitative results are presented in Figure 1. Figure 6 shows the Empirical Cumulative Density Function (ECDF) of the translation error, rotation errors, and inference time for different methods. The distributions indicate that the proposed method with ICP refinement has the best translation error across samples, closely matched by DGR [15], however with one order of magnitude smaller inference time. The inference time distributions show that the proposed method is the fastest among baselines, with an inference time of 320ms on average, with negligible standard deviation (17ms). The proposed method can operate in real-time considering data input frequencies up to 3Hz.

F. Ablation Study
We assess the impact of the proposed attention network into the registration performance, measured in terms of translation and rotation errors. This is achieved by removing the graph attention module, retraining the model and evaluating its performance on the CODD test set. The results, indicated in Table  III "Ours -Att", show that the graph-attention network plays a key role in improving the accuracy of the correspondences, resulting in lower translation and rotation errors. The benefits of the graph attention network is most significant for lowoverlapping point clouds, as indicated by the last range group in Table III, where the removal of the attention results in a 40% reduction of the registration recall and a significant increase in the mean translation and rotation errors.

VI. CONCLUSION
We proposed a novel point cloud registration method focusing on fast inference of partially overlapping lidar point clouds. The performance evaluation on the KITTI and CODD datasets indicates that the proposed model can operate with a latency lower than 410ms and 320ms, respectively. The results show that the proposed model outperform baseline methods in terms of rotation and translation errors for pairs of point clouds with low overlap. Furthermore, we show that the proposed graph attention module plays a key role in improving the quality of the correspondences in low overlapping point clouds, which results in higher registration performance.


Deformed Implicit Field: Modeling 3D Shapes with Learned Dense Correspondence

We propose a novel Deformed Implicit Field (DIF) representation for modeling 3D shapes of a category and generating dense correspondences among shapes. With DIF, a 3D shape is represented by a template implicit field shared across the category, together with a 3D deformation field and a correction field dedicated for each shape instance. Shape correspondences can be easily established using their deformation fields. Our neural network, dubbed DIF-Net, jointly learns a shape latent space and these fields for 3D objects belonging to a category without using any correspondence or part label. The learned DIF-Net can also provides reliable correspondence uncertainty measurement reflecting shape structure discrepancy. Experiments show that DIF-Net not only produces high-fidelity 3D shapes but also builds high-quality dense correspondences across different shapes. We also demonstrate several applications such as texture transfer and shape editing, where our method achieves compelling results that cannot be achieved by previous methods.

. Our DIF-Net can produce 3D shapes with dense correspondences for object categories containing complex geometry variation and structure differences. It enables high-quality texture transfer shown in the middle four columns, where the two smaller figures after each transfer result show the color-coded correspondences (top) and their uncertainty (bottom; blue and red indicates low and high uncertainty respectively). With our learned shape space and correspondence, shapes can be freely edited by simply moving one or a sparse set of points, as shown in the last two columns.

Abstract
We propose a novel Deformed Implicit Field (DIF) representation for modeling 3D shapes of a category and generating dense correspondences among shapes. With DIF, a 3D shape is represented by a template implicit field shared across the category, together with a 3D deformation field and a correction field dedicated for each shape instance. Shape correspondences can be easily established using their deformation fields. Our neural network, dubbed DIF-Net, jointly learns a shape latent space and these fields for 3D objects belonging to a category without using any correspondence or part label. The learned DIF-Net can also provides reliable correspondence uncertainty measurement reflecting shape structure discrepancy. Experiments show that DIF-Net not only produces high-fidelity 3D shapes but also builds high-quality dense correspondences across different shapes. We also demonstrate several applications such as texture transfer and shape editing, where our method achieves compelling results that cannot be achieved by previous methods. 1

Introduction
3D objects in a same class share some common shape features and semantic correspondences, which can be used to construct a deformable shape model beneficial for a diverse array of downstream tasks in 3D and 2D domains such as shape understanding [31,2], reconstruction [49,10,60], manipulation [8,26], and image synthesis [50,54,48].
Learning a 3D shape model with dense correspondences is a longstanding task in computer vision and graphics. However, existing works mostly focus on object classes with consistent geometric topologies such as human face and body [8,53,32,35,61]. Shapes in these object categories can be pre-aligned for 3D model construction. Recent deep learning based approaches directly learn a latent space of 3D objects [55,46,1,39]. Although these methods can model complex objects, they do not deal with dense correspondences between 3D shapes.
In this paper, we investigate learning model of 3D shapes and their dense correspondences for more generic objects such as cars and chairs. Compared to human face and body, these object classes exhibit much larger shape variations and structure changes, rendering correspondence con-struction extremely challenging. For these object categories, even human cannot reliably label the dense correspondences between two arbitrary shapes.
To achieve this goal, we act on recent advances in deep implicit fields, which have shown extraordinary power of representing complicated 3D geometry [37,39,14,42], and propose a novel Deformed Implicit Field (DIF) representation for joint shape latent space and dense correspondence learning. With DIF, a 3D shape is represented by a template implicit field, shared across the category, together with a 3D deformation field and a scalar correction field, dedicated for each shape instance. The output implicit field of a shape can be constructed by deforming the template implicit field and applying correction. The deformation field serves as a shape alignment function, with which dense correspondence between two shapes can be established by deforming their surfaces to the aligned 3D space. The correction field is introduced to enhance shape representation capability.
We apply a neural network called DIF-Net to learn these fields together given a collection of shapes. To achieve unsupervised correspondence learning without any label, our key observation is that the normal direction of a shape point is highly correlated to its semantic information and very useful for correspondence reasoning. In light of this, we simply enforce the normals of two corresponding surface points connected by deformation to be close. In addition, we impose a spatial smoothness constrain on the deformation fields and enforce the correction fields to be minimal. Thorough qualitative and quantitative evaluations show that our DIF-Net trained in this way can produce high quality correspondences. Moreover, correspondence uncertainty reflecting structure discrepancy between two shapes can be reliably measured by our method.
The contribution of this paper can be summarized as follows: 1) we propose DIF, a novel implicit field based 3D shape representation modeling shape deformations across an object category; 2) we propose DIF-Net, the first method devoted to 3D shape modeling with dense correspondences learned in an unsupervised fashion for objects with structure variation; 3) we show that our method can achieve high-quality dense correspondences and compelling texture transfer and shape editing results that cannot be achieved by previous methods. We believe our method can be applied in a wide range of 3D shape analysis and manipulation tasks.

Related Work
3D Shape Models with Correspondence. Building 3D shape model for a class of shapes has been actively studied in the past. Perhaps the most famous 3D shape model is the 3D morphable model (3DMM) introduced by Blanz et al. [8] for human faces. To build a 3DMM model, face scans are aligned by shape registration methods to derive correspondences, based on which shape deforma-tion bases can be obtained via PCA. The 3DMM model has brought a profound impact to human face related research [41,59,51,28,33,49,21,20,18,50,48,16]. Apart from face, Loper et al. [35] build a 3D morphable model for skinned human body which can control body shapes and poses. This model has been applied in various tasks such as pose estimation [9,3] and image manipulation [54]. Similarly, Zuffi et al. [61] propose a morphable model for animals. The object categories handled by these methods typically have consistent topologies where shapes can be aligned to build correspondences. They do not address more complex object classes containing structure variations.
Learning Shape Latent Space. A large volume of methods [55,46,1,7,6,25,58,45,19,39] have been proposed in recent years to model 3D shapes and learn a latent shape space using deep neural networks, especially generative adversarial networks (GANs) [24] and variational autoencoders [30,45]. However these methods do not explicitly model the dense correspondence among different shapes. Our method not only learns a shape latent space but also generates dense shape correspondence. Implicit Shape Representation. Recent studies show that learning implicit functions for 3D shapes excels at representing complicated geometry [39,37,14,23,22,4,5,42,17]. For example, Park et al. [39] use a neural network to approximate the signed distance field (SDF) of 3D shapes and show superior results compare to voxel and mesh based representations [46,25]. Sitzmann et al. [42] show that surfaces of complex scene can be represented by a simple 5layer MLP with periodic functions as activation. However, these methods mainly target at high-fidelity surface reconstruction and cannot reveal shape correspondence. Genova et al. [23] introduce an implicit template constructed with multiple RBF kernels. They can obtain a coarse dense correspondence between shapes by deforming and relocating RBF kernels to fit different shapes. However, their learning process is designed only for shape reconstruction thus the obtained correspondences are not reliable. Our representation in this paper is also based on implicit fields, but enables correspondence reasoning. Our new loss functions leads to high-quality correspondences learned without any label.

Overview
Given a collection of 3D objects {O i } from one category, our goal is to learn a latent shape space L as well as a neural shape model f that can generate these objects and provide dense shape correspondence. Each shape can be represented by a latent code α ∈ R k in L, and the shape model f maps the latent code to corresponding 3D shape, with a neural network. We adopt the auto-decoder framework presented in [39] to jointly learn the shape codes {α j } for the given objects and the weights of model f . This autodecoder framework can give rise to a decent latent space as shown in [39]. We can also apply proper latent space regularization as in VAE training [30] to learn a desired latent space, which will be discussed later. After training, new shapes can be generated by latent space sampling, and a shape can be embedded into the latent shape via inverse optimization.
Implicit field. To generate high-fidelity shapes, we use signed distance fields (SDF) which can faithfully represent surface geometry details using a neural network as the field function [39]. SDF is a continuous representation which assigns any point p ∈ R 3 a scalar value s ∈ R: where the magnitude of s represents the distance from p to its closest shape surface and the sign indicates whether p is inside (negative) the shape or outside (positive). With an SDF, shape surface can be implicitly represented by the isosurface of SDF (·) = 0. A 3D mesh can be extracted from this implicit surface using off-the-shelf algorithms such as Marching Cubes [36]. Using SDF to represent shapes, our neural shape model can be rewritten as Network components. The task in Eq.
(3) intertwines shape information decoding from latent codes and SDF prediction for 3D points. Inspired by [43], we employ two networks to decompose this task: a DIF-Net for SDF prediction and a Hyper-Net for shape information decoding, as shown in Fig. 2. Hyper-Net Ψ predicts the weights ω for the neurons in DIF-Net Φ, and the two networks can be written as (5) Our DIF-Net Φ consists of two sub-networks: a template SDF generation network T and a Deform-Net D. We will introduce these two sub-networks and our deformed implicit field representation in the next section.

p p'
Hyper-Net Figure 2. Overview of our proposed method. For a shape code α, Hyper-Net Ψ predicts (a part of) the weights of DIF-Net Φ, which further predicts the SDF for the shape. DIF-Net Φ consists of Deform-Net D which predicts a 3D deformation field and a correction field for the shape, and network T for generating a template implicit field shared across all shapes.

Deformed Implicit Field Representation
For a given object class, we assume that the object instances are mostly composed by a few common patterns or semantic structures. This is a mild assumption valid for many real-world object classes. For example, all cars consist of bodies and tires, and most chairs have back, seat, and legs. We seek to find a template implicit field which depicts common structures of the class and can derive the implicit fields for different object instances through 3D deformation.
Template implicit field and deformation field. To obtain an optimal template implicit field, we jointly learn it with deformations during training. Our template SDF generation network T and Deform-Net D can be written as The network weights of T is shared across the whole class, whereas weights of D are instance-specific and derived from the Hyper-Net, as shown in Fig. 2 and indicated by subscript ω in Eq. (7). With T and D ω , the SDF value of point p can be obtained via With the predicted deformation fields D ω (p), dense correspondences between two shapes can be established by deforming their surface to the aligned template space. In practice, one can densely sample surface points on two shapes, deforming them to the aligned 3D space, and find closest neighbors between two point sets.
Correction field. We also learn an SDF correction field to enhance our DIF representation. Specifically, we use Deform-Net D to predict not only a deformation vector but also a scalar correction term ∆s, Note that the template implicit fields are not valid shape SDFs and they characterize different shape structures within a category.
and we change the SDF value output in Eq. (8) to As illustrated in Fig. 4, the correction term is helpful to generate desired shapes. We empirically found that this correction term is also important to learn high-quality correspondence, as we will show in the experiments.
In summary, our neural shape model can be written as which is parameterized by the weights of network Ψ and T .
Discussions. It is crucial to note that our template implicit field is fundamentally different from a template 3D shape. In fact, Figure 3 shows that the learned template implicit fields are not valid shape SDFs (e.g., compare the lower part of chair's template field slice and the extracted iso-surface at s = 0). Instead, they characterize different shape structures within a category. In the experiments, we find that if under some circumstances (e.g., training with improper losses) they degenerate to a valid shape SDF representing a certain shape, both shape reconstruction accuracy and correspondence quality will significantly drop. This shows the advantage of our representation over previous works using template meshes or grids [25,56].

Learning Shape and Correspondence with DIF
We use the auto-decoder framework presented in [39] to jointly train weights of networks Ψ and T and learn latent codes {α j }. We design new loss functions for DIF to learn desirable dense correspondences.
Given a collection of shapes, we first apply an SDF regression loss similar to [42] to learn the SDFs of these shapes. Let Φ i (p) be the short-hand notation for Φ Ψ(αi) (p)  which is the predicted SDF value, we have wheres andn denote the ground-truth SDF value and surface normal respectively, ∇ denotes the spacial gradient of a 3D field, Ω is the 3D space and S i denotes shape surfaces. In practice, points will be sampled in the free space and on shape surface to calculate the loss. The second term in Eq. (12) is used to learn correct normals on shape surfaces -the gradient function of an SDF equals the surface normal given surface points as input and can be easily computed using network backpropagration. The third term is derived from the Eikonal equation which enforces the norm of spatial gradients ∇Φ i to be 1. The last term penalizes SDF values close to 0 for non-surface points through ρ(s) = exp(−δ · |s|), δ 1. We refer the readers to [42] for mode details about this loss. As in [39], we also apply a regularization loss to constrain the learned latent codes: Alternatively, we can apply stronger regularization on the latent space, such as minimizing the Kullback-Leibler divergence between the latent code posterior distribution and Gaussian distribution as in VAE training [30]. More details and results can be found in the suppl. material.
Normal consistency prior. To learn desired correspondences, our key observation is that the normal of a surface point is highly correlated with its semantic information. For example, normals on car hoods always point to the sky, and normals on the left doors always point to the left. In light of this, we encourage the normal directions of points in the template space to be consistent with their correspondences on all given shape instances: where ∇T is the spatial gradient of template field T , and n denotes the ground-truth normal of point p on object surface S i . Since the template field is used to derive all final shapes through deformation, this loss essentially enforces the normal consistency for correspondences across all the shapes generated by our network. Note that this loss is different with the normal term in Eq. (12): generating correct normals for each shape, as enforced by the latter, does not necessitate consistent correspondence normals between shapes, which is yet imposed by the former.
Deformation smoothness prior. To encourage smooth deformation and avoid large shape distortion, we add a simple smoothness loss on the deformation field: which penalizes the spatial gradient of the deformation field along X, Y and Z directions.
Minimal correction prior To encourage shape representation through implicit field deformation rather than correction, we minimize the correction field via In summary, the whole training process can be formulated as the following optimization problem: arg min {αj },Ψ,T L sdf +w 1 L normal +w 2 L smooth +w 3 L c +w 4 L reg (17) where w's are the balancing weights for different loss terms.

Correspondence Uncertainty Measurement
In practice, it is desirable to have a quality or uncertainty metric for the obtained shape correspondences, which can be used for structure difference analysis, bad correspondence removal etc. As mentioned previously, the correspondence between two objects O i and O j can be built by nearest neighbor search in the template space. Let p i be a point GT OccNet DeepSDF Ours Figure 6. Shape reconstruction results for unseen shapes by Occ-Net [37], DeepSDF [39] and our DIF-Net.
on O i and p j its corresponding points found on O j , we propose a simple yet surprisingly-effective uncertainty metric based on their distance in the template space: is the deformation vector (similarly for v j ) and γ is a scaling factor. The examples in Fig. 5 show that the regions with high uncertainty computed by Eq. (18) conform well to structure discrepancy between shapes.

Experiments
Implementation Details We implement Hyper-Net Ψ as ten MLPs to predict the weights of Deform-Net layers. Each MLP has a single hidden layer of dimension 256 and ReLU activations. The Deform-Net D and the template field network T are both MLPs with three hidden layers of dimension 128, equipped with sine activations advocated by [42]. We train our model on four categories in ShapeNet-V2 [11], including car, airplane, chair, and table. We use the first 3K, 3.5K, 4K, and 4K instances in the dataset as training set for the above categories, respectively. All parameters are trained end-to-end using the Adam [29] optimizer with learning rate 1e−4 for 60 epochs. Training takes about 4 hours on 8 NVIDIA V100 GPUs with batchsize 256 for one category. Due to space limitation, more details are presented in the suppl. material.

Ability of Shape Representation
To evaluate the representation power of DIF and the latent shape space learned by DIF-Net, we embed new shapes unseen in the training stage and measure the reconstruction accuracy. With our trained networks Ψ and T , we embed a test shape to the latent space by solving the following simplified optimization problem of Eq. (17): Figure 7. Qualitative evaluation of our learned dense correspondence for each category. For better visualization, we manually paint a generated shape (first column) with different colors on different semantic components. We also draw some strip patterns and texts on the shapes. Then we transfer these colors to other shapes generated by our method (last five columns) according to their correspondences. We also visualize the correspondences color-coded by spatial coordinates as well as the correspondence uncertainty for each shape.  Table 1. Reconstruction accuracy for unseen shapes. We use the first 100 shapes in the intersection of the test set splits from DeepSDF [39] and ours. OccNet [37] is trained for each category using our training data, and a per-category DeepSDF model trained by [34] is evaluated here. Reconstructed meshes are extracted at a resolution of 256 3 for all methods. CD and EMD are evaluated using 10K and 8K sampled points respectively.
We compare with two state-of-the-art shape modeling methods based on deep implicit filed: OccNet [37] and DeepSDF [39]. For OccNet, we train an individual model for each category using our training data for a fair comparison. For DeepSDF, we use a per-category model trained by [34] to evaluate its performance. Table 1 shows the shape reconstruction accuracy on 100 test shapes for each category, measured with chamfer distance (CD) and earth mover distance (EMD), and Fig 6 visually compares some results. It can be seen that all three methods perform well in representing unseen shapes, and our method is slightly better in terms of numerical error.
We also compare with a variant of our method which does not model dense correspondence. Specifically, we replace DIF-Net Φ with a MLP of three hidden layers that directly predicts the SDF of a shape. The numerical results are presented in Table 1, which are slightly better than our DIF-Net. It indicates that the deformation-based implicit design only leads to moderate decease of representation capability. However, high-quality dense correspondences can be achieved with this design, as we will show in the following sections.
Due to space limitation, more evaluations of our trained DIF-Net including latent space interpolation, sampling and retrieval are deferred to the suppl. materials.

Learned Dense Correspondence
Qualitative evaluation. Figure 7 visualizes the correspondences generated by our method, where we manually paint salient color patterns on the shapes to better check the correspondence quality. Visually inspected, our method produces convincing correspondences across various shapes despite their structure differences. It not only correctly matches shared semantic components between two shapes but also preserves the original color patterns. Moreover, the uncertain regions revealed by our method well reflect structure differences between two shapes.
Quantitative evaluation via Label Transfer. To our knowledge, there is no dataset offering ground truth dense correspondence for objects with structure variation. There-   Figure 9. Quatitative comparison on label transfer task.
fore, we resort to a semantic label transfer experiment for quantitative evaluation. We use the ShapeNet-Part dataset [57] which contains part labels for ShapeNet objects. For each of the four object categories, we selected 5 labeled shapes as source shapes, and transfer their labels to other shapes leveraging dense correspondences. This task can be viewed as few-shot 3D shape segmentation learning using 5 samples as training data. For this task, we first deform all labeled points on the 5 source shapes into our template 3D space, as shown in Fig. 8. For an unlabeled shape, we deform its surface points into the template 3D space, find 10 nearest labeled points for each, and then conduct simple label voting.
We compare our method with AtlasNet [25] and SIF [23], which can be used to build correspondences, as well as a naive closest point based method. For fair comparison, we train AtlasNet and SIF for each category individually with our training data. For label transfer with AtlasNet, we first label its template grid vertices which are

Correspond. Accuracy
Iso-surface (s = 0) Figure 10. Influence of different training losses and the correction field in our model. We use the chair category for evaluation, and measure shape reconstruction accuracy via F-score [47] at τ = 0.001 and correspondence accuracy via label IoU in the part label transfer task of Section 4.2. We also present visual results to further illustrate the effect of different components.
further used to label new points. For SIF, we follow a similar step to [23] to obtain template coordinates for all points to transfer labels. In the naive method, we label each point by direct label voting using its nearest labeled points in the original shape space. We use 10 nearest neighbors for all methods. More details can be found in the supp. material. Table 2 compares the accuracy measured by IoU between ground-truth and transferred labels, and Fig. 9 shows some visual results. Our method outperforms all others by a wide margin. AtlasNet and SIF may generate inconsistent correspondences (e.g., a point labeled as chair "arm" corresponds to "arm" regions for some shapes but to "seat" for others). Their results are worse than the naive closest point method for categories with large shape variation.

Ablation Study
In this section, we conduct ablation study to validate the efficacy of our training loss terms and the correction field. The main results of different settings are shown in Fig. 10. As can be seen, without the normal loss L normal , the learned correspondences are inferior as indicated by the significant IoU drop in label transfer. Without the deformation smoothness loss L smooth , the learned correspondences are highly distorted, as shown in the transferred textures. Without the correction field, shape representation ability of the model decreases significantly. Interestingly, Fig. 10 shows that the correction field is also crucial to obtain high quality correspondences. We hypothesis that with- out the correction field, more complex deformation fields are needed to represent the final shape with various structures, which significantly increases the learning difficulty. Under this situation, learning may get stuck into local minima, leading to an inferior template field lacking rich structural information (as shown by the iso-surfaces in Fig. 10), which further results in correspondence accuracy drop. The correction field helps to learn a better template field, hence improving the correspondence quality.

Texture Transfer
Using dense correspondence generated by DIF-Net, we are able to transfer textures from one object to another. Successfully transferring rich textures among various shapes necessitates high quality dense correspondence. Figure 11 shows texture transfer results between ground truth shapes in ShapeNet. Visually inspected, the rich texture patterns are well preserved and transferred to correct semantic areas in new shapes. Figure 1 contains two more texture transfer results of our method for embedded 3D shapes.

Shape Editing
With the learned latent shape and dense correspondence, our method can be used to manipulate 3D shapes by moving one or a sparse set of points. Specifically, give a shape with embedded latent code α, we can freely select one 3D point p 1 on the shape and specify its desired new position p 2 . Let be the deformed point of p 1 in the template 3D space, we achieve shape editing via solving for a new shape codeα minimizing the following equation: is the deformation vector for p 2 with the new shape codeα. In this equation, the first term enforces the original and new points on the shapes before and after editing to be a correspondence pair thus having same semantic meaning. The second term ensures the new point lies on the new shape surface. The third term requires the code change to be small. Figure 12 shows the editing results for two shapes and another two examples can be found in Fig. 1. We can even add new structures to a given shape via Eq. (20), where in this case we directly select p 1 in the template space and a free point p 2 in the shape space. An example is shown in Fig. 12. More details and results regarding shape editing can be found in the suppl. material.

Conclusion
We have presented Deformed Implicit Field, a novel implicit-based representation modeling a class of 3D shapes and providing dense correspondences. We also presented DIF-Net, a neural shape model that learns high-quality dense correspondences in an unsupervised manner through our proposed loss functions. Various experiments and applications collectively demonstrated the high quality shapes and correspondences generated by our method. In future, we plan to extend the DIF representation to handling more generic 3D objects and scenes.
A. More Implementation Details Data Preparation. To train DIF-Net via the SDF regression loss L sdf defined in Eq. (12) in the main paper, we randomly sample points on shape surface and in the free space. Specifically, we follow a similar step as in [39] to normalize each ground truth mesh into a sphere with radius of 1/1.03. Then, for surface points, we render 100 virtual images for each normalized mesh using 100 virtual cameras regularly sampled on the unit sphere. Surface points are obtained via back-projecting the depth pixels from these virtual images. Normals of these surface points are obtained in a similar way from virtual normal images. This sampling strategy helps us get rid of inner structures of each mesh that are invisible from outside (e.g car seats).
For free-space points, we uniformly sample them within a cube of [−1, 1] 3 , and calculate their distance to the nearest surface points sampled using the above strategy. To decide the sign of distance for a free-space point, we render it with the same virtual cameras used to obtain surface points, and check if its depth is smaller than the depth of a surface point falling into the same pixel. As long as it has a smaller depth value in any virtual image, it will be classified as an external point and get a positive sign. Otherwise, it will be classified as an internal point and get a negative sign.
In the end, we randomly sample 500K surface points along with normals and 500K free space points with their SDF values for each mesh.
Network Architecture. Figure I shows the structures of Hyper-Net Ψ and DIF-Net Φ used in our method. All networks are MLPs. The Hyper-Net consists of multiple MLPs each responsible for the weights ω i of a single fullyconnected layer i in the Deform-Net D ω . The weights of Template Field T are shared across the class.
Training Details. We jointly learn all latent codes {α i }, the Hyper-Net Ψ, and the Template Field T using the losses in Eq. (17) in the main paper. Similar to [39], we initialize all latent codes using N (0, 0.01 2 ). The weights of Hyper-Net Ψ are initialized using [27]. The weights of Template Field T are initialized as in [42].
We train an individual model on each category for 60 epochs. At each iteration, we randomly select 4K surface points and 4K free-space points for each shape in a batch to calculate the losses in Eq. (17) and update learnable parameters. During each epoch, there are in total 200K surface points and 200K free-space points used for each shape.
We set the balancing weights for four terms in the SDF regression loss L sdf in Eq. (12) to 3e3, 1e2, 5e1, and 5e2 respectively following [42]. w 1 and w 4 are set to 1e2 and 1e6 respectively for all categories. For w 2 , we set it to 5, 2, 5, and 1 for car, airplane, chair, and table respectively. For w 3 , we set it to 1e2, 1e2, 5e1, and 1e2 for each of the above categories, respectively. Models for all categories are trained using an Adam optimizer [29] with a learning rate of 1e − 4 and a batchsize of 256.
Inference Details. At inference time, given a test shape, we obtain its latent code via optimizing Eq. (19). Weights for different loss terms and point sampling strategy are identical to the training phase. We use Adam optimizer with a learning rate of 1e − 4, and update the latent code for 30 epochs in total.

B. Different Latent Code Regularization
In the main paper, we constrain the norm of learned latent code α to be small using L reg defined in Eq. (13). Alternatively, we can also constrain the posterior distribution of latent code to be close to a Gaussian distribution to learn a latent space for better sampling. Specifically, we can replace the L reg in Eq. (13) with the following loss: where KL denotes the Kullback-Leibler divergence and q(α i |O i ) is the posterior distribution of α i represented by a Gaussian distribution with mean equals to α i . To calculate L reg in Eq. (I), we have to obtain the standard deviation for q(α i |O i ) as well. Therefore, we introduce an extra learnable latent code σ with the same dimension of α to represent its standard deviation. Specifically, at training stage, the input latent code α to the Hyper-Net Ψ is replaced by a random variable α sampled from N (α, σ 2 I). Then, we train all learnable parameters using the following losses: where L reg is defined in Eq. (I) and all other losses are the same as in Eq. (17). In practice, we set the Gaussian distribution in Eq. (I) to N (0, 0.01 2 ), and set the weight w 5 to 1e2. Other balancing weights are the same as described in the previous section.

C. Learned Shape Latent Space
Latent Space Interpolation. In this part, we show latent space interpolation results using the model trained with Eq. (17) in Fig. IV. Shapes in even positions are interpolated from their two neighbors. We also show the colorcoded correspondence of them. As depicted, interpolated shapes are reasonable and correspondence between different shapes are consistent.
Latent Space Retrieval. Given a source shape, we can search for its nearest neighbors using the Euclidean distance between shape latent codes as metrics. Retrieval results are shown in Fig. V. Shapes on the right-hand side are nearest neighbors of the left-most shapes, and are sorted in an ascending distance order. Our learned latent space can capture the shape similarities in the original shape space-similar shapes are embeded close to each other.

D. More Details of Label Transfer
Source Shapes. Figure II shows the source shapes used to conduct the label transfer experiment. We manually select 5 sources for each category characterizing various structures in the shape distribution.
AtlasNet. For AtlasNet, we train individual model for each category using two different settings: one using a sphere mesh as the template, dubbed Atlas-sph., and the other using 25 square meshes as the template, dubbed Atlas-25. For each trained model, we first fit the template to all source shapes, and label each grid vertex on the template using label voting of its 5 nearest labeled points on 5 source shapes respectively. Then, given unlabeled points on a target shape, we fit the template to that shape, and label all points using label voting of 10 nearest grid vertices on the template.

SIF.
We train SIF for each category using a template of 100 implicit kernels as in [23]. To conduct label transfer, we follow [23] to first calculate the template coordinates for all point on different shapes. The template coordinates has a dimension of 300 (100 kernels each with a 3-dimensional coordinates). Then, given an unlabeled point on the target shape, we find its 10 nearest labeled points in the template coordinate system and conduct label voting.
E. More Details of Shape Editing Implementation Details. We achieve shape editing via Eq. (20) as described in the main paper. Given a shape with latent code α, we initialize the variableα to α at the beginning of the optimization. Balancing weight for three terms in Eq. (20) are set to 1, 1, and 5 respectively. We use Adam optimizer with a learning rate of 1e − 4, and updateα with 1,000 iterations. The optimization process takes about 10 seconds.
Adding New Structures. To add new structures to a given shape, we select p 1 defined in Eq. (20) in the template space, as is shown in Fig. III. Then, we follow the same optimization process to learnα for the new shape as described in the above paragraph.
More Editing Results. We show more shape editing results in Fig. VII. Our method is able to move selected points on the shape to desired position or add new structures. Features of the original shapes can be preserved after editing.