s2fieldsofstudy:
Computer Science
Engineering

text:
Non-Intrusive Load Monitoring Based on Deep Pairwise-Supervised Hashing to Detect Unidentiﬁed Appliances

: Non-intrusive load monitoring (NILM) is a fast developing technique for appliances operation recognition in power system monitoring. At present, most NILM algorithms rely on the assumption that all ﬂuctuations in the data stream are triggered by identiﬁed appliances. Therefore, NILM of identifying unidentiﬁed appliances is still an open challenge. To pursue a scalable solution to energy monitoring for contemporary unidentiﬁed appliances, we propose a voltage-current (V-I) trajectory enabled deep pairwise-supervised hashing (DPSH) method for NILM. DPSH performs simultaneous feature learning and hash-code learning with deep neural networks, which shows higher identiﬁcation accuracy than a benchmark method. DPSH can generate different hash codes to distinguish identiﬁed appliances. For unidentiﬁed appliances, it generates completely new codes that are different from codes of multiple identiﬁed appliances to distinguish them. Experiments on public datasets show that our method can get better F 1 - score than the benchmark method to achieve state-of-the-art performance in the identiﬁcation of unidentiﬁed appliances, and this method maintains high sustainability to identify other unidentiﬁed appliances through retraining. DPSH can be resilient against appliance changes in the house. appliance. The retrained model can also continue to detect other appliances.

Introduction
Information and Communication Technologies (ICT) and Intelligent Data Analytical Technologies (IDAT) have become the new trend for various industries' development [1][2][3]. Following this trend, ICT and IDAT are increasingly implemented in multiple industries [4][5][6]. Load Monitoring is one of the ICT and IDAT implementation cases in the power system, and it can disaggregate the whole electricity consumption signal into the signals of appliances in a residential, commercial, or industrial building. Load monitoring can identify appliances and report consumers consumption patterns to improve consumer behavior [7]. Furthermore, finding the detailed electricity consumption patterns of the customers helps energy suppliers to efficiently plan and operate power system networks.
Traditional load monitoring equipment is intrusive, that is, a sensor with communication function is installed for one monitoring equipment in the total load, and then, the power consumption information is received through the network for real-time monitoring. This method requires a large number of sensors, which increases installation and maintenance costs. Unlike it, non-intrusive load monitoring (NILM) installs a smart meter at the user's entrance to obtain the total current and terminal voltage. NILM can apply digital signal chemistry to the collected data, and then use algorithms to analyze and extract the power consumption information of various types of indoor appliances. The advantages of this method are as follows: low installation cost, little interference to users, and flexible application. Therefore, non-intrusive load monitoring technology has received widespread attention from scholars in recent years.
NILM was first proposed by Hart for residential load decomposition [8]. The operating states of appliances are divided into steady and transient. Therefore, the load monitoring methods can perform load decomposition based on steady or transient characteristics. The transient characteristics mainly include the change of the current or voltage waveform at the moment when the appliance starts. The duration of transient characteristics is short and unique, which can improve the recognition between loads. However, the transient feature extraction needs complex hardware, and the transient process of the load is affected by conditions such as grid voltage fluctuations, and the aging of electrical equipment. Steady-state load characteristics such as current harmonics [9], power harmonics [10,11], and current waveforms [12][13][14] have been successively applied to NILM. Steady-state characteristics are generally obtained by index quantification. They are less affected by noise, but the probability of similarity of the single steady-state characteristics of the load increases when the number of loads rises. In order to distinguish multiple appliances, a new load characteristic that is V-I trajectory has been developed for NILM in recent years. The V-I trajectory is plotted based on the steady-state voltage and current, and it is used to express appliances' electrical characteristics. The V-I trajectory in conjunction with many popular classification algorithms can offer better or generally comparable overall precision of prediction, robustness, and reliability [15]. In short, the V-I trajectory has advantages as a currently popular feature.
Based on different load characteristics, a variety of load identification algorithms have been proposed in NILM [16,17]. With the development of machine learning, the system results from the learning process can deliver the optimal predictive performance for appliance loads. Therefore, machine learning techniques have become a popular choice for NILM, since they showed significant disaggregation performance; in particular, Factorial Hidden Markov models (FHMMs) [18][19][20], Neural Networks (NN) [21][22][23][24], graph-based signal processing [25], Support Vector Machines (SVM) [26], k-Nearest Neighbours [26], and Decision Trees [27] have been successfully employed for NILM. Specifically, Reference [28] proposed a NILM algorithm based on features of the V-I trajectory. Ten V-I trajectory features were quantified based on physical significance, which accurately represented those appliances that had multiple built-in modes with distinct power consumption profiles, and the support vector machine multi-classification algorithm was employed for load identification. Reference [29] proposed a NILM algorithm based on the joint use of active and reactive power in the Additive Factorial Hidden Markov Models framework. In particular, in the proposed approach, the appliance model was represented by a bivariate Hidden Markov Model whose emitted symbols are the joint active-reactive power signals. The disaggregation was performed by means of an alternative formulation of the Additive Factorial Approximate Maximum a Posteriori (AFAMAP) algorithm for dealing with the bivariate HMM models. Reference [30] proposed an experimental design process for the application of energy disaggregation using multi-label classification. This paper took the electrical parameters of the current (I), real power (P), reactive power (Q), and power factor (PF) at every one-minute and employed RAndom k-labELsets (RAkEL) with Decision Tree as the multi-label classification algorithm together with the right model parameter configuration.
However, it is worth noting that most classification algorithms described in the literature cannot identify unidentified appliances in the consumer environment. In these algorithms, the unidentified appliance will be assigned a label and power consumption. They correspond to the identified appliance which have the most similar features. This leads to confusion between the identification of identified appliances and unidentified appliances. At the same time, the accuracy of appliance identification is reduced. Therefore, the household power consumption that is fed back to consumers and the power department is inaccurate.
Considering that the V-I trajectory is an image feature, its application enables NILM to be transformed into image retrieval. With the explosive growth of data in real applications like image retrieval, approximate nearest neighbor (ANN) search [31] has become a hot research topic in recent years. Due to its fast query speed and low memory cost, hashing [32] has become one of the most popular and effective techniques among existing ANN techniques. Existing hashing methods can be divided into data-independent methods and data-dependent methods. In data-independent methods, the hash function is typically randomly generated. It is independent of any training data. The representative data-independent methods include locality-sensitive hashing (LSH) [33] and its variants. Data-dependent methods try to learn the hash function from some training data, and they are also called learning to hash (L2H) [34] methods. Compared with data-independent methods, L2H methods can achieve comparable or better accuracy with shorter hash codes. Representative learning to hash methods include fast supervised hashing (FastH) [35], supervised discrete hashing (SDH) [36], column-sampling based discrete supervised hashing (COSDISH) [37], and column generation hashing (CGHash) [38].
Therefore, this paper proposes a V-I trajectory enabled deep pairwise-supervised hashing (DPSH) method for NILM. It contains simultaneous feature learning and hashcode learning. DPSH encodes the V-I trajectory images of identified appliances into compact binary hash codes. According to different coding results, we can identify various identified appliances in the environment, and DPSH can detect previously unidentified appliances in an automated way. When there is an unidentified appliance, DPSH will encode the V-I trajectory images of this appliance into brand new hash codes, which are different from other identified appliances. Thence, our proposed method can provide a scalable solution to energy monitoring for contemporary unidentified appliances.
The main contributions of this paper can be summarized as follows: Firstly, to the best of our knowledge, DPSH which can perform simultaneous feature learning and hash-code learning for applications with pairwise labels is first applied to NILM. This method transfers appliance identification to approximate nearest neighbor search, and improves the identification accuracy of identified appliances.
Secondly, the majority of the NILM approaches are sensitive to the replacement and addition of appliances in the house, and thus require regular retraining. In this paper, the focus lies on creating a classification algorithm that is able to detect unidentified appliances. Therefore, the algorithm can be resilient against the replacement and addition of appliances in the house. If an unidentified appliance is detected, labeling and retraining are requested to restore the identified environment and then identify the next unidentified appliance.
Thirdly, this paper also reflects the retraining results of our proposed method after identifying the unidentified appliance. The results show that the identification accuracy of DPSH can be restored to a high level through retraining, and when the next unidentified appliance appears, DPSH can still recognize it. In other words, DPSH maintains high sustainability. Experiments on public datasets show that DPSH can outperform the benchmark method to achieve state-of-the-art performance in NILM. This paper is organized as follows. Section 2 defines some symbols and issues in DPSH method. Section 3 explains the model and learning process of DPSH method as well as how it can be used for load disaggregation. Section 4 introduces benchmark datasets, the input of network, performance metrics, and selection of code length. The experimental results on publicly available datasets are presented in Section 5 to evaluate the performance of the proposed DPSH method for NILM. Moreover, the conclusions are given in Section 6.

Notation and Problem Definition
We convert the classification of electrical appliances to approximate nearest neighbor search of V-I trajectories in this paper.

Notation
The lowercase letters like z are used to denote vectors. We use uppercase letters like Z to denote matrices. Z T denotes the transpose of Z. The Euclidean norm of a vector is denoted as || · || 2 . sgn(·) is used to denote the element-wise sign function. If the element is positive, sgn(·) will return 1. Otherwise it will return −1.

Problem Definition
Suppose we have n V-I trajectory images X = {x i } n i=1 where x i is the i-th element in set X. Besides the set of V-I trajectories, the training set of supervised hashing with pairwise labels also contains a set of pairwise labels S = {s ij } n×n with s ij ∈ {0, 1}. s ij = 1 denotes V-I trajectory x i is similar to V-I trajectory x j . Otherwise, they are dissimilar. Here, the pairwise labels typically refer to semantic labels provided with manual effort.
The goal of supervised hashing with pairwise labels is to classify V-I trajectories correctly by learning hash function h(x). We can get a binary code b i ∈ {−1, 1} c for every trajectory , and c means the length of code. All of binary codes are collected in the set B = {b i } n i=1 . The similarity in S should be preserved in the binary codes B. If s ij = 1, it means that the Hamming distance between the binary codes b i and b j should be as small as possible. Otherwise s ij = 0, when the binary codes b i and b j have a high Hamming distance.

Deep Pairwise-Supervised Hashing
In this section, we introduce the DPSH model based on the V-I trajectory in detail. This section contains the model composition and learning algorithm.

Model Composition
The workflow of the proposed method that is able to detect unidentified appliances is shown in Figure 1. In the training phase, a hash function that can encode samples of the same appliance into the same binary hash codes is computed from the V-I trajectory images by training the feature learning part. The V-I trajectory images must be paired and labeled respectively as must-or cannot-links. This depends on if the images belong to the same class or not. The transformation does not depend on the appliance label. On the transformed input, DPSH determines the final encoding results of each identified appliance by minimizing pairwise loss. In the test phase, a V-I trajectory image is encoded to a kind of only binary hash codes. Next, we calculate the Hamming distance (d) to all representation codes. If the distance is equal to zero, the V-I trajectory is classified as one of the identified appliances. If the distance is not equal to zero, the trajectory is labeled as ''unidentified". DPSH model has an end-to-end deep learning architecture, which contains two essential parts: feature learning part and objective function part, and it is shown in Figure 2. Specifically, the feature learning part and objective function part feedback with each other during the training procedure. Feature learning part aims to learn a deep neural network which can extract multiple features from V-I trajectory images, and then features are encoded into compact binary hash codes. The goal of objective function part is to learn how to encode the features, which can reflect the supervised information (similarity) between two V-I trajectory images. The combination of the two parts achieves the purpose of load identification.

Feature Learning Part
Convolutional neural network (CNN) is a type of neural networks (NNs) that are often used in computer vision because they are highly suitable to classify images. We adopt a classical CNN model to extract the features of V-I trajectory images in this part. The feature learning part contains the Alexnet [39] model as a component, which has eleven layers and is good at extracting the features of images. The structure of the DPSH model is symmetrical. In other words, there are two Alexnets (top Alexnet and bottom Alexnet) in the feature learning part. These two Alexnets have the same structure and share weights.
That is to say, the input of the model are pairs of V-I trajectories and pairwise labels between the trajectories. The feature learning part is an indispensable foundation for NILM.

Pairs of V-I trajectory images
Feature learning

Final trajectory codes
Hash codes

Trajectory codes
Calculate Hamming distance to all representation codes and select minimum d  The network structure and parameters of the Alexnet model have been introduced in Table 1. More specifically, there are 5 convolutional layers (Conv 1-5), 3 max-pooling layers (MaxP 1-3), and 3 fully connected layers (FC 1-3) in the Alexnet model. The role of the convolutional layers is to extract local features of the trajectories. The progress of Alexnet is to imitate a large receptive field effect by using multiple convolution kernels in sequence. Alexnet uses convolution kernels and pooling kernels to deepen the network structure continuously and improve performance. The max-pooling layers are used to reduce the size of images, and fully connected layers are to reassemble the extracted local features into a complete graph through the weight matrix. The parameters of the Alexnet model are mainly concentrated in three fully connected layers. As shown in Table 1, the size of the convolution kernels in each segment gradually decreases, and the pooling kernels' size in each segment is the same.

Layer
Size of Filter Number of Channels Stride

. Objective Function Part
We can define the likelihood of pairwise labels S = {s ij } n×n as follows, when the binary codes B = {b i } n i=1 for all the V-I trajectory images are given. where −Ω ij . Ω ij means half of the inner product of two codes. When the two codes are the same, the inner product is the largest. In contrast, when two codes are completely different, the inner product is the smallest, so Ω ij can indicate the similarity of two codes. It is worth noting that By taking the negative log-likelihood of the pairwise labels observed in S, we can get the following optimization problem: The optimization problem in (2) fully reflects the goal of supervised hashing with pairwise labels. The Formula (2) ensures that the Hamming distance between two similar V-I trajectory images can be as small as possible, while the Hamming distance between two dissimilar V-I trajectory images can be as large as possible. Therefore, the purpose of distinguishing various appliances through the V-I trajectories is achieved. However, the problem in (2) is a discrete optimization problem that is hard to solve. Although it is solved by (1), directly relaxing B = {b i } n i=1 from discrete to continuous, satisfactory performance still cannot be obtained. Therefore, we adopt a novel strategy that can directly solve the problem in (2) in a discrete way. In other words, there is no need to give up the accuracy of B and convert it. We reformulate the problem in (2) as the following equivalent problem: To optimize the above problem, we can move the equality constraints in (3) to the regularization terms, so we reformulate the optimize problem (3) as the following equivalent one: where η is the regularization term that is a hyper-parameter.

DPSH Model
The DPSH model uses an end-to-end framework, which integrates the above feature learning part and objective function part together, and the end-to-end framework is expressed as where the network parameters of Alexnet model in the feature learning part are denoted as θ. Furthermore, We define the output of the Alexnet model with the network parameter θ as φ(x i ; θ), when the V-I trajectory image x i is used as the input of the Alexnet model. W ∈ R 1000×c is a weight matrix, and v ∈ R c×1 denotes a bias vector. That is to say, our DPSH model integrates the feature learning part and the objective function part into an end-to-end framework through a weight matrix and a bias vector. This method is similar to the function of a fully connected layer. After integrating the two parts, the final problem becomes Therefore, we get an end-to-end DPSH model. It can perform both feature learning and hash-code learning simultaneously in the same framework. The method can generate different hash codes to distinguish identified appliances and a completely new kind of hash codes for the unidentified appliance.
In conclusion, DPSH contains three key components. The first component is a deep neural network to learn image features from pixels. The second component is a hash function to map the learned image features to hash codes, and the third component is a loss function to measure the quality of hash codes guided by pairwise labels.

Learning Algorithm
In the DPSH model, known parameters include the pairwise labels S and the regularization term η. Other parameters including W, v, θ, and B need to be learned. In this paper, we adopt a minibatch-based strategy for learning. That is to say, in each iteration, we sample a mini-batch of V-I trajectory images from the whole training set, and then we can perform learning based on these sampled V-I trajectory images, and we design a method of alternating learning. More specifically, we optimize one parameter while other parameters are fixed. The hash codes b i can be directly optimized as follows: where sgn(·) can extract the sign of the input. we adopt the back-propagation (BP) algorithm to calculate the gradient in order to update other parameters W, v, and θ. In particular, we can calculate the derivative of the loss function with regard to u i according to the following formula: where a ij = σ( 1 2 u T i u j ). Then, we can calculate the gradient and use back-propagation algorithm to update the parameters W, v, and θ respectively:

Tool and Environment
All the experiments are implemented using Python 3.6 on a standard PC with an Intel Core i7-6700MQ CPU running at 3.40 GHz and with 16.0 GB of RAM. The CNN architecture is constructed based on Pytorch, and the pre-trained CNN model is migrated to DPSH.

Benchmark Datasets
The performance of the proposed algorithm is validated on the Reference Energy Disaggregation Data Set (REDD) and the Plug-Level Appliance Identification Dataset (PLAID).

REDD Dataset
REDD is a freely available data set containing detailed power usage information from several homes. It is aimed at furthering research on energy disaggregation [40]. The data contains power consumption from real homes over several months' time. They are the power consumption of the whole house as well as for each individual circuit in the house. All data in REDD is recorded with UTC time stamps. For each monitored house, REDD record the AC waveform itself in order to compute both real and reactive powers easily. This dataset includes low-frequency power data in each house, high-frequency voltage, and current data in house 3 and house 5. The list of appliances in the REDD dataset has been introduced in Table 2.

PLAID Dataset
The Plug-Level Appliance Identification Dataset is a public and crowd-sourced dataset for load identification research. PLAID dataset includes short voltage and current measurements for different residential appliances. The measurement equipment collects data in the order of a few seconds. The goal of PLAID is to provide a public library for high-resolution appliance measurements. It can be integrated into existing or novel appliance identification algorithms [41]. PLAID currently includes current and voltage measurements sampled at 30 kHz from 11 different appliance types present in more than 60 households in Pittsburgh, Pennsylvania, USA. Data collection took place during the summer of 2013 and winter of 2014. Measurements with significant noise in the voltage due to measurement errors were removed [42]. The list of appliances in the PLAID dataset has been introduced in Table 3.  Our paper adopts an event detection method and a trajectory extraction method, which are proposed in Reference [28]. An event is defined as the state-switching process of an appliance within a certain period of time. In this paper, the event is detected by comparing the variation in power during that durations with two predetermined thresholds.
Event detection is summarized by (12). Stride is represented by R in (12), and R is set to 1 s. The aggregated apparent power at t s is P t . The difference between two adjacent aggregated apparent powers is denoted by ∆P t (∆P t = P t+1 − P t ). The event begins when |∆P t | ≥ P on1 , and continues to calculate |∆P t+1 |, |∆P t+2 |, ..., until |∆P t+TR | < P on1 and |∆P t+TR+1 | < P on1 . If |P t+TR − P t | ≥ P on2 , the appliance has a state transition at t ∼ t + TR s. In other words, the integral event begins at t s and finishes at t + TR s. T is the number of strides that represents the duration of the event. P on1 and P on2 are set as 30W and 100W respectively in this paper. We believe that power fluctuations below P on1 are considered to be caused by noise, and it is considered a complete state switching process that the power difference before and after the event is greater than P on2 , and event detection is an essential step for V-I trajectory extraction. The voltage and current data must be processed before plotting the trajectory.
We extract the same number of voltage and current waveforms before and after the event. Four kinds of waveforms (the voltage waveforms before the event, the current waveforms before the event, the voltage waveforms after the event, and the current waveforms after the event) are interpolated and averaged separately. Then, the voltage waveforms before and after the event are averaged. The current waveform before and after the event takes the difference. Therefore, we can plot the V-I trajectory as a delta-form signature, which makes use of the difference between two consecutive snapshots and meets the feature-additive criterion [12]. We extracted 10 types of appliances' V-I trajectories from the REDD dataset to verify the effectiveness of our proposed method. According to the above method, the V-I trajectories for different appliances from REDD database are shown in Figure 3. Then, we directly use the raw images as input in DPSH model. We extract 4400 V-I trajectory images to train the proposed model. Each image belongs to one of the 10 classes. For these classes, the number of images of each class is at least 100, and we randomly select 1100 V-I trajectory images for the test set.
PLAID is a data set composed directly of the voltage and the current data of many appliances. It requires the same data processing to extract the trajectory. However, PLAID dataset is different from REDD dataset. PLAID currently includes current and voltage measurements sampled from different appliance types present in more than 60 households. We need a method to reduce the fluctuation of the image shape, so we convert every V-I trajectory image into a binary V-I image (n × n matrix) by meshing the V-I trajectory. Each cell of the mesh is assigned a binary value that denotes whether or not it is traversed by the trajectory. Binary V-I image can reduce the volatility of data generated by different appliance types present in different households, and we choose 6 binary V-I trajectory images to test DPSH algorithm, as shown in Figure 4. We select 757 V-I trajectory images to train the proposed model. Each image belongs to one of the 6 classes. For these classes, the number of images of each class is different. The category with the fewest numbers contains 26 images, and we randomly select 226 V-I trajectory images for the test set.   All of the V-I trajectory images are resized as 224 × 224. Then we directly use the raw image pixels as input in DPSH model. Please note that there are two Alexnets (top Alexnet and bottom Alexnet). These two Alexnets have the same structure and share the same weights. That is to say, both the input and the loss function are based on pairs of images. The extracted V-I trajectories are input into the Alexnets in pairs. The features of V-I trajectories, which are extracted after 5 convolutional layers, 3 max-pooling layers, and 3 fully connected layers, form many feature vectors of 1000 elements. DPSH model learns a hashing function can convert 1000-dimensional feature vectors to trajectory codes in the training period. The code length is a hyper-parameter. We can set it according to our own needs. The trajectory codes must ensure that the Hamming distance between two similar V-I trajectory images is as small as possible, while the Hamming distance between two dissimilar V-I trajectory images is as large as possible. In other words, the trajectory codes can reflect pairwise similarity. Therefore, we can identify the category information of V-I trajectories according to these codes in the testing period. Two V-I trajectories will belong to the same category if the Hamming distance between two trajectory codes is close to 0, and the proposed model can produce a lot of coding results that have never been seen in the training period when there are unidentified appliances in the consumer's environment. At this point, we can retrain the model quickly to accommodate the addition of a new appliance. The retrained model can also continue to detect other newer appliances.

Performance Metrics
In this paper, the problem of NILM is considered as an approximate nearest neighbor search task, which should assign each V-I trajectory image into one of predefined classes. We utilize a meaningful performance metric, mean average precision (MAP), which can illustrate the performance in many multi-class classification tasks [43]. We estimate the ranks of data samples in the calculation of average precision(AP). The discrete form of AP for class C k is where we use x i to represent the ith query V-I image. |C k | denotes the cardinality of set C k , and rank(s; S) is the rank of s in set S. A smoothed pair-wise rank function can be used to estimate ranking relation between two samples x i and x j . This function is defined in [42] as follows. rank(x i ; x j ) = 1 For M-class classification, we will generally use the mean of all APs of different classes to evaluate the overall performance.
We adopt three classification metrics to evaluate the effect of classification. They are shown in the following equations based on True Positive (TP), True Negative (TN), False Positive (FP), and False Negative (FN). These metrics analyze how well the algorithm can identify changes in the appliance's status.
Precision + Recall F 1 -score is a measure of the test's accuracy and is obtained by calculating the weighted average of the Precision and Recall. It calculates the percentage of energy correctly assigned to each appliance in the dataset. A higher F 1 -score value indicates a better identification of the appliance. To obtain the final test result, the average F 1 -score is taken: where N is the amount of different appliances, and F 1,i is the F 1 -score when appliance i is used as hold out appliance.

Selection of Code Length
We adopt the REDD dataset and PLAID dataset to verify the performance of DPSH. In the experiment on each dataset, both training and testing are done on all of the appliances. The number of V-I trajectory images produced by each kind of appliance has a different proportion in the total. The reason is that the frequency of opening and closing is different in the operation of various appliances. We adopt a pre-trained Alexnet model to reduce training times. Besides, we set the mini-batch size to be 64 and tune the learning rate among [10 −6 , 10 −2 ]. For DPSH method, the hyper-parameter η is set to be 50 by using a validation strategy. The most important parameter is the length of the output hash code.
The optimal code length determines the recognition accuracy and space complexity of DPSH algorithm. Therefore, the code length is set to be 12, 24, 32, 48 bits [35][36][37] so that we can verify the effect of this parameter on experimental results. As shown in Figure 5, the DPSH method based on the Alexnet model shows high accuracy in terms of MAP when the code length is set to be 12, 24, 32, 48 bits.  It can be seen that the performance of DPSH on REDD is better than that on PLAID. This is because we believe that there is only one V-I trajectory for each appliance when processing the REDD dataset, and the V-I trajectory shape of each appliance is different in the PLAID dataset. In other words, we assume that all 10 appliances in the REDD dataset are single-state appliances (Each appliance corresponds to a V-I trajectory). In the PLAID dataset, there are many types of V-I trajectories for each appliance. This may be a normal multi-state appliance, or it may be caused by measurement errors, and the part of trajectories of various appliances may be repeated. This is the reason why the results on REDD and PLAID are different. However, DPSH can maintain high accuracy in both two datasets, as shown in Figure 5. The minimum value of MAP in the PLAID dataset reaches 0.9169, when the code length is set to be 12 bits.
According to the above analysis, the MAP of DPSH on the two datasets is high and the performance is stable. Through calculation, we get the average MAP of each code length as 0.9581, 0.9616, 0.9627, and 0.9627. It can be seen that the MAP value of 0.9627 is the best result, but the average MAP of 32 bit and 48 bit are equivalent. At the same time, a too-long code length will increase memory cost, so we chose 32 bit as the code length for subsequent result analysis.

Results and Discussion
To define how well the DPSH model can identify identified appliances and unidentified appliances, we designed the following two scenarios. Scenario 1 simulates a normal household electricity environment where all of appliances are identified. Scenario 2 simulates a household environment where unidentified appliance exists.

Recognition of Identified Appliances
We train and test on two datasets to verify whether the proposed algorithm can distinguish identified appliances well. The number of appliances used for training and testing is the same, and Table 4 details the performance indicators achieved by running DPSH with 32-bit code length on different datasets. These are the best results of our proposed method on two datasets. As shown in Table 4, the Precision, Recall, and F 1 -score of every appliance are introduced in detail. This shows that our proposed algorithm is highly effective in appliance identification. It can accurately identify all appliances in REDD. DPSH also has a good ability to identify all appliances in the PLAID dataset. In conclusion, this method has a strong adaptability to different datasets. The 32-bit encoding results of DPSH method on the PLAID dataset are shown as Figure 6. It ensures that the Hamming distance between codes of different labels is as large as possible, but a kind of appliance label does not necessarily correspond to only one encoding result. This phenomenon illustrates the complexity of the PLAID dataset, and multiple encoding results of DPSH increase the stability of the method. It ensures that the algorithm can still maintain a high accuracy of appliance identification when the V-I trajectory images of the appliance are deviated due to fluctuations or the appliance has multiple states.
Ap4 Figure 6. The 32-bit encoding results of DPSH method on PLAID dataset.
The benchmark method in Reference [44] is based on siamese neural network and DB-SCAN clustering method (SN-DBSCAN). Figure 7 is the Precision and Recall for proposed DPSH and SN-DBSCAN [44] algorithm on REDD and PLAID datasets. The appliances in Figure 7 correspond to Tables 2 and 3. Figure 7a,b reflects the results generated by testing on REDD. It is obvious that the Precision and Recall for DPSH-32-bit are stable. The Precision for the SN-DBSCAN algorithm has a slight deviation. However, the Recall for the SN-DBSCAN algorithm has great fluctuation. It has the lowest Recall value for the third appliance, only 0.556. Meanwhile, as shown in Figure 7c,d, the results on the PLAID dataset are similar to the above. The Precision and Recall for DPSH-32-bit only fluctuate a little bit and are basically stable at very high values. Due to the complexity of PLAID, the Precision and Recall for DPSH-32-bit have some fluctuations on the second and third appliances. The Recall value of the second appliance is the lowest, but it has reached 0.882. The Precision and Recall of SN-DBSCAN not only have lower values on the second and third appliances but also have larger deviations on other appliances. This shows that DPSH is more stable on a complex PLAID dataset than SN-DBSCAN. In other words, this demonstrates that DPSH leads to performance improvements with respect to the SN-DBSCAN even in the presence of noise or measurement error. After the above analysis, it specifically reflects that DPSH is always more accurate than the SN-DBSCAN algorithm. We can conclude from Figure 7 that our proposed algorithm maintains high Precision and Recall on REDD and PLAID to ensure high accuracy of appliance identification. The radar chart in Figure 8 shows the F 1 -score for each appliance in the experiment including two datasets, and the area of each colored line is proportional to the F average of the related algorithm. It shows that the DPSH method gives better results for every appliance on F 1 -score, compared with the SN-DBSCAN algorithm. The F average of DPSH and SN-DBSCAN are 0.984 and 0.932 on REDD, respectively, and the indicators on the PLAID dataset are 0.969 and 0.815. The more complex the test environment, the greater difference in F average between DPSH and SN-DBSCAN. Compared with the SN-DBSCAN algorithm, the experimental results indicate that the proposed method significantly improves the accuracy and can be efficiently generalized when they are tested on the same database.

Recognition of Unidentified Appliance
To define how well the method can identify unidentified appliances, we complete the experiment on two datasets. We choose the Ap10 to be an unidentified appliance and call it Un10 in the REDD dataset. All other appliances are identified appliances, so training is done on 9 appliances and testing on 10 appliances. Similarly, we choose the Ap5 to be an unidentified appliance and call it Un5 in the PLAID dataset, and training is done on 5 appliances and testing on 6 appliances. The goal of our experiments is to detect unidentified appliances in user environments. Therefore, we choose Un10 and Un5, respectively, in two datasets as fixed unidentified appliances. This is called leave-one-appliance-out validation. In the REDD, each appliance corresponds to one V-I trajectory, but there is more than one V-I trajectory for each of these appliances in the PLAID, and they may be similar to each other, so we designed two experiments on two different datasets. It is validated whether (1) the selected appliances are properly separated by different codes with high Hamming distance, and (2) the unidentified appliance has its trajectory images classified as "unidentified".
The 32-bit encoding results of DPSH method on the PLAID are shown as Figure 9 when the last appliance is selected to be unidentified. Compared with Figure 6, we can see that the encoding results of the unidentified appliance are different from the encoding results of the identified appliances and include several forms. Therefore, our proposed method can achieve the purpose of detecting the unidentified appliance. Note that the encoding results of the same labels in Figures 6 and 9 are different. Because the samples used to learn the hash function are different (in Figure 9, the samples for the fourth appliance are not used).
Un5 Figure 9. The 32-bit encoding results of DPSH method with the unidentified appliance.
To know which appliances are mixed up, a confusion matrix is created: Figure 10a,b reports for each appliance type (row index) the number of labels that were correctly predicted or confused with other appliances (column index). The values in the matrix are percentages, and the colors represent the recall value per row (thus per appliance). It can be seen that the recognition accuracy of the appliances used for training is still very high. Due to measurement errors or similar shapes to other appliances, the unidentified appliance is encoded into a variety of discrete hash codes, so the unidentified appliance may be confused with other appliances. On the REDD dataset, only a small number of identified appliances are distinguished as unidentified. All the unidentified appliances are distinguished and marked correctly. Due to the complexity of the environment, the identified and unidentified appliances are confused on the PLAID, during the process of differentiation, but the number is small. In conclusion, we can discover that the DPSH algorithm also has high accuracy for the identification of unidentified appliances by analyzing the experimental results on the two datasets. According to Reference [44], the training appliances are used for learning form clusters. The samples belonging to the holdout appliance do not belong to any cluster and neither form a cluster as the siamese neural network is not trained on them. They have spread around and get the label "unidentified". We train and test SN-DBSCAN on the same datasets. The experimental results of the two methods on REDD and PLAID with the unidentified appliance are shown in Tables 5 and 6. For all appliances, the recognition accuracy of DPSH is basically higher than that of SN-DBSCAN. At the same time, we find that DPSH can maintain a high recognition capability for identified appliances. F 1 -score is an evaluation that comprehensively considers Precision and Recall and can more fully reflect the appliance identification. Therefore, we next focus on this indicator. The radar chart in Figure 11 shows the F 1 -score for each appliance in the experiment, including all the appliances in two datasets. In the REDD experiment, the F 1 -score of DPSH on Ap1 and Ap2 are slightly lower than SN-DBSCAN. On Ap1 and Ap2, especially on the unidentified appliance, DPSH's F 1 -score are significantly higher than SN-DBSCAN, and the F average values of DPSH and SN-DBSCAN are, respectively, 0.942 and 0.860. In the PLAID experiment, DPSH can better identify the unidentified appliance, but the F 1 -score of DPSH on Ap2 and Ap6 are slightly higher than SN-DBSCAN. The F average values of two algorithms are, respectively, 0.883 and 0.726. When unidentified appliances appear in the environment, although the SN-DBSCAN algorithm can identify them, the recognition accuracy of the identified and unidentified appliances decreases significantly. Concerning SN-DBSCAN algorithm, the proposed approach shows a higher improvement. The relative differences of F average are +0.082 and +0.157. In conclusion, the recognition ability of DPSH is very prominent for both identified appliances and the unidentified appliance. To further illustrate the effectiveness and practicality of DPSH, this paper conducts retraining experiments. We label the V-I trajectory images of unidentified appliances that are identified by two algorithms in the above experiments, and we add these V-I trajectory images to the respective training set. This new training set is used to retrain DPSH and SN-DBSCAN model. The indicators for evaluating the effectiveness of the two models are shown in Tables 7 and 8. By observing F 1 -score, it can also be concluded that the ability of DPSH to recognize the appliances is very strong. We calculate that the F average values of the two methods on REDD are 0.984 and 0.918, and the F average values of the two methods on PLAID are 0.967 and 0.763. We can see that the accuracy of DPSH can be restored to a very high level through the retraining process, although the number of unidentified appliances used for retraining is relatively small. In contrast, the accuracy of SN-DBSCAN is greatly affected by the amount of training data, and the accuracy after retraining is not satisfactory. When other unidentified appliances appear, the number of retraining increases, the accuracy of SN-DBSCAN will become lower and lower. However, DPSH will still accurately identify all appliances. Therefore, we can conclude that DPSH has good practicality. It is able to detect other unidentified appliances after retraining. The sustainability of this method performs well.

Conclusions
This paper has proposed a voltage-current trajectory enabled DPSH for NILM. Our major purpose is that different appliance loads including unidentified appliances can be distinguished by encoding their V-I trajectory images. DPSH model has an end-to-end deep learning architecture containing feature learning part and objective function part. Specifically, the feature learning part aims to learn a deep neural network which can extract multiple features from the original images, and then, features are encoded into compact binary hash codes. The purpose of the objective function part is to learn how to encode the features, which can reflect the similarity between query images and database images well. Experiments on real datasets have shown that DPSH model improves the accuracy and can outperform the benchmark method to achieve state-of-the-art performance in NILM. The F average of DPSH on REDD and PLAID dataset are 0.984 and 0.969, separately. Meanwhile, DPSH has solved a difficult problem that the accuracy of the recognition algorithm will drop a lot when the unidentified appliance is added to the environment. In other words, this method can identify the unidentified appliance, under the condition of ensuring the accuracy of identified appliances. The F average of two experiments to distinguish unidentified appliance are respectively 0.942 and 0.883, so we can add images of the unidentified appliance to the training set and retrain the model so that it can recognize all appliances well. The F average of two experiments increased to 0.984 and 0.967, respectively.


Learning by Aligning: Visible-Infrared Person Re-identification using Cross-Modal Correspondences

We address the problem of visible-infrared person re-identification (VI-reID), that is, retrieving a set of person images, captured by visible or infrared cameras, in a cross-modal setting. Two main challenges in VI-reID are intra-class variations across person images, and cross-modal discrepancies between visible and infrared images. Assuming that the person images are roughly aligned, previous approaches attempt to learn coarse image- or rigid part-level person representations that are discriminative and generalizable across different modalities. However, the person images, typically cropped by off-the-shelf object detectors, are not necessarily well-aligned, which distract discriminative person representation learning. In this paper, we introduce a novel feature learning framework that addresses these problems in a unified way. To this end, we propose to exploit dense correspondences between cross-modal person images. This allows to address the cross-modal discrepancies in a pixel-level, suppressing modality-related features from person representations more effectively. This also encourages pixel-wise associations between cross-modal local features, further facilitating discriminative feature learning for VI-reID. Extensive experiments and analyses on standard VI-reID benchmarks demonstrate the effectiveness of our approach, which significantly outperforms the state of the art.

Introduction
Person re-identification (reID) aims at retrieving person images, captured across multiple cameras, with the same identity as a query person. It provides a wide range of applications, including surveillance, security, and pedestrian analysis, and has gained a lot of attention over the last decade [40,47]. Most reID methods formulate the task as a single-modality retrieval problem, and focus on finding matches, e.g., between RGB images. Visible cameras are incapable of capturing appearances of persons, particularly * Equal contribution. † Corresponding author. Figure 1: An example of dense cross-modal correspondences between RGB and IR images from the SYSU-MM01 dataset [36]. For the visualization purpose only, we show the top 20 matches according to similarities between local person representations learned without (left) and with (right) our approach. Our person representations are robust to the cross-modal discrepancies, while being highly discriminative, especially for person regions. (Best viewed in color.) important for person reID, under poor-illumination conditions (e.g., at night time or dark indoors). Infrared (IR) cameras, on the other hand, work well, regardless of visual light, capturing an overall scene layout, while not taking scene details, such as texture and color. Accordingly, visible-IR person re-identification (VI-reID), that is, retrieving IR person images of the same identity as an RGB query and vice versa, has recently been of great interest [36].
VI-reID is extremely challenging due to intra-class variations (e.g. viewpoint, pose, illumination and background clutter), noisy samples (e.g. misalignment and occlusion), and cross-modal discrepancies between RGB and IR images. Visual attributes and statistics of RGB/IR images are significantly different from one another [36]. VI-reID methods based on convolutional neural networks (CNNs) alleviate the discrepancies using cross-modal metric losses [7,37,41] along with a modality discriminator [4] to learn person representations robust to the cross-modal discrepancies, and further refine the representations with self-attention [39] or disentanglement techniques [3]. These approaches focus on learning coarse image-level or rigid part-level representations, assuming that person images are roughly aligned.
Misaligned features from RGB and IR images, however, have an adverse effect on handling the cross-modal discrepancies, distracting learning person representations.
In this paper, we propose to leverage dense correspondences between cross-modal images during training for VI-reID. To this end, we encourage person representations of RGB images to reconstruct those from IR images of the same identity, which often depict different appearances due to viewpoint and pose variations, and vice versa. We achieve this by establishing dense cross-modal correspondences between RGB and IR person images in a probabilistic way. We incorporate parameter-free person masks to focus on the reconstructions of person regions, while discarding others including background or occluded regions. We also introduce novel ID consistency and dense triplet losses using pixel-level associations, allowing the network to learn more discriminative person representations. Dense crossmodal correspondences align pixel-level person representations from RGB and IR image explicitly, which is beneficial to person representation learning for VI-reID due to two main reasons. First, by enforcing semantically similar regions from RGB and IR images to be embedded nearby, we encourage the network to extract features invariant to the input modalities, even from misaligned RGB and IR person images. Second, by encouraging a local association, we enforce the network to focus on extracting discriminative pixel-wise local features, which further facilitates the person representation learning. The network trained using our framework is thus able to offer local features that are robust to cross-modal discrepancies and highly discriminative ( Fig. 1), which are aggregated to form a final person representation for VI-reID, without any additional parameters at test time. Experimental results and extensive analyses on standard VI-reID benchmarks demonstrate the effectiveness and efficiency of our approach. The main contributions of this paper can be summarized as follows: • We propose a novel feature learning framework for VI-reID using dense cross-modal correspondences that alleviates the discrepancies between multi-modal images effectively, while further enhancing the discriminative power of person representations. • We introduce ID consistency and dense triplet losses to train our network end-to-end, which help to extract discriminative person representations using cross-modal correspondences. • We achieve a new state of the art on standard VI-reID benchmarks and demonstrate the effectiveness and efficiency of our approach through extensive experiments with ablation studies.

Related work
In this section, we briefly describe representative works related to ours, including person reID, VI-reID, cross-modal image retrieval and dense correspondence.
ReID. Person reID methods typically tackle a singlemodality case, that is, RGB-to-RGB matching. They formulate the reID task as a multi-class classification problem [49], where person images of the same identity belong to the same category. A triplet loss is further exploited to encourage person representations obtained from the same identity to be embedded nearby, while those from the different identities to be distant in feature space [12]. Recent methods focus on extracting person representations robust to intra-class variations, exploiting attributes to offer complementary information [19], disentangling identity-related features [9,48], or incorporating attention techniques to see discriminative regions [18,46]. Many reID methods leverage part-based representations [8,32,33], which further enhance the discriminative power of person features. Specifically, they divide person images into multiple horizontal grids exploiting human body parts implicitly. Local features from the horizontal parts are more robust to intra-class variations, especially for occlusion, than the global one. However, when body parts from corresponding horizontal grids are misaligned, this rather distracts learning person representations. The works of [15,23,45,52] propose to align semantically related regions between person images, by employing auxiliary pose estimators [23,45] or human semantic parsing techniques [15,52]. While these auxiliary branches offer reliable estimations to guide the alignment, they have two main drawbacks: First, they typically require additional datasets during training. Second, the auxiliary predictions are required at test time, making the overall pipeline computationally heavy. On the other hand, we perform the alignment during training only, by leveraging dense correspondences, without additional supervisory signals except ID labels, while enabling an efficient pipeline at test time.
VI-reID. VI-reID has recently been explored compared to single-modality reID, according to the wide spread of RGB-IR cameras. VI-reID methods focus on handling cross-modal discrepancies between RGB and IR images, while learning discriminative person representations. Early works try to learn discriminative features generalizable across different modalities. They adopt classification and/or triplet losses, widely used in single-modality reID methods [36,40], which however do not mitigate the crossmodal discrepancies explicitly. To address this problem, recent methods use a cross-modal triplet loss, where positive/negative pairs and an anchor are sampled from person images with different modalities [7,37,41]. For example, RGB images are used as anchors, while exploiting IR ones as positive/negative samples. These approaches encourage the features, obtained from person images of the same identity but having different modalities, to be sim-ilar, providing person representations robust to the crossmodal discrepancies. More recently, DDAG [39] proposes to leverage a graph attention network in order to consider cross-modal relations between RGB and IR images explicitly. VI-reID methods based on generative adversarial networks (GANs) alleviate the cross-modal discrepancies in an image level. For example, they synthesize novel IR person images, with an identity-preserving constraint [34] or cycle consistency [35], given RGB inputs, in order to compare person images with the same modality. Other approaches to leveraging adversarial learning techniques for VI-reID are to disentangle identity-related features from person representations [3], or to exploit a modality discriminator to better align feature distributions of RGB/IR images [4]. Although GANs better capture discriminative factors for person reID, they require lots of parameters and heuristics to train networks [28]. In contrast to current VI-reID methods, we address the cross-modal discrepancies in a pixel level. To this end, we align semantically related regions explicitly via dense cross-modal correspondences, which also allows discriminative feature learning, even from misaligned person images.
Cross-modal image retrieval. VI-reID is closely related to cross-modal image retrieval that focuses on finding matches between images of different modalities, e.g., sketch/natural images [27,29], and RGB/IR images [1,20]. Existing works typically employ a siamese network [42] to learn a metric function between input image pairs [1,29], or disentangle feature representations into modality sharedand specific-embeddings [20,27]. They attempt to alleviate the cross-modal discrepancies between multi-modal images in an image-level. We instead address the discrepancies in a pixel-level by leveraging dense correspondences.
Correspondence. Establishing correspondences between images has long been of particular importance in many computer vision tasks, including depth prediction [13,43], optical flow [2,6], 3D scene reconstruction [16,51], and colorization [11,44]. In context of person reID, the work of [31] leverages dense correspondences to learn a metric function for single-modality person reID. The learned metric function, however, is required even at test time, demanding large computational power and memory. In contrast, we exploit the correspondences as explicit regularizers to guide the feature learning during training only, enabling a simple cosine distance computation between person representations at test time.

Approach
We describe in this section an overview of our framework for VI-reID (Sec. 3.1), and present detailed descriptions of a network architecture (Sec. 3.2) and training losses (Sec. 3.3). Pooling

Training time Te s t t i m e
Matching Probability Figure 2: Overview of our framework for VI-reID. We extract RGB and IR features, denoted by fRGB and fIR, respectively, using a two-stream CNN. The CMAlign module computes cross-modal feature similarities and matching probabilities between these features, and aligns the cross-modal features w.r.t each other using soft warping, together with parameter-free person masks to mitigate ambiguous matches between background regions. We exploit both original RGB and IR features and aligned ones (fRGB andfIR) during training, and incorporate them into our objective function consisting of ID (LID), ID consistency (LIC) and dense triplet (LDT) terms. At test time, we compute cosine distances between person representations, obtained by pooling RGB and IR features. See text for details.

Overview
We show in Fig. 2 an overview of our framework for VI-reID. We first extract RGB and IR features from corresponding person images, and then align the features with a CMAlign module. It establishes dense cross-modal correspondences between RGB and IR features, and warps these features w.r.t each other using corresponding matching probabilities. Note that we exploit the CMAlign module at training time only, enabling an efficient inference at test time. To train our framework, we exploit three terms: ID (L ID ), ID consistency (L IC ), and dense triplet (L DT ) losses. The ID loss applies to each feature from RGB or IR images, separately, similar to single-modality reID [12]. It enforces the features from person images of the same identity to be the same, while providing different ones for the images of different identities. The ID consistency and dense triplet terms exploit the matching probabilities, and encourage RGB and IR features from the same identity to reconstruct one another in a pixel-level, while those from different identities do not. The person representations obtained using these terms are thus robust to the cross-modal discrepancies between RGB and IR images. Note that we use identification labels alone to train our model, without exploiting auxiliary supervisory signals, such as e.g., body parts [15] or landmarks [23], for the alignment. Note also that all components in our model are fully differentiable, making it possible to train the whole network end-to-end.

Network Architecture
Feature extractor. We use a two-stream CNN to extract feature maps of size h×w×d from a pair of RGB/IR person images, where h, w and d are the height, width and number of channels, respectively. Assuming that the cross-modal discrepancies between RGB/IR images mainly lie in lowlevel features [36,40], we use separate parameters specific to the input modalities for shallow layers, while sharing the remaining ones for others.
CMAlign. The CMAlign module aligns RGB and IR features bidirectionally, i.e., from RGB to IR and from IR to RGB, using dense cross-modal correspondences in a probabilistic way. In the following, we describe an IR-to-RGB alignment. The other case can be performed similarly.
For the IR-to-RGB alignment, we compute local similarities between all pairs of RGB and IR features. Concretely, we compute cosine similarities between RGB and IR features, denoted by f RGB ∈ R h×w×d and f IR ∈ R h×w×d , respectively, as follows: where · 2 computes the L2 norm of a vector. We denote by f RGB (p) and f IR (q) RGB and IR features of size d at position p and q, respectively. Based on the similarities, we compute RGB-to-IR matching probabilities using a softmax function as follows: where we denote by P a matching probability, a 4D tensor of size h × w × h × w, and β is a temperature parameter. Note that we can establish dense correspondences explicitly from RGB to IR images, by applying an argmax operator to the matching probabilities for each RGB feature, i.e., argmax q P (p, q). This offers reliable crossmodal correspondences for semantically similar regions, but aligning IR and RGB features using the hard correspondences is problematic. The correspondences are easily distracted by background clutter and image-specific details (e.g., texture and occlusion), and appearance variations between RGB and IR images are even more significant. Moreover, we could not establish correspondences between different background regions, e.g., from person images captured with different surrounding environments. To alleviate these problems, we instead exploit the matching probabilities, and align IR and RGB features between foreground regions only, typically correspond to persons, via soft warping as follows: where we denote byf RGB ∈ R h×w×d and M RGB ∈ R h×w a reconstructed RGB feature by the IR-to-RGB alignment and a person mask, respectively. We denote by W a soft warping operator that aggregates features using the matching probabilities, defined as follows: The person mask ensures that the featuresf RGB , for person regions are reconstructed by aggregating IR features in a probabilistic way, while others come from original RGB features f RGB . This reconstruction together with ID consistency and dense triplet losses encourages our model to provide similar person representations, regardless of image modalities, for the corresponding regions. To infer the mask without ground-truth labels, we assume that features, learned with ID labels for the reID task, are highly activated on person regions than other parts, and compute an activation map based on L2 norms of the local feature vectors, denoted by g RGB ∈ R h×w for an RGB feature, as follows: With the activation map of an RGB feature, g RGB , at hand, we define a person mask for an RGB feature as follows: where f performs min-max normalization: The CMAlign, which is a non-parametric module that operates directly on the features obtained from the feature extractor, facilitates learning robust person representations by providing the following advantages in VI-reID: First, a cross-modal alignment helps to alleviate the discrepancies between RGB and IR images in a pixel-level, allowing to suppress modality-related features from person representations more effectively, even with misaligned person images; Second, a dense alignment allows our network to focus on learning local features, especially for person regions, further enhancing the discriminative power of person representations. Note that a pair of RGB and IR images does not have to be of the same identity in our framework, enabling exploiting both positive and negative pairs for training.

Loss
We exploit ground-truth ID labels of person images to train our model with an overall objective function as follows: where L ID , L IC and L DT are ID, ID consistency, and dense triplet losses, respectively. λ IC and λ DT are hyperparameters to balance corresponding terms. In the following, we present a detailed description of each term in the loss.
ID loss (L ID ). As an ID loss, we adopt a sum of classification and hard triplet losses [12] using image-level person representations, which have shown the effectiveness on learning discriminative person features in single-modality person reID. We denote by φ(f RGB ) ∈ R d and φ(f IR ) ∈ R d image-level person representations for the RGB and IR features, respectively, which are obtained by applying a GeM pooling operation [26] to each feature. To compute the classification term, we feed each image-level feature, φ(f RGB ) and φ(f IR ), into a same classifier to predict class probabilities, that is, likelihoods of being particular identities for the image-level feature, where the classifier consists of a Batch Normalization layer [14], followed by a fully-connected layer with a softmax activation [22]. We then compute a cross-entropy between the class probabilities and groundtruth identities. The hard triplet term is also computed using image-level person representations, obtained from anchor, positive, and negative images, where the anchor and positive ones share the same ID label, while other pairs do not. Note that the ID loss does not address the cross-modal discrepancies between RGB and IR images explicitly.
ID consistency loss (L IC ). We design a term to consider the cross-modal discrepancies between RGB and IR features in an image-level. Suppose that we have a positive pair with the same identity but having different modalities, i.e., RGB and IR images are of the same identity. The featuresf RGB for person regions are reconstructed by aggregating IR features f IR , suggesting that the identity of the reconstructionf RGB should be the same as ground-truth identity for the original features f RGB and f IR . More specifically, image-level representations of φ(f RGB ) and φ(f IR ) should have the same ID labels as corresponding positive counterparts of different modalities, f IR and f RGB , respectively. To implement this idea, we define an ID consistency loss as a cross-entropy using image-level representations, similar to the classification term in the ID loss. We instead exploit reconstructed features, φ(f RGB ) and φ(f IR ). Note that we use the same classifier as in the ID loss. The ID consistency loss enforces ID predictions from person images of the same identity but with different modalities to be consistent, allowing to suppress modality-related features from person representations. Moreover, the reconstructions, φ(f RGB ) and φ(f IR ) provide an effect of offering additional samples to train the classifier, further guiding the discriminative person representation learning.
Dense triplet loss (L DT ). The ID loss facilitates learning discriminative person representations, and the ID con- Figure 3: Visualization of person masks for IR and RGB images, MIR(left) and MRGB(right), and a corresponding co-attention map, AIR(middle). We overlay the masks and the attention map over corresponding images from SYSU-MM01 [36]. We can see that the IR image depicts a person with fully visible body parts, whereas the person of the same identity in the RGB image is partially occluded (lower body). The co-attention map highlights image regions that are mutually visible in both images and suppresses others using dense cross-modal alignments via soft warping. (Best viewed in color.) sistency term alleviates the cross-modal discrepancies explicitly. They, however, focus on learning image-level person representations, which prohibits discriminative feature learning, especially when the person images are occluded or misaligned. To address this problem, we introduce a dense triplet loss. It locally compares original features and reconstructed ones using the features of different modalities, encouraging final image-level person representations to be discriminative, while alleviating the cross-modal discrepancies in a pixel-level. A straight-forward approach is to compute L2 distances between local features, which is, however, suboptimal in that this does not take occluded regions into consideration. This is particularly problematic when each of person images in a pair depicts disassociated human parts. Enforcing local alignments between the entire person regions in this case is infeasible, and maybe even harmful. To circumvent this issue, we incorporate a coattention map highlighting person regions visible in both RGB and IR images. This considers feature alignments within mutually visible foreground regions only to compute the dense triplet loss. We define a co-attention map, denoted by A RGB ∈ R h×w for an RGB image, as follows: For W(M IR (p)), in this case, we compute the matching probabilities P between f RGB and f IR , similar to (4), whereas the person masks are exploited for soft warping. Namely, the co-attention map A RGB is an intersection of the RGB person mask M RGB (p) and the warped IR one, w.r.t the RGB image, W(M IR (p)). We compute a coattention map for an IR image similarly, and show in Fig. 3 an example of a co-attention map. Note that we define coattention maps between positive pairs of the same identity only. Note also that we perform min-max normalization f on the obtained co-attention map, which we omit for notational brevity.
To facilitate training with the dense triplet term, we sample a triplet of anchor, positive, and negative images, where the anchor and other two images have different modalities e.g., an RGB image for the anchor, and IR images for a pair of positive and negative samples. We use the superscripts, a, p, and n to indicate features from anchor, positive, and negative images, respectively. For example, we denote byf p RGB a reconstructed RGB feature using an anchor f a RGB and a positive pair f p IR with the same identity as the anchor f a RGB . Similarly,f n IR is a reconstructed IR feature using an anchor f a IR and a negative pair f n RGB with the different identity from the anchor f a IR . With co-attention maps at hand, we define the dense triplet loss as follows: where α is a pre-defined margin and the operation [·] + indicates max(0, ·). d + i (p) and d − i (p) compute local distances between an anchor feature and reconstructed ones from positive and negative images, respectively, as follows: Note that the reconstructions,f p i andf n i , are the aggregations of similar features w.r.t the anchor f a i from positive and negative images, respectively. We can thus interpret that our loss enforces an aggregation of similar features from negative images to be distant in the embedding space, compared to its positive counterpart by a margin. This is similar to the typical triplet loss [12,30], but ours penalizes incorrect distances for all local features visible in both anchor and positive images in a soft manner. Note that this local association is possible due to the CMAlign module that performs the dense cross-modal alignment between RGB and IR person images in a probabilistic way.

Experiments
In this section, we present a detailed analysis and evaluation of our approach including ablation studies on different losses and network architectures.

Implementation details
Dataset. We use two benchmarks for evaluation: 1) The RegDB dataset [24] contains 412 persons, where each person has 10 visible and 10 far-infrared images collected by dual camera systems. Following the experimental protocol in [24], we divide the dataset into training and test splits randomly, each of which includes non-overlapping 206 identities. We test our model in both visible-to-IR and IR-tovisible settings, which correspond to retrieving IR images from RGB ones and RGB images from IR ones, respectively, and report the results averaged over 10 trials with different training/test splits. 2) SYSU-MM01 [36] is a largescale dataset for VI-reID, consisting of RGB and IR images obtained by four visible and two near-infrared sensors, respectively. Concretely, it contains 22,258 visible and 11,909 near-infrared images with 395 identities for training. The test set contains 96 identities with 3,803 near-infrared images for a query set and 301 visible images for a gallery set. We adopt the evaluation protocol in [36], which uses all-search and indoor-search modes for testing, where the gallery sets for the former and the latter contain images captured by all four and two indoor visible cameras, respectively. Note that all our results are obtained by taking an average value over 4 training and test runs.
Training. Following the previous VI-reID methods [3,21,39], we adopt ResNet50 [10], trained for ImageNet classification [5], as our backbone network. The backbone networks for visible and infrared images share the parameters, except for the first residual blocks that take images of different modalities, and the stride of the last convolutional block is set to 1. We resize each person image to the size of 288 × 144, and apply horizontal flipping for data augmentation. We set the size of a person representation d to 2,048. For a mini-batch, we randomly choose 8 identities from each modality and sample 4 person images for each identity. We train our model for 80 epochs with a batch size of 64, using the SGD optimizer with momentum of 0.9 and weight decay of 5e-4. We use a warm-up strategy [22], gradually raising learning rates for the backbones and other parts of the network up to 1e-2 and 1e-1, respectively, which are then decayed by a factor of 10 at the 20th and 50th epochs. We use a grid search to set hyper-parameters: λ IC = 1, λ DT = 0.5, α = 0.3, β = 50. Note that we employ BNN trick [22] during training only, exploiting ResNet50 [10] at test time without any additional parameters. We implement our model with PyTorch [25] and train it endto-end, taking about 6 and 8 hours for RegDB [24] and SYSU-MM01 [36], respectively, with a Geforce RTX 2080 Ti GPU.

Results
Comparison with the state of the art. We present in Table 1 a quantitative comparison of our method with the state of the art for VI-reID [3,4,17,21,34,35,36,37,38,39]. We report mean average precision (mAP) (%) and rank-1 accuracy (%) for a single-shot setting on RegDB [24] and SYSU-MM01 [36]. From the table, we can see that our model sets a new state of the art for VI-reID, except for an indoor-search mode on SYSU-MM01 [36], where DDAG [39] shows better results. This method, however, requires additional parameters, other than the ResNet50 [10] backbone, for a feature refinement with self-attention at test Table 1: Quantitative comparison with the state of the art for VI-reID. We measure mAP (%) and rank-1 accuracy (%) on the RegDB [24] and SYSU-MM01 [36] datasets and report the average and standard deviations over 4 training and test runs. Numbers in bold indicate the best performance and underscored ones indicate the second best.

Methods
RegDB [24] SYSU-MM01 [36] Visible  time, while being outperformed by ours in other benchmarks. We can also see that our model achieves better results than cm-SSFT 1 [21] by a significant margin on both datasets. Note that cm-SSFT [21] uses multiple RGB and IR images to extract person representations, even at test time. That is, it exploits additional images of different modalities, e.g., multiple IR images to extract features from an RGB input. cm-SSFT [21] is thus computationally expensive, and requires a lot of memory. Overall, the experimental results on the standard benchmarks demonstrate that our approach provides person representations robust to the cross-modal discrepancies and intra-class variations across RGB and IR images. Qualitative comparisons along with rank-10 accuracy (%) can be found in the supplementary material.
Parameter and runtime analysis. We compare in Table 2 the average runtime to extract a final person representation. For fair comparison, we measure the average runtime over 50 executions, for person images of the size 288 × 144 on the same machine with a Geforce RTX 2080 1 For cm-SSFT [21], we report in Table 1 the results obtained without using a random erasing technique [50] and a BNN trick [22], similar to ours, for fair comparison. The results are taken from Table 4   Ti GPU. Table 2 also compares the number of network parameters required at test time. Our method is fastest among the state of the art, and uses the smallest number of parameters, as it does not use any additional parameters, except the ones for a backbone network, at test time. Other methods on the contrary exploit additional layers or networks.

Discussion
Ablation study. We show in Table 3 an ablation analysis on training losses and the CMAlign module. We train variants of our model using different combinations of loss terms, L IC and L DT , and co-attention map A, while adding CMAlign modules to different layers of the backbone network. We compare the performance in terms of mAP and rank-1 accuracy on SYSU-MM01 [36] under the all-search mode. For the baseline model in the first row, we exclude the CMAlign module and train it using the ID loss alone.
Overall, we can see that the baseline shows the worst performance, indicating that incorporating the CMAlign module is beneficial for VI-reID. For example, exploiting the CMAlign module with either the ID consistency term (the second row) or the dense triplet term coupled with the co-attention map (the fourth row) boosts the performance significantly. This is because the ID consistency term mainly addresses cross-modal discrepancies in an image-level and the dense triplet term handles them in a pixel-level, while further enhancing the discriminative power of person representations. From the second, fourth, and last rows, we can observe that using all losses and the co-attention map gives the best results, suggesting that they are complementary to each other. Note that the co-attention map is particularly important for the dense triplet term, as shown in the fifth and last rows.
Computing the loss on distractive regions (e.g., occlusions and background clutter) may hinder learning discriminative representations. We also compare in the last three rows our models involving the CMAlign modules in different layers of the backbone network, where the modules are added on top of conv4-6 and/or conv5-3 of ResNet50 [10].
We can see that adding the modules to both conv4-6 and conv5-3 gives the best results, as this allows to consider cross-modal discrepancies in multiple levels of features.
Visualizations of dense correspondences. We show in Fig. 4 examples of cross-modal correspondences between RGB and IR images on SYSU-MM01 [36]. We can see that the matches are established well between persons of the same identity, and they are not influenced by cross-modal discrepancies, appearance variations, and background clutter. Specifically, our model provides local features that are robust against scale variations ( Fig. 4(a)) and occlusions ( Fig. 4(b)). This implies that our model is able to extract discriminative person representations with rich semantics, which are important for the person reID task, while alleviating the cross-modal discrepancies. In particular, our model offers local features that are robust to viewpoint variations (Fig. 4(c)), where a person's sweatshirt or trousers often matches to its pair regardless of front or side view. This indicates that our network provides local person features that are robust to viewpoint variations, which is particularly useful for VI-reID. This aspect of correspondences for reID is in contrast to typical correspondence tasks, e.g., stereo matching and optical flow estimation, that favor viewpoint-specific matches. We also provide in Fig. 5 a visual comparison of correspondences for different configurations of losses. Our model trained with the ID loss alone is unable to establish reliable matches between cross-modal images and easily distracted by background clutter (Fig. 5(a)), mainly due to cross-modal discrepancies and a lack of discriminative power in local feature representations, particularly for person regions. The ID consistency loss handles the cross-modal discrepancies, establishing correspondences between local person representations from different modalities (Fig. 5(b)). The dense triplet loss further encourages each local feature to be discriminative, which in turn offers matching results focusing on person re-

Conclusion
We have introduced a novel feature learning framework for VI-reID that exploits dense correspondences between cross-modal person images, allowing to learn person representations that are robust to intra-class variations and crossmodal discrepancies across RGB and IR person images. We have also proposed ID consistency and dense triplet losses exploiting pixel-level associations, enabling our model to learn more discriminative person representations. We set a new state of the art on standard benchmarks, outperforming other VI-reID methods by a significant margin. Extensive experimental results clearly demonstrate the effectiveness of our approach.
In this supplementary material, we present additional performances under different quantitative metrics (Sec. S1) and results on applying our method to single-modality person re-identification (reID) (Sec. S2). We also present qualitative comparisons with the state-of-the-art methods for visible-infrared person reID (VI-reID) (Sec. S3), along with additional visualizations of co-attention maps (Sec. S4), and cross-modal correspondences (Sec. S5).

S1. Quantitative comparisons
We report in Table S1 the rank-10 accuracy (%) on RegDB [3] and SYSU-MM01 [6] datasets, as these metrics provide additional insights in evaluating the robustness of a VI-reID model. The results are obtained by taking an average value over 4 training and test runs.

S2. RGB reID
With minor effort, our feature learning framework is applicable to other reID tasks. To verify the generalization ability, we apply our method to single-modality person reID. To this end, we train our baseline using ResNet50 [2] as a backbone and PCB [5], a representative work for person reID, with and without a CMAlign module and our losses. Table S2 shows the results on DukeMTMC-reID [4]. We can see that the CMAlign module and our losses bring performance improvements for both cases in terms of mAP and rank-1 accuracy. This demonstrates that our approach is also effective to the single-modality setting, and it boosts performance for other reID methods. Note that our approach does not require any additional parameters or computational overhead at test time.

S3. Qualitative comparisons
We provide in Figs. S1 and S2 the visual comparisons of retrieval results with the state-of-the-art methods [1,7] * Equal contribution. † Corresponding author.  Table S2: Quantitative comparison of reID methods trained with and without our components (i.e., a CMAlign module, and ID, ID consistency and dense triplet terms). We use the ID loss alone for the baseline and PCB [5]. We measure mAP (%) and rank-1 accuracy (%) on DukeMTMC-reID [4]. on SYSU-MM01 [6] and RegDB [3], respectively. We sort the retrieval results according to cosine distances w.r.t each query image. In contrast to other methods, our model successfully retrieves person images with the same identity as the input query, even under severe intra-class variations across person images. We also show in Fig. S3 failure cases on SYSU-MM01 [6]. Current VI-reID methods including ours fail to discriminate persons with similar silhouettes. Retrieving person images of different modalities is still extremely challenging. The main reason is that IR sensors do not capture scene details, such as cloth patterns and color. In particular, our method tends to fail when a person image depicts an object of which correspondences are ambiguous, e.g., a backpack in the last row of Fig. S3. Training with the CMAlign module in this case would be ineffective in enhancing the discriminative power of local features.

S4. Co-attention map
We incorporate co-attention maps to focus on reconstructions between image regions that are visible in both RGB/IR person images for discriminative feature learning with the dense triplet loss L DT . We show in Fig. S4 examples of co-attention maps on SYSU-MM01 [6]. Our coattention map ignores disassociated human parts for a given pair of RGB/IR person images. For example, in Fig. S4(a), when a person's feet are not visible in the IR image, the co-attention map ignores these parts, while highlighting regions that are mutually visible.

S5. Cross-modal correspondences
For each pair of RGB/IR person images, we pick top 20 matches according to the matching probabilities on SYSU-MM01 [6], and show the results in Figs. S5 and S6. We categorize image pairs into three groups depending on the type of intra-class variations within each pair: scale ( Fig. S5(a)), occlusion ( Fig. S5(b)) and viewpoint (Fig. S5(c)). We can see that our model offers local features robust to the crossmodal discrepancies between RGB and IR images, even under severe intra-class variations across person images. Figure S6 compares correspondences obtained from models trained with different configurations of training losses. Our full model (Fig. S6(c)) establishes dense correspondences between regions that are semantically related, while being robust to the cross-modal discrepancies between RGB and IR images.  Figure S1: Retrieval results of (a) Hi-CMD [1], (b) DDAG [7], and (c) ours on SYSU-MM01 [6]. For each row, the person image in the left is given as a query. The retrieved images in green boxes indicate correct results and the red ones are the opposite. We also show retrieval results, obtained with an RGB image as a query, although we do not evaluate this case quantitatively following the experimental protocol of SYSU-MM01 [6]. The model trained with LID alone tends to establish incorrect correspondences, e.g., matches between background and person regions, as this term does not handle cross-modal discrepancies explicitly. Incorporating LIC somewhat alleviates this problem. Our full model in (c) offers more discriminative features, providing more correct matches between semantically related regions, particularly for person regions. (Best viewed in color.)