Direct Current (DC) Motor Speed and Direction Controller

The direct current motor is an important drive configuration for many applications across a wide range of power and speeds. It has variable characteristics and is used extensively in variable-speed drives. The goals of this project are to control the direction and speed of a Direct Current (DC) motor. Due to the advancement of wireless technology, there are several communication devices introduced such as GSM, Wi-Fi, ZIGBEE and Bluetooth. Each of the connections has its own unique specification and application. Among these wireless connections, Bluetooth technology is often implemented and can be sent from the mobile phone at a distance of 10 meters. The speed control was implemented using Bluetooth technology to provide communication access from a smartphone. Instead, the ARDUINO UNO platform can be used to quickly promote electronic systems. And an electronics technique is called Pulse Width Modulation (PWM) is used to achieve speed control, and this technique generates high and low pulses, then these pulses vary the speed in the motor. In order to control this PWM pulse, variable resistors are used and depend on it the speed of the DC motor will increase or decrease. The variable resistor is adjusting to varying the speed of the motor, and the higher the resistance the lower the speed of the motor rotates. The direction of the motor is controlled by the relay by giving and giving a command on the virtual terminal. The speed of the motor is directly proportional to the resistance as the speed increased after the resistance also increased and vice versa. The significance of this study is practical and highly feasible from the economic point of view and has the advantage of running the motor at a higher rating in term of a reliable, durable, accurate and efficient way of controlling speed and direction control.

Introduction
The direct current (DC) motor means electrical energy converts to mechanical energy and is always used in industries. The application of speed controllers used on a large scale, easy and used in many ways. In most applications, speed and direction are very important. The purpose of a motor speed controller is to take a signal representing the demanded speed and to drive a motor at that speed. There are a lot of applications where control of speed is needed, as in drilling machines, conveyor, lift and elevators. These applications may demand high-speed control accuracy and good dynamic responses. In-home appliances, washers, dryers and mixers are good examples of the usage of the motor.
There are numerous applications in our life that require DC motor speed control. DC motors have the ability to change speed, torque and direction at any time to meet the new condition. At low speed, DC motors are able to provide a high starting torque, and it is possible to obtain speed control over a wide range. DC motors are suitable in many applications, for example, conveyors, turntables and others for which adjustable direction and constant or low-speed torque are required. The DC motor can also work in dynamic braking and reversing applications, which are usually used in many industrial machines. So, the study of controlling the DC motor is more practical and important [1]. Nowadays the  [2]. The past years have brought standards for the use and conservation of power, leading engineers all around the world to build low-power consumption. Nowadays, our day-to-day life becomes very vast, so it is not possible for industries to go to the workplace and control motor working. All home appliances are preferred to control wired and wireless mechanism. For example, the previous technology only used a remote control to control the speed of the motor such as a ceiling fan or a switch to operate the machines. By using wireless control, eliminates the need for the operator to be in direct contact with running machines. This brings to the importance of the control of DC motors.
The motor needs to vary the speed because some machines need to operate more than one speed. Thus, the following challenges are encountered: speed control of DC motor needs to vary the value of the resistor to get a different value of speed. Next, the rotation of the motor is important because it makes it more functional. The previous technology like conventional exhaust a fan is able to rotate in one direction only [3]. So, the project needs to drive a DC motor for the required application in clockwise and anti-clockwise directions. In this project, the aim is to simulate the speed and direction of the DC motor using Proteus software. Next, it is to measure the speed of the DC motor after it is varied by using a variable resistor and to control the rotation of the motor using a mobile phone. Figure 1 shows a block diagram of DC motor speed and direction controller. In general, terminal mode chooses to write the instruction from the Bluetooth application "Arduino Bluetooth'' and the signal will be sent to Arduino via Bluetooth Module HC-06. Then Arduino will send the signal to the motor driver, and the motor will rotate according to the command sent through the Bluetooth as shown in Figure 1.

Software Development Process
Proteus was designed to be easy to use, readable and consistent. Proteus has everything needed such as to develop, test and virtually prototype designs. This design allows engineers to develop their projects more quickly and empowering them with the flexibility to make hardware and reduce time to design. For this project, Proteus software needs to be used by connecting the entire circuit. The circuit comprises the triggering circuit, supply and virtual load connections [4][5]. Figure 2 shows the circuit diagram drawn by using Proteus 8 Professional. The Variable resistor is connected with the Arduino and depends on the value of that resistor given; the DC motor speed will either increase or decrease. The relay is simulated in a simple circuit in which when you run the simulation, the relay will automatically activate and control the relay using logic. For example, when you provide 5 V to it and then the relay will go activated, and when you give GND, then it will de-energize and vice versa. But this simulation is connected to PWM and GND. The relay acts as a switching as the command is given to the virtual terminal.

Hardware Development Process
Hardware development is developed after the simulation is working successfully. The circuit was tested on a breadboard first before fully transfer to the printed circuit board (PCB) because to check the circuit whether it is working well or not. After a few tests and troubleshoot, the circuit is transferred to PCB for the last progress. Figure 3(a) shows the components used in hardware development. The circuit first, as in Figure 3   After being satisfied with the circuit and it is functioning well, then the circuit transferred to PCB. On PCB, the interconnection between the components is made through copper tracks instead of using a number of current-carrying wires. It makes the interconnections less bulky and better presentation. A typical PCB offers a simple platform to arrange the electronic components in a compressed and efficient way. This compactness allows the creation of big and complicated electronic circuits in small form factors. This, in turn, takes less space in devices. For the communication device between a mobile phone and Bluetooth, and apps, Arduino Bluetooth Controller is installed at the play store. As the Bluetooth used HC-06, so it will connect to a device. After a successful connection, there are a few modes popping up on the screen, and Terminal mode is chosen as the command [6][7][8]. Figure 4 shows the final product of hardware development with different views. The circuit, power supply and motor are put in a Perspex box. The blade of the motor is used to show the motor rotating in which direction. The variable resistor is put at the side of the box, so it is easier to control the motor without opening the box.

Results and Discussion
Theoretically, the motor is measured using the tachometer to get the speed of the motor in rpm. As the resistance is increased, the speed of the motor decreased. This is because PWM is produced by the transistor driving the motor with 'ON-OFF' pulses and varying the duty cycle, the fraction of time that the output voltage is 'ON' compared to when it is 'OFF', of the pulses while keeping the frequency constant. The longer the pulse is 'ON', the faster the motor will rotate and, likewise, the shorter the pulse is 'ON' the slower the motor will rotate. Two types of variable resistor were used to compare the speed of the motor by using a different value of resistance. Table 1 is the information of the motor and variable resistor 10 Kȍ and 1 Mȍ. Figures 5 and 6 show the result of the speed of the motor by using a variable resistor 10 Kȍ and 1 Mȍ respectively.  Figure 5 shows the variety of behaviour of the graph after obtaining the data from the hardware. The higher the resistance, the lower the speed of the motor. The resistor limits torque and protects the motor from being damaged by a high current. If the armature resistance is an increase, the drop across armature increases, which causes the speed to decrease. Increasing the total voltage will increase the speed of the motor. The current and voltages also drop as the resistance increases as shown in Figure  5.  Figure 6 shows the various behaviour of the graph after obtaining the data from the hardware. The graph shows the higher the resistance, the lower the speed of the motor. But this resistance is higher, so the speed drops slower. The resistor limits torque and protects the motor from being damaged by a high current. If the armature resistance is an increase, the drop across armature increases, which causes the speed to decrease. Increasing the total voltage will increase the speed of the motor. The current and voltages also drop as the resistance increased. The current is smaller than the previous current, and it is almost 0 A. Table 2 shows the result of the direction control using a mobile phone. The commands are typed on the terminal mode either C, A or S to change the direction. The LEDs also light up according to the rotation of the motor. Both LEDs will light up if the motor is signaling the LED to stop. The speed of the motor depends on the variable resistor when the shaft of the variable resistor is turned during the command sent. The motor takes time to change the direction from clockwise to anti-clockwise or vice versa because the motor needs to slow down until 0 rpm, then the direction will change. Yellow and Green

Conclusion
The control of the DC motor has been developed by using Bluetooth technology as one of the wireless connection technologies, which is widely available in mobile phones and computers. The proposed project is to simulate the speed and direction of a DC motor using Proteus software. The simulation is successfully done after all the LEDs and motor functioned as expected. The speed of the motor is also successfully done after varying the speed by using a variable resistor but by using a 9 V power supply and 6 to 24 Vdc motor. Two variable resistors were used to see the difference between speed, voltage and current produced. The direction of the motor by using Bluetooth and a mobile phone is barely achieved because it is not working as expected as in the simulation. The Bluetooth was connected with the phone, but the transmission is not received well between Bluetooth and mobile phone.


Sign Language Recognition via Skeleton-Aware Multi-Model Ensemble

Sign language is commonly used by deaf or mute people to communicate but requires extensive effort to master. It is usually performed with the fast yet delicate movement of hand gestures, body posture, and even facial expressions. Current Sign Language Recognition (SLR) methods usually extract features via deep neural networks and suffer overfitting due to limited and noisy data. Recently, skeleton-based action recognition has attracted increasing attention due to its subject-invariant and background-invariant nature, whereas skeleton-based SLR is still under exploration due to the lack of hand annotations. Some researchers have tried to use off-line hand pose trackers to obtain hand keypoints and aid in recognizing sign language via recurrent neural networks. Nevertheless, none of them outperforms RGB-based approaches yet. To this end, we propose a novel Skeleton Aware Multi-modal Framework with a Global Ensemble Model (GEM) for isolated SLR (SAM-SLR-v2) to learn and fuse multi-modal feature representations towards a higher recognition rate. Specifically, we propose a Sign Language Graph Convolution Network (SL-GCN) to model the embedded dynamics of skeleton keypoints and a Separable Spatial-Temporal Convolution Network (SSTCN) to exploit skeleton features. The skeleton-based predictions are fused with other RGB and depth based modalities by the proposed late-fusion GEM to provide global information and make a faithful SLR prediction. Experiments on three isolated SLR datasets demonstrate that our proposed SAM-SLR-v2 framework is exceedingly effective and achieves state-of-the-art performance with significant margins. Our code will be available at https://github.com/jackyjsy/SAM-SLR-v2

I. INTRODUCTION
Sign language [1] is a visual language performed with the dynamic movement of hand gestures, body posture, and facial expressions. It is a widely-used alternative approach for deaf and mute people to communicate effectively. Understanding and performing sign language requires a substantial time of learning which is beyond feasible for the public, which leads to a barrier between deaf-mute people and others. Moreover, sign language is dependent on language [2], [3], [4] (e.g., English and Chinese) and culture [5] (e.g., American and British) that further limits its popularity. Sign Language Recognition (SLR) aims to help deaf-mute people communicate smoothly with others in their daily life by automatically interpreting sign language. As machine learning and computer vision make great progress in the past decade, SLR has drawn much research attention.
SLR contains two tasks, isolated SLR and continuous SLR. The isolated setting is a fine-grained and fully supervised classification at word (gloss) level, while the continuous setting maps whole videos into sentences (i.e., sequences of glosses) in a weakly supervised manner. SLR is a more challenging problem compared with conventional action recognition for the following reasons. First, sign language requires both delicate hand gestures and global body motion to distinctly and accurately express its meaning. Facial expressions may also be used to express emotions and emphasis. Similar gestures may sometimes express different and even opposite meanings. Second, different signers may perform sign language differently (e.g., left-hander, right-hander, different speed, body shape, and localism), which makes SLR more challenging. Collecting more samples from as many signers as possible is desired yet expensive. To this end, it requires a finer-grained model to capture the delicate dynamics of the whole human body. Besides, we expect the model to effectively merge the information from different cues when making the final recognition.
Recently, in human action recognition, skeleton-based approaches have become more popular and drawn increasing attention due to their strong adaptability to dynamic circumstances and complicated backgrounds [19], [20], [21], [22], [23]. Besides, the skeleton-based methods provide complementary information to RGB-based modalities, hence their ensemble results further improve the overall recognition rates. However, there exist some barriers that hinder the extension of skeleton-based methods to the SLR task. The skeleton-based methods for action recognition rely on ground-truth annotations of human bodies provided by motion capture systems, which restrict themselves to limited datasets captured in labcontrolled environments. Apart from that, most motion capture systems only focus on the fewer body keypoints excluding hand keypoints. The provided keypoints are insufficient to recognize sign language performed with delicate hand gestures and motions. Some researchers attempt to obtain hand poses using separate hand detectors with keypoint estimators and propose to use an RNN-based model to recognize the sign language [24]. Unfortunately, their estimated hand keypoints are unreliable, and the RNN-based model cannot learn the dynamics of the human skeleton properly.
In this work, we focus on the isolated SLR task. We propose a Skeleton Aware Multi-modal SLR framework with a Global Ensemble Model (SAM-SLR-v2) to explore the potential of skeleton-based SLR and fuse with other modalities in both RGB and RGB-D scenarios to further improve the recognition rate. Specifically, we design a new spatio-temporal skeleton graph for SLR using whole-body keypoints extracted by a pretrained whole-body pose estimator. Then we propose a multi-stream Sign Language Graph Convolution Network (SL-GCN) to model the embedded dynamics. To fully exploit the information in whole-body keypoints, we propose a novel Separable Spatial-Temporal Convolution Network (SSTCN) to learn from the whole-body skeleton features. Moreover, studies on action recognition reveal that data from different modalities complement each other, provide knowledge of latent correlation, and further improve the final performance. Although we can simply add predictions from all modalities together to achieve higher accuracy, we desire a method that tunes the best weight for each modality in a data-driven way. Hence, we propose a Global Ensemble Model (GEM) to automatically learn the multi-modal ensemble and improve the overall recognition rate. Our main contributions can be summarized as follows: • We construct novel 2D and 3D skeleton graphs designed for SLR using a pretrained whole-body pose estimator and graph reduction, which requires no extra annotation effort. • We propose a novel SL-GCN to model dynamics in the skeleton graphs. To our best knowledge, this is the first successful attempt to tackle the SLR task using 2D/3D whole-body skeleton graphs that surpasses RGB-based methods. • We propose a novel SSTCN to further exploit wholebody skeleton features. It can significantly improve the accuracy compared with the traditional 3D convolution.
• We propose an ensemble model GEM for both RGB and RGB-D based SLR, which learns weights from seven modalities and achieves state-of-the-art performance on three isolated SLR datasets with significant margins. A preliminary version of this work [25] has been reported in the corresponding workshop of the SLR challenge [26], during which we won the championships in both RGB 1 and RGB-D tracks 2 . Compared with our workshop version, we have made the following improvements: (1) We introduce a new modality Keypoint3D, which considers 3D coordinates in space and deal with occlusions. It improves the overall recognition rate of RGB-D ensemble. (2) We propose a new learning-based late-fusion ensemble method named GEM for multi-modal ensemble, which achieves higher recognition rates than our previous version and saves efforts on weights tuning.
(3) The test labels of the AUTSL dataset has been released, so we update our performance from the validation set to the test set. (4) Beyond the challenge dataset (AUTSL), we report our performance on two additional large-scale datasets for isolated (i.e., SLR500 and WLASL2000) compared with the recent state-of-the-art methods. Our approach significantly surpasses their performance with notable margins. (5) We update our figures, provide more details of models, analyze ensemble sensitivity, and discuss challenging cases, which may inspire future research on SLR.
II. RELATED WORK Sign Language Recognition (SLR) achieves significant progress in obtaining high recognition accuracy in recent years due to the development of practical deep learning architectures and the surge of computational power [15], [10], [13], [27]. Researchers have been modeling the spatio-temporal information in the videos using 2D CNN with RNN or 3D CNN. A 3D-convolutional SLR network associated with attention modules is proposed in [15] to learn the spatio-temporal features from raw videos. CNN, Feature Pooling Module, and LSTM Networks associated with adaptive weights are utilized in [10] to obtain distinctive representations. Besides, researchers have been extending isolated SLR to weakly supervised continuous SLR and Sign Language Translation (SLT) by incorporating sequence learning methods [28], [29], [14], [30], [31]. Recently, multi-cue methods using upperbody poses, hand keypoints, and mouth features have been developed to improve the recognition rate. For example, hand pose priors are introduced in hand-aware frameworks for further performance improvement in [17], [16]. A spatialtemporal multi-cue network for continuous SLR and SLT is proposed in [18]. However, these methods are still not effective enough to capture the complete motion information for robust sign language recognition. Skeleton-based Action Recognition mainly focuses on learning dynamic patterns from the motion of the human skeleton and effectively recognizing human action and activities [32], [33], [34], [35], [36], [37]. In the meanwhile, aiming for a higher recognition accuracy, it can collaborate with other Fig. 2. Construction of 2D and 3D graphs. A pretrained whole-body pose estimator is applied on an input RGB video (a) to obtain 133-point whole-body keypoints (b). We then construct a spatio-temporal 2D graph following the natural connections of the human body. The graph is reduced to 27-node (c) using graph reduction to mitigate noises. Each node in the 2D graph is represented by (x, y, s) where x-y are 2D coordinates and s is the confidence score. The 3D graph is obtained by overlaying the 2D coordinates on the depth video, reading the depth encoding z at the corresponding location, and treating it as an additional dimension as (x, y, z, s).
modalities (e.g., RGB, depth, and EMG) and benefit from multi-modal learning [38], [39], [40], [41], [42]. Recurrent neural networks (e.g., RNN and LSTM) are once popular in modeling the temporal information of skeleton data [33], [37], [36], [43]. Recently, a graph-based approach that implements a Graph Convolutional Network (GCN) to model the dynamic patterns in skeleton data is first explored by ST-GCN [19]. It draws much research attention that a few improved models are developed to further improve the performance [35], [44], [45], [46], [21], [22], [23]. Specifically, AS-GCN [35] digs the latent joint connections to boost the recognition performance. A twostream approach is presented in [45] and further extended to four streams in [21]. DecoupleGCN [22] increases the GCN capacity while introducing no extra computational cost. Inspired by ResNet [47], ResGCN [23] incorporates a bottleneck architecture to boost model capacity, reduce parameters, and avoid gradient vanishing. However, skeleton-based SLR methods are still under exploration. Simply applying the ST-GCN to SLR has been unsuccessful, which only reaches around 60% top-1 accuracy on 20 classes (much worse than RGB-based approaches) [48]. Multi-modal Approach aims to explore data captured from different resources, by different devices, and from distinctive views to improve the overall performance. Its motivation lies in an assumption that different modalities contain unique and view-specific information that complements each other.
The assumption suggests that multi-modal information can be fused together to further boost performance. Frameworks that learn robust and view-invariant feature representation for downstream tasks are proposed in [49], [50]. A weight-sharing model is developed in [51] to obtain modality hallucination for multi-modal image classification. DA-Net [52] adopts a view-independent module collaborated with a view-specific module to capture the multi-modal information and effectively merges the prediction scores. PM-GANs [53] utilizes a crossview generative strategy associated with a novel fusion to learn the prediction correlations effectively. A feature factorization network is proposed in [54] which learns the specific viewshared information for RGB-D action recognition. A cascaded residual autoencoder is designed to handle incomplete view classification settings [55]. Researchers also propose to use a super vector to fuse the multi-modal representations [56]. En-couraged by the success of those multi-modal approaches, we aim to incorporate more modalities (e.g., visual, depth, body gesture, and hand gesture) to capture both view-dependent and view-specific information from all aspects. We aim to learn via a universal framework to achieve higher performance.

III. METHODOLOGY
This section first gives an overview of the proposed SAM-SLR-v2 framework. Then we introduce the SL-GCN for keypoint graphs and the SSTCN for skeleton features, respectively. After that, we present an effective 3DCNN baseline model for the other modalities. Last, we describe the latefusion GEM, for the multi-modal ensemble.

A. SAM-SLR-v2 Framework Overview
An overview of the proposed SAM-SLR-v2 is shown in Figure 1. Seven modalities processed from RGB and depth videos are considered in the proposed framework. Three different architectures (i.e., SL-GCN, SSTCN, and 3DCNN) are used to extract features from the seven modalities and make predictions of sign language glosses independently. The latefusion ensemble model (i.e., GEM) takes predictions from all the modalities and outputs final predictions in the RGB and the RGB-D scenarios.

B. SL-GCN for Skeleton Keypoints
Graph Construction and Reduction. For skeleton-based action recognition, researchers rely on ground-truth skeleton keypoints annotated by motion capture systems such as Kinect v2 [57], which unfortunately is not capable of providing hand and finger annotations. Since hand gestures play a crucial role in performing sign language, we use a pose estimator pretrained with whole-body annotations to predict whole-body keypoints, which include 133 landmarks of face, body, hands, and feet. A spatial 2D graph can then be constructed by connecting every pair of adjacent keypoints according to the human-body natural connections. This graph is further extended to a spatio-temporal graph by connecting all the nodes to themselves in the temporal dimension. Mathematically, the , the minimum number of nodes) between Node v i and v j in the shortest path. However, contrasting with the graph used in skeleton-based action recognition which contains around 17 nodes, the wholebody skeleton graph contains too many nodes and edges which introduce high-level unexpected noise. Besides, if the distance between two nodes is too far (i.e., have many nodes in between), it is inaccurate to explore their interactions. Our experiment shows that simply using the whole-body skeleton graph results in lower accuracy. Therefore, we operate a graph reduction on the whole-body skeleton graph which trims down the 133 nodes to 27 nodes based on our observations of the videos and visualizations of the GCN activation. The resulted graph consists of seven nodes for the upper body (nose, eyes, shoulders, and elbows) and ten nodes for each hand, as illustrated in Figure 2(c). Graph reduction leads to faster model convergence and significantly higher recognition rates. Each node in the 2D graph is represented by (x, y, s) where x-y are 2D coordinates and s is the confidence score. When depth information is available, we construct a 3D graph by reading the corresponding depth z at keypoint locations x-y and treating it as an additional dimension as (x, y, z, s), as illustrated in Figure 2(d). Graph Convolution. We adopt the spatio-temporal graph convolution with the spatial partitioning strategy [19] to capture the embedded dynamics in the whole-body skeleton graph. We implement the spatial graph convolution as where A represents an adjacent matrix of intra-body connections, I represents an identity matrix of self-connections, D stands for the diagonal degree of (A + I), and W is the trainable weights of the convolution. Practically, Equation 2 is implemented by performing standard 2D convolution and multiplying the results by D − 1 2 (A + I)D − 1 2 . To perform temporal graph convolution, we implement a standard 2D convolution on the temporal dimension with a kernel size k t ×1 as the reception field. We adopt an extended variation of the spatial GCN called decoupling GCN [22]. In a decoupling GCN layer, to further boost the model capacity, the extracted features are grouped into G groups that each group has its trainable adjacent matrix A. We then concatenate the outputs of all groups back together as the output features. Multi-stream SL-GCN. Inspired by [21] which adopts a multi-stream workflow for action recognition, we find that it is also worth investigating 1st-order coordinates (joints), 2ndorder vector (bone vector), and their motion vectors for sign language recognition, as shown in Figure 3(a). Following the natural connections of the human body, we generate the bone vectors pointing from the starting joints to their ending joints. Thus the bone stream is represented by a tree graph where the nose acts as the root node. Mathematically, the starting and ending joints can be represented as where (x, y, [z], s) represents 2D coordinates, optional depth, and the confidence score. The bone vectors v B are calculated by subtracting Equation 3 from Equation 4 as where the set H contains all natural connections of the human body. Motion streams are obtained by subtracting the difference between adjacent frames. The joint motion vectors v JM and the bone motion vectors v JM are represented as We train every stream separately, multiply their predictions by assigned weights, and summing up the results as the final prediction. SL-GCN Structure. The structure of our proposed SL-GCN is presented in Figure 3(b). The input stream is permuted and normalized before feeding into N instances of SL-GCN units for spatio-temporal graph convolution. An average pooling is then applied to the temporal dimension of the resulted features. The result is reshaped and fed into a fully connected layer for classification. Our proposed SL-GCN unit is illustrated in Figure 3(c). We find that deep graph models are easier to overfit on the video classification task and the ordinary dropout layer works poorly in GCNs. We propose to construct the basic SL-GCN Unit with a decoupled spatial convolutional layer (DecoupleSCN) [22] to mitigate overfitting. We also introduce an STC (spatial, temporal, and channel-wise) self-attention mechanism inspired by [21]. As illustrated in Figure 3(d), the STC module consists of modules of spatial attention, temporal attention, and channel attention connected in a cascaded configuration. In our experiment, we use N = 10 such SL-GCN units in the proposed SL-GCN.

C. SSTCN for Skeleton Features
We propose an SSTCN model to exploit the whole-body features in addition to the keypoint coordinates. We can learn from ResNet2+1D [58] that the performance of an action recognition model can be further improved via factorizing the network into a temporal part and a spatial part. Therefore, our SSTCN model is separated into four stages to handle the features from different dimensions. The pipeline is shown in Figure 4. We save the features of 33 keypoints including 1 nose keypoint, 4 mouth keypoints, 6 upper-body keypoints, and 22 hands keypoints before the argmax operation in the pose estimator. We then uniformly sample 60 frames in each video for SSTCN for recognition. The saved features are resized to 24 × 24 using maximum pooling. We process the input features with a 2D separable convolution layer that reduces the parameters and converges easily. At Stage one, features are reshaped from 60×33×24×24 to 60×792×24, and fed to 1×1 convolution layers for temporal convolution. At Stage two, the extracted features are then shuffled and grouped into 60 groups for 3×3 grouped convolution to extract spatial features of each frame. At the next stage, we start processing the features in the feature dimension. Specifically, the features are shuffled again and grouped into 33 groups. We use grouped 3×3 convolution on spatial and temporal dimensions for each keypoint. At the last stage, a few fully connected layers are used to make final predictions. We adopt a residual path for the first three stages to avoid gradient vanishing. Besides, random dropouts are deployed in every module to avoid overfitting [59].
Researchers have found that one-hot labels and crossentropy loss may be easy to overfit with limited data [60]. So we use the technique of label smoothing to avoid overfitting. Mathematically, label smoothing is defined as where k is the number of classes, q (k|x) is a label-smoothed predicted distribution, is a hyper-parameter in (0, 1), and u(·) represents the uniform distribution. The cross-entropy loss is then modified as where H(·) is the cross-entropy function and q is the real data distribution, and p stands for the predicted distribution. The modified cross-entropy can be explained as the penalty to the difference between the predicted distribution and the real distribution combined with the difference between the predicted distribution and a prior distribution (e.g., uniform distribution). To further improve the performance, we replace all activations to the Swish [61] activation function as

D. 3DCNN Baselines for the Other Modalities
As mentioned in Section II, studies reveal that ensembles from multiple cues and modalities could further improve the overall performance. To benefit from the other modalities (i.e., RGB frames, optical flows, HHA, and depth flow), we hence build a simple yet effective baseline using 3D CNNs. Most 3D CNN architectures are easy to overfit, especially on smaller datasets. In our experiment, The ResNet2+1D [58] architecture that separates the temporal and spatial convolution of 3D CNNs, provides the best result compared with other popular 3D CNN networks. We also notice that deeper model depth does not always lead to better performance while making the network to overfit easier. So we choose the ResNet2+1D-18 variation pretrained on the Kinectics dataset [62] as our backbone. Apart from that, to improve the accuracy further, we pretrain the model on the largest available SLR dataset SLR500 [63] for RGB frames. Pretraining increases the final accuracy by around 1% and improves the model convergence. Similar to the SSTCN presented in Section III-C, we adopt the Swish activation described in Equation 10 instead of using ReLU. Besides, to mitigate overfitting, we implement the label smoothing technique in Equation 8 with the corresponding cross-entropy loss in Equation 9.

E. Multi-modal Late-fusion Ensemble
Model-free Late Fusion. In our previous version, we use a simple late-fusion approach to fuse predictions from all modalities. To be specific, for all modalities, we save the predictions from the last fully connected layers before the softmax. We assign weights to all modalities manually and add them up as the final prediction q RGB-D =α 1 q skel3D + α 2 q RGB + α 3 q flow + α 4 q feat where q represents the predictions before softmax, α 1,2,3,4,5,6 are hyper-parameters to be tuned based on validation accuracy.  as illustrated in Figure 6. The whole process can be described as where q represents the results of each modality, G represents the prediction procedure of GEM, and [·] stands for the concatenation operation. The first layer of the model is used to down-sample the modalities effectively with a global convolution. Therefore, the kernel of the first layer is C × 1.
Then we use 1 × 1 fully connected layers to predict the final weights of all modalities. The final weights are multiplied with the inputs respectively and the weighted modalities are summed up as the final prediction.

IV. EXPERIMENTS
In this section, we report the evaluation results of our proposed SAM-SLR-v2 on three major sign language recognition datasets of different sign languages. We show the singlemodality performance of our proposed multi-stream SL-GCN, SSTCN, and 3DCNN models, as well as different combinations of their ensembles. We demonstrate the effectiveness of proposed approaches via ablation studies on the components of SL-GCN, SSTCN and 3DCNN networks. We also study the proposed GEM compared with simple model-free late fusion. Last, we show some challenging cases and discuss the limitations of SAM-SLR-v2.
A. Datasets AUTSL Dataset [64] is a Turkish SLR dataset collected using Kinect V2 sensor [57], [67]. Statistically, 226 different sign glosses are performed by 43 signers with 20 backgrounds. The dataset contains 38,336 videos that split into training, validation, and testing subsets. We use their currently released balanced test set in our experiments and report both top-1 and top-5 recognition rates. SLR500 Dataset [63] is a balanced Chinese sign language dataset for isolated sign language recognition (sometimes referred to as CSL isol.) that contains 500 words performed by 50 signers. Each word is performed by all 50 signers 5 times, so there are 125,000 videos in total. This dataset is captured in a controlled lab environment with a solid-color background. The former 36 signers are used for training and the later 14 signers are used for testing. WLASL2000 Dataset [11] is a American Sign Language with a vocabulary size of 2000 words. It is a challenging dataset collected from web videos performed by 119 signers. It contains 21,083 samples with unconstrained recording conditions. The dataset is imbalanced and the average samples per video are much lower than the above two datasets. We follow the signer-independent settings for all the above datasets. Besides, we are also aware of other large-scale isolated sign language datasets. Microsoft American Sign Language dataset (MS-ASL) [65] is an American sign language dataset that is no longer publicly available due to the deletion of the YouTube-hosted videos. A new large-scale British Sign Language dataset (BSL-1K) [66] has not been released to the public yet at the moment of writing. A statistical summary of the mentioned datasets is provided in Table I.

B. Multi-modal Data Preparation
Whole-body Pose Keypoints and Features. MMPose [68] provides a whole-body pose estimator pretrained with wholebody keypoint annotations and HRNet [69] as its backbone. We use the pretrained model to obtain 133 keypoints from RGB videos, based on which we construct the 27-node 2D graph in Section III-B and process the 3D graphs using depth videos. Keypoint coordinates are normalized to [-1,1]. Random sampling, mirroring, rotating, scaling, jittering, and shifting are applied as data augmentations. We use a sample duration of 150 frames. We repeat videos with lesser than 150 frames to 150 frames. To obtain skeleton features for each video, we downsample the estimated heatmaps, choose 33 joint channels, and uniformly sample 60 frames. RGB Frames and Optical Flow. We extract all frames from RGB videos to load and process them faster in parallel. Besides, we use the Denseflow API implemented with OpenCV and CUDA that provided by OpenMMLab [70] to obtain the optical flows using the TVL1 algorithm [71]. The output x and y flow maps are concatenated in the channel dimension. We cropped all RGB and optical flow frames to the bounding boxes from the pose estimator, and then resize them to 256 × 256. When we train the model, we sample 32 consecutive frames randomly from the input video. When testing, we sample five such clips uniformly from the input videos and average on their predicted scores. Depth HHA and Depth Flow. HHA stands for the horizontal disparity, height above the ground, and angle normal that encode a depth map into a three-channel RGB image. Because HHA features enable better scene understanding, we extract HHA features from depth videos as another modality instead of inputting gray-scale depth maps directly. The depth videos of the AUTSL dataset come with masks, so we mask out those regions and fill them with zeros when generating HHA. An HHA example is shown in Figure 5(c), where black regions are masked out pixels. Cropping and resizing operations are performed the same way as RGB frames. We also apply the same data augmentations to HHA features as the RGB modality. Apart from that, we use Denseflow to extract optical flow from the depth modality as well. A sample of depth flow is shown in Figure 5(e). Compared with optical flows (e.g., Figure 5(d)), depth flows contain less noise and capture distinctive motion information.

C. Performance of Multi-stream SL-GCN
Both top-1 and top-5 recognition rates of the proposed multi-stream SL-GCN are reported in Table II. Among the four streams, the joint stream provides the best accuracy. The multi-stream approach further improves the overall accuracy, which demonstrates that our proposed whole-body skeleton graph and multi-stream SL-GCN are very effective. Table IV shows that SL-GCN (Keypoints 2D and 3D) perform the best in all single-modality methods. Since the skeleton graphs are less complicated, the graph-based method is lighter-weight and much faster to inference compared with large-capacity 3DCNN-based models, which is another advantage of the proposed SL-GCN. Table III presents ablation studies on the proposed SL-GCN. We find that the graph reduction technique contributes the most to the recognition rate. Without that, it can barely learn from the noisy dynamics in the 133-node skeleton graph. Due to the limitation of annotated data, the model is easy to overfit, thus the data augmentation techniques are also important in learning the embedded dynamics. Besides, we find that the DropGraph module, the decoupling GCN module, and the STC attention mechanism all contribute to the final performance.

D. Multi-modal Performance on AUTSL Dataset
The results of all single-modal methods on the AUTSL balanced test set are reported in Table IV. The Keypoints 2D and Keypoints 3D method represents our proposed multistream SL-GCN using 2D and 3D skeleton graphs. They perform the best compared with the other methods. We also find that the depth flow achieves a little better recognition rate compared with the RGB flow resulted from lesser noisy data. The fused ensembles in both RGB and RGB-D scenarios using different choices of modalities are summarized in Table V as three groups. The skeleton-based ensemble (Skel 2D and Skel 3D) achieves better accuracy than "RGB + Flow" and Depth ensemble (HHA + Depth Flow), which demonstrates that the proposed skeleton-based methods are very effective in SLR. The RGB and RGB-D ensemble results that use all available modalities show that the skeleton-based methods also complement RGB and depth modalities. Their collaboration further improves the overall performance. Comparisons with other players in the challenge are reported in Table VI. Our improved SAM-SLR-v2 achieves better recognition rates compared with its previous version SAM-SLR that won the championships of the SLR challenge. During the challenge, we are allowed to use the validation labels to fine-tune our models.
Our results with fine-tuning on the validation set are shown at the bottom. Our results without fine-tuning still surpass the other methods fine-tuned with validation labels. Our method achieves the state-of-the-art with significant margins in both RGB and RGB-D scenarios.

E. Multi-modal Performance on SLR500 Dataset
Performance on the isolated Chinese sign language dataset (SLR500) is reported in Table VII. Since this dataset is collected in a lab-controlled environment with a large number of repetitions and invariant background, all modalities obtain higher recognition rates compared with the other two datasets. For the same reason, compared with skeleton-based methods, RGB-based 3DCNN achieves a higher Top-1 recognition rate but a lower Top-5 recognition rate. Since the SLR500 dataset

Modality
Top-1 Top-5 GLE-Net [73] 96.80 -Hand + RGB [16] 98. provides only a validation set instead of train-val-test splits, we use the weights learned from the AUTSL dataset to obtain the final ensemble results. Compared with the recent state-ofthe-art method GLE-Net [73] and Hand+RGB [16], our twomodality methods and all-modality SAM-SLR-v2 all achieve much better recognition rates.

F. Multi-modal Performance on WLASL2000
The WLASL2000 dataset is the most challenging variation of the WLASL dataset with unconstrained backgrounds and camera conditions. Since the dataset is not balanced, we report both per-instance accuracy as well as per-class accuracy following [65], [16]. As reported in Table VIII, the keypoints modality performs the best (51.50% per-instance) among all the single-modal methods due to its independence on backgrounds and resistance to noises. The keypoints and RGB modalities complement each other and boost the twomodal performance. Using all the available modalities, the top-1 performance is improved to 57.55% per-instance. Our proposed late ensemble model is capable of raising the recognition rates by around 0.7%. Compared with other state-ofthe-art methods, our proposed SAM-SLR-v2 achieves the best recognition rates in both per-instance and per-class metrics with large leading gaps of around 8%.

G. Ablation Study on SSTCN and 3DCNN
An ablation study on SSTCN is provided in Table IX. Compared with ResNet3D [74], [75] and ResNet2+1D [58] on  the same feature size, SSTCN is more effective on recognizing sign glosses using skeleton features. Our study also reveals that the proposed SSTCN can achieve even higher accuracy with larger input feature sizes. Table X shows an ablation study of 3D CNN architecture using the RGB frames. It shows that both swish activation and label smoothing are effective, where they improve the top-1 accuracy by 2% and 1%, respectively. Pretraining on the CSL dataset [63] improves the overall accuracy by 1.4%. The ResNet2+1D-18 backbone provides 1.7% better performance than ResNet3D-18.

H. Comparison between Multi-modal Ensembles
The model-free simple ensemble method requires a lot of effort in tuning the weights of different modalities. With our proposed late-fusion GEM, those weights are automatically learned from the data. We study the effectiveness of GEM compared with the simple ensemble method. In Table VI, VII, and VIII, performance of model-free simple ensemble is shown as SAM-SLR. Compared with the simple ensemble, our proposed late-fusion GEM can improve the overall performance as well as avoid hard tuning on modality weights.

I. Sensitivity Analysis of Ensemble Methods
A sensitivity study on ensemble methods is conducted on the AUTSL test dataset. In the experiment, we use the manually tuned parameters in Section III-E as our basis and change one parameter at a time (varying from 0.0 to 2.0), while keeping the other parameters fixed. We plot the resulted top-1 accuracy to analyze the sensitivity of those ensemble parameters. In the RGB scenario, we have four modalities and we analyze the sensitivity of α 1 to α 4 , as shown in Figure 7(a). In the RGB-D scenario, we analyze the sensitivity of α 1 to α 6 , as shown in Figure 7(b). In this experiment, the ensemble parameters tuned on the validation set do not provide the highest performance on the test set. The ensemble performance is sensitive to the values of the parameters that small changes may lead to large variations in the ensemble accuracy. Moreover, when there are more modalities (e.g., in the RGB-D scenario), it is hard to optimize all those parameters manually. Our proposed late-fusion GEM solves this problem by automatically learning these parameters from data. Figure 7(c) shows the top-1 accuracy during training the proposed model. Initially, the ensemble model gives sensitive results due to its large search space. The model converges after 100 epochs and delivers steady and accurate ensemble results. In summary, the proposed late-fusion GEM is very effective in learning the multi-modal ensemble, gives robust higher performance than simple ensemble, and saves a lot of effort to tune the best parameters. Figure 8 shows some challenging cases of sign language recognition from the AUTSL dataset. The offline full-body pose estimator may fail due to off-screen or occlusion, especially for fingers, which is a common issue for pose estimation methods. However, fingers play a critical role in expressing signs. Some of those failures can be corrected by RGBbased features. That is the reason why multi-modal ensemble significantly boosts the recognition rate. Moreover, a same sign may be performed very differently by different signers (e.g., signers may use their left hands, right hands, or both hands to perform the same gloss). Hence, mirror augmentation is very important in model training, but more data collected from more signers are desired. Last, opposite signs could be visually similar in a single frame (e.g., heavy vs light-weight). Distinguishing between those signs requires delicate modeling of spatio-temporal dynamics. In our experiment, we find that skeleton-based methods are smarter choices over RGB-based methods.

V. CONCLUSION
In conclusion, we propose a novel SAM-SLR-v2 framework to learn multi-modal feature representations from RGB-D videos towards more effective and robust isolated SLR. Among those modalities, our proposed skeleton-based methods are the most effective in modeling motion dynamics due to their signer-independent and background-independent characteristics. Specifically, we construct novel 2D and 3D spatio-temporal skeleton graphs using pretrained whole-body keypoint estimators and propose a multi-stream SL-GCN to model the embedded motion dynamics. Our method does not require any additional effort on annotating hands and provides more reliable hand keypoint estimation than off-line hand detectors. Besides modeling motion dynamics of keypoint coordinates via graphs, we propose SSTCN to predict using skeleton features. Furthermore, we study the multi-modal fusion problem based on the other modalities (i.e., RGB frames, optical flow, HHA, and depth flow) via a learning-based late-fusion ensemble model named GEM. Experimentally, we show that our proposed SAM-SLR-v2 framework achieves the state-of-the-art performance on three challenging datasets for isolated SLR (i.e., AUTSL, SLR500, and WLASL2000) as well as won the championships in both RGB and RGB-D tracks during the CVPR 2021 challenge on isolated SLR. We hope our work could facilitate and inspire future research on SLR.


extfieldsofstudy:
Computer Science
Engineering

text:
AISHELL-3: A Multi-speaker Mandarin TTS Corpus and the Baselines

In this paper, we present AISHELL-3, a large-scale and high-fidelity multi-speaker Mandarin speech corpus which could be used to train multi-speaker Text-to-Speech (TTS) systems. The corpus contains roughly 85 hours of emotion-neutral recordings spoken by 218 native Chinese mandarin speakers. Their auxiliary attributes such as gender, age group and native accents are explicitly marked and provided in the corpus. Accordingly, transcripts in Chinese character-level and pinyin-level are provided along with the recordings. We present a baseline system that uses AISHELL-3 for multi-speaker Madarin speech synthesis. The multi-speaker speech synthesis system is an extension on Tacotron-2 where a speaker verification model and a corresponding loss regarding voice similarity are incorporated as the feedback constraint. We aim to use the presented corpus to build a robust synthesis model that is able to achieve zero-shot voice cloning. The system trained on this dataset also generalizes well on speakers that are never seen in the training process. Objective evaluation results from our experiments show that the proposed multi-speaker synthesis system achieves high voice similarity concerning both speaker embedding similarity and equal error rate measurement. The dataset, baseline system code and generated samples are available online.

INTRODUCTION
Speech synthesis, or Text-To-Speech(TTS), is the automated process of mapping input text specifications to target utterances [1]. In recent years, neural network based TTS synthesis systems have achieved marvelous results in terms of audio quality and perceptual naturalness [2]. This flourishing research progress is made largely due to the introduction of attention based sequence-to-sequence modeling architectures such as Tacotron [2,3] or Transformer-TTS [4], and neural vocoders that map the lower dimensional acoustic representation to waveforms [5,6,7].
A key characteristic of TTS is the lack of constraint, which renders the task essentially a one-to-many mapping [1]. Since given only textual content, speeches uttered by either male or female, with voices agitated or neutral, are equally valid outputs. But real-world application of such systems requires robust and consistent behaviors. This begs the question of whether we could provide further specification to the system to gain more flexibility over conventional approaches. There is a growing interest within the field in designing TTS systems that are more flexible and admits stronger constraints on its behaviors. Recent publications on expressive or prosodic TTS systems tend to associate the acoustic model with explicit control signals (e.g., pitch/energy for supervised settings [8] and learned embeddings for unsupervised variants [9,10,11]) as augmented input besides normalized texts. A more prominent and intuitive feature of speech is the speaker identity, and multi-speaker acoustic models give TTS systems the ability to disentangle perceptual speaker identity from the textual contents of the synthesized utterance by explicitly conditioning the model on the desired speaker [12,13,14,15].
Training such systems naturally requires significant amount of annotated data. VCTK [16] is a freely available multi-speaker corpus that could be used to train such systems. However, VCTK only contains recordings in English. As suggested by previous studies [17,18], despite the cultural influence of English language as a lingua franca in the academia, language specific subsystems and model modifications are indeed an area of active research. TTS systems targeted on tonal languages such as Chinese Mandarin and Japanese face difficult situations considering their complex tonal and prosodic structures [19]. The lack of a publicly available multi-speaker Mandarin dataset suitable for TTS system training makes researches in this area more difficult and costly, and lacks objective indicators that are comparable across studies.
To this end, we introduce the AISHELL-3 corpus in this paper to fill this vacancy in open resources. AISHELL-3 contains roughly 85 hours of high fidelity Mandarin speech recordings from 218 native speakers, with manually transcribed Chinese characters and pronunciations in the form of pinyin notation. Furthermore, we present a multi-speaker TTS system trained with this dataset as a baseline system. Objective evaluations on the synthesized samples show consistent behavior with previous studies conducted on a VCTK system with the same architecture.

THE AISHELL-3 DATASET
The AISHELL-3 dataset is a multi-speaker Mandarin Chinese audio corpus, which could be used to train multi-speaker TTS systems. There are in total 88035 recordings from 218 native speakers reading off text from given scripts with neutral emotions. All utterances are recorded in a quiet indoor environment using high fidelity microphones (44.1 kHz, 16 bit depth), which are 20 cm away from the speaker. The topics of the textual content spread a wide range of Each recording contains accompanying transcripts in both Chinese characters and pinyins, which is the official latinized notation for marking Chinese pronunciations. The pinyin transcripts are obtained through human listening tests and directly corresponds to the speaker's actual readings, which addresses four key difficulties in automatically deriving them from the Chinese characters via dictionary lookups: 1. Homograph. Some characters could be pronounced in multiple ways depending on the textual context they resides [20]. 2. Tone sandhi. Some tones shift under certain phonological contexts, a good example being, normally characters in the initial part of consecutive third tones shift to the second tone, e.g. guan3 li3 (to manage) should be pronounced guan2 li3, but this rule does not apply to all such circumstances. 3. Erization (erhua). The Chinese character for son (pronounced er2) behaves like a normal character in some contexts, but it also acts as an erization marker indicating the preceding character has an erized final. 4. Accents and Mispronunciations. The above difficulties make the phonetizing process non-trivial and the manually labeled transcripts valuable for obtaining a pure dataset for training high quality TTS models and further development of more sophisticated conversion systems.  Table 1 and table 2 show the distribution of some basic attributes across the entire dataset. It is worth mentioning that speaker genders are biased towards females, and the age distribution centers on young adults of around 20 years old.

BASELINE SYSTEM
The baseline system is a composition of three independently designed components: a textual-frontend, which performs necessary analysis and conversion on the textual input, an acoustic model, which maps textual and speaker specifications to a series of feature vectors, and a neural vocoder to decode the feature sequence into audio waveforms.

The Speaker-agnostic Subsystem
The textual-frontend and vocoder modules are implemented as speaker-agnostic considering the textual analysis and acoustic feature inversion processes involved in the TTS pipeline have lower correlations to speaker identities than that of the acoustic model. The textual-frontend, which is used to preprocess the datasets' raw labels, is composed of a pinyin-parser, a phonetizer, and a LSTM based prosody prediction model. We use a MelGAN [7] model trained with recordings from the AISHELL-3 dataset regardless of speaker labels as the neural vocoder.
In addition, we've found that Tacotron-2 alone, when naively trained with only Chinese phoneme or pinyin inputs, tends to produce synthetic samples with monotonic prosody. Considering the prosody have a high correlation with the naturalness in mandarin synthesis, we apply an RNN based prosodic label prediction model trained on a corpus labeled with prosodic taggings [21]. Then the prosodic annotations are added into the phoneme sequence [22] as the final input sequence to the acoustic model. The prosody label prediction task is formulated as a sequence tagging problem: the input sequence is a concatenation of learnable character embeddings, BMES-tagged [23] word segmentations and part-of-speech tags. The input sequence is first encoded with 2 layers of bi-directional LSTMs and then decoded by 2 fully-connected output-layers to produce BMES-tagged prosodic predictions. The prediction is made in a hierarchical fashion where the prosodic phrase is conditioned on wording predictions in light of the hierarchical nature of the structure of the prosodic units.

The Speaker-aware Subsystem
We follow a recently published work on multi-speaker TTS incorporating speaker-embedding feedback constraint [15] to build our speaker-aware acoustic model. It is a Tacotron-2 based architecture with an extra speaker-encoder module as illustrated in figure 2.

Tacotron-2 backbone
Tacotron-2 follows the basic structure of a sequence-to-sequence architecture, which is composed of an encoder, a decoder and an attention mechanism in between. The encoder module of the tacotron-2 model is a CNN-BiLSTM based neural network that maps input texts into a sequence of hidden states. This hidden state sequence is then used as values in an attention mechanism to calculate the context vector for every step of the decoder module. The attention mechanism used by Tacotron-2 is the hybrid attention, which is a form of stateful additive attention that leverages both content and location information of the decoding process. The decoder module of Tacotron-2 can be decomposed into three parts: prenet, postnet, and the decoder RNN. for every time-step in the decoding process, the mel-spectrum from the previous step is mapped to a lower dimensional hidden representation by prenet, which is a series of reluactivated dense layers. The processed frame and context vector are then passed to two layers of zone-out LSTMs [24], whose hiddenstate is used by the attention mechanism to produce the current context vector. Finally, the hidden state and context vector is concatenated and fed into two projection layers to form the output acoustic feature frame for the current time-step and the stop-token, which signals whether the last feature frame has been produced in inference stage. The postnet is itself two layers of 1d-convolution, and the final output feature frames are obtained by passing the decoded frames through postnet together with a residual connection.

Speaker encoder
The speaker encoder is used to extract discriminative speaker information from variable length reference audio inputs, which in our  Fig. 2. The speaker-aware subsystem case is always the ground-truth audio during training. We follow [25] to implement a Resnet-based speaker verification network with global mean-std pooling as the speaker encoder for our baseline multi-speaker system. The speaker embedding vector used to condition our acoustic model is a linear-projection of the pooled mean and standard deviation feature vectors.

Feedback constraint
We've learned from previous studies that concatenating speaker embedding vectors on encoder hidden states alone does not impose a strong enough incentive on the acoustic model to produce high fidelity results in terms of speaker similarity, especially when the reference audio is never seen in the training data [14]. Recent work on speaker identity feedback constraint shows that by adding an additional loss term between reference embedding and the extracted embedding of the synthesized signal, the robustness and speaker similarity of the produced speeches are effectively increased [15]. We follow the system description from the aforementioned study to modify our baseline multi-speaker acoustic model. As shown in Figure 2, each optimization step in the training process includes an additional speaker embedding cosine similarity loss term. The parameters in the speaker encoder module are pre-trained, and frozen during the training process of the acoustic model. The final loss function includes several terms: Where M SE mel refers to the mean square error between the predicted and ground-truth mel-spectrograms, the BCEstop term denotes the binary cross-entropy on the stop-token, and the last term denotes the cosine similarity between synthesized and ground-truth speaker embeddings. The weight hyper-parameter α is set to 1.0.

EXPERIMENTS
We implemented and trained the baseline synthesis system and performed evaluations on the synthetic samples. The acoustic model and neural vocoder are trained using the presented dataset, and the speaker encoder module is trained separately using 775289 utterances from the aidataTang [26] and MAGICDATA Mandarin [27] corpus. The detailed data preparation process for the acoustic model and evaluation results are described in this section.

Down-sampling
To match the acoustic feature used for training the pre-trained speaker encoder, we downsample the data to 16kHz sample-rate, and uses window and hop lengths of 50 ms and 12.5 ms respectively.

Train-test separation.
The training and testing data division is in broad terms made randomly based on speaker identities. Of the presented 218 speakers, 44 were drawn randomly from the population to form the unseen speakers test-set. The remaining 174 speakers are used to train the multi-speaker model described in chapter 3. It is worth noting that not all samples from the training set are used in model training, a portion of the utterances for each speaker in the training set are kept for seen speakers validation, The resulting train-set contains 64773 utterances, which is around 60 hours long.

Silence trimming
In our proposed system, the attention alignment formed between encoder hidden states and decoder decoding steps directly controls the timing and rhythms of the synthesized utterance. However, since the alignment itself is an unsupervised by-product of the model optimization process, the stage in which a good alignment is formed could effectively affect the overall quality of the trained synthesis model. This correlation is demonstrated through experiments in [28]. In our preliminary studies, we've found, by controlled experiments, that trimming the silence segments at the initial positions of train samples exceedingly speeds up the alignment formation during acoustic model training. So we performed energy-based VAD on all train samples' Mel-spectrogram and trimmed all initial silence segments. This led to an alignment formation speedup of 10 times in terms of optimization steps under 2 GTX-1080Ti GPUs.

Long-form sentence augmentation
It is observed that when the target sentence is too long during the inference phase, the synthesized utterance displays unstable attention alignment or unnatural speaking rate. This is due to the poor generalization ability of the hybrid-attention mechanism in long-formed synthesis commonly seen in Tacotron-2 models [9,29]. Preliminary experiments show that by leveraging purely location-based attention mechanisms [28], this generalization problem is effected lifted at the cost of prosodic naturalness. However in our case, we turn towards data augmentation as a solution to this problem.
The average sentence length of the entire dataset is 11.3 characters, and a fair number of samples cover up to more than 20 characters long. We concatenate the acoustic feature and textual labels of N randomly selected sentences drawn from the same speaker to produce a longer training sample, where N is a discrete r.v. drawn from the distribution (P2 = 0.6, P3 = 0.2, P4 = 0.2). Here Pn denotes the probability mass function fN (·) of N . By this augmentation, we generated 30000 samples as the augmented training data.
The augmented data is used to fine-tune the TTS model converged on the original dataset.

Objective Evaluation
We perform objective evaluations to assess synthetic samples produced by the baseline model in terms of speaker similarity. The evaluation is performed on two groups: the validation set that contains the same speakers as the training set, which reflects the model's ability to mimic the voice of speakers seen in the training process, and the test-set, which shows how well the model generalizes on reference speakers that are unseen during training. In both trials, we synthesize 20 text-dependent and 20 textindependent utterances for every speaker, where text-dependent means the ground-truth audio with the same textual content is used to extract the speaker embedding, while text-independent is using the speakers' respective mean embedding vectors as the reference. The textual content of the synthesized samples are not seen in the training stage. In addition, we synthesize 3 versions per each sample to account for the randomness introduced to the inference stage by the dropout layers in the prenet module. By assessing the resultant attention maps, we eliminate the mis-aligned samples. This is done by calculating cosine similarities of the attention score vectors between output steps, since a natural alignment would conform to a nearly diagonal trend, resulting in lower inter-step similarities. The speaker representation of each selected synthetic sample is extracted for the following measurements.
T-SNE Plot. We visualize the extracted speaker representation space as 2-D scatter plots using t-Distributed Stochastic Neighbor Embedding (t-SNE). Since t-SNE plot models relative distance be-  figure 3 represent groups of utterances that are close in terms of speaker identities. We observe clearly that embeddings from the same speaker forms dense clusters that are well separated from other speakersubspaces. Cosine Similarity. We use cosine function to measure speaker embedding vector similarity, which is a common method used in speaker verification systems. We measured the cosine similarity between synthesized speechs' embeddings and embeddings extracted from ground truth audio samples. A higher value means higher similarity between synthesized and recorded samples as is considered by the speaker verification system. SV-EER. We also make use of the concept of Speaker Verification (SV) Equal-Error-Rate (EER) as an objective evaluation index. EER is ubiquitously seen in SV tasks as a pointer to system performance, and it could also be interpreted as a measurement for the quality of a multi-speaker speech synthesis system. Perfect TTS systems would produce results that are indistinguishable from real data. To evaluate the system using the EER measure, we draw 10,000 pairs of samples from a pool of audio samples per trial. Table 3 presents our results and a similar experiment performed on the VCTK dataset published in [15]. It's observed that both validation-set and test-set speakers achieved high average cosine similarities between synthesized and real samples. And the EERs for validation speakers are only 0.65% and 0.60% higher than the SV-baseline. There is a performance drop of around 5% for EER between test-set and validation-set speakers, which is consistent with results obtained on VCTK although the two speaker embedding networks are trained using different data.

CONCLUSION
In this paper, a new publicly available Mandarin speech corpus that could be used to train multi-speaker TTS systems is presented. The dataset contains 88035 high-fidelity utterance recordings collected from 218 native speakers and includes hand-labeled full pinyin annotations. Moreover, a baseline multi-speaker TTS system based on speaker embedding feedback constraint is presented and described at length. We analyzed the characteristics of the presented dataset and incorporated a number of augmentation procedures, such as prosodic label prediction, silence trimming and long-form sentence concatenation to the data preparation process. The trained model is further evaluated in terms of speaker similarity and generalization capacity. And we found our presented multi-speaker TTS system is capable of producing natural speeches that imitates the voice of reference speakers. We draw the conclusion based on both objective and perceptual experimental results, that the presented corpus is valuable for the multi-speaker TTS research in Mandarin.