extfieldsofstudy:
Computer Science
Engineering

text:
Classification of Audio Segments in Call Center Recordings using Convolutional Recurrent Neural Networks

Detailed statistical analysis of call center recordings is critical in the customer relationship management point of view. With the recent advances in artificial intelligence, many tasks regarding the calculation of call statistics are now performed automatically. This work proposes a neural network framework where the aim is to correctly identify audio segments and classify them as either customer or agent sections. Accurately identifying these sections gives a fair metric for evaluating agents' performances. We inherited the convolutional recurrent neural network (CRNN) architecture commonly used for such problems as music genre classification. We also tested the same architecture's performance, where the previous class information and the gender information of speakers are also added to the training data labels. We saw that CRNN could generalize the training data and perform well on validation data for this problem with and without the gender information. Moreover, even the training was performed using Turkish speech samples; the trained network was proven to achieve high accuracy for call center recordings in other languages like German and English.

Introduction
Telephony has been an essential tool used by companies for marketing and customer satisfaction. Moreover, if the services or products already meet customer needs, telephone calls guarantee customer satisfaction and loyalty for any company [10]. A thorough analysis of telephone calls and improving overall call performance is the most critical job for quality managers of companies that use telephony as a significant component of their customer relationship management (CRM) processes. Although it seems that the call center agents may not be replaced by artificial intelligence (AI) soon, the majority of call center quality managers seem to be so, since most of both quantitative and qualitative analysis tasks can now be performed by artificial intelligence.
In this study, we propose a method to analyze phone calls between a call agent and a customer to classify a given speech segment as belonging to either a customer or an agent. As of being another deep supervised learning procedure, also this solution method needs numerous labeled training data. Since we had no readily available training data, we also propose a convolutional neural network (CNN) based procedure that can be used to produce labeled data for the training. The corresponding procedure will further be explained in the following sections.
Big tech companies like Google, Amazon, and Facebook are now leading AI research. It is not unusual to meet a new product deployed by these companies, which solves a critical AI problem. However, it is still not affordable for the vast majority of small and medium-sized enterprises (SMEs) to use them in their daily business process cycles. AdresGezgini Inc. (from now on referred to as the company ) is an example located in Turkey. It is an internet ads and software development company serving for the last 13 years. The company mostly uses telephony for its CRM processes. For the last six years, the company manages its call operations by using a third-party call center software that keeps the basic call statistics and all the call recordings. However, call center software cannot measure more detailed statistics like individual speaker durations. The manuscript is organized as follows: We start with review of previous studies that either directly or indirectly inspired us in this study. Then we give the details of the proposed method. Creating an accurate training data set was the most challenging and time-consuming process; hence, we give as much detail as possible about the training data. Then we explain the proposed convolutional recurrent neural network (CRNN) model architecture and its training process. The performance of the model on both training and validation data is reported in the results section. We spared another subsection for an extension of the model. Even if we primarily aim to classify a given audio segment as either customer or agent, we also have gender information for each data. Hence, we increased the number of target classes to four, namely female customer, male customer, female agent, and male agent. We also report the results in the results section. We conclude the manuscript with concluding remarks and our insights on relevant future studies.

Related Work
Our primary intuition was that the agent speech and customer speeches have specific rhythm and style. Hence, we hypothesize that identifying agents and customers from given audio segments resemble the music genre classification problem. While building up our solution method, we made an analogy with this problem. The music genre prediction problem has been drawing significant attention in audio related literature for the last few decades.
The idea of using Mel-frequency cepstral coefficients (MFCCs) in audio processing research has been much appreciated since 1980 [5]. It is still a vital feature extraction method for state of the art deep learning methods to solve various audio processing problems. Likewise, we use MFCCs to extract features to be used as training data. In the 2000s, to perform music genre classification, radial basis functions (RBFs) with MFCC and STFT features were proven to achieve close to human-level performance (HLP) [25]. In [19], temporal information integration of audio features is performed using multivariate autoregressive (MAR) features, and HLP comparable accuracies were achieved. On the contrary, these previous studies needed carefully tailored handcrafted features.
With the recent advances in machine learning (ML) deep learning (DL), audio-related problems are now solved with neural networks if there is a sufficient number of training data samples. Since audio signals have implicit temporal information, using recurrent neural networks (RNNs) gives better results in audio-related problems like speaker verification [26]. Moreover, CRNNs perform well in some problems since they can learn deep features and relate temporal information embedded in sequential data like audio signals [14].
In [3], music tagging is performed using CRNNs, and it is reported that it outperforms three different CNN architectures built with convolutional and fully connected layers. A similar CRNN architecture is used for music artist classification in [20]. In this study, we use a slightly modified version of the architecture proposed in [3]. In the recurrent layer, we tested both gated recurrent (b) Gender detection architecture Figure 1: (a) The depicted architecture addresses the speech/non-speech segment detection problem [9] (b) The depicted architecture addresses gender detection of speech segments [8].
unit (GRU) and long short term memory (LSTM) units. The structural details and differences of these units can be seen in [11]. The effect of using either GRU or LSTM on performance is reported in the Results section. This study needs preprocessing of the files to separate customer and agent speeches since the call center conversations are recorded and compressed as single-channel .gsm (i.e., GSM 6.10 prI-ETS 300 036 13kbit/s Standard) files with 8 kHz sampling frequency for disk usage optimization. A thorough analysis of some state-of-the-art methods for speech separation in machine learning literature can be found in [27]. In [16], an end-to-end convolutional time-domain audio separation framework shows significant performance in speaker separation for single-channel audio.
On the other hand, since the call center recordings are dialogues, i.e., the speaker voices do not overlap, speaker diarization can also be effectively used to solve the "who spoke when" problem for them. In [30], Google researchers proposed a speaker diarization system that uses generalized end to end (GE2E) loss, which is proposed initially for speaker verification in [26]. The GE2E loss utilizes d-vectors [12], which are embeddings generated using LSTM based deep neural networks (DNNs).
Since this study primarily aims to classify given segments into two classes, namely customer and agent, we do not perform a thorough speaker diarization, which requires a neural network welltrained explicitly for this purpose. We instead use publicly available pre-trained neural networks for classifying given audio segments into four classes, i.e., speech, music, silence, and noise. It is also possible to identify the gender of speech segments. Both problems are addressed and solved using CNNs in the joint studies [9] and [8]. inaSpeechSegmenter is an open-source implementation of these works where one can use these pre-trained CNNs. We use these networks to preprocess our audio recordings and create a training data set for the proposed purpose. These CNNs are depicted in Figure 1.
Prepocessing the call center recordings with the CNNs in Figure 1 enables us to extract speech segments and their gender information accordingly. The first preprocessing step is to segment the single-channel audio signal into four groups representing "silence", "noise", "speech" (with gender info), and "music". inaSpeechSegmenter Tool [9], [8] is used to perform this segmentation. segmented audio signal annotated audio signal Figure 3: Our Database based annotation scheme (DBAS) is depicted. Since we consider only the calls with two opposite gender segments, we mainly have two types of calls with the female agentmale customer, and the male agent -female customer pairs.

Data Set
Collecting and labeling data for deep supervised training is the most challenging and time-consuming step, and many studies in artificial intelligence literature are mostly about proposing new architectures for solving specific problems using publicly shared benchmark data sets like ImageNet [6].
The data used in this study is the call center recordings of the company. There are hundreds of thousands of recordings of varying lengths. For these recordings to be used as training data for our cause, we first filtered out very long and very short conversations. We only considered recordings of length between 60 and 600 seconds. Hence we obtained 48,462 files.
The inaSpeechSegmenter library can detect speech and non-speech segments also detect the gender of speech segments for a given audio sequence as depicted in Figure 2.
The audio records where there exist two opposite genders are of our concern. Because we already have the agent id and agent gender information of any recording in our database and we can label speech segments with four different labels: "female customer", "male customer", "female agent", and "male agent". We call this phase "Database based Annotation". The database based annotation scheme (DBAS) with two possible cases, female agent -male customer, and male agent -female customer, is depicted in Figure 3.
This annotation scheme theoretically finalizes our data picking procedure. On the contrary, we came up with a significant amount of ambiguous gender information for numerous speakers. inaSpeechSegmenter library randomly mislabels some male speakers' genders with aurally perceptible dominant high frequencies and some female speakers with dominant low frequencies in their voices. Hence we also filtered out the speakers where they are tagged with multiple gender labels in different conversations. We considered a speaker's conversations and discarded that speaker if she/he is labeled with multiple gender labels in those conversations, and kept the speakers who were consistently labeled with a single consistent gender label. Finally, the segments in conversations with two opposite genders can be labeled with female/male and customer/agent labels once the agent gender is known. We obtained 23,308 utterances from 377 different speakers, which total up to approximately 65 hours. Since we want the proposed network to learn speaker type, not the speaker voices, we spared 38 speakers for validation and did not use their utterances in the training epochs. These statistics are summarized in Table 1.
We kept each utterance length fixed at 10 seconds. All the records are originally kept in .gsm format with an 8kHz sampling frequency. We used log-amplitude mel-spectrograms as input since they have proven to outperform STFT, MFCCs, and linear-amplitude mel-spectrograms [2], [7]. As proposed in [3], we used 96 mel-bins. The window and hop lengths are 200 and 80 samples, respectively. With the above configuration, each utterance is transformed into a log-amplitude mel-spectrogram array with a size (96,1000).

Proposed Model Architecture
For this study, we use a slightly modified version of the CRNN model proposed in [3] (see Figure  4). In the architecture, four similarly structured convolution layers are followed by two recurrent layers whose output is connected to a fully connected dense layer and a softmax layer for the final classification.
Each convolutional layer has a cascaded max-pooling and dropout layer. Convolutional layers perform "same convolution", and they have varying numbers of filters. Except for the first convolutional layer, the max-pool layers have a kernel size of (3,3). The dropout layer probabilities are kept at 0.1 for all four convolutional layers.
The convolutional layers' output is a 2D array with a size (32,42), where the second dimension represents the temporal dimension. This output is directly fed to a recurrent neural network with two layers. GRU has become the most common choice for structuring RNNs. However, we tested both GRU and LSTM in the recurrent layer and reported their performances.  Figure 4: Inherited CRNN architecture for segment classification proposed in [3], and originally designed for music genre classification is depicted. The last layer output of the recurrent layer is connected to a dense layer, which is succeeded by a softmax layer for final classification. For our two-class classification, the softmax layer has two nodes to classify customer and agent segments. The architecture performs well on both training and validation data sets for this problem. We report the results in Section 4.

Gender Extension
Our main goal is to classify audio segments into two classes, namely customer and agent. Hence, we prepared the training and validation data sets with these labels accordingly. However, we inherently have gender data either and we can re-label the data sets with four labels: "female customer", "male customer", "female agent", and "male agent". With these new labels, the data statistics in Table  1 can be modified as Table 2.
We trained our network with four labels and observed that the model also performs well on the training and validation datasets. The only modification in the model is at the softmax layer, where we increase the number of output nodes from two to four. The corresponding training performances are reported in Section 4.

Implementation Details
We implemented our codes in Python [22]. While structuring our CRRN, we used Keras [4] interface for Tensorflow 2.1 [1] library. Other common tools such as Pandas [18] and Numpy [23]are also used for managing data, Matplotlib [13] and Scikit [21] libraries are used for data visualization. The experiments are performed on a Dell Precision 5820 workstation with a 16 core 3.70GHz Intel(R) Xeon(R) W-2145 CPU, 64GB 2666MHz DDR4 RAM, and a GeForce RTX 2080 8GB GPU. Since it is not possible to load and process the whole data, we used the data pipeline procedure of Tensorflow. For convenience, we kept every training sample in this folder structure: <train/validation>/<agent/customer>/ <female/male>/<speakerId>/<utteranceNo>.npy Tensorflow's data pipelines make it possible to read each data file and extract labels from the folder names at processing time. We used Adam optimizer [15] and early stopping criteria [29] for the optimization.

Results
We dedicate this section to the experimental results. Recalling Figure 4, we construct our CRNN first using GRUs in the recurrent layer and tested it on two problems, presented in Section 3. We then replaced GRUs with LSTMs and similarly tested the network with the same data and problems. We call our original problem as "2 Class Case" and the gender extended version of the problem defined in Section 3.3 to as "4 Class Case".
In Figure 5, we list experiments' results using GRUs in the recurrent layer. Figures 5a, 5b, 5c, and 5d present the network performance for problems. Similarly, Figures 6a, 6b, 6c, and 6d present the same results for the network with LSTM units in the recurrent layer.
We see that both network architectures perform quite well on both training and validation data sets for either problem. However, it is essential to see the network's classification performance for each class separately. For this reason, we plotted confusion matrices for the experimental results in Figure 7. The labeling conventions for the experiments can be summarized in Table 3.
The classification performance of the network can also be measured with a numerical metric called F1 Score. This score is calculated by using precision and recall rates. These sub-metrics are calculated by using true-positive (TP), false-positive (FP), and false-negative (FN) values, which can be deduced from the confusion matrices. Precision, Recall, and F1 Score can be defined as Equations 1, 2, and 3, respectively. F1 score can be calculated for each class separately.
We summarize the experimental results in Table 4. This table shows us that the GRU network's number of parameters is less than the LSTM network. By looking at the epoch number at which we achieve maximum accuracy, we see that the GRU network tends to be trained more before it overfits. By looking at the maximum accuracy values achieved for the validation data, we can say that both networks perform similarly. We show F1 Scores calculated for each class for the corresponding problems. Even the training data set is not well-balanced, the F1 scores are high enough to deduce that the network achieves a performance close to human-level performance (HLP) for the given problems.

Can the network generalize to other languages?
We showed that the proposed network architecture gives significant results on both training and validation data sets. The data set has thousands of different utterances from different speakers. Nevertheless, all of the utterances belong to native Turkish speakers. On the other hand, we thought that the call center agents somehow have a universal style of speaking regardless of their language. The same intuition can also be extended to the customers. Hence, we prepared a testbed for testing our network's generalization capability to other languages.
In the first test, we have chosen two sales call between one of the company's native German speaker female agent and two native German speaker male customers, for which we can diarize the opposite gender speakers by using inaSpeechSegmenter. Segmentation results can be seen in Figures 8a and 8c. We filter out non-speech segments using these labels and only consider speech segments shown as green in Figures 8b and 8d. We bundle each speaker's segments, and create two separate speaker streams from these bundles. We slide a 10 seconds wide window on this stream and by shifting it 1 second at a time, we created a data sample for each second of the speaker segments.
In the second test, we preferred to use two sales call samples from a completely different source. We found two YouTube videos [17], [24] of call center calls compatible with our problem definition. The calls are between English speaking female agents and customers. After converting these videos to mp3 format, we processed them in the same way as the German test sample.  Even the training is performed using Turkish audio samples; we see that the network can also generalize for German and English conversations. If we assume y i represents one of the classes, we want to find class probabilites P (y i |x k j,S l ) for each given utterance x k j,S l . i is either 0 or 1 for 2 Class Case and 0, 1, 2, or 3 for 4 Class Case with the labeling convention in Table 3. k represents the actual label of the utterance, j is the utterance number, and S l represents lth speaker, which is either 0 or 1 for the majority of call center calls. Since we want to predict the most probable class for a speaker (S l ), we first find the probability for a speaker to be in either class (y i ). We find the mean of the class probabilities for each utterance of the speaker as in Equation 4. The ultimate label prediction can be predicted by using Equation 5.
We present the calculated class probabilities of the above tests in Table 5. In the last column of this table, we show the calculated class probabilities. For example, for the German Sample 1, we see that female agent speaker audio achieves the maximum probability for class 2, considering Table  3, representing "female agent". Hence, if we predict labels according to Equation 5, all speakers are correctly classified with the correct labels. We observed a 78% match for the male customer in English Sample 1. When we closely analyze the segments detected by inaSpeechSegmenter, we observed that this slight inconsistency is due to faulty gender labels. These results support the idea that our network can generalize for the same problem in different languages. The network, with its optimized parameters, can work for other languages. It is also possible to fine-tune using transfer learning [28] with some training data from the desired language.

Concluding Remarks
In this study, we presented a CRNN based solution to the speaker classification problem specific to call centers. A typical call center conversation takes place between a customer and an agent. Conventional call center software stores the calls with maximum possible compression rates as single-channel audio records. Hence, it is not possible to directly measure the individual speaker speech length for these records. If the speaker segments are a priori separated and fed to the proposed network, it can classify speakers into four classes, female customer, male customer, female agent, and male agent, with very high accuracy.
We used the company's agents' gender information and the calls between two opposite genders to prepare the training data. Hence, we obtained a data set of utterances from 377 different speakers with known class labels. Thirty-eight speakers' utterances were never used in training and kept for validation. We also diversified the network structure by replacing GRUs with LSTM units in the recurrent layer. Training and validating with the data set yielded HLP comparable accuracies shown in Figures 5 and 6 for both versions of the network. These results are summarized in Table  4.
We also tested the network's generalization capability by using a couple of call samples from the German and English languages. Even without a thorough transfer learning, the trained network performed well on the selected samples. Since the study only focuses on the classification of speaker segments, we did not perform a thorough speaker separation or diarization process. With the current configuration, the proposed network can only analyze conversations between two opposite genders. Moreover, the system accuracy is inherently limited with the accuracy of inaSpeechSegmenter, which tends to mislabel speaker gender. Hence as a future study, we want to train another network for speaker separation for our single channel call center recordings, which can perform speaker separation regardless of gender and be used before the proposed network. Such a system can analyze the conversations between opposite genders and conversations between the same genders, i.e., female customer -female agent and male customer -male agent pairs. Calculating the individual speaker times gives a valuable metric to evaluate call center agents' performances. Also, unusual classprobability values may be interpreted to detect possible emotional abnormalities in the conversations between the customers and agents.


LRSCnet: Local Reference Semantic Code learning for breast tumor classification in ultrasound images

Purpose This study proposed a novel Local Reference Semantic Code (LRSC) network for automatic breast ultrasound image classification with few labeled data. Methods In the proposed network, the local structure extractor is firstly developed to learn the local reference which describes common local characteristics of tumors. After that, a two-stage hierarchical encoder is developed to encode the local structures of lesion into the high-level semantic code. Based on the learned semantic code, the self-matching layer is proposed for the final classification. Results In the experiment, the proposed method outperformed traditional classification methods and AUC (Area Under Curve), ACC (Accuracy), Sen (Sensitivity), Spec (Specificity), PPV (Positive Predictive Values), and NPV(Negative Predictive Values) are 0.9540, 0.9776, 0.9629, 0.93, 0.9774 and 0.9090, respectively. In addition, the proposed method also improved matching speed. Conclusions LRSC-network is proposed for breast ultrasound images classification with few labeled data. In the proposed network, a two-stage hierarchical encoder is introduced to learn high-level semantic code. The learned code contains more effective high-level classification information and is simpler, leading to better generalization ability.

Page 2 of 14 Zhang et al. BioMedical Engineering OnLine (2021) 20:127 speaking, the breast ultrasound CAD system mainly includes three stages: lesion segmentation, feature representation and lesion classification. Feature representation is the essential basis for breast lesion classification. Texturebased features [6][7][8][9][10][11] are commonly used features for ultrasound image classification because they can represent the scattering properties of breast tissues reasonably. However, they ignore the accurate classification on images from different ultrasound scanners. In order to handle texture variations of BUS images acquired with distinct ultrasound devices, Yang et al. [12] firstly decomposed each BUS image into multiple ranklet images. And then, gray-level co-occurrence matrix (GLCM)-based texture features were extracted from ultrasound images via multi-resolution ranklet transform. Gómez-Flores et al. [13] processed the input image using the ranklet transform, and then extract the auto-mutual information-based texture features from each ranklet image. Ranklet-based features is helpful to improve the robustness. However, Ranklet transformation ignores the local details of the images. In addition, the dimension of Rankletbased features is high, which may cause curse of dimensionality. In order to tackle these two problems, Xi et al. [14] proposed LSP-Ranklet transformation method by exploring the neighbor information. Based on the LSP-Ranklet-based texture features, multi-task learning is also used to select the robust features. However, it's difficult to train an effective multi-task learning model by using small data.
In recent years, deep learning has achieved great success in object classification due to its powerful feature learning abiltiy. Huynh et al. [15] extracted features using pretrained convolutional neural network (CNN) and then used to support vector machine (SVM) classifiers to distinguishing benign and malignant lesions. Xie et al. [16] selected four pre-trained classification network structure, such as LeNet, AlexNet [17], ZFNet and ResNet [18] for automated feature extraction and classification. Alexander Ciritsis et al. [19] use deep convolutional neural network for automatic classification of breast lesions. Razmjooy et al. proposed a new derivative-free meta-heuristic algorithm, which achieved good performance on some optimization problems [20]. Xu et al. proposed an automatic computer-assisted method based on satin bower bird optimization (SBO) algorithm to optimize the rationality of the number of hyperparameters in CNN [21]. The Razmjooy et al. proposed to apply the imperialist competition algorithm to optimize the neuro-fuzzy system and used five new features to train the system, which resulted in a higher classification accuracy [22]. In deep learning framework, sufficient training data with annotation is one of the key factor to improve the performance. However, it's difficult to collect sufficient ultrasound images with annotation. Moreover, the small data problem may cause reduction of generalization ability of the model. Shi et al. [1] extracted the texture feature firstly, and then developed the stacked deep polynomial network to learn high-level representation for small data classification. However, developing a deep CNN framework may further improve class performance for image classification task because CNN has advantages on effective local information learning than other networks.
However, two challenges arise for existing methods. (1) The intra-class variance occurs, result in performance degradation. Considering texture has ability to capture the discriminative scattering properties of breast tissue, some texture-based features have been proposed and achieved satisfactory performance. However, they ignore the accurate classification on images from different ultrasound devices. The texture variance is occurred between the same class due to different imaging principle of ultrasonic devices, result in the large intra-class variance. Figure 1 gives four images from different devices. The four images belong to benign lesion. As shown in this figure, the texture of the second image is smooth while others are coarse, result in the large intra-class variance.
(2) It's difficult to collect sufficient labeled training data, which causes the reduction of generalization ability. Sufficient training data with labels is important to improve the classifier performance. Unfortunately, it's difficult to collect sufficient ultrasound images with annotation in the real world due to the difficulty of biopsy label collection, limiting performance improvement. Based on the above idea, this paper develops a novel Local Reference Semantic Code (LRSC) network for lesion classification. In this paper, local reference refers to the common local structure of benign and malignant lesions. Effective local structure is helpful to distinguish class and reduce the intra-class variance [23][24][25][26][27][28]. The local structure extractor is firstly developed to learn local references via optimizing the developed class guided distance-based object function. The generated local references contain class information. After that, a two-stage hierarchical encoder is developed to encode the local structures of lesion into the high-level semantic code. The proposed encoder contains similarity learning module and semantic transfer module. In the first stage, the encoder learns the local similarity code via the similarity learning module. The similarity learning module is developed based on a deep learner which can learn effective relation between the local patch and local references. In the second stage, based on the learned similarity code, the encoder further generated the semantic code via semantic transfer module. The obtained semantic code can be seen as the high-level class description. It has ability to measure the relation between the whole image and certain common local characteristics of tumors. Therefore, the global code is robust to intra-class variance.
In addition, it contains more effective high-level classification information and is simpler, leading to better generalization ability. At last, the self-matching layer is proposed to obtain the final classification result, eliminating time-consuming traditional distance matching, improving the matching speed. The experimental results on our database demonstrate the effectiveness and robustness of the proposed method. The main contribution of this paper can be summarized as follows: 1. In order to solve the problem of intra-class variance and small data, a novel LRSCnet is proposed. On one hand, the common local structure is learned to tackle the problem of intra-class variance. On the other hand, a novel high-level semantic feature is learned to avoid overfitting. 2. Local reference is proposed to describe the common local characteristics of lession.
The local structure extractor is developed to learn local reference in the proposed network. 3. In order to encode the local structures of lesion into semantic code, a novel twostage encoder which contains similarity learning module and semantic transfer module is proposed. 4. The self-matching layer is introduced for final matching, eliminating traditional timeconsuming distance matching and classifier, improving the matching speed.
This paper is organized as follows. The results are given in "Results" section. The discussion based on the experimental results is give in "Discussion" section. This paper is summarized in "Conclusion" section. The proposed method is given in "Method" section.

Data description
To verify the effectiveness of the proposed method, experiments are conducted on our breast ultrasound images dataset [14]. The BUS images database contains 186 cases which are collected from 135 benign cases and 51 malignant cases provided by the Qianfushan Hospital of Shandong Province. The images were captured from four ultrasonic devices whose types are ALOKA α 10, AplioXG, GE LOGIQ E7 and SIEMENS Sequoia 512 respectively, as shown in Table 1. In addition, lesion boundaries were manually demarcated by experienced radiologists, and lesion ROI can be selected according to the lesion boundary labeled.

Experience setting
In our experiment, the values of learning rate, momentum and weight decay are setting to 0.001, 0.9 and 10 −6 respectively. Eighty percent of benign lesions and malignant lesions are selected as the training data, and remain images are regarded as the testing data. In addition, test metrics are used such as area under the ROC curve (AUC), accuracy (ACC), sensitivity (Sen), specificity (Spec), positive and negative predictive values (PPV and NPV) [12] to evaluate the diagnostic performance of the proposed method. Specifically, each index of every mammography-ultrasound pair is defined as: where TP, TN, FP and FN are shown in Table 2.

Experimental results
Firstly, the performance of LRSCnet with the textural features extracted by wavelets, Ranklet-TF [12], LSP-Ranklet-TF-MTL [14] and the Google inception_v3 model [29] are compared. The inception_v3 model was initialized by the ImageNet pre-training model and fine-tuned on our dataset. In addition, data augmentation was also used in the inception_v3 model training process. The experimental results of the different methods are shown in Fig. 2.
As shown in this figure, it can be observed that LRSCnet achieves the best performance in each of the evaluated metrics, AUC, ACC, Sen, Spec, PPV, and NPV are 0.9540, 0.9776, 0.9629, 0.93, 0.9774 and 0.9090, respectively.
The results on different training sets whose size are 60%, 70%, 80% and 90% of the original dataset are reported respectively, shown in   the number of train data increasing, LRSCnet has achieved better performance. LRSCnet can outperform traditional methods by using only 70% training data. Table 3 lists computation time of LSP-Ranklet-TF-MTL and LRSCnet. As show in this table, LRSCnet spent less match time because the obtained similarity code is short and contains class information. The self-matching layer can generate

Discussion
The proposed method improves the effectiveness and efficiency due to its several advantages: 1. Local reference is proposed to describe the tumor' common local characteristics which is robust to local intra-class variance. In addition, a novel two-stage encoder is developed to encode the local structures of lesion into shot high level semantic code which has ability to overcome the over-fitting problem. 2. The self-matching layer is introduced for final matching, eliminating traditional timeconsuming distance matching and classifier, improving the matching speed.
The experimental results demonstrate the effectiveness of the proposed method. As shown in Fig. 2, compared with state-of-the-art classification algorithms, Accuracy, AUC, Sen, Spec, PPV, and NPV have improved respectively. LSP-Ranklet-TF achieves the better performance among the features extracted by shallow model. The proposed LRSCnet outperforms LSP-Ranklet-TF. Multi-task learning model can select robust information from the complex LSP-Ranklet-based features. However, it's difficult to train an effective multi-task learning model by using small data. On the contrast, LRSCnet has ability to learn high-level class description. The learned semantic code contains more knowledge about lesions and is simplier, which is helpful to improve the generalization ability. Compared with deep learning method, LRSCnet also outperforms inception_v3 model. Compared with traditional deep learning method, LRSCnet focus on effective learning of local structure which is robust to intra-class variance.
As shown in Fig. 3, LRSCnet can learn similarity code which represents the relation between the target class and common local structure, containing more knowledge of the lesions. In addition, the learned code is shot. Therefore, LRSCnet has better generalization ability which is helpful to tackle the problem of small data.
The performance of LRSnet was tested with different number of local references. The number is set as 30,40,50,60 and 70 respectively, shown in Fig. 4. It was observed that with the number of local references increasing, the performance is improved. However, when the number is larger than 50, the performance is degraded. The reason may be that more local details can be learned with the number of local references increasing. However, noise may be introduced when the number of local references is too large, result in the performance degradation.
The main disadvantage of this study is that it has more hyper-parameters. Compared with traditional neural network, the extra hyper parameter for local references is introduced. In this paper, the number of local references are given via experimental results. However, the number of references has affected the classification performance. Therefore, develop a new method to automatically achieve the optimal parameters for local references is a direction of the future work.

Conclusion
This paper proposes LRSC-network for breast lesion classification with few labeled data. In the proposed network, the local structure extractor is firstly developed to learn the local reference which is robust to large intra-class variance. Moreover, a two-stage hierarchical encoder is introduced to learn high-level semantic code. The learned code contains more effective high-level classification information and is simpler, leading to better generalization ability. The experimental results on our database demonstrate the effectiveness of the proposed method. AUC, ACC, Sen, Spec are 0.9540, 0.9776, 0.9629, 0.93, respectively. In the future work, we will extend the proposed method to the semi-supervised learning framework.

Method
This paper proposes a novel Local Reference Semantic Code (LRSC) network for breast lesion classification. Figure 5 shows the architecture of the LRSCnet. In the proposed network, the local structure extractor is firstly developed for local references learning and local patches generation. Then two pipeline CNN feature extractors are used to learn the features of the acquired local patch and local reference. After that, the two-stage encoder which contains similarity learning module and semantic transfer module is introduced to learn the global semantic code. The learned code has ability to measure the similarity among the input image and local references. Finally, the self-matching layer is developed for faster classification.

Local structure extractor
Local information plays an important role in the breast ultrasound images classification [8]. Moreover, capturing effective local structure is helpful to distinguish class and reduce the intra-class variance [6,23]. Based on this idea, local structure extractor is developed to capture the effective local information of the lesions. In the proposed network, local structure extractor aims to learn local references and generate the local patches. Local references are learned by optimizing the class guided distance-based objective function. In addition, the local patches are generated by using a slide window of size m × m.
According to the definition of local reference in this paper, local reference should have following two properties: (1) It represents common local characteristics of the lesion, such as, internal echo, posterior acoustic behavior [8,30] and so on. (2) It contains class information, i.e. the local reference belongs to benign or malignant lesions.
According to the two properties, it can be inferred that the local reference can be seen as the local structure atoms with class information. For example, the local patches extracted from the posterior echo of benign lesion should be similar with local reference which describes the posterior echo of the benign lesion. Based on the above idea, the local reference is learned by minimizing the class guided distance-based objective function as follows: In Eq. (1) [31], variable x and z denotes the local patch extracted from the benign lesion and the malignant lesion respectively, p(t) is the indicate function whose value is 1 or 0. If input t belongs to the benign lesion, its value is 0, and vice versa. k and r denote the number of local references from benign lesion and malignant lesion respectively. µ i and ϑ i denotes the i-th local reference of benign lesion and malignant lesion respectively. The class guided distance-based objective function minimization can be achieved by the iterative optimization: (1) the initial value of the local references is given randomly.
(2) similarity calculation. Euclidean distance is used to measure the similarity between the local reference and the local patch.M i denotes the local patch set which is more similar with the i-th local reference from benign lesion while B i denotes the local patch set which is more similar with the i-th local reference from malignant lesion. The step of similarity calculation and local references updating are repeated. The iteration is stopped when the value of the local reference is not updated, and the final local references can be obtained.

Feature extraction
The flexible two pipeline network architecture which shares the same architecture and tied parameters is used for feature extraction. In the proposed network, the CNN pipeline is composed of three convolutional block, each of which is a 3 × 3 convolution with 32 filters followed by batch normalization, ReLU non-linearity and 2 × 2 max-pooling. At the end of the last dense block, a global average pooling is performed, as shown in Fig. 6, (a) is the network structure of the CNN feature extractor, and (b) is the network structure of the Conv block in (a).

Two-stage Encoder
In the proposed network, the two-stage encoder is proposed to encode the local structure of lesion into the high-level semantic code. It mainly contains similarity learning module and semantic transfer module. Similarity learning module is developed to learn the similarity between local references and local patches. Based on the obtained local similarity feature, semantic transfer module is developed to generate the semantic code 14 Zhang et al. BioMedical Engineering OnLine (2021) 20:127 which measures the relation between the image and the local references with classification information.

Local similarity learning
In the first-stage encoder, similarity learning module is developed to learn the similarity between local references and local patches. The module is constructed by using a neural network which contains three layers. The first two layers are used to further learning effective feature while the last layer is used to learn the similarity between local references and local patches. The input of the similarity learning module is a vector, thus the extracted CNN features of these two samples should be combined into one feature vector. Inspired by learning to rank framework [32], the similarities between the two samples are compared by combining two features with subtracted operation, as show in Eq. (4).
In Eq. (4), f ϕ denote the feature extractor, v i denotes the combined feature, x i is local patch, l i is local reference. Compared with traditional concat methods, the proposed method can reduce the dimension of the combined features.
The first two layers which are fully connected layer further learn effective features via hierarchical nonlinear feature transformation, as shown in following equation: where v ( x i ,l i ) is the combined feature, F is the learned feature, W and b are the parameters to be learned.
Based on the learned features, the similarity between local patch and local reference can be learned by using the last layer, as shown in Eq. (7).   20:127 where is the similarity between the local patch and the local reference. If the local patch and the local reference are related, the similarity between them is higher. Therefore, for arbitrary local patch p, its local similarity feature F ls can be learned. The i-th element in F ls denotes the similarity between p and i-th local reference. The length of the feature vector F ls is the number of the local references. For example, the number of local references is 100, the dimension of F ls is 100. Since the local reference contains class information, the learned feature also contains certain class information.

Semantic code generation
The semantic transfer module is developed to transform the obtained local similarity codes into the high-level semantic code, show in Fig. 7.
The semantic code is composed of semantic bits, arbitrary semantic bit is obtained based on certain local similarity code, as shown in following equations. In Eq. (8), K 1 is the number of local reference, f is the local similarity feature, f k denotes the k-th element of the feature. B is the semantic bit. In this paper, if the value of B is 1, it represents that the local patch belongs to benign lesion, and vice versa. The generated semantic code is the high-level class description of the image I. Each bit of the code has ability to measure the relation between the lesion and the certain local structure. Therefore, the generated code can describe more knowledge of lesion by using all bits, which is more robust to noise bit caused by intra-class variance. In addition, each bit contains high-level class information. Moreover, the learned similarity code is simpler, which is helpful to improve the generalization ability. The self-matching layer is introduced for the final lesion classification. Considering that the semantic code contains the class information, the final classification result can be obtained via simple statistical analysis of each bit, shown in Eq. (11). In this equation, n is the number of dimension of generated semantic code for the given image. B k is the k-th bit in the global semantic code. 0.5 is set as the threshold. If l is larger than 0.5, the image belongs to benign lesion, and vice versa.
Compared with traditional classification method, self-matching layer can generate the classification result without training, eliminates traditional time-consuming classification method, such as classifier training, distance calculation, improving matching speed.