extfieldsofstudy:
Computer Science
Engineering

text:
Environment-dependent denoising autoencoder for distant-talking speech recognition

In this paper, we propose an environment-dependent denoising autoencoder (DAE) and automatic environment identification based on a deep neural network (DNN) with blind reverberation estimation for robust distant-talking speech recognition. Recently, DAEs have been shown to be effective in many noise reduction and reverberation suppression applications because higher-level representations and increased flexibility of the feature mapping function can be learned. However, a DAE is not adequate in mismatched training and test environments. In a conventional DAE, parameters are trained using pairs of reverberant speech and clean speech under various acoustic conditions (that is, an environment-independent DAE). To address the above problem, we propose two environment-dependent DAEs to reduce the influence of mismatches between training and test environments. In the first approach, we train various DAEs using speech from different acoustic environments, and the DAE for the condition that best matches the test condition is automatically selected (that is, a two-step environment-dependent DAE). To improve environment identification performance, we propose a DNN that uses both reverberant speech and estimated reverberation. In the second approach, we add estimated reverberation features to the input of the DAE (that is, a one-step environment-dependent DAE or a reverberation-aware DAE). The proposed method is evaluated using speech in simulated and real reverberant environments. Experimental results show that the environment-dependent DAE outperforms the environment-independent one in both simulated and real reverberant environments. For two-step environment-dependent DAE, the performance of environment identification based on the proposed DNN approach is also better than that of the conventional DNN approach, in which only reverberant speech is used and reverberation is not blindly estimated. And, the one-step environment-dependent DAE significantly outperforms the two-step environment-dependent DAE.

Introduction
In a distant-talking environment, channel distortion drastically degrades speech recognition performance because of mismatches between the training and test environments. There are two different approaches, namely, front-and back-end-based methods, for dealing with this problem [1]. Many front-end-based approaches [1][2][3][4][5][6][7][8] have been proposed to reduce the effect of reverberation *Correspondence: wang@vos.nagaokaut.ac.jp 2 Nagaoka University of Technology, 1603-1 Kamitomioka, Nagaoka 940-2188, Japan Full list of author information is available at the end of the article in the observed speech signal. The back-end-based methods, on the other hand, attempt to modify the acoustic model and/or decoder to suit the respective reverberant environment [9,10]. In this paper, we focus on front-endbased approaches for distant-talking speech recognition.
Many single-and multi-channel dereverberation methods have been proposed to suppress reverberation [2][3][4][11][12][13][14]. Single-channel dereverberation approaches are much easier and cheaper to implement in real applications than multi-channel ones. In this paper, dereverberation is performed using a single-channel speech signal. Cepstral mean normalization (CMN) can be considered the most general single-channel approach [15][16][17]. Having been extensively examined, it has been shown to be a simple and effective way of reducing reverberation by normalizing cepstral features. However, the dereverberation of CMN is not completely effective in environments with late reverberation. Several studies have focused on mitigating this problem [3,4,12]. A reverberation compensation method for speaker recognition using spectral subtraction [18], where late reverberation is treated as additive noise, was proposed in [3]. A method based on multi-step linear prediction (MSLP) was proposed for both single and multiple microphones [4,12]. The method first estimates late reverberations using long-term MSLP, and then suppresses these with subsequent spectral subtraction. Wolfel proposed a joint compensation of noise and reverberation by integrating an estimate of the reverberation energy derived by an auxiliary model based on MSLP, into a framework, which so far, tracks and removes nonstationary additive distortion by particle filters in a low-dimension logarithmic power frequency domain [19].
Neural network (NN)-based approaches have been proposed for feature transformation [20,21]. Bottleneck features extracted by a multi-layer perceptron (MLP) can be used for nonlinear feature transformation [20]. However, deep networks of MLPs with many hidden layers have a high computational cost, the lower layers in a DNN architecture are hard to train because of vanishing gradients. Deep belief networks, which employ an unsupervised pre-training method using a restricted Boltzmann machine (RBM), have been proposed to train better initial values of deep networks [22]. Deep neural networks (DNNs) with pre-training have been shown to achieve better performance than the conventional MLP without pre-training [22]. There are many DNN, recurrent neural network (RNN) and long short-term memory (LSTM) [23][24][25][26][27][28][29] based speech enhancement and feature enhancement approaches that have been proposed for speech enhancement for human listening and robust speech recognition and that have shown good performance for the REVERB challenge task [40]. Recently, the denoising autoencoder (DAE), one type of DNN, has been shown to be effective in many noise reduction applications because higher-level representations and increased flexibility of the feature mapping function can be learned [30][31][32][33]. Ishii et al. applied a DAE to spectral-domain dereverberation resulting in improved word accuracy of large-vocabulary continuous speech recognition (LVCSR) [34]. Previously, we found that cepstral domain DAE-based dereverberation is efficient for distant-talking speech recognition [35]. As shown in [35], DAE worked well especially with strong reverberation. However, the results of DAE with small reverberation are not good compared to other methods. Typically, in the training of a DAE [26,34,35], data incorporating various environmental conditions are used.
Although this training method is suitable for training models in various environments, the nonlinear transformation ability of DAE training using various conditions that do not match those of the test data is lower for certain acoustic conditions of the test set. Thus, the performance of a DAE cannot be sufficiently improved for an unknown test reverberant condition.
To improve robustness of speech recognition, the idea of using side information from the environment as additional features, such as speaker-specific side information (e.g., i-vectors) and room information etc. has been proposed previously [36][37][38]. In this paper, two environmentdependent DAEs are proposed to reduce the influence of mismatches between training and test environments, that is, DAEs are trained and used corresponding to the different environments. In the first approach, various DAEs are trained using speech from different acoustic environments, and the DAE with the condition that best matches the test condition is automatically selected using a DNN (that is, a two-step environment-dependent DAE). The performance of our proposed two-step environmentdependent DAE is dependent on the precision of the automatic environment identification. In this paper, to achieve higher environment identification performance, a DNN using both reverberant speech and reverberation estimated by MSLP is also proposed. In the second approach, reverberation features estimated by MSLP are directly used as an input of the DAE (that is, a one-step environment-dependent DAE or a reverberation-aware DAE). By simultaneously estimating and suppressing the environment-dependent reverberation with a onestep environment-dependent DAE (that is, reverberationaware DAE), the mismatch between the training data and test data will be reduced. Therefore, better estimation of the clean speech can be expected. In previous work, conventional DAE was trained using speech data under various environments, and the test reverberant speech is transformed using a conventional environmentindependent DAE that cannot deal with environmental variation when there is limited training data. In the proposed approach, the test reverberant speech is transformed using an environment-dependent DAE that can estimate the environment-dependent reverberation. Thus, the environment-dependent DAE is more robust to environmental changes than a conventional DAE. The proposed methods are evaluated in both simulated and real reverberant environments.
The remainder of this paper is organized as follows: Section 2 describes the DAE for cepstral-domain dereverberation. The methods for one-step and two-step environment-dependent DAEs are described in Section 3, while the experimental results and a discussion thereof are presented in Section 4. Finally, Section 5 summarizes the paper.

Topology of DAE
An autoencoder is a type of artificial neural network (NN), whose output is reconstruction of input, and is often used for dimensionality reduction. DAEs share the same structure as autoencoders, but input data are a noisy version of the teacher signal. In this paper, we use the clean reference speech signal as teacher signal. Autoencoders use feature mapping to convert noisy input data into clean output and have been used for noise removal in the field of image processing [30]. Ishii et al. applied a DAE for spectral-domain dereverberation [34]. However, the suppressed spectral-domain feature needs to be converted to a cepstral-domain feature, and this improvement is not sufficient. In this paper, we apply a denoising autoencoder for cepstral-domain dereverberation because there are many LVCSR systems that adopt a cepstral-domain feature as the direct input. Given a pair of speech samples, clean speech and corresponding reverberant speech, DAE learns the non-linear conversion function that converts reverberant speech features into clean speech. In general, reverberation is dependent on both current and several previous observation frames. In addition to the vector of the current frame, vectors of past frames are concatenated to form input.
For cepstral feature X i of observed reverberant speech of the i−th frame, cepstral features of N −1 frames before the current frame are concatenated with the current frame to form a cepstral vector of N frames. Output O i of the non-linear transformer based on the DAE is given by where f l is the non-linear transformation function in layer l and N is the number of frames to be used as the input features.
Topology of the cepstral-domain DAE for dereverberation is shown in Fig. 1. In this paper, the number of hidden layers is set to three. In Fig. 1, W i (i = 1, 2) shows the weighting of the different layers, and W T i shows the transposition of W i 1 . That is to say, W 1 and W 2 are the encoder matrices and W T 1 and W T 2 are the decoder matrices, respectively.

Restricted Boltzmann machine
To train a deep neural network, deep belief networks (DBNs) [22] are used for pre-training because they can obtain accurate initial values of the deep-layer neural networks.
RBM is a bipartite graph shown in Fig. 2. It has a visible and hidden layer in which visible units that represent observations are connected to hidden units that learn to represent features using weighted connection. An RBM is restricted such that there are no visible-visible or hidden-hidden connections. Different types of RBMs are used in the case of binary or real-valued input. Bernoulli-Bernoulli RBMs are used to convert binary stochastic variables to binary stochastic variables. Gaussian-Bernoulli RBMs are used to convert real-valued stochastic variables to binary stochastic variables. Details of RBM are obtained in [22].
To obtain a pre-trained RBM, we trained all the hidden layers by using the Bernoulli-Bernoulli RBM. DBNs are hierarchically configured by connecting these pre-trained RBMs. Here, W 1 and W 2 are learned automatically, and W T 1 and W T 2 are generated from W 1 and W 2 in Fig. 1.

Backpropagation algorithm
After pre-training, a backpropagation algorithm was applied to adjust the parameters. Backpropagation modifies the weights of the network to reduce the error of the teacher signal and the output value when a pair of signals (input signal and the ideal teacher signal, the cepstral feature of clean speech) are given. We scaled the cepstral feature value of the input data and teacher signal to between 0 and 1 using a sigmoid function. The minimization in this paper is carried out by minimizing the cross entropy using conjugate gradients [22].

Environment-dependent denoising autoencoder
A conventional DAE trained using data under various acoustic conditions is effective for noise reduction and dereverberation. However, it is impossible to deal with mismatched conditions of the training and test data or unseen data with limited training data. We deal with this problem by two approaches. In the first approach, multiple DAEs for each environment are trained and selectively used. However, the reverberation environment is unknown in the test stage. Here, we use a DNN for environment identification because it is more effective than other classifiers such as the Gaussian mixture model (GMM) and support vector machine for audio classification [39]. In the second approach, reverberation features estimated by MSLP are directly used as an input of the DAE (that is, a one-step environment-dependent DAE or a reverberation-aware DAE). In the following, we describe the proposed environment-dependent DAE.

Environment-independent and environment-dependent DAEs
For a conventional DAE (environment-independent DAE), parameters are trained using pairs of reverberant speech and clean speech under various acoustic conditions. The environment-independent DAE is not robust for mismatches between training and test conditions. To address this problem, we propose two environmentdependent DAEs to mitigate the influence of mismatch in the training and test environments. In the first approach of environment-dependent DAE (that is, two-step environment-dependent DAE), various DAEs are trained using speech from different acoustic environments, and the DAE with the condition that best matches the test condition is automatically selected as shown in Fig. 3.
In the second approach (that is, one-step environmentdependent DAE), we add the environmental information (e.g., estimated reverberation features) to input of DAE as shown in Fig. 4. These approaches are expected for reducing the influence of mismatch between training and test environment, and improving LVCSR performance.

Environment identification
First, we divide the training data according to the environment. This is performed manually because the training data environment is known. Next, each DAE, with its respective data and the environment identification model, is trained. The training approach for the environment identification model is the same as that for the DAE with the exception of the input data and giving the reverberation environment as the correct label. In the DAE, the second half of weights of DAE is generated from Although reverberant speech features are used as input in the training of a DAE, this is not sufficient in the training of the identification model. As shown in Fig. 4, reverberation is estimated from reverberant speech, with reverberation features also used as input. By doing this, we can expect the performance of environment identification to be improved. In this paper, estimation of reverberation is based on MSLP [12]. The original MSLP algorithm estimates the reverberation from reverberant speech and suppresses reverberation in the spectral domain. In this study, MSLP is used for reverberation estimation only, and reverberation suppression is not performed.
where y(n) is the observed signal, D is the step size, N is the filter length, w(p) are prediction coefficients, and e(n) is the prediction error. Prediction filter w(n) is estimated by minimizing the mean square energy of the prediction error. Late reverberation is estimated using reverberant speech and the estimated prediction coefficients. Figure 5 shows the flowchart for environment-dependent dereverberation using the environment identification technique and multiple DAEs. First, we identify the reverberation environment of the input speech by applying an identification model to the speech. Here, the reverberation features estimated by MSLP are used as input for the model as well as for the training thereof. Next, the DAE corresponding to the identified environment is selected, Fig. 5 Flowchat of environment-dependent DAE and automatic environment identification and dereverberation is applied by it. Since dereverberation is applied by a DAE suited to the environment of the speech, we expect an improvement in performance.

One-step environment-dependent DAE
The second approach is almost the same as that for the environment identification in Section 3.2.1.
In this approach, estimated reverberation and reverberant speech are directly used as inputs of the reverberation-aware (DAE) as shown in Fig. 4. The one-step environment-dependent DAE can estimate and suppress the environment-dependent reverberation automatically. So, it is expected to reduce the influence of mismatches between training and test environments.
On the other hand, the conventional DAE does not use environment-dependent reverberation, so its performance will not be robust for mismatches between training and test conditions.

Experimental setup 4.1.1 Training dataset
We used the training dataset provided by the "REVERB challenge" (reverberant voice enhancement and recognition benchmark) [40]. This dataset consists of the clean WSJCAM0 [41] training set and a multi-condition (MC) training set. Reverberant speech is generated from the clean WSJCAM0 training data by convolving the clean utterances with measured room impulse responses and adding recorded background noise. The reverberation times of the measured impulse responses range from approximately 0.1 to 0.8 s. The training data of the "REVERB Challenge" were used to train the environment identification DNN. The environment labels depended on room type and the distance between the microphone and speaker. The training data include three types of rooms and two types of distances between the microphone and speaker, so in total, six types of environments with distinct rooms and distances were used. This training dataset were also used to train the DAEs and acoustic models. It should be noted that the recording rooms used for the multi-condition training data and test data were different.

Development and evaluation test sets
It is important to note that the proposed dataset consists of real recordings (RealData) and simulated data (SimData), part of which has similar characteristics to the RealData in terms of reverberation time and microphonespeaker distance. This setup allowed us to perform evaluations in terms of both practicality and robustness of various reverberant conditions. Specifically, the development (Dev.) and final evaluation (Eval.) test sets each contained the following SimData and RealData; SimData was generated from the WSJCAM0 corpus [41], and RealData from the MC-WSJ-AV corpus [42]. This development dataset was used to determine the optimal parameters for dereverberation and speech recognition. Details of the training and test datasets are given in Tables 1 and 2.

Experimental conditions for LVCSR and dereverberation
In this study, Mel-frequency cepstral coefficients (MFCCs) were used as features for LVCSR. The dimension of the MFCCs was 39 including 12 MFCCs plus power and their Delta and Delta-Delta coefficients. MFCC features were normalized using the mean of the entire multi-condition training set. DAE training was carried out using mini-batch conjugate gradients with a mini-batch size of 128 samples. In this paper, the number of hidden layers is set to three. The number of units in each layer is 512, and each unit uses a sigmoid function as an activation function. Because reverberation affects multiple frames, we supply multiple frames at the same time as the input and teacher signals of the DAE. The dimensions of the input data and output are 39 (the dimensions of MFCC per frame) * 9 (the number of segments to supply at the same time) = 351. These parameters were empirically determined. Please refer to [35] for a more detailed description. Fifty epochs with a learning rate of 0.002 were used for all layers during pre-training, and 100 epochs with a learning rate of 0.1 were used for all layers during fine-tuning. Training of the environment identification model architecture was almost the same as for the DAEs. The number of hidden layers is 5, the number of units in each layer is 1024, and 20 epochs were used for all layers during fine-tuning. The number of classes (the number of units at the final softmax layer) is 6, which is determined by the number of environments in the training data (i.e., Room 1, Room 2, and Room 3, each with near and far conditions). The MSLP algorithm generates an inverse filter through the prediction coefficients to estimate the inverse system [12]. We estimated the late reverberation components using the inverse filter and applied dereverberation by power spectral subtraction. For MSLP-based dereverberation, the step size and the order of linear prediction were set to 500 and 750, respectively. We used MSLP to estimate the late reverberation of both the training and test data with the same parameters as for MSLP-based dereverberation.
We used a subspace GMM with maximum mutual information-based discriminative training (MMI-SGMM) [43] and a cross-entropy training DNN for the acoustic model. The KALDI toolkit [44] was used as a decoder for LVCSR. In this study, the numbers of hidden layers and units were set to 3 and 1024 for the DNN acoustic model. The final results were obtained from a confusion network combination of MMI-SGMM with 7000 states and DNN-HMM with 2500 states. Details can be found in [44]. Standard Wall Street Journal 5000-word trigram language model was used for decoding. We used word error rate (WER) to evaluate the speech recognition performance for each method.

Results of environment-dependent DAE
In this section, we compare the following four dereverberation methods:  Tables 3 and 4 show the speech recognition results for each method on the Dev. and Eval. datasets, respectively. For the Dev. dataset, DAE-based cepstral-domain dereverberation shows a remarkable improvement when compared with CMVN-and MSLP-based dereverberation.  The DAE works especially well with strong reverberation, i.e., far-field microphone in "Room 2" and "Room 3" of SimData. The cepstral domain environment-independent DAE outperformed CMN and MSLP under almost all conditions. Although the performance of the two-step environment-dependent DAE is better than the conventional environment-independent DAE in some environments, it is worse in some environments. The reason may be that the performance of environment identification and environment special DAE depended on the training data, environment label, and also the environment of test data. The proposed one-step environment-dependent DAE (that is, reverberation-aware DAE) outperformed the conventional environment-independent DAE and two-step environment-dependent DAE on both SimData and Real-Data in the Dev. and Eval. datasets using SGMM, DNN, and SGMM+DNN. The improvement of the one-step environment-dependent DAE under large reverberation conditions is greater than that under small reverberation conditions. The reason is that the conventional DAE is not effective enough when there are large environmental mismatches between the training and test conditions, and the one-step environment-dependent DAE can reduce the influence of mismatch by estimating the late reverberation and adding it to the input DAE. For SimData in the Dev. dataset, using the one-step environment-dependent DAE with reverberation features estimated by MSLP, the average WER was reduced from 6.36 % with the conventional DAE to 5.77 % using SGMM+DNN, i.e., a relative error reduction rate of 9.28 %. For RealData in the Dev. dataset, the average WER was reduced from 27.46 % with the conventional DAE to 26.66 % using SGMM+DNN,  i.e., a relative error reduction rate of 2.91 %. For the Eval. dataset, a similar trend was observed. The proposed one-step environment-dependent DAE (that is, reverberation-aware DAE) with reverberation features estimated by MSLP outperformed all the other methods. For SGMM+DNN acoustic model, compared with the conventional DAE, relative error reduction rates of 9.97 % for SimData and 12.21% for RealData were achieved. The results show that the proposed one-step environmentdependent DAE is also robust to variations of speaker and speech context.

Comparison of different environment identification models
In this section, we investigate the effect of the environment identification method for the two-step environmentdependent DAE. We compare the performance of the environment identification model, using and not using reverberation estimated by MSLP, for training. Table 5 shows the speech recognition results on the Dev. dataset for these two methods. The results are based on a system combination of MMI-SGMM and DNN. Bigram training was used for the language model. These results indicate that the performance of environment identificat7ion is improved by using estimated reverberation in training the DNN. By blindly using reverberation estimated by MSLP, the DNN can identify an unknown test environment precisely. Without using the estimated reverberation, the two-step environmentdependent DAE performs worse than the conventional environment-independent one owing to poor environment identification performance. That is to say, the two-step environment-dependent DAE is sensitive to the environment identification performance.

Comparison with results of the other participants in the REVERB-challenge
We compared our results with those of the other participants under the same conditions for the training data and language model. A single-channel dataset provided by the REVERB-challenge was used. Table 6 shows the speech recognition results using the trigram language model for each participant. The WER of Alam et al. [45] was 11.1 % on SimData and 32.4 % on RealData. Tachioka et al. [46] achieved a WER of 10.05 % on SimData and 28.06 % on RealData. In our study, for the Eval. dataset, WER was 7.04 % on SimData and 28.66 % on RealData using MMI-SGMM, and 6.41 % on SimData and 26.83 % on RealData using SGMM+DNN.
The results indicate that the performance of our proposed environment-dependent DAE is better than almost all the other participants' methods using the same training data and language model.

Conclusions
In this paper, we proposed two environment-dependent DAE for robust distant-talking speech recognition. The proposed method was evaluated using simulated and real distant-talking speech. DAE-based cepstral-domain dereverberation achieved a remarkable improvement compared with CMN-and MSLP-based dereverberation in both environments. Furthermore, speech recognition performance was improved by the environment-dependent DAE compared with the conventional environment-independent DAE. For SimData in the Eval. using the one-step environment-dependent DAE with reverberation features estimated by MSLP, the average WER was reduced from 7.12 % with the conventional DAE to 6.41 % using SGMM+DNN, i.e., a relative error reduction rate of 9.97 %. For RealData in the Eval. dataset, the average WER was reduced from 30.56 % with the conventional DAE to 26.83 % using SGMM+DNN, i.e., a relative error reduction rate of 12.21 %. The results of our proposed dereverberation method are better than almost all of those of the other participants in the REVERB-challenge for single-channel speech and trigram language model conditions. Endnote 1 W i and W i T 1 correspond to f L in Eq. 1.

Competing interests
The authors declare that they have no competing interests.


extfieldsofstudy:
Computer Science
Medicine

text:
Face Liveness Detection Using Defocus

In order to develop security systems for identity authentication, face recognition (FR) technology has been applied. One of the main problems of applying FR technology is that the systems are especially vulnerable to attacks with spoofing faces (e.g., 2D pictures). To defend from these attacks and to enhance the reliability of FR systems, many anti-spoofing approaches have been recently developed. In this paper, we propose a method for face liveness detection using the effect of defocus. From two images sequentially taken at different focuses, three features, focus, power histogram and gradient location and orientation histogram (GLOH), are extracted. Afterwards, we detect forged faces through the feature-level fusion approach. For reliable performance verification, we develop two databases with a handheld digital camera and a webcam. The proposed method achieves a 3.29% half total error rate (HTER) at a given depth of field (DoF) and can be extended to camera-equipped devices, like smartphones.

Introduction
At present, many people deal with personal business using portable devices. From unlocking cellular phones to financial business transactions, people can easily conduct their individual business tasks through such a device. Due to this trend, personal authentication has become a significant issue [1]. Instead of using a simple PIN code, industries have developed stronger security systems with biometric authorization technology [2]. Biometric traits, such as face, iris and fingerprint, are very powerful factors to protect one's private information.
However, attempts to invade security systems and steal personal information have been increasing. One type of these attacks involves using fake identities. Spoofing faces and fingerprints are threatening security systems and privacy. This would not matter if current face recognition (FR) systems were secure, but current systems cannot distinguish fake faces from real faces. In some cases, the FR system embedded in cellular phones gives approvals to forged faces. This phenomenon is an example of weakness in the biometric system. If this problem remains unsolved, anyone will be able to easily obtain others' personal information in order to commit identity-related crimes. For this reason, technological defense against spoofing attacks is necessary, so as to protect personal systems and users' private data. Over the last decade, researchers have shown steady progress in developing anti-spoofing technologies [3]. Most of these methods concentrate on exploiting features obtained from the analysis of textures, spectrums and motion in order to detect face liveness.
In this paper, we propose a new method to secure face identification systems from forged 2D photos. The key factor of our methods is that we utilize the camera function, variable focusing. In shape-from-focus, it is possible to construct 3D images using focus measures [4,5]. Even though we need not recover the 3D depth images, we use the characteristics of the defocusing technique in order to predict the existence of the depth information. By adjusting the focusing parameters, parts of the image that are not in focus become blurry. With this function, we can evaluate differences in the degree of focus between real faces and fake faces and use this information to detect face liveness. To evaluate our method, we organized two databases using a handheld digital camera and a webcam.
The remainder of this paper is organized as follows. In Section 2, we discuss previous studies on face liveness detection and the theoretical background of camera focusing. Our proposed methodologies are stated in Section 3. In Section 4, experimental results are shown and the details are discussed. Finally, concluding remarks are provided in Section 5.

Countermeasures against Spoofing Faces
Numerous approaches to minimize vulnerability to attacks using spoofing faces have been proposed. In early research, intrusive methods that request user cooperation, such as speaking phrases and shaking one's head [6], were developed. However, these approaches cause users inconvenience and rely on users' cooperation. For this reason, many researchers have attempted to develop non-intrusive methods.
Depending on the type of attack, methods can be categorized into three groups: 2D static attacks (facial photographs), 2D dynamic attacks (videos) and 3D attacks (masks). Skills and devices for disguising one's identity have evolved gradually. Masks and videos are examples of advanced spoof attacks. Some studies have focused on protecting FR systems from these advanced attacks [7,8]. However, due to the difficulty and cost of obtaining such advanced tools, 2D static attacks, such as photographs, have been widely used by attackers. In this chapter, we review studies for detecting 2D facial photo-based spoof attacks.
There are three main spoof detection approaches, depending on the characteristics of input faces. The first approach is based on textures. Real and fake faces have different texture characteristics. Some studies have used texture to detect forged faces. Kim et al. [9] applied local binary patterns (LBP) for texture analysis and power spectrum for frequency analysis. Määttä et al. [10] and Bai et al. [11] also detected face liveness by examining micro texture with multiscale LBP. Peixoto et al. [12] proposed a method to detect and maintain edges (high-middle frequencies) with different Gaussian characteristics under poor illumination conditions. In [13], the authors extracted essential information for discrimination using a Lambertian model. Singh et al. [14] proposed a method to classify real faces based on a second-order gradient. This approach focuses on differences between skin surfaces of real and fake faces. Kant et al. [15] presented a real-time solution using the skin elasticity of the human face. Approaches with a single image have advantages in terms of low capacity and simplicity.
The second approach uses motion information. Signs of liveness, such as eye blink and head movements, are clues to distinguish motionless spoofing faces. Image sequences can be used to perceive movements. These factors are exploited intuitively [16][17][18][19][20]. In addition, optical flow and various illumination approaches are helpful to analyze the differences between real and fake faces [21][22][23][24][25]. Applying the entropies of RGB color spaces is one factor in face liveness detection [26]. To make a robust system, several methods use a combination of static and dynamic images [18,27].
The last approach is based on 3D facial information. The obvious difference between a real face and a fake face is the presence or absence of depth information. Human faces have curves, while photos are flat. By considering this feature, researchers have classified spoofing attacks. Wang et al. [28] suggested an approach to detect face liveness by recovering sparse 3D facial models, and Lagorio et al. [29] presented a solution based on 3D facial shape analysis.

Background Related to Focusing
Unlike previous research, our method utilizes the effect of defocus. Defocusing is exploited to estimate the depth in an image [4,5,30]. The degree of focus is determined by the depth of field (DoF), the range between the nearest and farthest objects in a given focal plane. Entities in the DoF are perceived to be sharp. In order to emphasize the effect of defocus, the DoF should be narrow. There are three parameters that modulate DoF, and Figure 1 shows those conditions for shallow DoF [31]. The first factor is the distance between the camera and the subject; a short distance produces a shallow DoF. The second factor is the focal length, which is adjusted to be longer for a shallow DoF. The last factor is the lens aperture of the camera, which is made wider to produce a shallow DoF. Using these options, we can achieve images with a narrow DoF and a large variation in focus [31].

Previous Work with Variable Focusing [32]
In the previous work [32], a method for face liveness detection using variable focusing was suggested. Two images sequentially taken at different focuses are used as input images, and focus features are extracted. The focus feature is based on the variation of the sum modified Laplacian (SML) [33] that represents the degrees of focusing. With the focus feature and a simple classifier, fake faces are detected. 2D printed photos are used as spoofing attacks, and a database composed of images with various focuses is produced for evaluation. When DoF is shallow enough to make the only partial area blurred, this method shows good results. However, at a deep DoF, the performance is deteriorated. In order to make up for the weakness of the previous work, we propose an improved method in this paper. Extracting local feature descriptors and frequency characteristics, as well as the focus feature from the defocused images, we detect spoofing faces. Moreover, the quantity of the database is increased, and various experiments are performed to achieve the best result. A detailed explanation will be described in the following sections.

Proposed Methodology
In this section, we introduce new FR anti-spoofing methods using defocusing techniques. From partially defocused images, we extract features and classify fake faces. The most significant difference between real and fake faces is the existence of depth information. Real faces have three dimensions, with the nose and ears being relatively far from each other. This distance can be used to adequately represent the depth information. Depending on the object or place of focus, the ear area might or might not be clear, as shown in Figure 2a. Unlike real faces, 2D spoofing faces are flat. There is little difference in clarity, regardless of the focus (Figure 2b). We emphasize this characteristic in order to discriminate real faces from 2D faces. In order to maximize the effect of defocus, we must adjust the DoF to be shallow, as mentioned in Section 2. However, according to the type of camera, the adjustment of DoF may not be possible. Therefore, we obtain input images using two cameras, a handheld digital camera and a webcam. We will explain image acquisition in the following section.
Our system is composed of three steps: image acquisition and preprocessing, feature extraction and classification ( Figure 3).

Image Acquisition and Preprocessing
In our method, image acquisition is an important factor in performance. As mentioned in the previous section, a narrow DoF increases the effect of defocus and assists with detecting fake faces. However, not every camera can easily change its DoF and focal plane. If people use handheld digital cameras, such as DSLR (digital single lens reflex) and mirrorless cameras, the DoF can be made shallow by directly controlling camera settings and the areas of desired focus can be manually selected. However, when users utilize webcams and cameras embedded in cellular phones, they cannot accurately manipulate the DoF. Moreover, the position of the focal plane is inexact with such cameras. Therefore, the process of image acquisition needs to vary with the type of camera. We will introduce two methods appropriate for a handheld digital camera and a webcam, respectively.

Using a Handheld Digital Camera
With handheld digital cameras (DSLR camera, mirrorless camera, compact digital camera, etc.), it is possible to manually control the focal plane and DoF. Hence, two sequential focused facial images are obtained for use in these experiments: one is focused on a nose and the other on ears ( Figure 2). When we set the focus on the ears and nose, we can tap on the LCD panel or turn a focus ring in accordance with the type of handheld digital camera. In this paper, a mirrorless camera (SONY-NEX5) is used, and it has a focus ring. Therefore, we acquire the focused images, turning the focus ring and checking the sharpness in the regions of the ears and nose with our eyes.
In the preprocessing step, we geometrically normalize images based on the location of the eyes [34]. In every image, the positions of faces are slightly different. For accurate comparison, faces must be aligned. Based on the coordinates of the eyes, we translate, rotate and crop facial images. The eyes can be automatically detected by using feature templates. In this paper, however, we select the correct positions of the eyes manually in every image and save the coordinates. Figure 4 shows the normalized images produced in the present study. Figure 4a,c is focused on the nose (I N ) and Figure 4b,d on the ears (I E ).

Using a Webcam
The focus in a webcam is controlled by adjusting the plastic lens in and out. However, the DoF is unknown, and it is difficult to select the focus area without the use of a supplemental program. Therefore, unless the program is used, it is not easy to obtain images focused on either the nose or ears. In order to acquire input images with a webcam, we approach the problem in a different way.
Although it is not possible to accurately take images focused on either the nose or ears when using a webcam, it is possible to obtain image sequences by changing the lens motor step. Depending on the adjustment of the lens, the focal plane varies, producing images with different focal planes. From the image sequence collected here, we select two images, I N and I E . I N and I E denote the normalized images for which the nose and ear area are in focus, respectively. In order to determine these images, we detect the nose and ears and calculate the degrees of focus in those areas [4]. As mentioned before, the centers of the eyes and the regions of the ears and nose are selected manually in this paper. When the value of a specific area is at a maximum at the k-step, that region is in focus. Figure 5 depicts the changes in focus values in accordance with the lens step. In Figure 5a, the nose area is in focus at the 20th step and the ears area at the 16th step. With fake faces, the steps of the maximum focus values for the nose and ears are same, as shown in Figure 5b. This allows one to distinguish between real and fake faces. Through this procedure and normalization, we can choose two images as I N and I E ( Figure 6).

Feature Extraction
To detect forged faces, features are extracted from normalized images. In this paper, we use three feature descriptors: focus, power histogram and gradient location and orientation histogram (GLOH) [35].

Focus Feature
The focus feature is related to the degree of focusing. In the previous study [32], this feature was suggested and used for classifying fake faces. Figure 7 shows the flowchart for extracting focus features. Using several focus measures [4], we can numerically calculate the focus levels in each pixel. There are various focus measures, such as Laplacian-based measures and gradient-based measures. We will show the performance in accordance with the focus measures.
The images in Figure 8 are the results of modified Laplacian (LAPM) focus measure calculations. LAPM is one of the focus measures introduced in [4,33]. This is presented as the sum of transformed Laplacian filters. Figure 8a,b shows the LAPMs of a real facial image focused on the nose and ears, and Figure 8c,d shows the LAPMs of a fake facial image focused on the nose and ears. We denote the LAPM of nose-focused images by LAPM N and the LAPM of ear-focused images by LAPM E . In LAPM N and LAPM E , bright pixels represent high values of LAPM, and those regions are in focus with sharp edges. On the contrary, out-of-focus regions have severe blurring, lose edge information and have low values of LAPM. In the case of real faces, the nose area in LAPM N (Figure 8a) is brighter than that in LAPM E (Figure 8b). However, there is little difference between the LAPM N and LAPM E of fake faces (Figure 8c,d). Consequently, by computing the variations in focus measures, we can determine the degree of focusing. In order to maximize the LAPM difference between regions of the nose and ears, we subtract LAPM E from LAPM N (= LAPM N − LAPM E ). To analyze the differences in LAPMs (DiF, difference in focus measures) in a single dimension, we add all of the DiF in the same column. In Figure 9, blue lines describe the cumulative sums of the DiF of real and fake faces. However, these distributions are not appropriate to be used for liveness detection without any refinement. The existence of noise affects the results. Therefore, curve fitting is performed to extract meaningful features. The sum of the DiF of real faces has a similar shape to the curvature of a quadratic equation, y = ax 2 + bx + c. In the quadratic equation, there are three coefficients, A = a b c T , and these are exploited as a feature for classification. To calculate the values of these coefficients, we perform error minimization [32]. Figure 9 presents the results of curve fitting (red circles). The curve for the cumulative sum of DiF of the real face is convex, as shown in Figure 9a, while that of the fake face is flat. In Figure 10, coefficients of quadratic equations are plotted. Blue circles are features of real faces, and red crosses are those of spoofing faces. Depending on the range of DoF, the degree of feature overlap will change.

Power Histogram Feature
Out-of-focus images have few edge components because the blurring filter eradicates the boundary. This affects the frequency characteristics of such images. We analyze this feature to identify forged faces. In this section, we introduce another feature, the power histogram feature, which contains spatial frequency information. The process of extracting this feature is presented in Figure 11.
In the first step, we divide a normalized image into three subregions, as shown in Figure 12. When a picture is taken focusing on the ears, we adjust the focal plane to include the ear area. Not only ears, but other components in the DoF are in focus. To analyze those components, we divide the images radially. The first subregion (subR1, Figure 12b) is the nose area, the second subregion (subR2, Figure 12c) includes the eyes and mouth, and the third subregion (subR3, Figure 12d) contains the ears and the contour of the chin.  Using a Fourier transform, we convert subregions from the spatial domain to the frequency domain. Figure 13 illustrates center-shifted Fourier spectrums of the three described subregions with power being concentrated at the center of each spectrum. According to the subregion, the distributions of power are different. In order to analyze those distributions, we calculate the percentage of power in circular regions. We divide the frequency spectrum into several circles by allowing it to be superimposed. The percentage of power within a circular region is computed by Equation (1) [36], where C is a circular region and real(u, v) and imag(u, v) are the real and imaginary parts of the frequency component, respectively. Each spectrum has a histogram, and the value of each bin is the percentage of power in each circular area. By concatenating three histograms, we can obtain a combined histogram from one image. The dimensionality of the histogram is determined by the radii of the circular regions in the frequency spectrum. With real faces, power histograms vary depending on the focus area. However, those of fake faces do not vary. We use the differences in the power histograms as a feature for liveness detection.

GLOH Feature
We extract another feature descriptor, the gradient location and orientation histogram (GLOH) [35], which is an extended version of scale-invariant feature transform (SIFT) [37] and makes it possible to consider more spatial regions, as well as making feature descriptors robust and distinctive. In this paper, we modify and apply this feature locally. Figure 14 shows the flowchart of extracting the GLOH feature. For each Gaussian smoothed image, the gradient magnitude, GM ag, and orientation, GOri, are computed by Equation (2).
GOri(x, y) = tan −1 I(x,y+1)−I(x,y−1) I(x+1,y)−I(x−1,y) GM ag(x, y) = (I(x + 1, y) − I(x − 1, y)) 2 + (I(x, y + 1) − I(x, y − 1)) 2 (2) Next, we divide the image into P × Q patches in order to draw features locally. Figure 15 shows how to separate the image into patches. GLOH descriptors are derived from polar location grids in patches. As shown in Figure 15, each patch is divided into 17 subregions (three bins in each radial direction and eight bins in each angular direction). Note that the central subregion is not split. In a subregion, the gradient orientations are quantized into 16 bins (Figure 16). From one patch, 17 histograms are created. We reshape these histograms into one column vector, whose dimensionality is 272 (=17 × 16), as illustrated in Figure 17. Finally, a 272 × P × Q-dimensional column vector is extracted from P × Q patches.

Classification
For classification, the support vector machine-radial basis function (SVM-RBF) is used [38]. The SVM classifier learns normalized focus, power histogram and GLOH features. Furthermore, we carry out fusion-based experiments by concatenating normalized features. Figure 18 shows the flowchart of the feature-level fusion approach. Depending on the training data and the development data, the parameter of the SVM classifier is determined. Figure 18. Flowchart of the feature-level fusion approach.

Experimentation
Before evaluating the performances of our approaches, we collected frontal facial images from 24 subjects, because there is no open facial database that has various focusing areas. Although there are some databases for liveness detection, they do not satisfy our requirements. Therefore, we created two databases, one composed of images taken by a mirrorless camera (SONY-NEX5) and the other containing images taken by a webcam (Microsoft LifeCam Studio). The difference between the two cameras is the possibility of the accurate and delicate control of focus. With the mirrorless camera, it is possible to focus precisely on the nose or ear area. However, the webcam makes it difficult to adjust focus in detail, and users are not able to determine what is in focus. We will explain the processes of acquiring databases in the next section. We printed photos for fake faces with a Fuji Zerox ApeosPort-II C5400 printer.
For evaluations, the following measures are used.
• False acceptance rate (FAR): the proportion of fake images misclassified as real.
• False rejection rate (FRR): the proportion of real images misclassified as fake. The performance of the proposed method is evaluated with our own databases. Databases are randomly categorized into 3 groups: training, development and testing sets.
• Training set (30%): to be used for training the classifier.
• Development set (30%): to be used for estimating the threshold of the classifier.
• Testing set (40%): to be used for evaluating the performance.
Thirty percent of the subjects are used for training and development, and forty percent of the subjects are used for testing. Three groups are disjoint. That is, if images of subject 'A' are used for training, they cannot be utilized for development or testing.

Data Acquisition
With the mirrorless camera, the nose and ear areas are able to be in focus, and the DoF is manually controlled. In order to obtain images with various DoFs, we adjusted the distance between the camera and the subject, focal length and F-stop. Figure 19 shows the ranges of the parameters.   Table 1. The number of males is 17 and that of females is 7. The size of each normalized image is 150 by 150 pixels, and the distance between the eyes is 70 pixels. Figure 20 shows real (a) and fake (b) samples from the database.

Experimental Results
We carry out experiments in accordance with the types of features, and the detailed results are described in Appendix A. The following shows the performance of the concatenated features. The process of combining features is carried out in the feature level. For high performance, we choose features based on the above results. Modified Laplacian (LAPM) and wavelet sum (WAVS) are used as focus features. In the case of the power histogram feature, the radii of the circular regions are 5, 15, 30, 50 and 75. GLOH features are extracted using 75 × 75 patches without allowing overlap. In order to reduce the dimensionality of the GLOH features, we apply PCA and use several eigenvectors whose variances are 90%. Table 2 shows the denotations of the features.  Table 3 and Figures 21 and 22 illustrate the results of the fusion-based methods. When the DoF is shallow (within 4 cm and 6 cm), the performances of focus features (LAPM and WAVS) are better than those of other features. However, as the DoF becomes deeper, the performances of focus features deteriorate. In the case of the GLOH and fusion-based features, the performances are maintained compared to other features. Especially, the HTERs of the fusion-based features under 16-cm DoF are lower than those of other features (6.27% and 6.08%). These numerical results demonstrate that the fusion-based methods are prominent when the effect of defocusing is low.

Data Acquisition
For evaluations, we gathered facial data using Microsoft LifeCam Studio. Using the provided program, we could control the lens motor step from 0 to 40. Therefore, one input sequence is composed of 41 images. Among those, we choose I N and I E , as mentioned in Section 3.1.2. The distance between the webcam and the subject is about 20 cm, so that the image can contain a whole face. The number of real face sequences is 94. Normal prints and an HD tablet (iPad 2) are used as spoofing attacks, and the number of sequences is 240 and 120 respectively. Five-fold cross-validation is applied for the evaluation.

Experimental Results
Numerical results are listed in Table 4. The good performance is maintained, even though the webcam database cannot express depth information well compared to the mirrorless camera database. The results of the combined features are the best, and the HTERs of them are 3.02% under normal print attack and 3.15% under HD tablet attack. These experiments show the possibility that our proposed method can be used in security systems at a low cost and with low specification devices. Furthermore, if detailed adjustment of the focus is possible in the device, our method can improve the performance more.

Discussion
Due to the characteristic of our proposed method, it is impossible to apply our method to open databases, such as the CASIA database [39] and the Replay-Attack database [40]. Therefore, we conducted comparative experiments by applying other methods to our own database. Table 5 demonstrates the performance comparison between our proposed method and other methods.
Other methods [9,41,42] detect the liveness based on textural analysis (local binary patterns) or frequency components (difference of Gaussian, power spectrum). Even though they have an advantage in terms of using a single image, the performances for our own database are not remarkable, regardless of the DoF; whereas the previous work [32] shows a good result relatively at the within 4-cm DoF. However, when the DoF is deep, the performance of [32] deteriorates. This represents that the performance of the previous system is determined depending on the method of input picture collection with great effects on defocus. In order to overcome this limitation, we propose our system by considering two factors. The first is by supplementing features. By adding other feature descriptors, we try to maintain good performance, even though the DoF becomes deeper. In the case of the GLOH feature [35], it has high matching scores for images with severe blur, whereas local features used in other methods [9,41,42] are not proper for the defocused images to be compared to the GLOH feature descriptor. The influence of the GLOH feature can be confirmed in the previous Section 4.1.2. In Figure 21, the performances of the focus and power histogram features are deteriorated in accordance with the increase of the DoF. However, the performance of the GLOH feature is maintained. As a result, we can achieve 6.51% HTER (feature-fusion) at a DoF within 16 cm by using additional features specialized for the defocused images. These results are better than the HTERs of other methods and the previous method [32], which uses only a focus feature of 20.8%.
The second way to mitigate the weakness of the previous study [32] is the use of the webcam database. Digital cameras, such as DSLR and mirrorless cameras, have high specifications and make it possible to manually adjust the DoF and focusing areas. However, due to their high cost, people might be unwilling to use digital cameras for image acquisition in anti-spoofing algorithms. Webcams are cheaper than digital cameras and are utilized broadly. With the webcam, we created a database and conducted experiments. As a result, we accomplish 3.02% HTER with the combined feature. The performance with the webcam database is similar to that with the mirrorless camera database.
Even though we show the good performance for liveness detection, our method has a disadvantage in the process of acquiring and normalizing images. In this paper, we set the focus on the ears and nose and find the centers of the eyes manually. In order to apply our proposed method to the security systems at low cost and with low specification devices, like smartphones, facial components must be detected automatically. Recently, many studies for feature point extraction have been in progress, and most cameras and smartphones have a face priority auto focusing function [43][44][45], which helps to obtain face-focused images by automatically controlling the lens actuator. If these technologies are utilized, the limitation of our method will be settled and applicable to the devices. Moreover, it will strengthen the security of smartphones.

Conclusion and Future Work
We proposed a face liveness detection method based on the characteristics of defocus. Our method pays attention to the difference between the properties of real and 2D fake faces. We use focus, power histogram and GLOH as feature descriptors and classify spoofing faces in terms of the feature-level fusion processes. Our experimental results show 3.29% HTER when the DoF of images is within 4 cm. Moreover, by applying various features, we overcome the limitation of DoF without adding any other sensors. Furthermore, through experiments with a webcam, we confirm that the good performance of our method is maintained.
Even though our proposed method yields good results, it has a limitation for being applied to camera-embedded security systems, such as smartphones, because of the manual processes to acquire the focused images and to detect facial components. Therefore, in future work, we will improve our method in order for it to operate automatically in the image acquisition and preprocessing and to make it possible to embed our method on a smart devices. Furthermore, we will consider more robust countermeasures against videos and 3D attacks by analyzing textural and temporal characteristics. Furthermore, we will advance our method using a light-field camera, which can acquire various focusing information in the spatial domain using a microlens array.

A. Appendix: Experiments According to the Type of Features
We carry out experiments in accordance with the types of features. The following shows the performance of our proposed methods.

A.1. Focus Feature
We conduct experiments with eight types of focus features. Eight focus features are categorized into four groups: statistic-based, Laplacian-based, gradient-based and wavelet-based operators [4]. The focus features are listed in Table A1. Related equations are organized in [4]. Table A2 and Figure A1 show HTERs and receiver operating characteristic (ROC) curves of focus features according to the range of the DoF. In general, the performance of the Laplacian group is better than those of other groups. As depicted in Figure A1, focus features in the Laplacian group swarm in the upper side. Especially, modified Laplacian (LAPM) has stable and prominent results all over the DoF (1.64% HTER under the within 4-cm DoF and 8.93% HTER under the within 16-cm DoF). The sum of wavelet coefficients (WAVS) also shows good performance. When the DoF is shallow, the effect of defocusing is great. This makes the focus features of real and fake faces more discriminative. As a result, focus features, except gray-level variance (GLVA), yield the best performances under the within 4-cm DoF. The GLVA focus feature, unusually, has the best performance when the DoF is within 16 cm (12.4% HTER). GLVA is the simple variance of the gray-scale image. Compared to other focus features, it is inadequate to represent the difference between the focused and defocused regions regardless of the DoF.   Figure A1. ROC curves of the focus features (DoF (a) within 4 cm, (b) within 6 cm, (c) within 10 cm and (d) within 16 cm).

A.2. Power Histogram Feature
In order to find that how to divide the frequency spectrum that yields good performance, we carry out experiments taking the radii of circular regions in Table A3. The dimensionality is the length of the concatenated histograms of the three subregions.  Table A4 describes the numerical results, and Figure A2 illustrates the distributions of the HTERs and ROC curves. When the averages of HTERs are calculated respectively, Rad.ver6 shows a good performance: 7.69% HTER. Even though the dimensionality of the power histogram feature is low, it yields the best performance compared to the others.

A.3. GLOH Feature
We perform experiments by altering the size of the patch, the energy percentage in PCA and whether the patches are overlapped or not. In Tables A5-A10, numerical results are listed, and Figure A3 describes the ROC curves.
As shown in Figure A3, the performances with and without allowing the overlap of patches are similar. In terms of the computational cost, the overlap of patches is not effective. Therefore, it is better not to overlap the patches to extract the GLOH features. In the case of the energy percentage in PCA, when the percentage is 98%, the performance is worse than those under 90% and 95%.
Additionally, experiments are carried out depending on the size of the patch. When the GLOH features are extracted from the whole image, the performance is the worst, and those features cannot represent the spatial properties sufficiently. As the size of the patch is 75 × 75 and the energy percentage in PCA is 90% without allowing the overlap, the performance is the best (7.75% HTER under the within 16-cm DoF).

Conflicts of Interest
The authors declare no conflict of interest.


extfieldsofstudy:
Computer Science
Biology
Medicine

text:
MetaboAnalyst 3.0—making metabolomics more meaningful

MetaboAnalyst (www.metaboanalyst.ca) is a web server designed to permit comprehensive metabolomic data analysis, visualization and interpretation. It supports a wide range of complex statistical calculations and high quality graphical rendering functions that require significant computational resources. First introduced in 2009, MetaboAnalyst has experienced more than a 50X growth in user traffic (>50 000 jobs processed each month). In order to keep up with the rapidly increasing computational demands and a growing number of requests to support translational and systems biology applications, we performed a substantial rewrite and major feature upgrade of the server. The result is MetaboAnalyst 3.0. By completely re-implementing the MetaboAnalyst suite using the latest web framework technologies, we have been able substantially improve its performance, capacity and user interactivity. Three new modules have also been added including: (i) a module for biomarker analysis based on the calculation of receiver operating characteristic curves; (ii) a module for sample size estimation and power analysis for improved planning of metabolomics studies and (iii) a module to support integrative pathway analysis for both genes and metabolites. In addition, popular features found in existing modules have been significantly enhanced by upgrading the graphical output, expanding the compound libraries and by adding support for more diverse organisms.

INTRODUCTION
MetaboAnalyst is a widely used, web-based system that supports comprehensive metabolomic data analysis, visual-ization and interpretation. The first release of MetaboAnalyst (introduced in 2009) contained just a single module focusing on metabolomic data processing and statistical analysis (1). The second version of MetaboAnalyst (released in 2012) contained four functional modules that supported expanded capabilities in metabolomic functional analysis and data interpretation (2). Since its introduction, MetaboAnalyst 2.0 has been continuously updated by improving existing functions, adding minor new features, upgrading the underlying design framework as well as the server hardware. These enhancements made the server increasingly popular within the metabolomics community. Indeed, the number of data analysis jobs submitted to the server has grown from ∼800/month (in 2010) to ∼3200/month (in 2013) to a current ∼40 000/month (in 2014). At the same time there has also been a significant shift in the type and complexity of metabolomics studies that are being routinely performed. In particular, the last several years have seen an increasing number of metabolomics-based biomarker studies in agricultural, biomedical and clinical settings (3)(4)(5)(6)(7)(8)(9)(10). Additionally there are now growing numbers of complex, multi-omic studies being performed that integrate metabolomics data with genomics, epigenomics or proteomics data over large populations (11)(12)(13)(14)(15). As a consequence, user-friendly tools to support biomarker analysis, population-based experimental design and multi-omic data integration have been among the most requested features by MetaboAnalyst's users. Likewise, with continuing advances in web-based technologies such as HTML5 and AJAX (asynchronous JavaScript and XML), the demands for more interactive visualization tools and larger-scale data visualizations have never been stronger.
Given the demand for new functions to support emerging applications in metabolomics, the need for a more computationally efficient implementation to accommodate the tremendous growth in jobs submitted to MetaboAnalyst, and the growing expectation for more powerful data visualization features we decided to undertake a near complete During the re-implementation of MetaboAnalyst, we made every effort to maintain the same 'look and feel' of the earlier versions in order to reduce the learning curve for current users. We only introduced interface changes if they led to significant performance gains, if they were more intuitive to use, or if they offered extra functionalities. Perhaps the most obvious change is the appearance of eight independent modules when users start a MetaboAnalyst session. MetaboAnalyst 2.0 originally offered four functional modules that shared the same navigation tree, allowing users to traverse to different modules during a given session. However, this design required the server to load all analysis modules, which resulted in large memory footprint. With the 50X increase in user traffic and server workload, this large memory consumption became increasingly burdensome, leading to a substantial slow down in performance. In version 3.0, each module is now an independent component with its own navigation tree. The new design not only reduces memory usage but also simplifies the navigation panel making it less prone to operational errors. It also makes it more straightforward to add new modules to future versions of MetaboAnalyst. The eight functional modules in MetaboAnalyst 3.0 can be grouped into three general categories--Category 1: exploratory statistical analysis (Statistical Analysis and Time Series Analysis); Category 2: functional analysis (Enrichment Analysis, Pathway Analysis and Integrated Pathway Analysis) and Category 3: advanced methods for translational studies (Biomarker Analysis and Power Analysis). In addition, there is also an Other Utilities module currently containing a compound ID conversion tool and a specialized function for lipidomic data analysis (19).
A flow chart describing the overall design, structure and functional modules for MetaboAnalyst 3.0 is given in Figure 1. Depending on the selected module and the type of data uploaded, different processing methods will be applied to convert the data into a data table (or data matrix) with samples in rows and features (compounds, peaks or spectral bins) in columns. Advanced data processing steps such as missing value estimation and data filtering are also available. Most of the functions implemented in MetaboAnalyst 3.0's data processing steps have largely remained the same as in the previous version. Some small improvements have been made. For instance, a modern Color Picker is now available to allow users to freely select any color to label specific clusters or groups. The Data Editor now allows users to exclude samples, features or groups (for multi-group data only) during an analysis. In addition to the changes (both large and small) in MetaboAnalyst's data processing functions, there have also been a number of updates and additions to MetaboAnalyst's other functional modules. These changes are grouped according to their functional categories and described in further detail below.

Category 1: exploratory statistical analysis
Besides completely re-engineering and consolidating MetaboAnalyst's user interface, significant effort was put into updating many of the exploratory statistical analysis elements in the Statistical Analysis and Time Series Analysis modules. The most significant enhancements have occurred with the scores/loadings plots, heatmaps and the feature details table.
i. Scores/loadings plots. The two common clustering and classification methods--principal component analysis (PCA) and partial least squares -discriminant analysis (PLS-DA) continue to be the most popular methods used in MetaboAnalyst. The most frequently viewed outputs from these two analyses are the scores and loadings plots. The scores plot provides an intuitive summary of the sample clustering patterns by projecting high-dimensional metabolomics data into two or three dimensions in a way that explains the maximal variance (PCA) or co-variance (PLS-DA) of the data; while the loading plot shows the underlying compounds responsible for such separation patterns. In developing MetaboAnalyst 3.0, we experimented with several different visualization approaches to improve both the quality and information content of the plots. In particular, the interface for two-dimensional (2D) scores plot now allows users to adjust a number of parameters to customize the graphical output. The corresponding loadings plot and the three-dimensional (3D) scores plot now support interactive visual exploration featuring point-and-click selection, zooming and rotating (3D only). An example graphical output is given in Figure 2A. These features run natively on all modern web browsers with JavaScript enabled. The 2D interactive visualization support has been extended to volcano plots as well as results from t-tests and ANOVA. The 3D feature has also been implemented for the interactive PCA (iPCA) method in the Time-series Analysis module. ii. Heatmaps. Clustered heatmaps are another very popular visualization tool in MetaboAnalyst. Heatmaps allow users to easily visualize changing patterns in metabolite concentrations across samples and across experimental conditions. In contrast to the scores plots, heatmaps display the actual data values using carefully chosen color gradients. In the previous version of MetaboAnalyst, the heatmap was restricted to a fixed size, which led to nearly illegible graphs when very large datasets were being visualized. In MetaboAnalyst 3.0, we re-implemented the heatmap visualization tool using the R pheatmap package (version 0.7.7). In addition to the fixed size 'Overview', users can also choose the 'Detail View' which will automatically adjust the output image size based on the actual uploaded data size to ensure that the resulting heatmaps will be easily readable (up to 2000 features). This new implementation takes advantage of the auto-scroll capability of newer web frame-work components for displaying overflowing content without distorting the user interface. This enhancement has been applied to all heatmaps generated via hierarchical clustering, correlation analysis and two-factor clustering analysis in the Time-series Analysis module. An example heatmap is shown in Figure 2B.

Category 2: functional analysis
There are two main approaches in functional analysis in MetaboAnalyst-metabolite set enrichment analysis (20) and metabolic pathway analysis (21). Both approaches work by comparing the significant compounds identified from the uploaded data to those pre-defined functional groups. The required inputs for these two analyses are metabolite concentration data or a list of metabolite names. Both inputs require a name-mapping step to standardize the compound IDs. In version 3.0, we have updated the underlying metabolite library based on the latest version of HMDB (16). This led to a 5X increase in the number of compounds or metabolite names available in MetaboAnalyst. We have also re-implemented the algorithm to improve the performance of fuzzy string matching, which has often been a significant computational bottleneck when the uploaded data contains many non-standard compound names. Based on user requests, we have also added support for more organisms for metabolic pathway analysis, such as Schistosoma mansoni, Plasmodium falciparum, Trypanosoma brucei, Synechococcus elongatus.

New module #1: integrated pathway analysis
This is a new module introduced in MetaboAnalyst 3.0 to allow users to integrate data from two commonly performed omic experiments--gene expression experiments and metabolomics experiments. This approach exploits the models from KEGG metabolic pathways to complete the analysis. The underlying assumption behind this module is that by combining the evidence based on changes in both gene expression and metabolite concentrations, one is more likely to pinpoint the pathways involved in the underlying biological processes. To this end, users need to supply a list of genes and metabolites of interest that have been identified from the same samples or obtained under similar conditions. The metabolite list can be selected from the results of a previous analysis downloaded from MetaboAnalyst. Similarly, the gene list can be easily obtained using many excellent web-based tools such as GEPAS (22) or INVEX (23). After users have uploaded their data, the genes and metabolites are then mapped to KEGG metabolic pathways for over-representation analysis and pathway topology analysis (21). Topology analysis uses the structure of a given pathway to evaluate the relative importance of the genes/compounds based on their relative locations. Clicking on the name of a specific pathway will generate a graphical representation of that pathway highlighted with the matched genes/metabolites. Users need to keep in mind that unlike transcriptomics where the entire transcriptome is routinely mapped, current metabolomic technologies capture only a small portion of the metabolome. This difference can lead to potentially biased results. To address this issue, the current implementation of this omic integration module allows users to explore the enriched pathways based either on the joint evidence or on the evidence obtained from one particular omic platform for comparison.

Advanced methods for translational or clinical studies
In recent years, there has been an increasing interest in the application of metabolomics for clinical or translational medicine studies such as biomarker identification for disease diagnosis, prognosis or monitoring. These studies often require specialized statistical analyses that are very different from the methods normally used in most metabolomic data analyses (which tend to focus on biological interpretation). In MetaboAnalyst 3.0, we have introduced two new modules to address these more pragmatic clinical or translational needs--the Biomarker Analysis module for biomarker identification and performance evaluation, and the Power Analysis module to support sample size estimation for clinical study design.

New module #2: biomarker analysis
Biomarkers are objectively measurable biological characteristics that can be used to indicate certain conditions or disease states. The primary goal of biomarker analysis is to build a predictive model from one or more variables, which can be used to classify new subjects into specific groups (e.g. healthy versus diseased) with optimal sensitivity and specificity. Biological understanding is not a prerequisite for biomarker development. The procedures for biomarker analysis are formalized into three major steps: (i) biomarker selection, (ii) performance evaluation and (iii) model creation. Many different approaches are available for each step. For instance, the performance of a biomarker model can be assessed in several ways. The two most important performance measures are sensitivity (true positive rate) and specificity (true negative rate). For any test, there is usually a trade-off between these two measures. Choosing a different threshold may increase the sensitivity at the expense of lowering the specificity or vice versa. One of the best ways to observe how a decision threshold affects sensitivity and specificity is through an ROC curve. A ROC curve can be created by plotting the sensitivity against 1-specificity at various threshold settings. It depicts the performance of a biomarker test over the complete range of possible decision boundaries. ROC curve analysis is widely considered to be the most objective and statistically valid method for biomarker performance evaluation. ROC curves are often summarized into a single metric known as the area under the ROC curve (AUC), which is widely used to compare the performance of different biomarker models. This new Biomarker Analysis module supports all common ROCcurve based biomarker analysis. These include: i. Classical ROC curve analysis. The section allows users to perform univariate ROC curve analysis for each compound. Users can generate ROC curves, to calculate the full AUC or partial AUC as well as their 95% confidence intervals, or to compute optimal cutoffs. ii. Multivariate ROC curve explorer. This section allows users to explore the performance of different biomarker models automatically created through the built-in feature selection and performance evaluation procedures. Users can choose among three well-established multivariate algorithms including PLS-DA, support vector machines and random forests to perform ROC curve analyses. iii. ROC curve based model creation and evaluation. This section allows users to manually select a subset of features and then test their performance using any of the three algorithms mentioned above. An example ROC curve output is shown in Figure 2C. This module also allows users to hold out a subset of samples for separate validation in addition to the built-in cross validation. The significance of the biomarker model can also be evaluated using permutation-based approaches.
Biomarker analysis often involves a number of very complex statistical procedures. More technical details about different algorithms are provided by the corresponding FAQs in MetaboAnalyst's online documentations. A more de-W256 Nucleic Acids Research, 2015, Vol. 43, Web Server issue tailed introduction on how to use and interpret ROC curves within the context of metabolomics biomarker analysis can also be found in a recent comprehensive tutorial (24).

New module #3: power analysis and sample size estimation
Statistical power is defined as the probability of detecting an effect, when the effect is present. For instance, let us assume a study that compares a specific effect or feature between a control population and a diseased population has a power of 0.8. Assuming we can conduct this study many times, then 80% of the time, we would get a statistically significant difference for that effect/feature between the two groups. The power of a study depends on three factors: (i) the magnitude of the effect in the population (effect size), (ii) the statistical significance criterion used in the test and (iii) the sample size used to detect the effect. In practice, the effect size can be estimated from a pilot data; the significance criterion (alpha level) is usually the P value in traditional univariate power analysis. Given these two constraints, researchers are usually most interested in knowing the sample size (number of subjects) required in order to obtain sufficient power in a given study.
However, estimating sample size for high-throughput omics studies is more complicated. Omics datasets are characterized by tens of thousands of features and a relatively small number of samples. Both the effect sizes and variances will have many values. A number of different methods have been proposed during the past years to deal with this issue (25)(26)(27)(28)(29). A general approach is that for high-dimensional omics data, the average power should be used instead of power, and significance levels need to take multiple testing into account using standard methods such as false discovery rate instead of raw P values.
The power analysis algorithm used in MetaboAnalyst 3.0 is based on the Bioconductor package SSPA as described by van Iterson et al. (26,29). This method uses the entire set of test statistics computed from the pilot data to estimate the effect size distribution, the power and the minimal sample size. Users first need to upload their pilot metabolomic data and perform the data processing and normalization steps as usual. Several diagnostic plots are then presented to allow users to check whether the test statistics follow an approximately normal distribution, and whether there are relatively a large number of P values that are close to zero (i.e. the effect indeed exists). When these assumptions are reasonably met, users can proceed to estimate the statistical power with regard to different sample sizes. The current implementation allows users to interactively explore the predicted powers for sample sizes ranging from 3 to 1000 samples per group. An example output from MetaboAnalyst's power calculations is shown in Figure 2D.

Implementation
MetaboAnalyst 3.0 was implemented using the PrimeFaces 4.0 component library (http://primefaces.org/). The majority of the backend computations are carried out by R functions (www.r-project.org) based on the R programming language (v3.03). The application is currently hosted on a Linux server with 16G RAM and eight-core 2.6 GHz CPUs. For researchers who are routinely generating large volumes of metabolomic data or for those requiring secure data handling, MetaboAnalyst 3.0 is also available for download and local installation. Detailed installation instructions are available on the 'Resources' page. To further facilitate collaborative research and future development, all of MetaboAnalyst's source code is available as an Apache Maven project upon request. Due to our limited hardware resources, MetaboAnalyst currently offers only basic support for raw spectral data processing. Users are encouraged to use other dedicated and freely available tools such as XCMSOnline (30) and MZmine (31) for such tasks. The peak lists generated by these tools can be easily uploaded into our server for further downstream analysis.

CONCLUSION
The development of MetaboAnalyst 3.0 has been driven primarily by user demands for new statistical methods to support emerging trends in metabolomics applications, by the demands for more efficient implementations and by requests for better data visualization to accommodate the tremendous increase in data analysis workloads. In redesigning and re-writing MetaboAnalyst 3.0, we directed a significant amount of effort and thought toward improving its computational efficiency. For instance, we made each module independent to reduce the large memory footprint and we significantly improved the efficiency in name mapping by adding several heuristic rules to the fuzzy search algorithm. We also looked for efficiencies in other areas. For instance, based on the Google Analytics and our own experience, we estimated that an average MetaboAnalyst analysis lasts for ∼20 min for new users. Therefore to maximize available computational resources and to reduce the load from zombie processes or abandoned analyses, we implemented a function to scan and kill processes that have run for more than 2.5 h to reclaim computational resources. We have also substantially increased the number (50X) of concurrent connections that are allowed on the application server. These procedures have dramatically increased the overall efficiency, performance and stability of the system. We are currently in the process of setting up a mirror site using the cloud service offered by Google Compute Engine.
In terms of overall scope and capabilities, MetaboAnalyst continues to be the most complete, freely available webbased resource for metabolomic data analysis. The main strengths of this web-based server are its user-friendly interface, its comprehensive data processing options, its wide array of univariate and multivariate statistical methods and its extensive data visualization and functional analysis support. In addition to these features, MetaboAnalyst 3.0 now offers a number of advanced statistical methods for multiomic data analysis and clinical or translational research. Our intention is to continue to upgrade MetaboAnalyst over the coming years and to be as responsive to user requests as possible. Several areas of likely upcoming development include new modules for metabolite-wide association studies and environment-wide association studies as well as new modules for spectral analysis and compound identification.