extfieldsofstudy:
Engineering
Computer Science
Mathematics
Medicine

text:
Unsupervised and Semi-supervised Anomaly Detection with LSTM Neural Networks

We investigate anomaly detection in an unsupervised framework and introduce Long Short Term Memory (LSTM) neural network based algorithms. In particular, given variable length data sequences, we first pass these sequences through our LSTM based structure and obtain fixed length sequences. We then find a decision function for our anomaly detectors based on the One Class Support Vector Machines (OC-SVM) and Support Vector Data Description (SVDD) algorithms. As the first time in the literature, we jointly train and optimize the parameters of the LSTM architecture and the OC-SVM (or SVDD) algorithm using highly effective gradient and quadratic programming based training methods. To apply the gradient based training method, we modify the original objective criteria of the OC-SVM and SVDD algorithms, where we prove the convergence of the modified objective criteria to the original criteria. We also provide extensions of our unsupervised formulation to the semi-supervised and fully supervised frameworks. Thus, we obtain anomaly detection algorithms that can process variable length data sequences while providing high performance, especially for time series data. Our approach is generic so that we also apply this approach to the Gated Recurrent Unit (GRU) architecture by directly replacing our LSTM based structure with the GRU based structure. In our experiments, we illustrate significant performance gains achieved by our algorithms with respect to the conventional methods.

I. INTRODUCTION A. Preliminaries
Anomaly detection [1] has attracted significant interest in the contemporary learning literature due its applications in a wide range of engineering problems, e.g., sensor failure [2], network monitoring [3], cybersecurity [4] and surveillance [5]. In this paper, we study the variable length anomaly detection problem in an unsupervised framework, where we seek to find a function to decide whether each unlabeled variable length sequence in a given dataset is anomalous or not. Note that although this problem is extensively studied in the literature and there exist different methods, e.g., supervised (or semisupervised) methods, that require the knowledge of data labels, we employ an unsupervised method due to the high cost of obtaining accurate labels in most real life applications [1] such as in cybersecurity [4] and surveillance [5]. However, we also extend our derivations to the semi-supervised and fully supervised frameworks for completeness.
In the current literature, a common and widely used approach for anomaly detection is to find a decision function This work is supported in part by Outstanding Researcher Programme Turkish Academy of Sciences and TUBITAK Contract No 117E153.
The authors are with the Department of Electrical and Electronics Engineering, Bilkent University, Bilkent, Ankara 06800, Turkey, Tel: +90 (312) 290-2336, Fax: +90 (312) 290-1223, (contact e-mail: {ergen, mirza, kozat}@ee.bilkent.edu.tr). that defines the model of normality [1]. In this approach, one first defines a certain decision function and then optimizes the parameters of this function with respect to a predefined objective criterion, e.g., the One Class Support Vector Machines (OC-SVM) and Support Vector Data Description (SVDD) algorithms [6], [7]. However, algorithms based on this approach examine time series data over a sufficiently long time window to achieve an acceptable performance [1], [8], [9]. Thus, their performances significantly depend on the length of this time window so that this approach requires careful selection for the length of time window to provide a satisfactory performance [8], [10]. To enhance performance for time series data, neural networks, especially Recurrent Neural Networks (RNNs), based approaches are introduced thanks to their inherent memory structure that can store "time" or "state" information [1], [11]. However, since the basic RNN architecture does not have control structures (gates) to regulate the amount of information to be stored [12], [13], a more advanced RNN architecture with several control structures, i.e., the Long Short Term Memory (LSTM) network, is introduced [13], [14]. However, neural networks based approaches do not directly optimize an objective criterion for anomaly detection [1], [15]. Instead, they first predict a sequence from its past samples and then determine whether the sequence is an anomaly or not based on the prediction error, i.e., an anomaly is an event, which cannot be predicted from the past nominal data [1]. Thus, they require a probabilistic model for the prediction error and a threshold on the probabilistic model to detect anomalies, which results in challenging optimization problems and restricts their performance accordingly [1], [15], [16]. Furthermore, both the common and neural networks based approaches can process only fixed length vector sequences, which significantly limits their usage in real life applications [1].
In order to circumvent these issues, we introduce novel LSTM based anomaly detection algorithms for variable length data sequences. In particular, we first pass variable length data sequences through an LSTM based structure to obtain fixed length representations. We then apply our OC-SVM [6] and SVDD [7] based algorithms for detecting anomalies in the extracted fixed length vectors as illustrated in Fig.  1. Unlike the previous approaches in the literature [1], we jointly train the parameters of the LSTM architecture and the OC-SVM (or SVDD) formulation to maximize the detection performance. For this joint optimization, we propose two different training methods, i.e., a quadratic programming based and a gradient based algorithms, where the merits of each different approach are detailed in the paper. For our gradient based training method, we modify the original OC-SVM and SVDD formulations and then provide the convergence results of the modified formulations to the original ones. Thus, instead of following the prediction based approaches [1], [15], [16] in the current literature, we define proper objective functions for anomaly detection using the LSTM architecture and optimize the parameters of the LSTM architecture via these well defined objective functions. Hence, our anomaly detection algorithms are able to process variable length sequences and provide high performance for time series data. Furthermore, since we introduce a generic approach in the sense that it can be applied to any RNN architecture, we also apply our approach to the Gated Recurrent Unit (GRU) architecture [17], i.e., an advanced RNN architecture as the LSTM architecture, in our simulations. Through extensive set of experiments, we demonstrate significant performance gains with respect to the conventional methods [6], [7].

B. Prior Art and Comparisons
Several different methods have been introduced for the anomaly detection problem [1]. Among these methods, the OC-SVM [6] and SVDD [7] algorithms are generally employed due their high performance in real life applications [18]. However, these algorithms provide inadequate performance for time series data due to their inability to capture time dependencies [8], [9]. In order to improve the performances of these algorithms for time series data, in [9], the authors convert time series data into a set of vectors by replicating each sample so that they obtain two dimensional vector sequences. However, even though they obtain two dimensional vector sequences, the second dimension does not provide additional information such that this approach still provides inadequate performance for time series data [8]. As another approach, the OC-SVM based method in [8] acquires a set of vectors from time series data by unfolding the data into a phase space using a time delay embedding process [19]. More specifically, for a certain sample, they create an E dimensional vector by using the previous E − 1 samples along with the sample itself [8]. However, in order to obtain a satisfactory performance from this approach, the dimensionality, i.e., E, should be carefully tuned, which restricts its usage in real life applications [20]. On the other hand, even though LSTM based algorithms provide high performance for time series data, we have to solve highly complex optimization problems to get an adequate performance [1]. As an example, the LSTM based anomaly detection algorithms in [10], [21] first predict time series data and then fit a multivariate Gaussian distribution to the error, where they also select a threshold for this distribution. Here, they allocate different set of sequences to learn the parameters of the distribution and threshold via the maximum likelihood estimation technique [10], [21]. Thus, the conventional LSTM based approaches require careful selection of several additional parameters, which significantly degrades their performance in real life [1], [10]. Furthermore, both the OC-SVM (or SVDD) and LSTM based methods are able to process only fixed length sequences [6], [7], [10]. To circumvent these issues, we introduce generic LSTM based anomaly detectors for variable length data sequences, where we jointly train the parameters of the LSTM architecture and the OC-SVM (or SVDD) formulation via a predefined objective function. Therefore, we not only obtain high performance for time series data but also enjoy joint and effective optimization of the parameters with respect to a well defined objective function.

C. Contributions
Our main contributions are as follows: • We introduce LSTM based anomaly detection algorithms in an unsupervised framework, where we also extend our derivations to the semi-supervised and fully supervised frameworks. • As the first time in the literature, we jointly train the parameters of the LSTM architecture and the OC-SVM (or SVDD) formulation via a well defined objective function, where we introduce two different joint optimization methods. For our gradient based joint optimization method, we modify the OC-SVM and SVDD formulations and then prove the convergence of the modified formulations to the original ones. • Thanks to our LSTM based structure, the introduced methods are able to process variable length data sequences. Additionally, unlike the conventional methods [6], [7], our methods effectively detect anomalies in time series data without requiring any preprocessing. • Through extensive set of experiments involving real and simulated data, we illustrate significant performance improvements achieved by our algorithms with respect to the conventional methods [6], [7]. Moreover, since our approach is generic, we also apply it to the recently proposed GRU architecture [17] in our experiments.

D. Organization of the Paper
The organization of this paper is as follows. In Section II, we first describe the variable length anomaly detection problem and then introduce our LSTM based structure. In Section III-A, we introduce anomaly detection algorithms based on the OC-SVM formulation, where we also propose two different joint training methods in order to learn the LSTM and SVM parameters. The merits of each different approach are also detailed in the same section. In a similar manner, we introduce anomaly detection algorithms based on the SVDD formulation and provide two different joint training methods to learn the parameters in Section III-B. In Section IV, we demonstrate performance improvements over several real life datasets. In the same section, thanks to our generic approach, we also introduce GRU based anomaly detection algorithms. Finally, we provide concluding remarks in Section V.

II. MODEL AND PROBLEM DESCRIPTION
In this paper, all vectors are column vectors and denoted by boldface lower case letters. Matrices are represented by boldface uppercase letters. For a vector a, a T is its ordinary transpose and ||a|| = √ a T a is the 2 -norm. The time index is given as subscript, e.g., a i is the i th vector. Here, 1 (and 0) is a vector of all ones (and zeros) and I represents the identity matrix, where the sizes are understood from the context.
We observe data sequences {X i } n i=1 , i.e., defined as where x i,j ∈ R p , ∀j ∈ {1, 2, . . . d i } and d i ∈ Z + is the number of columns in X i , which can vary with respect to i. Here, we assume that the bulk of the observed sequences are normal and the remaining sequences are anomalous. Our aim is to find a scoring (or decision) function to determine whether X i is anomalous or not based on the observed data, where +1 and −1 represent the outputs of the desired scoring function for nominal and anomalous data respectively. As an example application for this framework, in host based intrusion detection [1], the system handles operating system call traces, where the data consists of system calls that are generated by users or programs. All traces contain system calls that belong to the same alphabet, however, the co-occurrence of the system calls is the key issue in detecting anomalies [1]. For different programs, these system calls are executed in different sequences, where the length of the sequence may vary for each program. Binary encoding of a sample set of call sequences can be X 1 = 101011, X 2 = 1010 and X 3 = 1011001 for n = 3 case [1]. After observing such a set of call sequences, our aim is to find a scoring function that successfully distinguishes the anomalous call sequences from the normal sequences.
In order to find a scoring function l(·) such that one can use the OC-SVM algorithm [6] to find a hyperplane that separates the anomalies from the normal data or the SVDD algorithm [7] to find a hypersphere enclosing the normal data while leaving the anomalies outside the hypersphere. However, these algorithms can only process fixed length sequences. Hence, we use the LSTM architecture [14] to obtain a fixed length vector representation for each X i . Although there exist several different versions of LSTM architecture, we use the most widely employed architecture, i.e., the LSTM architecture without peephole connections [13]. We first feed X i to the LSTM architecture as demonstrated in Fig. 2, where the internal LSTM equations are as follows [14]: where  input vector and h i,j ∈ R m is the output vector for the j th LSTM unit in Fig. 2. Additionally, s i,j , f i,j and o i,j is the input, forget and output gates, respectively. Here, g(·) is set to the hyperbolic tangent function, i.e., tanh, and applies to input vectors pointwise. Similarly, σ(·) is set to the sigmoid function. is the operation for elementwise multiplication of two same sized vectors. Furthermore, W (·) , R (·) and b (·) are the parameters of the LSTM architecture, where the size of each is selected according to the dimensionality of the input and output vectors. After applying the LSTM architecture to each column of our data sequences as illustrated in Fig. 2, we take the average of the LSTM outputs for each data sequence, i.e., the mean pooling method. By this, we obtain a new set of fixed length sequences, i.e., denoted as {h i } n i=1 ,h i ∈ R m . Note that we also use the same procedure to obtain the state informationc i ∈ R m for each X i as demonstrated in Fig. 2.
Remark 1. We use the mean pooling method in order to obtain the fixed length sequences ash i = 1 di di j=1 h i,j . However, we can also use the other pooling methods. As an example, for the last and max pooling methods, we useh i = h i,di andh i = max j h i,j , ∀i ∈ {1, 2, . . . n}, respectively. Our derivations can be straightforwardly extended to these different pooling methods.

III. NOVEL ANOMALY DETECTION ALGORITHMS
In this section, we first formulate the anomaly detection approaches based on the OC-SVM and SVDD algorithms. We then provide joint optimization updates to train the parameters of the overall structure.

A. Anomaly Detection with the OC-SVM Algorithm
In this subsection, we provide an anomaly detection algorithm based on the OC-SVM formulation and derive the joint updates for both the LSTM and SVM parameters. For the training, we first provide a quadratic programming based algorithm and then introduce a gradient based training algorithm. To apply the gradient based training method, we smoothly approximate the original OC-SVM formulation and then prove the convergence of the approximated formulation to the actual one in the following subsections.
In the OC-SVM algorithm, our aim is to find a hyperplane that separates the anomalies from the normal data [6]. We formulate the OC-SVM optimization problem for the sequence {h i } n i=1 as follows [6] min θ∈R n θ ,w∈R m ,ξ∈R,ρ∈R where ρ and w are the parameters of the separating hyperplane, λ > 0 is a regularization parameter, ξ is a slack variable to penalize misclassified instances and we group the LSTM parameters Since the LSTM parameters are unknown andh i is a function of these parameters, we also minimize the cost function in (7) with respect to θ.
After solving the optimization problem in (7), (8) and (9), we use the scoring function to detect the anomalous data, where the sgn(·) function returns the sign of its input. We emphasize that while minimizing (7) with respect to θ, we might suffer from overfitting and impotent learning of time dependencies on the data [22], i.e., forcing the parameters to null values, e.g., θ = 0. To circumvent these issues, we introduce (9), which constraints the norm of θ to avoid overfitting and trivial solutions, e.g., θ = 0, while boosting the ability of the LSTM architecture to capture time dependencies [22], [23].
Remark 2. In (9), we use an orthogonality constraint for each LSTM parameter. However, we can also use other constraints instead of (9) and solve the optimization problem in (7), (8) and (9) in the same manner. As an example, a common choice of constraint for neural networks is the Frobenius norm [24], i.e., defined as for a real matrix A, where A ij represents the element at the i th column and j th row of A. In this case, we can directly replace (9) with a Frobenius norm constraint for each LSTM parameter as in (11) and then solve the optimization problem in the same manner. Such approaches only aim to regularize the parameters [23]. However, for RNNs, we may also encounter exponential growth or decay in the norm of the gradients while training the parameters, which significantly degrades capabilities of these architectures to capture time dependencies [22], [23]. Thus, in this paper, we put the constraint (9) in order to regularize the parameters while improving the capabilities of the LSTM architecture in capturing time dependencies [22], [23].
1) Quadratic Programming Based Training Algorithm: Here, we introduce a training approach based on quadratic programming for the optimization problem in (7), (8) and (9), where we perform consecutive updates for the LSTM and SVM parameters. For this purpose, we first convert the optimization problem to a dual form in the following. We then provide the consecutive updates for each parameter.
We have the following Lagrangian for the SVM parameters where ν i , α i ≥ 0 are the Lagrange multipliers. Taking derivative of (12) with respect to w, ξ and ρ and then setting the derivatives to zero gives Note that at the optimum, the inequalities in (8) become equalities if α i and ν i are nonzero, i.e., 0 < α i < 1/(nλ) [6]. With this relation, we compute ρ as By substituting (13) and (14) into (12), we obtain the following dual problem for the constrained minimization in (7), (8) and (9) min θ∈R n θ ,α∈R n where α ∈ R n is a vector representation for α i 's. Since the LSTM parameters are unknown, we also put the minimization term for θ into (16) as in (7). By substituting (13) into (10), we have the following scoring function for the dual problem where we calculate ρ using (15). In order to find the optimal θ and α for the optimization problem in (16), (17) and (18), we employ the following procedure. We first select a certain set of the LSTM parameters, i.e., θ 0 . Based on θ 0 , we find the minimizing α values, i.e., α 1 , using the Sequential Minimal Optimization (SMO) algorithm [25]. Now, we fix α as α 1 and then update θ from θ 0 to θ 1 using the algorithm for optimization with orthogonality constraints in [26]. We repeat these consecutive update procedures until α and θ converge [27]. Then, we use the converged values in order to evaluate (19). In the following, we explain the update procedures for θ and α in detail.
Based on θ k , i.e., the LSTM parameter vector at the k th iteration, we update α k , i.e., the α vector at the k th iteration, using the SMO algorithm due to its efficiency in solving quadratic constrained optimization problems [25]. In the SMO algorithm, we choose a subset of parameters to minimize and fix the rest of parameters. In the extreme case, we choose only one parameter to minimize, however, due to (17), we must choose at least two parameters. To illustrate how the SMO algorithm works in our case, we choose α 1 and α 2 to update and fix the rest of the parameters in (16). From (17), we have We first replace α 1 in (16) with (20). We then take the derivative of (16) with respect to α 2 and equate the derivative to zero. Thus, we obtain the following update for α 2 at the k th iteration where K ij h T ihj , M i n j=3 α k,j K ij and α k,i represents the i th element of α k . Due to (17), if the updated value of α 2 is outside of the region [0, 1/(nλ)], we project it to this region. Once α 2 is updated as α k+1,2 , we obtain α k+1,1 using (20). For the rest of the parameters, we repeat the same procedure, which eventually converges to a certain set of parameters [25]. By this way, we obtain α k+1 , i.e., the minimizing α for θ k .
Following the update of α, we update θ based on the updated α k+1 vector. For this purpose, we employ the optimization method in [26]. Since we have α k+1 that satisfies (17), we reduce the dual problem to For (22) and (23), we update W (·) as follows where the subscripts represent the current iteration index, µ is the learning rate, k G T k and the element at the i th row and the j th column of G, i.e., G ij , is defined as Remark 3. For R (·) and b (·) , we first compute the gradient of the objective function with respect to the chosen parameter as in (25). We then obtain A k according to the chosen parameter.
Using A k , we update the chosen parameter as in (24).
With these updates, we obtain a quadratic programming based training algorithm (see Algorithm 1 for the pseudocode) for our LSTM based anomaly detector.
2) Gradient Based Training Algorithm: Although the quadratic programming based training algorithm directly optimizes the original OC-SVM formulation without requiring any approximation, since it depends on the separated consecutive updates of the LSTM and OC-SVM parameters, it might not converge to even a local minimum [27]. In order to resolve this issue, in this subsection, we introduce a training method based on only the first order gradients, which updates the parameters at the same time. However, since we require an approximation to the original OC-SVM formulation to apply this method, we also prove the convergence of the approximated formulation to the original OC-SVM formulation in this subsection.
Considering (8), we write the slack variable in a different Using θ k , obtain {h} n i=1 according to Fig. 2 7: Find optimal α k+1 for {h} n i=1 using (20) and (21) 8: Based on α k+1 , obtain θ k+1 using (24) and Remark 3 9: while κ(θ k+1 , α k+1 ) − κ(θ k , α k ) 2 > 10: Detect anomalies using (19) evaluated at θ k and α k form as follows where β w,ρ (h i ) ρ − w Th i . By substituting (26) into (7), we remove the constraint (8) and obtain the following optimization problem min w∈R m ,ρ∈R,θ∈R n θ s.t.:W (·)T W (·) = I, R (·)T R (·) = I and b (·)T b (·) = 1. (28) Since (26) is not a differentiable function, we are unable to solve the optimization problem in (27) using gradient based optimization algorithms. Hence, we employ a differentiable function S τ (β w,ρ (h i )) = 1 τ log 1 + e τ βw,ρ(hi) (29) to smoothly approximate (26), where τ > 0 is the smoothing parameter and log represents the natural logarithm. In (29), as τ increases, S τ (·) converges to G(·) (see Proposition 1 at the end of this section), hence, we choose a large value for τ . With (29), we modify our optimization problem as follows min w∈R m ,ρ∈R,θ∈R n θ where F τ (·, ·, ·) is the objective function of our optimization problem and defined as To obtain the optimal parameters for (30) and (31), we update w, ρ and θ until they converge to a local or global optimum [26], [28]. For the update of w and ρ, we use the SGD algorithm [28], where we compute the first order gradient of the objective function with respect to each parameter. We first compute the gradient for w as follows Using (32), we update w as where the subscript k indicates the value of any parameter at the k th iteration. Similarly, we calculate the derivative of the objective function with respect to ρ as follows Using (34), we update ρ as For the LSTM parameters, we use the method for optimization with orthogonality constraints in [26] due to (31). To update each element of W (·) , we calculate the gradient of the objective function as ij e τ βw,ρ(hi) 1 + e τ βw,ρ(hi) .
We then update W (·) using (36) as where Remark 4. For R (·) and b (·) , we first compute the gradient of the objective function with respect to the chosen parameter as in (38). We then obtain B k according to the chosen parameter.
Using B k , we update the chosen parameter as in (37).

Remark 5.
In the semi-supervised framework, we have the following optimization problem for our SVM based algorithms [29] min θ,w,ξ,η,γ,ρ where γ ∈ R and η ∈ R are slack variables as ξ, C is a trade-off parameter, l and k are the number of the labeled and unlabeled data instances, respectively and y i ∈ {−1, +1} represents the label of the i th data instance. For the quadratic programming based training method, we modify all the steps from (12) to (25) with respect to (39)-(43). In a similar manner, we modify the equations from (26) to (38) according to (39)-(43) in order to get the gradient based training method in the semi-supervised framework. For the supervised implementation, we follow the same procedure with the semi-supervised implementation for k = 0 case.
Proof of Theorem 1. We have the following Hessian matrix of F τ (w, ρ, θ) with respect to w which satisfies v T ∇ 2 w F τ (w, ρ, θ)v > 0 for any nonzero column vector v. Hence, the Hessian matrix is positive definite, which shows that F τ (w, ρ, θ) is strictly convex function of w. Consequently, the solution w τ is both global and unique given any ρ and θ. Additionally, we have the following second order derivative for ρ e τ βw,ρ(hi) 1 + e τ βw,ρ(hi) 2 > 0, which implies that F τ (w, ρ, θ) is strictly convex function of ρ. As a result, the solution ρ τ is both global and unique for any given w and θ.

B. Anomaly Detection with the SVDD algorithm
In this subsection, we introduce an anomaly detection algorithm based on the SVDD formulation and provide the joint updates in order to learn both the LSTM and SVDD parameters. However, since the generic formulation is the same with the OC-SVM case, we only provide the required and distinct updates for the parameters and proof for the convergence of the approximated SVDD formulation to the actual one.
In the SVDD algorithm, we aim to find a hypersphere that encloses the normal data while leaving the anomalous data outside the hypersphere [7]. For the sequence {h i } n i=1 , we have the following SVDD optimization problem [7] min θ∈R n θ ,c∈R m ,ξ∈R,R∈R W (·)T W (·) = I, R (·)T R (·) = I and b (·)T b (·) = 1, where λ > 0 is a trade-off parameter between R 2 and the total misclassification error, R is the radius of the hypersphere andc is the center of the hypersphere. Additionally, θ and ξ represent the LSTM parameters and the slack variable respectively as in the OC-SVM case. After solving the constrained optimization problem in (46), (47) and (48), we detect anomalies using the following scoring function Obtain w k+1 , ρ k+1 and θ k+1 using (33), (35), (37) and Remark 4 8: Detect anomalies using (10) evaluated at w k , ρ k and θ k 1) Quadratic Programming Based Training Algorithm: In this subsection, we introduce a training algorithm based on quadratic programming for (46), (47) and (48). As in the OC-SVM case, we first assume that the LSTM parameters are fixed and then perform optimization over the SVDD parameters based on the fixed LSTM parameters. For (46) and (47), we have the following Lagrangian where ν i , α i ≥ 0 are the Lagrange multipliers. Taking derivative of (50) with respect toc, ξ and R and then setting the derivatives to zero yields Putting (51) and (52) into (50), we obtain a dual form for (46) and (47)  In order to solve the constrained optimization problem in (53), (54) and (55), we employ the same approach as in the OC-SVM case. We first fix a certain set of the LSTM parameters θ. Based on these parameters, we find the optimal α using the SMO algorithm. After that, we fix α to update θ using the algorithm for optimization with orthogonality constraints. We repeat these procedures until we reach convergence. Finally, we evaluate (56) based on the converged parameters.
Remark 6. In the SVDD case, we apply the SMO algorithm using the same procedures with the OC-SVM case. In particular, we first choose two parameters, e.g., α 1 and α 2 , to minimize and fix the other parameters. Due to (54), the chosen parameters must obey (20). Hence, we have the following update rule for α 2 at the k th iteration where S = n j=3 α k,j and the other definitions are the same with the OC-SVM case. We then obtain α k+1,1 using (20). By this, we obtain the updated values α k+1,2 and α k+1,1 . For the remaining parameters, we repeat this procedure until reaching convergence.

Remark 7.
For the SVDD case, we update W (·) at the k th iteration as in (24). However, instead of (25), we have the following definition for G at the k th iteration. For the remaining parameters, we follow the procedure in Remark 3.
Hence, we obtain a quadratic programming based training algorithm for our LSTM based anomaly detector, which is also described in Algorithm 3 as a pseudocode. Find optimal α k+1 for {h} n i=1 using the procedure in Remark 6 8: Based on α k+1 , obtain θ k+1 using Remark 7 9: while π(θ k+1 , α k+1 ) − π(θ k , α k ) 2 > 10: Detect anomalies using (56) evaluated at θ k and α k 2) Gradient Based Training Algorithm: In this subsection, we introduce a training algorithm based on only the first order gradients for (46), (47) and (48). We again use the G(·) function in (26) in order to eliminate the constraint in (47) as follows min θ∈R n θ ,c∈R m ,R∈R Since the gradient based methods cannot optimize (57) due to the nondifferentiable function G(·), we employ S τ (·) instead of G(·) and modify (57) as min θ∈R n θ ,c∈R m ,R∈R where F τ (·, ·, ·) is the objective function of (59). To obtain the optimal values for (59) and (60), we updatec, R and θ till we reach either a local or a global optimum. For the updates ofc and R, we employ the SGD algorithm, where we use the following gradient calculations. We first compute the gradient ofc as Using (61), we have the following updatẽ where the subscript k represents the iteration number. Likewise, we compute the derivative of the objective function with respect to R 2 as ∂F τ (c, R, θ) With (63), we update R 2 as For θ, the gradient calculation is as follows .
(65) Using (65), we have the following update where Remark 8. For R (·) and b (·) , we first compute the gradient of the objective function with respect to the chosen parameter as in (67). We then obtain B k according to the chosen parameter.
Using B k , we update the chosen parameter as in (66).

Remark 9.
In the semi-supervised framework, we have the following optimization problem for our SVDD based algorithms [30] min θ,c,R,ξ,γ,η where η ∈ R is a slack variable as ξ, γ ∈ R is the margin of the labeled data instances, C 1 , C 2 and C 3 are tradeoff parameters, k and l are the number of the labeled and unlabeled data instances, respectively and y j ∈ {−1, +1} represents the label of the j th data instance. For the quadratic programming based training method, we modify all the steps from (50) to (56), Remark 6 and Remark 7 with respect to (68)-(71). In a similar manner, we modify the equations from (57) to (67) according to (68)-(71) in order to obtain the gradient based training method in the semisupervised framework. For the supervised implementation, we follow the same procedure with the semi-supervised implementation for l = 0 case.
The complete algorithm is provided in Algorithm 4. In the following, we provide the convergence proof as in the OC-SVM case.
Theorem 2. Letc τ and R 2 τ be the solutions of (59) for any fixed θ. Then,c τ and R 2 τ are unique and F τ (c τ , R τ , θ) converges to the minimum of F (c, R, θ), i.e., defined as Proof of Theorem 2. We have the following Hessian matrix of F τ (c, R, θ) with respect toc where Ω i = e τ Ψc ,R (hi) , which implies v T ∇ 2 c F τ (c, R, θ)v > 0 for any nonzero column vector v. Thus, the Hessian matrix is positive definite, which shows that F τ (w, ρ, θ) is strictly convex function ofc. As a result, the solutionc τ is both global and unique given any R and θ. In addition to this, we have the following second order derivative for R 2 1 + e τ Ψc ,R (hi) 2 > 0, which implies that F τ (c, R, θ) is strictly convex function of R 2 . Therefore, the solution R 2 τ is both global and unique for any givenc and θ.
The convergence proof directly follows the proof of Theorem 1.

IV. SIMULATIONS
In this section, we demonstrate the performances of the algorithms on several different datasets. We first evaluate the performances on a dataset that contains variable length data Using θ k , obtain {h} n i=1 according to Fig. 2 7: Obtainc k+1 , R 2 k+1 and θ k+1 using (62), (64), (66) and Remark 8 8: Detect anomalies using (49) evaluated atc k , R 2 k and θ k sequences, i.e., the digit dataset [31]. We then compare the anomaly detection performances on several different benchmark real datasets such as the occupancy [32], Hong Kong Exchange (HKE) rate [33], http [34] and Alcoa stock price [35] datasets. While performing experiments on real benchmark datasets, we also include the GRU based algorithms in order to compare their performances with the LSTM based ones. Note that since the introduced algorithms have bounded functions, e.g., the sigmoid function in the LSTM architecture, for all the experiments in this section, we normalize each dimension of the datasets into [−1, 1]. Throughout this section, we denote the LSTM based OC-SVM anomaly detectors that are trained with the gradient and quadratic programming based algorithms as "LSTM-GSVM" and "LSTM-QPSVM", respectively. In a similar manner, we use "LSTM-GSVDD" and "LSTM-QPSVDD" for the SVDD based anomaly detectors. Moreover, for the labels of the GRU based algorithms, we replace the LSTM prefix with GRU.

A. Anomaly Detection for Variable Length Data Sequences
In this section, we evaluate the performances of the introduced anomaly detectors on the digit dataset [31]. In this dataset, we have the pixel samples of digits, which were written on a tablet by several different authors [31]. Since the speed of writing varies from person to person, the number of samples for a certain digit might significantly differ. The introduced algorithms are able to process such kind of sequences thanks to their generic structure in Fig. 2. However, the conventional OC-SVM and SVDD algorithms cannot directly process these sequences [6], [7]. For these algorithms, we take the mean of each sequence to obtain a fixed length vector sequence, i.e., two dimensional in this case (two coordinates of a pixel). In order to evaluate the performances, we first choose a digit as normal and another digit as anomaly. We emphasize that randomly choose digits for illustration and we obtain similar performances for the other digits. We then divide the samples of these digits into training and test parts, where we allocate 60% of the samples for the training part and 40% for the test part. In both the training and test parts, we select the samples so that 10% of the samples are anomalies. Then, using the training part, we optimize the parameters of each algorithm using two fold cross validation, where we also select certain crucial parameter, e.g., µ. This procedure results in µ = 0.05, 0.001, 0.05 and 0.01 for LSTM-GSVM, LSTM-QPSVM, LSTM-GSVDD and LSTM-QPSVDD, respectively. Furthermore, we select the output dimension of the LSTM architecture as m = 2 and the regularization parameter as λ = 0.5 for all the algorithms. For the implementation of the conventional OC-SVM and SVDD algorithms, we use the libsvm library and their parameters are selected in a similar manner via built in optimization tools of libsvm [36].
Here, we use area under ROC curve as a performance metric [37]. In a ROC curve, we plot true positive rate (TPR) as a function of false positive rate (FPR). Area under this curve, i.e., also known as AUC, is a well known performance measure for anomaly detection tasks [37]. In Fig. 3a and 3b, we illustrate the ROC curves and provide the corresponding AUC scores, where we label digit "0" and "9" as normal and anomaly, respectively. For the OC-SVM and SVDD algorithms, since we directly take the mean of variable length data sequences to obtain fixed length sequences, they achieve significantly lower AUC scores compared to the introduced LSTM based methods. Among the LSTM based methods, LSTM-GSVM slightly outperforms LSTM-QPSVM. On the other hand, LSTM-GSVDD achieves significantly higher AUC than LSTM-QPSVDD. Since the quadratic programming based training method depends on the separated consecutive updates of the LSTM and SVM (or SVDD) parameters, it might not converge to even a local minimum. However, the gradient based method can guarantee convergence to at least a local minimum given a proper choice of the learning rate [28]. Thus, although these methods might provide similar performances as in Fig. 3a, it is also expected to obtain much higher performance from the gradient based method for certain cases as in Fig. 3b. However, overall, the introduced algorithms provide significantly higher AUC than the conventional methods.
Besides the previous scenario, we also consider a scenario, where we label digit "1" and "7" as normal and anomaly, respectively. In Fig. 4a and 4b, we illustrate the ROC curves and provide the corresponding AUC scores. As in the previous scenario, for both the SVM and SVDD cases, the introduced algorithms achieve higher AUC scores than the conventional algorithms. Among the introduced algorithms, LSTM-GSVM and LSTM-GSVDD achieve the highest AUC scores for the SVM and SVDD cases, respectively. Furthermore, the AUC score of each algorithm is much lower compared to the previous case due to the similarity between digit "1" and "7".

B. Benchmark Real Datasets
In this section, we compare the AUC scores of each algorithm on several different real benchmark datasets. Since our approach in this paper is generic, in addition to the LSTM based algorithms, we also implement our approach on the recently introduced RNN architecture, i.e., the GRU architecture, which is defined by the following equations [17]: where h i,j ∈ R m is the output vector and x i,j ∈ R p is the input vector. Furthermore, W (·) and R (·) are the parameters of the GRU, where the sizes are selected according to the dimensionality of the input and output vectors. We then replace (1)-(6) with (72)-(75) in Fig. 2 to obtain GRU based anomaly detectors.
We first evaluate the performances of the algorithms on the occupancy dataset [32]. In this dataset, we have five features, which are relative humidity percentage, light (in lux), carbon dioxide level (in ppm), temperature (in Celsius) and humidity ratio, and our aim is to determine whether an office room is occupied or not based on the features. Here, we use the same procedure with the previous subsection to separate the test and training data. Moreover, using the training data, we select µ = 0.05, 0.05, 0.001 and 0.01 for LSTM-GSVM, LSTM-QPSVM, LSTM-GSVDD and LSTM-QPSVDD, respectively. Note that, for the GRU based algorithms in this subsection, we use the same parameter setting with the LSTM based algorithms. Furthermore, we choose m = 5 and λ = 0.5 for all of the experiments in this subsection in order to maximize the performances of the algorithms.
As can be seen in Table I, due to their inherent memory, both the LSTM and GRU based algorithms achieve considerably high AUC scores compared to the conventional SVM and SVDD algorithms. Moreover, GRU-GSVDD achieves the highest AUC score among all the algorithms, where the LSTM based algorithms (LSTM-GSVM and LSTM-QPSVM) also provide comparable AUC scores. Here, we also observe that the gradient based training method provides higher AUC scores compared to the quadratic programming based training method, which might stem from its separated update procedure that does not guarantee convergence to a certain local minimum.
Other than the occupancy dataset, we also perform an experiment on the HKE rate dataset in order to examine the performances for a real life financial scenario. In this dataset, we have the amount of Hong Kong dollars that one can buy for one US dollar on each day. In order to introduce anomalies to this dataset, we artificially add samples from a Gaussian distribution with the mean and ten times the variance of the training data. Furthermore, using the training data, we select µ = 0.01, 0.005, 0.05 and 0.05 for LSTM-GSVM, LSTM-QPSVM, LSTM-GSVDD and LSTM-QPSVDD, respectively.
In Table I, we illustrate the AUC scores of the algorithms on the HKE rate dataset. Since we have time series data, both the LSTM and GRU based algorithms naturally outperform the conventional methods thanks to their inherent memory, which preserves sequential information. Moreover, since the LSTM architecture also controls its memory content via output gate unlike the GRU architecture [17], we obtain the highest AUC scores from LSTM-GSVM. As in the previous cases, the gradient based training method provides better performance than the quadratic programming based training.
We also evaluate the AUC scores of the algorithms on the http dataset [34]. In this dataset, we have 4 features, which are duration (number of seconds of the connection), network service, number of bytes from source to destination and from destination to source. Using these features, we aim to distinguish normal connections from network attacks. In this experiment, we select µ = 0.01, 0.05, 0.001 and 0.01 for LSTM-GSVM, LSTM-QPSVM, LSTM-GSVDD and LSTM-QPSVDD, respectively. We demonstrate the performances of the algorithms on the http dataset in Table I. Even though all the algorithms achieve high AUC scores on this dataset, we still observe that the LSTM and GRU based algorithms have higher AUC scores than the conventional SVM and SVDD methods. Overall, GRU-QPSVDD achieves the highest AUC score and the quadratic programming based training methods performs better than the gradient based training method on this dataset. However, since the AUC scores are very high and close to each other, we observe only slight performance improvement for our algorithms in this case.
As the last experiment, we evaluate the anomaly detection performances of the algorithms on another financial dataset, i.e., the Alcoa stock price dataset [35]. In this dataset, we have daily stock price values. As in the HKE rate dataset, we again artificially introduce anomalies via a Gaussian distribution with the mean and ten times the variance of the training data. Moreover, we choose µ = 0.01, 0.001, 0.001 and 0.005 for LSTM-GSVM, LSTM-QPSVM, LSTM-GSVDD and LSTM-QPSVDD, respectively.
In Table I, we illustrate the AUC scores of the algorithms on the Alcoa stock price dataset. Here, we observe that the GRU and LSTM based algorithms achieve considerably higher AUC scores than the conventional methods thanks to their memory structure. Although the LSTM based algorithms have higher AUC in general, we obtain the highest AUC score from GRU-QPSVDD. Moreover, as in the previous experiments, the gradient based training method provides higher performance compared to the quadratic programming based method thanks to its learning capabilities.
V. CONCLUDING REMARKS In this paper, we study anomaly detection in an unsupervised framework and introduce LSTM based algorithms. Particularly, we have introduced a generic LSTM based   structure in order to process variable length data sequences. After obtaining fixed length sequences via our LSTM based structure, we introduce a scoring function for our anomaly detectors based on the OC-SVM [6] and SVDD [7] algorithms. As the first time in the literature, we jointly optimize the parameters of both the LSTM architecture and the final scoring function of the OC-SVM (or SVDD) formulation. To jointly optimize the parameters of our algorithms, we have also introduced gradient and quadratic programming based training methods with different algorithmic merits, where we extend our derivations for these algorithms to the semi-supervised and fully supervised frameworks. In order to apply the gradient based training method, we modify the OC-SVM and SVDD formulations and then provide the convergence results of the modified formulations to the actual ones. Therefore, we obtain highly effective anomaly detection algorithms, especially for time series data, that are able to process variable length data sequences. In our simulations, due to the generic structure of our approach, we have also introduced GRU based anomaly detection algorithms. Through extensive set of experiments, we illustrate significant performance improvements achieved by our algorithms with respect to the conventional methods [6], [7] over several different real and simulated datasets.


Examining Service Innovation Competencies of Turkish Deposit Banks With Fuzzy ANP, Fuzzy TOPSIS and Fuzzy VIKOR Methods

It is aimed to rank the service innovation issues of deposit banks in Turkey. In this context, 26 deposit banks of Turkey are analyzed by considering 16 different balanced scorecard-based criteria. Regarding the methodology, first of all, fuzzy ANP method is used to measure the significance of dimensions and criteria. It is defined that the most significant dimension is performance. It is also seen that organizational compliance (D3) is on the last rank. Additionally, return on investment (C1), profitability (C2) and customer expectations (C5) have highest significance. These alternatives are ranked by considering fuzzy TOPSIS and fuzzy VIKOR methods. It is understood that the results of both these two methods are very similar. It is concluded that there is not a comparative advantage among the banks regarding the ownership type. For example, the best bank (F15) is a foreign bank whereas the worst bank (F13) is also another foreign bank. Similar to this situation, some private banks (P1, P7, P8) have successful performance while some others (P4, P5) have weak performance.

INTRODUCTION
Service innovation is a very signi icant issue for almost all companies. The main reason is that globalization led to high competition between the companies in all industries (Ayuningrat, Noermijati., & Hadiwidjojo, 2016;Lamberth-Cocca & Meiren, 2017). Because of this situation, it can be said that in order to increase competitive power, companies should take some actions, such as service innovation. Otherwise, these companies cannot be preferred by the customers. Hence, it will not possible for these companies to survive in the market (Bernik, Azis, Kartini, & Harsanto, 2015;. While considering this situation emphasized below, it is understood that evaluating the performance of the companies regarding service innovation is crucial. Most of the methods to achieve the performance consider only inancial aspects. However, it can be said that these kinds of methods are not suf icient to achieve this objective. Within this framework, the popularity of balanced scorecard method increases since non-inancial issues are also considered in this process (Dinçer, Hacıoglu, & Yuksel, 2017;Jingnan, Yunus, & Kamal, 2018). This study aims to analyze Turkish banking sector for service innovation. For this purpose, 16 different criteria considering balanced scorecard dimensions are determined. Additionally, fuzzy ANP, TOPSIS and VIKOR methods are used. The criteria are assessed by using fuzzy ANP. Moreover, fuzzy TOPSIS and VIKOR approaches are considered for ranking different banking groups (state, foreign and private) regarding service innovation. This study has six different parts. After the introduction part, balanced scorecard approach will be detailed. In this part, four different balanced scorecard dimensions are explained. In the third part, literature is reviewed. The fourth part fuzzy ANP, TOPSIS and VIKOR methods are identi ied. Moreover, in the ifth part, the details of analysis results are explained. In the conclusion part, suggestions are shared based on these results.

Balanced Scorecard Approach
Especially in the last decades, competition among the companies increased very much due to the globalization. Therefore, measuring the performance of the companies play a more signi icant role to recognize any de iciency before it causes bigger problems. Within this framework, it is understood that classical performance measurement methods are not suf icient to satisfy this condition. The main is that these classical methods mainly focus on the inancial situation of the companies while measuring the performance. The signi icant difference between the balanced scorecard method from the others is that it also includes non-inancial issues in addition to the inancial factors. The term "balanced" refers to the condition that this method gives balanced importance to inancial and non-inancial aspects. This method mainly includes following four different dimensions (Kaplan & Norton, 2007;Tangpornpaiboon & Puttanapong, 2016). • Finance: The inancial performance is analyzed. It helps to learn whether there is a problem in the company regarding inancial aspect . • Customer: It analyzes the quality of the image of the companies. Within this framework, it concentrates on the expectations of the customers to increase customer satisfaction (Dincer, Yuksel, & Martinez, 2019;Kuo & Chen, 2015). • Internal Process: In this dimension, employee participation plays an important role. This situation contributes both the performance of the employees and the companies. When employee participate any aspects in the company, it increases the motivation of them. On the other side, by considering different ideas of the companies, company can reach its objectives more easily (Dinçer & Yuksel, 2018;Kozina, 2017). • Learning and Growth: This dimension de ines quali ication of the employees and adaption of the companies to any changes in the market (Alahoul, Azizan, & Alwi, 2016;Dincer, Yuksel, & Cetiner, 2019).

LITERATURE REVIEW
Service innovation concept is considered for many different aspects. Grif in and Page (1996) identi ied new services as investments. Therefore, they analyzed the returns of these investments. Y. Liu and Yang (2009) emphasized the aspect that successful service innovation contributes the pro itability of the companies. Moreover, Tajeddini (2011) underlined that service innovation provided cost ef iciencies. Furthermore, Grif in (1997) also determined the same conclusions. Furthermore, some other studies emphasized the effect of new service development on customer relationship. De Brentani (1995) identi ied that if new service development process could performed successfully, it helps to meet customer demands. Furthermore, Wu, Tzeng, and Chen (2009) determined that effective service innovation process increases customer satisfaction. Similarly, Y. Liu andYang (2009), Wu et al. (2009) showed that service innovation provides long term effective relationship with the customers. Additionally, the in luence of service innovation on organizational compliance is also emphasized in many different studies. For example, Homburg and Kuehnl (2014), Y. Liu and Yang (2009) and Perks and Riihela (2004) underlined that it has a positive effect on personnel motivation. Schilling and Hill (1998) de ined that with service innovation, employees can understand the goals of the organization. Edvardsson, Meiren, Schäfer, and Witell (2013) stated that employees should be considered in this aspect. Similar to these studies, Melton and Hartline (2010) and Stevens and Dimitriadis (2004) also underlined that employee participation in this process improves the quality of this process. Also, some researchers also considered the importance of technological improvement in new service development. Brown and Eisenhardt (1995) determined that in service innovation projects, market-based databases should be considered and that underlined that for this purpose, technological improvement is crucial. Similar to these studies, S. Liu (2012), Van Riel and Lievens (2004), Van Den Ende (2003) and Ittner and Larcker (1997) also identi ied that with the help of technological improvement, the data low in new service development process can be provided effectively. On the other side, Kuczmarski (1992) focused on the importance of professional training activities for the employees. Consequently, it is identi ied that many different studies focus on new service development concept in different aspects. For instance, some of them underline the importance of inancial situation while some others concentrate on the customer relationship in this process. In addition to these subjects, organizational compliance and the importance of training are also taken into the consideration. Therefore, a new study which analyzes all these concepts together in service innovation process. Within this context, it is obvi-ous that balanced scorecard approach helps to achieve this objective since it has all these concepts. Table 1 gives information about key performance indicators of service innovation competencies.  Page (1996), Oktar and Yuksel (2015), Yuksel (2017)

METHODOLOGY Fuzzy ANP Method
People face many dif iculties in order to make decision. The main reason is that they have to consider lots of dif icult situation at the same time in this process. ANP aims to make decisions in complex conditions. In addition to this situation, clusters can also affect each other in ANP method (Saaty, 1990). The details of ANP system are given below ISSN: 2414-309X DOI: 10.20474/jabs-5.1.3 (Chang, Kuo, Wu, & Tzeng, 2015). First of all, the problem should be explained in detail. Within this scope, the purpose, criteria and sub criteria related to this problem should be identi ied. Moreover, interaction between the elements is taken into the consideration. As a result, a supermatrix can be created. After that, a weighted matrix is developed by using the importance of the elements. These matrixes are illustrated below. In these matrixes, "a" represent criteria whereas "w" shows the weights (Dincer, Hacioglu, & Yuksel, 2016).
In this process, Chang's fuzzy extent analysis is considered which is also detailed below (Chang et al., 2015). a-"Fuzzy Synthetic Extent" (Si) is identi ied. The calculation is shown below.

Fuzzy TOPSIS Method
This approach is used to make decisions in complex environment. Hwang and Yoon developed this method in 1981. It refers to the irst letters of "technique for order of preference by similarity to ideal solution". The main advantage of this method is to use and make comments to the about results easily. In the analysis process, both positive and negative ideal solutions are generated (Dincer et al., 2016). In the equation below, X ij shows the value for criterion "j" and alternative "i". Additionally, it can be said that there are K decision makers.
Where i = 1, 2, 3, ..m and j = 1, 2, 3..n W j = 1 K + W 1 j + W 2 j + W 3 j + · · · + W K j While using these fuzzy weights, fuzzy decision matrix can be generated. The details are shown below. In this equation, "C" represents criteria while "A" gives information about the alternatives.
In the following step, fuzzy decision matrix is normalized by using the following equations.
Just then, positive and negative solutions are calculated with the formulas below.
In this context, "A+" demonstrates the fuzzy positive ideal solution whereas "A-" shows the fuzzy negative ideal solution. Furthermore, "Di-" refers to the distance from negative ideal solution. Moreover, "Di+" shows the distance from positive ideal solution. Hence, the closeness coef icient can be calculated by using following equations. people face many dif iculties when they try to select the best alternative. In VIKOR method, C refers to the criteria whereas A shows the alternatives. The different steps of this method are detailed below (Dincer et al., 2016). In the irst step, the criteria are identi ied as following. f * i = max f ij and f − i = min f ij On the other side, for the cost issue, the following equations are considered. f * i = min f ij and f − i = max f ij In the second step, Sj and Rj values are calculated as below. In these equations, w represents weights of the criteria.
In the third step, the index value (Qj) is calculated as below.
In the fourth step, the values are ranked.

ANALYSIS RESULTS
The irst phase of the analysis includes the fuzzy ANP methodology to measure the relative importance of dimensions and criteria. The project team has been also been deined as a decision-maker group to appoint the linguistic terms. The summary results can be seen in Table 2.  Table 2 shows that performance dimension (D1) has the highest importance in the balanced scorecard perspectives while the organizational compliance (D3) has the weakest weight of new service development competencies. Accordingly, the return on investment (C5) and customer expectation (C8) are the most importance criteria as the commitment (C8) has the weakest criterion with 0.02.
Following phase continues with the ranking the alternatives by using the fuzzy TOPSIS and fuzzy VIKOR methods comparatively. Both methods used the linguistic terms provided from the group decision. The following steps of the fuzzy TOPSIS have been applied to rank alternatives by calculating the negative D − 1 and positive D * 1 ideal solution as well as the closeness coef icient CC 1 .  In Table 3, "A" shows the alternatives. With respect to the ownership, "S" demonstrates state banks, "P" refers to the private banks and "F" focuses on the foreign banks. A26 is the best bank in service innovation competencies, and A24 has the worst degree of the service innovation performance. The fuzzy VIKOR is used to provide the comparative results of the banks' new service development competencies as well. The results are demonnstrated in Table 4.    S3  11  11  A4  P1  3  3  A5  P2  20  19  A6  P3  15  14  A7  P4  22  20  A8  P5  24  24  A9  P6  7  5  A10  P7  2  2  A11  P8  5  4  A12  F1  16  16  A13  F2  25  25  A14  F3  17  17  A15  F4  18  21  A16  F5  13  13  A17  F6  9  9  A18  F7  12  12  A19  F8  10  10  A20  F9  14  15  A21  F10  21  22  A22  F11  8  8  A23  F12  19  18  A24  F13  26  26  A25  F14  23  23  A26  F15  1  1 In Table 4, A26 is the best in the new service development competencies while A24 has the weakest performance in the banking sector. Table 5 presents the comparative results of two integrated decision making approaches under the fuzzy environment. FANP-FTOPSIS and FANP-FVIKOR models give the same re-sults for selecting the best and the worst bank in the performance measurement. Thus, both approaches could provide the coherent results for ranking the banks in the performance of service innovation competencies.

CONCLUSION
Nowadays, banks all around the world have to give very much importance to service innovation concept to increase the competitive advantage. Otherwise, they cannot ind a chance to survive in such a severe competition. This study aimed to assess the service innovation performance. In the analysis process, 26 Turkish deposit banks are examined. Also, 16 different criteria are identi ied by considering 4 different dimensions of balanced scorecard approach. Moreover, fuzzy ANP, fuzzy TOPSIS and fuzzy VIKOR are considered to achieve this purpose. It is concluded that performance (D1) has the highest importance (0.34). Furthermore, it is also determined that internal factors have the lowest weight (0.18). Furthermore, it is also identi ied that return on investment (C1), proitability (C2) and customer expectations (C5) are the most important criteria. However, commitment (C8), contribution (C11) and encouragement (C12) are accepted as the least important criteria. It is also de ined that the results of fuzzy TOPSIS and fuzzy VIKOR are very similar. In other words, both of these two methods give similar ranking results for the banks with respect to new service development competencies. In spite of this situation, by analyzing the performance of the banks according to the ownership type, it is identi ied there is not a comparative advantage of one type to another. For instance, the best bank in both methods is a foreign bank (F15). However, another foreign bank (F13) is also determined as the worst bank. Similarly, a private bank (P7) has the second highest performance while another private bank (P5) has a very bad performance. The concept of service innovation has a high popularity in banking sector. Banks have to consider this situation to increase power. Additionally, it can also be said that new service development concept will probably become more important in the future. Hence, this study has signi icant results by focusing on a very important topic for banking sector.


Grammatical Sequence Prediction for Real-Time Neural Semantic Parsing

While sequence-to-sequence (seq2seq) models achieve state-of-the-art performance in many natural language processing tasks, they can be too slow for real-time applications. One performance bottleneck is predicting the most likely next token over a large vocabulary; methods to circumvent this bottleneck are a current research topic. We focus specifically on using seq2seq models for semantic parsing, where we observe that grammars often exist which specify valid formal representations of utterance semantics. By developing a generic approach for restricting the predictions of a seq2seq model to grammatically permissible continuations, we arrive at a widely applicable technique for speeding up semantic parsing. The technique leads to a 74% speed-up on an in-house dataset with a large vocabulary, compared to the same neural model without grammatical restrictions

Introduction
Executable semantic parsing is the task of mapping an utterance to a logical form (LF) that can be executed against a data store (such as a SQL database or a knowledge graph), or interpreted by a computer program in some other way. 1 Various authors have tackled this task via sequenceto-sequence (seq2seq) models, which have already led to substantial advances in machine translation. These models learn to directly map the input utterance into a linearised representation of the corresponding LF, predicting it token by token. Seq2seq approaches have yielded state-ofthe-art accuracy on both classic (e.g., Geoquery (Zelle and Mooney, 1996) and Atis (Dahl et al., 1994)) and more recent semantic parsing datasets (e.g., WebQuestions, WikiSQL and Spider) (Liang et al., 2017;Lapata, 2016, 2018;Yin and Neubig, 2018;Yu et al., 2018). The recent datasets are of much larger scale, which not only enables the use of more data-hungry models, such as deep neural networks, but also provides more complex challenges for semantic parsing.
The material presented in this paper was motivated by a question-answering dataset for equity search in a financial data and analytics system. We will refer to this dataset as "the EQS dataset" going forward (and we will refer to "equity search" as EQS for short). The queries in the dataset pertain to equity stocks; they are usually of the form Show me companies that satisfy suchand-such criteria, or What are the top 10 companies that · · ·?, and so on. The dataset pairs such queries with logical forms that capture their semantics. These logical forms are designed to be readily translatable into an executable query language in order to retrieve the corresponding answers from a data store in the back end. Questions can involve a large number of diverse search criteria, such as price, earnings per share, country of domicile, membership in indices, trading in specific exchanges, etc., applied to a large set of equities for which the system offers information.
The large number of search criteria and entities is reflected in the LFs, leading to a problem common with newer, more complex semantic-parsing datasets: having to deal with a large LF vocabulary size. In the EQS dataset the LF vocabulary has a size that exceeds 50,000. Since seq2seq models apply some operation over the whole vocabulary -usually the softmax operation -when deciding what symbol to output next, large LF vocabularies can slow them down considerably. For example, we observe in our EQS experiments with seq2seq models that it takes on average between 250 and 300 milliseconds to parse a query, which is too slow for one single component in a larger, real-time question-answering pipeline. This is consistent with observations made previously in the neural language modelling literature; see for example Bengio et al. (2003); Mikolov et al. (2010), where the authors show that when the vocabulary size exceeds a certain threshold, the softmax calculation becomes the computational bottleneck.
Our proposal for tackling this bottleneck is based on the fact that there generally exist grammars, which we call LF grammars, specifying the concrete syntax of valid logical forms (LFs). This is usually the case because LFs need to be machine-readable. We further note that, for a given LF prefix, one can usually use the LF grammar to look up the next grammatically permissible tokens (i.e., tokens that are part of a grammatically valid completion of the prefix). For example, if the language of valid LFs can be expressed by a context-free grammar (CFG), as is almost always the case, then look-ups could be performed with an online version of the Earley parser (Earley, 1970). If it is possible to efficiently look up the permissible next tokens for a given prefix, then restricting the softmax operation to those permissible tokens should improve efficiency, and because only nonpermissible tokens are ruled out, this will only ever prevent the system from producing invalid LFs.
If the number of grammatically permissible tokens at some prediction step is substantially smaller than the LF's vocabulary size, the integration of the LF grammar may reduce prediction time for that step significantly. In semantic parsing problems a grammar can naturally lead to prediction steps with few choices. To see why this might be the case, consider our LFs in Figure 1, which involve atomic constraints of the form: (field operator value).
While there are many grammatically permissible choices for field and value, the choices for operator are rather limited. 2 LFs for many applications will contain "structural" elements with a limited number of choices in grammatically predictable positions, and we can use grammars to exploit this fact.
In order to make the computation of permissible next tokens efficient, we propose to use a finitestate automaton (FSA) approximation of the LF grammar. Finite-state automata can capture local 2 Equality, less than and so on. relations that are often quite predictive of the admissible tokens in a given context, and can therefore lead to considerable speed improvements for our setting, even if we use an approximate grammar. Moreover, approximations can be designed in such a way that a FSA accepts a superset of the actual LF language, preserving the guarantee that only ill-formed LFs will ever be ruled out.
In this paper we therefore work with a grammar for which the next permissible tokens can be computed efficiently, and show how such a grammar can be combined with a seq2seq model in order to substantially improve the efficiency of inference. While we focus on using FSAs to restrict a recurrent neural network with attention in the EQS dataset, our approach is generic and could be used to speed up any sequential prediction model with any grammar that allows for efficient computation of next-token sets. Our experiments show that in our domain of interest we obtain a reduction in parsing time by up to 74%. Some sample queries: • What are the top five Asian tech companies?
• Show me all auto firms traded in Nysex whose market cap last quarter was over $1 billion • Top 10 European non-German tech firms sorted by p/b ratio Queries may also be expressed in much more telegraphic style, e.g., the second query could also be phrased as auto nysex last quarter mcap > $1bn. The two queries in Figure 1 are additional examples of tersely formulated queries, the first one asking to display the return-on-capital for all companies in the S&P 500 index, and the second one asking for all Western European companies in the steel sector except for German companies. The LF language we use was designed to express the search intent of a query in a clear and non-ambiguous way. In the following section we describe the abstract grammar and concrete syntax of a subset of this LF (we cannot treat every construct due to space limitations).

LF Abstract Grammar and Concrete Syntax
As with many formal logical languages, the abstract grammar of our LF naturally falls into two classes: atomic LFs corresponding to individual logical or operational constraints; and complex LFs that contain other LFs as proper parts. The former constitute the basis case of the inductive definition of the LF grammar, while the latter correspond to the recursive clauses.

Relational Atomic Constraints
The main atomic constraints of interest in this domain are relational, of the form where typically field is either a numeric field (such as price); or a so-called "enum field," that is, an enumerated type. An example here would be a field such as a credit rating (say, long-term Fitch ratings), which has a a finite number of values (such as B+, AAA, etc.); or country of domicile, which also has a a finite number of values (algeria, belgium, and so on); or an index field, whose values are the major stock indices (such as the S&P 500). The value is a numeric value if the corresponding field is numeric, though it may be a complex numeric value, e.g., one that has currencies or denominations attached to it (such as "5 billion dollars"). The operator op is either equality (EQ), inequality (NEQ), less-than (LS), greaterthan (GR), less-than-or-equal (LE), etc. 4 Note that all fields, both numeric and enum, are indexed by a time expression t, representing the value of that field at that particular time. For example, the atomic constraint (price(June 23, 2018) = $100) states that the (closing) price on June 23, 2018 was 100 USD. We drop the time t when it is either immaterial or the respective field is not time sensitive. We omit the specification of the grammar and semantics of time expressions, since we will not be using times in what follows in order to simplify the discussion.
Display Atomic Constraints Some of our atomic constraints are operational in the sense that they represent directives about what fields to display as the query result, possibly along with auxiliary presentation information such as sorting order. For instance, for the query Show me the market caps and revenues of asian tech firms, two of the resulting constraints would be the display directives (display FLD MKT CAP) and (display FLD SALES REV TURN).
Complex Constraints Complex constraints are boolean combinations of other constraints, obtained by applying one of the operations NOT, OR, AND, resulting in recursively built constraints of the form (NOT c), (AND c 1 c 2 · · · c n ), and (OR c 1 c 2 ). 5

Encoding LF grammar in FSAs
For efficient incremental parsing and computation of the next permissible tokens, we encode our grammar using finite state automata (FSAs). As FSAs can only produce regular languages that are strictly less expressive than context free languages such as the one recognized by our LF grammar, our strategy is to use automata to build a superset for our LF language. Some of the automata involved in building this superset are shown in Figure 2. Note that, while we defined our FSA approximation manually, there exist general techniques to construct an automaton whose language is a superset of a CFG's language for any given CFG (Nederhof, 2000). This means that the approach could easily be used for any LF language that can be described by a CFG. For all automata, we take the start states to be 0 and indicate the final states with double circles. The "|" stands for the union operation over automata. On each arc, we either specify as labels LF tokens that the FSA can consume in order to transition to its next state(s); or else we specify a previously defined machine (automaton) noted with "M :machine name" 6 where the source state of the arc coincides with the start state of the automaton and its target state coincides with the final state(s) of the automaton. 7 Relational Atomic Constraint machines The automaton RCM ("Relational Constraint Machine") in the top part of Figure 2 generates relational atomic constraints of the form 6 In that case the "arc" is just a concise representation of the entire automaton that goes by machine name. 7 In the case of multiple final states, one simply replicates the target state to coincide with each of the final states.
(field op value); the FPNM ( floating point number machine) is an automaton recognizing restricted floating point numbers. Note that some extra-syntactic information about fields is explicitly built into the machine. For example, if a constraint begins with an unordered enum field that only admits equality, such as FLD DOMICILE, then the operator (on the arc from state 2 to state 3) is always EQ, whereas if the field is ordered (as all numeric fields are, and some enum fields such as ratings), then any operator may follow (such as LE, GR, etc.). The automaton constrains what follows a num field in a similar fashion.
Complex Constraint machines Unlike their atomic counterparts, logically complex constraints can be arbitrarily nested, thereby forming a nonregular context-free language that cannot be characterized by FSAs. We get around this limitation by constructing FSAs for such complex constraints that accept a regular language forming a superset of the proper context-free LF language. The automaton "OR Machine" in Figure 2 illustrates such a construction. This machine recognizes LFs of the form (OR RCM RCM) along the topmost horizontal path of the automaton (state sequence 0-1-2-3-4-5). But if one or two of these relational constraints are replaced by logically com-plex constraints, the automaton can recognize the result by taking one or two of the vertical paths (state sequences 2-6-7 and 3-8-9, respectively). These paths can also accept strings that are not syntactically valid LFs. However, we are only using these automata to restrict the softmax application to a subset of the LF vocabulary, and for that purpose these automata are conservative approximations. An alternative approach would be to use FSAs for logically complex constraints that essentially unroll nested applications of logical operators up to some fixed depth k, e.g., say k = 2 or 3, as logically complex constraints with more than 3 nested logical operations are exceedingly uncommon, though possible in principle. But the present approach is simple and already leads to considerable reductions in the number of permissible tokens at each prediction step, thereby significantly accelerating our neural semantic parser.
The final automaton representing the entire LF language, which we write as M LF , is the union of atomic machines such as RCM with three "approximation" machines for the three logical operators (negation, conjunction and disjunction).

Grammatical continuations by Automata
We now show how to use the automaton M LF that represents the LF grammar in order to (a) compute the set of valid next tokens, and (b) update the current prefix by appending an RNN-predicted token. We present very simple algorithms for both operations, nextTokens and passToken, which can be used with any grammar that is represented as a DFA. 8 nextTokens: This function returns a list of the permissible next tokens based on the current automaton state, which corresponds to the current LF prefix (note that because the grammar is a DFA, there is a unique resulting state for any prefix accepted by the automaton). The function simply enumerates all the outgoing arcs from the current state and returns the corresponding labels in a list. This function is called before the token prediction model (RNN + softmax), so that its result can be 8 For convenience, of course, the grammar could be represented by non-deterministic automata (NFAs). The algorithms we present here would still be applicable via a simple preprocessing step that would convert the NFAs to DFAs using standard algorithms for that purpose (Rabin and Scott, 1959). used to restrict the application of softmax; the actual integration model is discussed in detail in subsection 4.2.
passToken: For any model that predicts the output in an incremental and sequential manner (e.g., RNN), we want to compute the DFA state corresponding to a partial output in a similar and lock-step fashion, so that computations in previous steps do not need to be repeated. We achieve this by maintaining a global state, called current state, which is the state reached after reading the prefix that has been produced by the neural model up to this point. To update the global state, the function passToken is called, which simply searches for the arc ('the' again due to the DFA property) that has the currently predicted token as a label, and then transitions to the next state via that arc. Once this is done, the new global state will represent all the predictions made so far.

Time Efficiency Concerns
The functions next-Tokens and passToken need to be called on every step of the output's generation, and therefore need to be efficient, so that the reduction of prediction space for the token-prediction model (e.g., RNN + softmax) can lead to runtime gains. In our case, nextTokens returns the labels of all outgoing arcs and passToken performs a simple label search in addition to carrying out a state transition. All of these operations can be performed with O(1) time complexity.

Integrating Grammar into Neural Models
After calculating the permissible next tokens, we can restrict predictions in order to improve both prediction time and accuracy. We apply this general strategy to the prediction layer of our RNN-based neural network (a linear layer + softmax operation, which can be seen as a log-linear model ), although it should be applicable to other prediction models, such as multi-class SVMs (Duan and Keerthi, 2005) or random forests (Ho, 1995). Figure 3 illustrates a concrete example of integrating the grammar (represented as an automaton in our case) into the token prediction model at a particular prediction step. We focus on the prediction layer of the model, which consists of one linear layer followed by the softmax operation. The linear layer involves a matrix of size |V |×d, where V is the LF vocabulary and d is the dimension of the vector passed from the previous layer; the linear layer predicts scores for each of the V tokens before they are passed to softmax operation.
To integrate the grammar, first, the function nextTokens is called to return a list of tokens allowed by the grammar at this prediction step; the valid tokens are then translated into a list of indices, denoted by l c , which is passed to the loglinear model. Supposing there are k indices in the list l c , we can dynamically construct another matrix of size k × d where the i th row in the new matrix corresponds to the j th row in the original matrix, for j = l c [i]. Figure 3 illustrates this process of choosing rows from the original matrix to construct the new matrix.
Then the new matrix-vector product will result in scores only for those k LF tokens that are permissible, and will then be passed to the softmax operation. The decision function (e.g., argmax in Figure 3) will then be applied based on the softmax score, whose results will finally be passed to passToken function to update the current DFA state.

Time Efficiency Concerns
We implement nextTokens to directly return a list of indices to avoid the cost of converting tokens to indices. We implemented our token prediction model in PyTorch, which supports slicing operations so that our on-the-fly matrix construction does not need to copy the original matrix data, but can instead just point to it. However, we find in our experiments that even matrix construction using slicing tends to be costly (see section 5).
To overcome this, we observe that we can enumerate the lists returned by nextTokens for every DFA state, and then cache the corresponding matrices. For example, consider RCM (the Relational Constraint Machine) in 2. We can cache the value of nextTokens for state 1 by precomputing the matrix corresponding to all the enum/num fields. Doing this caching for every DFA state can be expensive in memory; in practice, one may consider tradeoffs between memory consumption and prediction time.

EQS Dataset
Our experiments are conducted on the EQS dataset. The dataset consists of queries paired with their LFs, which were obtained in a semiautomated manner. The dataset contains 1981 (NL, LF) pairs as training data and 331 (NL, LF) pairs as test data. The LF vocabulary size is 56209, most of which consists of enum field names and values. All the LFs can be accepted by the FSA discussed in Section 3.
The dataset is too small to effectively learn a model that can reliably predict rare fields or values. However, as most of the queries involve only common fields and entities, we find in our experiments that our neural semantic parser is able to parse a large number of those queries correctly; orthogonal research is being conducted on how to handle more rare fields or entities.

Baseline Neural Model
We use a seq2seq neural architecture as our baseline. For our encoder, we initialize the word embeddings using Glove vectors (Pennington et al., 2014); then a Bi-LSTM is run over the question where the last output is used to represent the meaning of the question. For the decoder, we again use an LSTM that runs over LF prefixes, where the LF token embeddings are learned during training. Our decoder is equipped with an attention mechanism (Luong et al., 2015) used to attend over the output of the Bi-LSTM encoder. We use greedy decoding to predict the LFs.
We choose hyperparameters based on our previous experience with this dataset. The word and LF token embeddings have 150 dimensions. The Bi-LSTM encoder is of dimension 150 for its hidden vector in each direction, therefore the decoding LSTM is of dimension 300 for its hidden vector. We train the model with RMSprop (Tieleman and Hinton, 2012) for 50 epochs.
Our baseline neural model achieves 80.33% accuracy on the test set. Most of the errors made by our model are due to unseen fields or values; we observe that our model also fails on queries involving compositionality patterns that have not been seen in training, a problem similar to those reported by (Lake and Baroni, 2018).

Experimental Setups
All our experiments were conducted on a server with 40 Intel Xeon@3.00GHz CPUs and 380 GB of RAM. We monitor the server state closely while conducting the experiments.
Our models are implemented in Py-Torch (Paszke et al., 2017), which is able to exploit the server's multi-core architecture. The peak usage for both CPU load and memory consumption for all our models is far below the server's capacity.
We run all the models over the entire test dataset (331 sentences) and report the average prediction time for each sentence. For each model, we conduct 5 such runs to calculate the standard deviations of different runs. The standard deviations are small in absolute and relative value.

Results
Integrating the LF grammar into prediction at decoding time eliminates all grammatical errors and can therefore improve accuracy. This has been shown, for example, by ; Yin and Neubig (2018), and indeed we obtain similar accuracy improvements. By incorporating the grammar at decoding time at all decoding steps (using its superset represented as an automaton), our parser is able to eliminate some grammatical errors, achieving 80.67% accuracy on the test set, which improves our baseline model by 0.30%. Table 1 shows the main results of our experiments. Our baseline neural semantic parser (NSP) takes on average 0.260 seconds to predict the LF for a given query. When we use the model that  -G(all) 4.416 ± 0.029 6336 NSP-GC(500) 0.074 ± 0.000 9643 NSP-GC(10 4 ) 0.069 ± 0.000 6981 NSP-GC(all) 0.067 ± 0.000 6336 Table 1: Prediction time (in seconds) and number of permissible tokens per query on average, for our baseline neural semantic parser (NSP) and various models using grammar integration with caching (NSP-GC) or without (NSP-G).
integrates the LF grammar but constructs the reduced matrices on the fly (GSP-G), we find that despite the reduction of average permissible tokens (from 56209 to 6336), the prediction time actually increases drastically to 4.416 seconds.
To shed some light on this, we integrate the grammatically permissible next tokens only when their number is (a) less than 500 and (b) less than 10 4 . We observe that when the number of permissible next tokens is small, as in case (a), integrating the grammar can indeed reduce prediction time, indicating that the slowing is due to the dynamic matrix construction that uses the PyTorch slicing operation, as nextTokens and passToken are called at every prediction step in all cases.
To avoid this, we cache the reduced matrices (subsection 4.2, NSP-GC in Table 1) and observe that prediction time decreases in this case when more grammar integration is applied. The best prediction time (0.067 second per query) is achieved by NSP-GC when the grammar is used at every step. But similar speed-ups can be achieved when we are using cached matrices only for states with a small nextTokens set.

Related Work
Speeding up neural models that have a softmax bottleneck is an ongoing research problem in NLP. In machine translation, some approaches tackle the problem by moving from the prediction of word-level units to sub-word units (Sennrich et al., 2016) or characters (Chung et al., 2016). This approach can reduce the dimensionality of the softmax significantly, at the price of increasing the number of output steps and thus requiring the model to learn more long-distance dependencies between its outputs. The technique could easily be combined with the one described here; the only adaptation required would be to change the grammar so that it uses smaller units to define its language. In a finite-state context, this would mean replacing transitions corresponding to a single LF token with a sequence of transitions that construct the token from characters. This creates potential for memory savings as well, if states in these expanded transitions can be shared in a trie structure.
Another approach for ameliorating a softmax bottleneck is the use of a hierarchical softmax (Morin and Bengio, 2005), which is based on organizing all possible output values into a hierarchy or tree of clusters. A token to be emitted is chosen by starting at the root cluster and then picking a child cluster until a leaf is reached. A token in this leaf cluster is then selected. Our approach could be combined with the hierarchical softmax method by creating a specific version of the cluster hierarchy to be associated with every state. We would filter all impossible tokens for a state from the leaf clusters and then prune away empty clusters in a bottom-up fashion to obtain a specific cluster.
While they have not been used in order to speed up predictions, grammars describing possible output structures have been combined with neural models in a number of recent papers on semantic parsing Neubig, 2017, 2018;Krishnamurthy et al., 2017;Xiao et al., , 2017. These papers use grammars to guide the training of the neural network model and to restrict the decisions the model can make at training and prediction time in order to obtain more accurate results with less data. Our approach is focused on speed improvements and does not require any changes to the underlying model or training protocols.
Like our approach, the one presented by L'Hostis et al. (2016) for machine translation tries to limit the decoding vocabulary. Their approach relies on limiting the tokens allowed during decoding to those that co-occurred frequently with the tokens in the input. Because this might rule out tokens that are needed to construct the correct output, this may decrease model performance. Our approach is guaranteed to never rule out correct outputs. For additional performance gains it should be possible to combine both approaches.

Future Work
We have used superset approximations based on finite-state automata instead of directly using the grammar of the LF language, which will usually be context-free. This choice is driven by the need for an efficient implementation of passToken and nextTokens, which could be expensive for longer sequences when using a general contextfree grammar. However, for those context-free grammars that are LR (Knuth, 1965), recognition can be performed in linear time, and it is easy to see that both passToken and nextTokens can then be implemented with O(1) time complexity on average. Furthermore, the caching mechanism we have proposed for nextTokens in this work is applicable in the case of LR grammars. Therefore, it would be possible to implement the methods proposed here for any LR grammar, and such grammars cover most LF languages in practical use. 9 For most LF languages there will be restrictions on the logical types of expressions that can occur in certain positions. We can detect some of these restrictions in our finite-state automata, but in general a type system could capture well-formedness conditions that cannot be easily expressed with FSAs, or even in context-free grammars. It would be interesting to investigate how more expressive type checking can be integrated into our present framework in a more general setting.

Conclusion
We propose a method to improve the time efficiency of seq2seq models for semantic parsing using a large vocabulary. We show that one can leverage a finite-state approximation to the LF language in order to speed up neural parsing significantly. Given a context-free grammar for the LF language, our strategy is general and can be applied to any model that predicts the output in a sequential manner.
In the future we will explore alternatives to finite-state automata, which potentially characterize the relevant LF languages exactly while still allowing for efficient computation of admissible next tokens. We also plan to experiment with additional datasets.