Principal Component Density Estimation for Scenario Generation Using Normalizing Flows

Neural networks-based learning of the distribution of non-dispatchable renewable electricity generation from sources such as photovoltaics (PV) and wind as well as load demands has recently gained attention. Normalizing flow density models are particularly well suited for this task due to the training through direct log-likelihood maximization. However, research from the field of image generation has shown that standard normalizing flows can only learn smeared-out versions of manifold distributions. Previous works on normalizing flow-based scenario generation do not address this issue, and the smeared-out distributions result in the sampling of noisy time series. In this paper, we exploit the isometry of the principal component analysis (PCA), which sets up the normalizing flow in a lower-dimensional space while maintaining the direct and computationally efficient likelihood maximization. We train the resulting principal component flow (PCF) on data of PV and wind power generation as well as load demand in Germany in the years 2013 to 2015. The results of this investigation show that the PCF preserves critical features of the original distributions, such as the probability density and frequency behavior of the time series. The application of the PCF is, however, not limited to renewable power generation but rather extends to any data set, time series, or otherwise, which can be efficiently reduced using PCA.

Introduction
The renewable electricity generation technologies photovoltaics (PV) and wind depend on natural occurrences and are therefore non-dispatchable. Additionally, the realization of these renewable power generation outputs exhibits uncertain and volatile behavior, which poses new challenges for the design and operation of energy systems compared to dispatchable fossil power generation (Agora Energiewende and Sandbag, 2020; Mitsos et al., 2018). To account for the uncertainty in renewable electricity sources and other relevant energy system parameters like electricity demands system operators require information about possible realizations. This information can either be provided via short-term point-forecasting (Tascikaraoglu and Uzunoglu, 2014) or via consideration of multiple scenarios (Morales et al., 2013). Here, the term scenario refers to a possible realization of the uncertain and volatile parameter over a certain time span. A set of scenarios thus refers to a collection of time series of equal length. Decision-making based on scenarios covers the broad spectrum of possible realizations and is, therefore, highly relevant for design problems (Morales et al., 2013). Scenarios can be applied in stochastic programming formulations for the design and operation of energy systems as well as energy-related scheduling tasks (Kaut and Wallace, 2003;Birge and Louveaux, 2011). The distributions of time series intervals are often unknown and consist of many non-independent dimensions due to the correlation between time steps. These distributions typically do not follow standard distribution models like multivariate Gaussians. Thus, sampling these distributions for scenario generation remains an open research question.
There are many contributions to the literature on how to generate scenarios. In general, the published approaches can be classified as univariate modeling approaches, i.e., step-by-step models, and multivariate modeling approaches, where multiple time steps are modeled in parallel (Ziel and Weron, 2018). Multivariate modeling approaches are trained on sets of scenarios created from historical time series. The equidistant segments of univariate scenarios are viewed as multidimensional points, e.g., a scenario of one day with hourly recordings is viewed as a 24-dimensional data point. Contributions on univariate modeling approaches include the traditional Box-Jenkins approach (Box et al., 1967) for sampling stochastic processes (Sharma et al., 2013) and artificial neural network (ANN)-based autoregressive models (Vagropoulos et al., 2016). Examples of multivariate modeling approaches are Gaussian mixture models , Copula methods (Pinson et al., 2009;Kaut and Wallace, 2011), and moment matching techniques (Chopra and Selvamuthu, 2020).
With the increase of computational power and the development of advanced machine learning algorithms, it is now possible to train specialized ANNs, so-called deep generative models (DGMs), to learn high-dimensional probability distributions without any statistical assumptions about the data. Thus, DGMs are powerful tools for multivariate modeling. In 2018, Chen et al. (2018c) proposed the use of generative adversarial networks (GANs) (Goodfellow et al., 2014) for scenario generation. Since then, scenario generation using GANs and other DGMs has become a popular topic (Chen et al., 2018c;Jiang et al., 2018Jiang et al., , 2019Wei et al., 2019;. Chen et al. (2018c) later extended their original work by introducing a scenario selection procedure (Chen et al., 2018b) and by using Bayesian GANs (Chen et al., 2018a). Other examples of GANs for wind and PV power output scenarios are presented in (Jiang et al., 2018;Jiang et al., 2019). For more consistent convergence in training, the authors in Chen et al. (2018c); Jiang et al. (2018); ; Jiang et al. (2019) used Wasserstein GANs (Arjovsky et al., 2017) where they enforce a Lipschitz constraint on the critic network. Besides the generation of wind and PV scenarios, GAN-based scenario generation was also applied to residential load forecasts (Gu et al., 2019) and hydro-wind-solar hybrid systems (Wei et al., 2019). Schreiber et al. (2019) study different loss functions for GANs and found the Wasserstein distance to be superior to the binary-cross-entropy. Besides GANs, a popular type of DGMs are variational autoencoders (VAEs) (Kingma and Welling, 2014). Examples of VAEs for scenario generation include electric vehicle load demand (Pan et al., 2019), hydro wind-solar hybrid systems (Zhanga et al., 2018), as well as hydro concentrated solar power hybrid systems (Qi et al., 2020).
Despite their recent success in scenario generation for renewable power generation, both GANs and VAEs show inconsistencies in training. Specifically, the Nash Equilibria obtained through GAN training are reported to be unstable, and there is no guarantee for the generator to sample from the target distribution (Arjovsky et al., 2017). The VAE uses the Evidence Lower Bound (ELBO) loss function in training which gives no concrete measure on the actual quality of the fit (Kingma and Welling, 2014).
In contrast to GANs and VAEs, a third DGM structure, normalizing flows (Dinh et al., 2017;Papamakarios et al., 2019), can directly fit the probability density function (PDF) of the unknown distribution by log-likelihood maximization. Similar to the more established transport map approach (Marzouk et al., 2016), normalizing flows model the distribution of a data set as a deterministic transformation of a Gaussian. While transport maps typically use triangular maps (Marzouk et al., 2016) to set up convex fitting problems, normalizing flows are designed using invertible neural networks to build more flexible designs. In theory, using a sufficiently expressive normalizing flow network and enough training data, the trained distribution will converge to the true distribution. However, normalizing flows are not as well established as GANs and VAEs in scenario generation yet. To our knowledge, the only works using normalizing flows in the context of energy time series are  and Ge et al. (2020) focusing on demand time series and Dumas et al. (2021) generating PV and wind electricity time series.
Recently, Brehmer and Cranmer (2020) and Behrmann et al. (2021) showed that normalizing flows are by design unable to fit distributions that lie on lower-dimensional manifolds. Instead, fitting a full dimensional normalizing flow to a manifold distribution results in exploding likelihood functions from numerically singular Jacobians (Behrmann et al., 2021) and a smeared-out version of the true distribution (Brehmer and Cranmer, 2020). Sampling from the smeared-out fit then leads to the samples outside of the true distribution, e.g., noisy time series. In 2016, Gemici et al. (2016) presented the necessary foundations to build normalizing flows on manifolds, i.e., to perform density estimation in a lower-dimensional space. However, they do not further elaborate on such an approach and do not report on any numerical experiments. Brehmer and Cranmer (2020) build normalizing flows in lower-dimensional space by fixing some of the latent space dimensions to a constant.
We find manifolds to be frequently present in energy time series due to the temporal correlation between time steps. However, , Ge et al. (2020), and Dumas et al. (2021) employ standard normalizing flow structures to such data, i.e., real non volume preserving transformation (RealNVP) (Dinh et al., 2017) or nonlinear independent component estimation (NICE) (Dinh et al., 2015). As a result, their works suffer from complications resulting from data manifolds. For instance, the training of Ge et al. (2020) results in extremely high density values (log p(x) ≈ 4000), which indicates that the model is trying to describe an infinite density. The validation loss shown by the authors indicates strong overfitting. However, the authors do not declare the exploding likelihood and the overfitting as issues and present the spurious results without further discussion.  acknowledge that their scenarios exhibit noisy behavior that does not match the expected results. However, they attribute this noise to their conditional training approach. Dumas et al. (2021) draw a comparison between normalizing flows, GANs, and VAEs at the example of PV, wind, and load time series. The authors find that their normalizing flow models are unable to recover the autocorrelation within the time series and instead exhibit very high fluctuations. However, they still conclude that scenarios generated from normalizing flows show good results despite the unrealistic characteristics. Based on our observation of the frequent complications resulting from data manifolds, we argue that normalizing flows should be set up with a lower-dimensional latent space to avoid generating unrealistic energy time series scenarios.
As part of this paper, we provide the following contributions: We elicit the contradiction between manifolds and the diffeomorphic transformations required for normalizing flows. Further, we use simple two-dimensional examples to highlight how data manifolds lead to the generation of unrealistic and outof-distribution data. In addition, we show how the principal component analysis (PCA) can be used to reduce the dimensionality of energy time series effectively and show theoretically that the resulting principal component flow (PCF) does not affect the density estimation procedure, which avoids the need for balancing reconstruction and likelihood maximization losses like in Brehmer and Cranmer (2020).
In numerical experiments, we train the PCF on data of PV and wind power generation as well as load demand in Germany in the years 2013 to 2015. For reference, we compare our results with scenarios generated using the alternative methods Copulas (Pinson et al., 2009) and Wasserstein GANs (Chen et al., 2018c). The results show that the PCF learns the distributions better than the full space normalizing flow (FSNF) and also maintains the frequency behavior of the original time series. The PCF-generated scenarios perform equally good or better compared to the Copula and W-GAN scenarios in all considered metrics.
The remainder of this paper is organized as follows: In Section 2, we introduce the general concept of normalizing flows and review the RealNVP affine coupling layer (Dinh et al., 2017) which will serve as the underlying flow structure in this paper. In Section 3, we continue the discussion started by Brehmer and Cranmer (2020) and Behrmann et al. (2021) about the effects of manifolds on normalizing flows through a clarifying toy example and show theoretically that PCA does not affect the density estimation in lower-dimensional latent space. In Section 4, we present the results of simulation studies on data of PV and wind power generation as well as load demand in Germany (Open power systems data, 2019). Finally, in Section 5 we conclude our work.

Density Estimation using Normalizing Flows
Normalizing flows are invertible transformations f between a complex target distribution and a welldescribed base distribution, e.g., a multivariate Gaussian (Papamakarios et al., 2019). Analog to the inverse transport map methodology (Marzouk et al., 2016), normalizing flows aim to model a complex distribution as a transformation of a simple one instead of manually deriving a complex model that fits the data. Thus, the sampling takes place in the known base distribution, and the transformation does not have to consider the randomness in the data.
A normalizing flow transformation must be set up as a diffeomorphism, i.e., both the forward and the inverse transformation must be continuously differentiable (Papamakarios et al., 2019). In their  (Dinh et al., 2017) with two coupling layers with alternating identity and affine transformations. Arrows point in generative direction. Compositions may include more than two coupling layers. Functions s I θ (z 1:d ), t I θ (z 1:d ), s II θ (z I d+1:D ), and t II θ (z I d+1:D ) are trainable ANNs with parameters θ.
standard form, normalizing flows require equal dimensionality of base and target distributions, in which case the density of the target distribution is well described using the change of variables formula (CVF) (Papamakarios et al., 2019) p where p X (x) and p Z (z) are the densities of the samples x and z of the target distribution X and the base distribution Z, respectively, and J f −1 (x) is the Jacobian of the inverse transformation f −1 . To fit complex distributions, the diffeomorphic transformation f needs to be flexible and expressive. However, expressive and yet easily invertible functions with tractable Jacobian determinants are often difficult to engineer. Fortunately, diffeomorphisms are composable, and therefore normalizing flow models can be built using compositions of simple transformations, i.e., where f 1 to f K are simple diffeomorphisms and the operator • denotes function composition. In logarithmic form, the CVF of compositions is given by . In practice, the transformation is often set up as a trainable function f θ with parameters θ. By using the CVF, the transformation f θ can be trained via direct likelihood maximization and for numerical reasons, the log form in Equation (3) is maximized in training: Here, the likelihood p X (x; θ) is parameterized by the trainable parameters θ of the transformation f θ and the historical samples of the target distribution x take the role of training data. In the last six years, many flow construction methods have been proposed, e.g., Dinh et al. (2015) and Dinh et al. (2017). A prominent normalizing flow model is RealNVP (Dinh et al., 2017) which uses an affine coupling layer and has shown promising results in a prior application to time series data . The idea of the RealNVP affine coupling layer is to split the full input vector z = z 1:D of dimension D and apply an affine transformation to one part of the input vector z d+1:D conditioned on the remaining part of the input vector z 1:d that is kept constant. Here, d is usually set to d = D/2 to allow for maximal interaction between dimensions but can take other values 1 < d < D, e.g., if D is uneven. The standard forward transformation f CL : z → x is given by where the functions s θ (z 1:d ) and t θ (z 1:d ) are feed-forward ANNs called conditioner networks with parameters θ and input and output dimensions d and D − d, respectively. The operator denotes element-wise multiplication. Note that when applied as a composition in alternating form RealNVP can build flexible and easily invertible transformations with tractable Jacobian determinants. A visual description of a composition of two affine coupling layers is presented in Figure 1. The affine coupling layer in Equations (5) and (6) has the advantage that the Jacobian of the transformation f CL is a lower triangular matrix, i.e., which means that the log of the absolute value of its determinant is simply given by and the Jacobian of the inverse transformation f −1 according to the inverse function theorem (Papamakarios et al., 2019). With Equations (8) and (9), the log-Jacobian determinant is computed from a simple evaluation of the forward or inverse transformation. For a more detailed introduction and a review of other normalizing flow designs, the interested reader is referred to the original RealNVP paper by Dinh et al. (2017) and the review articles by Papamakarios et al. (2019) and Kobyzev et al. (2020).

Principal Component Density Estimation
In this section, we first discuss the effects of manifolds on the Jacobians and the contradiction between manifolds and diffeomorphic transformations and visualize how distributions get smeared out at the example of a one-dimensional distribution embedded in two-dimensional space. Secondly, we show how unlike many other dimensionality reduction techniques PCA can be used in combination with standard normalizing flows for manifold density estimation with tractable and direct likelihood computation.

Normalizing Flows and Manifolds
Normalizing flows are transformations between the space of observable variables (target distribution) and a latent space with independent variables, i.e., a space with zero covariance (base distribution) (Papamakarios et al., 2019). As a consequence, normalizing flows disentangle the information contained in the observable variables, e.g., the power generation over a given period, to the set of independent latent variables, i.e., the normalizing flow transformation eliminates the correlation between the dimensions (Papamakarios et al., 2019;Kobyzev et al., 2020). For the disentanglement to function properly, the normalizing flow transformation f and its inverse f −1 must be continuously differentiable, i.e., the Jacobian of f must be non-singular for f −1 to be differentiable and vise versa (Papamakarios et al., 2019). The distributions of many data sets do not occupy the entire space of observable variables, i.e., the distributions lie on lower-dimensional manifolds (Fefferman et al., 2016). A transformation from a latent space Gaussian with equal dimensionality to the space of observable variables must compress the Gaussian samples to the lower-dimensional space and, therefore, have a singular Jacobian (Hyvärinen and Pajunen, 1999;Behrmann et al., 2021). Consequently, a Gaussian with equal dimensionality cannot be matched to a manifold distribution using a diffeomorphic transformation. However, normalizing flow transformations like RealNVP (Dinh et al., 2017) are designed to be diffeomorphic between spaces of equal dimensionality. When applied to manifold data, they have to include all latent space variables in the transformation with a non-zero gradient even if the data does not show any related variance. This leads to a contradiction between a problem that cannot be solved using a diffeomorphism and a strictly diffeomorphic normalizing flow. In practice, fitting a normalizing flow to manifolds leads to numerically singular Jacobians (Behrmann et al., 2021) as the training attempts to compress the Gaussian to the lower-dimensional space. For sampling, the learned distribution is then smeared out around the true distribution (Brehmer and Cranmer, 2020), causing skewed distribution densities and generation of out-of-distribution data.
To illustrate this fundamental problem, we train two RealNVP models with identical architecture on samples of a one-dimensional, uniform distribution embedded in a two-dimensional curve and a twodimensional distribution. Figure 2 and Figure 3 show the encoded training data (Data latent) and samples of a two-dimensional Gaussian (Gauss latent) on the left, the training data (True), and samples generated using the RealNVP (RealNVP) in the center, and training and validation loss on the right for the two different distributions, respectively. The example in Figure 2 highlights that the points from the one-dimensional curve remain on a one-dimensional curve after the transformation with the diffeomorphism. Accordingly, any point in the latent space outside of this curve has no counterpart in the true data distribution. Thus, the samples drawn in the Gaussian latent space (left) are transformed to out-of-distribution data, as can be seen by the RealNVP samples outside of the curve in the center of Figure 2. Nevertheless, the generated distributions resemble the shape of the true distributions and the out-of-distribution samples may not be noticed for higher-dimensional distributions, e.g., time series, that cannot be viewed in their ambient space.
The absolute values of the loss functions are not comparable as they describe different distributions. However, while the validation loss of the one-dimensional example in Figure 2 indicates overfitting and unstable training, the validation loss of the two-dimensional example in Figure 3 indicates good generalization, stable training, and convergence in fewer epochs. Besides the loss functions, the data transformed to the latent space in the left of Figure 3 and the RealNVP generated data in the center of Figure 3 show good matches of the Gaussian and true data distribution, respectively.

Principal Component Flow Layer
Given an available injective map ψ : R M → R D , x = ψ(x) with D > M transforming data points from the lower-dimensional latent spacex to their full space embedding x, a flow on a Riemannian manifold can be built (Gemici et al., 2016). Since the injective map expands the dimensionality, the typical CVF-relation (Equation (3)) of infinitesimal volumes using the Jacobian determinant does no longer apply. Instead, the relation between the two infinitesimal volumes is given by with Jacobian J ψ of the the mapping ψ. The generalized form of the CVF for the injective flow then is: For the injective map ψ to be applicable for the use in combination with normalizing flows, its inverse ψ −1 must be available and easy to compute, and the Jacobian determinant term in Equation (11) must be computationally tractable during training. Brehmer and Cranmer (2020) propose to build an injective map by fixing some of the latent space dimensions of a normalizing flow to a constant. However, their approach still results in expensive Jacobian computations and requires balancing a reconstruction loss with the likelihood maximization. To avoid expensive Jacobian computations and achieve easily invertible dimensionality reduction, we exploit the isometric, i.e., distance preserving, characteristic of the affine PCA (Pearson, 1901). PCA is based on the singular value decomposition of the sample covariance matrix K X,X of the data distribution X, i.e., where N is the number of data points, x are the data points, and µ X is the empirical mean vector of the distribution. The covariance matrix K X,X is decomposed into a diagonal matrix of singular values Σ and two unitary matrices U and V of left and right singular vectors, respectively. The columns of the matrix of right singular vectors V corresponding to the largest singular values are called the principal components of the data distribution X. By truncating the columns with small or zero singular values, we can use the resulting semi-orthogonal matrix V P for the affine embedding function: with V P as the truncated matrix of right singular vectors and µ X as the mean value of the distribution (Pearson, 1901).
In general, isometries are defined via: Here, I M is the M × M identity matrix. For PCA, Equation (14) holds as the Jacobian of ψ PCA is J ψ PCA (x) = V P , where V P is semi-orthogonal. Thus: The isomertric property of PCA results in two major advantages for the combination with normalizing flows. First, the PCA (pseudo-) inverse can be computed using the transpose of V P : which makes the inverse explicit and computationally efficient. Second, the PDF described by the CVF is invariant to the PCA dimensionality reduction, as the expression in Equation (15) enters in the Jacobian determinant term of Equation (11), i.e., When we build a composition of a standard normalizing flow and the PCA following Equation (3) it becomes apparent that the PCA transformation does not influence the PDF described by the CVF: In Equation (18),x = ψ −1 PCA (x) is the lower dimensional representation (Equation (13)) and f is a standard normalizing flow model, e.g., RealNVP (Dinh et al., 2017). Because PCA does not influence the CVF, it can be solved prior to fitting the normalizing flow and the PCF layer does not include trainable variables for the log-likelihood maximization. Hence, we avoid composite loss functions that balance reconstruction loss and likelihood maximization as in Brehmer and Cranmer (2020). The available inverse and separation of PCA fit and normalizing flow training make the use of PCA computationally very efficient, as neither the inverse nor the Jacobian have to be computed at any point. Note that the efficient inverse and the omittable Jacobian determinant do not transfer to most other dimensionality reduction techniques, e.g., local linear embeddings (Roweis and Saul, 2000) or diffusion maps (Coifman and Lafon, 2006). A graphical description of the combination of the PCA and RealNVP is shown in Figure 4.

Numerical Experiments
In this section, we train the PCF on real-world time series data. The three different data sets considered contain data of PV power generation, wind power generation, and load demand, where each represents the total values in Germany in the years 2013 to 2015 (Open power systems data, 2019). In the analysis below, we treat each of the time series independently. Prior to any processing, we clean the data of any days with missing values. To our knowledge, there are no curtailment effects in the data. PV and wind scenarios are scaled by the installed capacity at each time step. Thus, the networks are trained on the so-called capacity factor. The load demand data is scaled to the [0, 1] interval resulting in what we refer to as the demand factor in the following. Learning the capacity factor distribution avoids skewness of the distribution due to the addition of further generation capacity in the given time frame. We have not observed long-term trends in the load demand data of the selected period. To be processed by the normalizing flow, the univariate time series are cut into intervals of one day each. With sampling intervals of 15 min, this leads to a set of 96-dimensional time series fragments that will serve as training data, i.e., a set of 96-dimensional vectors, which is a typical approach in the field of DGM scenario generation (Chen et al., 2018c;. This approach is known as multivariate prediction (Ziel and Weron, 2018) as multiple time steps are predicted at the same time.
For a comparison with more established scenario generation approaches, we also generate scenarios using Copulas and Wasserstein GANs (W-GANs). In contrast to autoregressive models (Sharma et al., 2013) and similar to normalizing flows, Copulas, and W-GANs describe the full distribution independent of conditional inputs. We use the linear interpolation of the inverse cumulative distribution function to fit the Copula to the data (Pinson et al., 2009), and for the W-GANs, we follow the approach by Chen et al. (2018c). Note that both Copulas and W-GANs use the same idea of multivariate scenario generation, i.e., treating scenarios as a vector rather than a sequence of values.
First, we investigate the manifold dimensionality of the data sets by looking at the explained variance ratio of the principal components. The explained variance ratio is the value of the singular values of the covariance matrix scaled by their sum, i.e., the relative amount of variance in the data, which is described by the right singular vector (the principal component) corresponding to the given singular value (Tipping and Bishop, 1999). As an indicator of the manifold dimensionality, we look at the cumulative explained variance (CEV) which describes how much of the information in terms of variance is maintained when the data is compressed to the space of principal components. Table 1 lists some relevant CEV values and the corresponding latent space dimensionality for our data sets. The results show that a significantly reduced dimensionality is sufficient to maintain close to all of the variance information in the data. The 100 % threshold is reached when the latent dimensionality is equal to the numerical rank of the covariance matrix. The PV data reaches this threshold at a latent dimensionality of 62. Note that the 100% CEV appears with fewer principal components than dimensions of the data, if either one or more of the dimensions has zero variance or if there is an exact linear dependency of one of the dimensions on one or more of the other dimensions. Another important thing to consider is that the PCA is fitted using the empirical covariance matrix, i.e., with a finite number of samples. Hence, the numerical rank might be different from the rank of the covariance matrix of the process and the low variance components identified by the PCA may not be fully representative of the underlying process. Hence, the low variance and high-frequency components identified by the PCA may not be desired.
We train full-space normalizing flow (FSNF) and PCF models on each of the three different data sets and compare the generated to the historical data (target). For each data set, we select two latent dimensionalities, namely, 16 and 62 for PV to represent 99.99% and 100% of the variance, and 6 and 10 for wind as well as 5 and 16 for demand to represent 99% and 99.9% of the variance, respectively. In each training, 20% of the 1096 scenarios are set aside as validation sets for hyperparameter tuning and to avoid overfitting. All of the following analysis is performed using the full historical scenario sets in Comparison of the probability density function (PDF) from kernel density estimation (Parzen, 1962). Top: historical data (target, solid lines), generated data from full space normalizing flow (FSNF, dashed lines), and generated data from PCF (PCF, dotted line and dash-dotted lines) Bottom: historical data (target, solid lines), generated data from the Copula (Copula, dotted lines), and generated data from W-GAN (W-GAN, dash-dotted lines). Left: PV capacity factor, center: wind capacity factor, right: demand factor.
comparison to sets of generated scenarios, to avoid random errors in the evaluation as a result of too few data points. We find that, for all networks, five RealNVP affine coupling layers with fully connected conditioner networks with two hidden layers of the same size as the input dimension each is sufficient and additional complexity does not improve the representation. All normalizing flow models are implemented using the Python-based machine learning libraries Tensorflow version 2.4.0 (Abadi and Agarwal, 2015) and Tensorflow-Probability version 0.12.1 (Dillon et al., 2017). We use the PCA routine from the open-source scikit-learn Python library version 0.24.0 (Pedregosa et al., 2011) for dimensionality reduction.
We estimate the probability density function (PDF) of the data using kernel density estimation (KDE) with Gaussian kernels (Parzen, 1962) as shown in Figure 5. The top line of Figure 5 shows the PDF plots of the historical scenarios, FSNF-generated scenarios, and scenarios generated using the proposed PCF for each of the three data sets. For reference, Figure 5 compares the Copula and W-GAN-generated scenario PDFs with the historical scenarios, in the second line.
For the PV data in the left of Figure 5, the data from all three normalizing flow structures appear to match the distribution. The FSNF data exhibits a higher density than the target for capacity factors smaller than 0.05, between 0.1 and 0.45, and a lower density above a certain capacity factor of 0.45. The PCF62 data shows an overestimation between a capacity factor of 0.1 and 0.4, however, the PCF62 density is closer to the density of the target distribution than the FSNF density, in particular, for capacity factors below 0.1. Finally, the PCF16 data distribution is closest to the target distribution with good fits for capacity factors smaller than 0.2 and similar deviations from the target as observed with the FSNF and the PCF62.
For the wind capacity factor (center in Figure 5) and the demand factor (right in Figure 5), we observe a more drastic overestimation of the PDF for the highest densities and an underestimation of the tails by the FSNF. As for the PV scenarios, the density estimates for wind and load demand scenarios show an improved approximation of the distribution through dimensionality reduction with PCA, although the covariance matrices are not rank deficient. It appears that the FSNF tends to ignore rare events during the Tab. 2. P-values (p-value ∈ [0, 1]) of Kolmogorov-Smirnov test (Hodges, 1958): Statistical comparison of historical and generated data. High p-values (≥ 0.1) indicate high probability that generated scenarios have same distribution as the historical scenarios. Results indicating a good match of the distributions are marked with an asterisk * .

FSNF
W-GAN Copula PCF5 PCF6 PCF10 PCF16 PCF62 Solar 2.67×10 −22 1.21×10 −04 0.4911 * 0.2483 * 2.07×10 −5 Wind 6.69×10 −20 0.268 * 0.4782 * 0.9502 * 0.0537 Demand 3.85×10 −81 6.35×10 −21 0.7727 * 0.2380 * 0.0361 likelihood maximization and thus overestimates the areas of high density. With a lower dimensionality, rare events occupy a relatively larger space and are therefore more likely to be considered by the PCF likelihood maximization. Furthermore, the PDF fits of both wind and demand appear to improve by considering fewer principal components. By adding more principal components to get from 99% to 99.9% CEV, the training problem becomes less well-conditioned as the additional dimensions are only narrowly occupied by data, i.e., the distribution comes closer to a lower-dimensional manifold. The bottom line of Figure 5 shows good matches of PV and wind capacity factor PDFs for both Copulas and W-GANs. The Copula scenarios also show a matching demand factor PDF, while the W-GAN results in a skewed PDF. A comparison of the top and bottom lines in Figure 5 highlights that PDF representations by the PCF are as good or better compared to the literature approaches.
Note that an exact evaluation of the KDE plot can be misleading since the KDE results are not exact, in particular, at the tails of the distribution and in low-density parts (Wied and Weißbach, 2012). However, the trends for all three data sets show improved fits of the PCF setups compared to the FSNF, which indicates better distribution fits due to the dimensionality reduction.
To support the results from the visual comparison of KDE plots, we run a Kolmogorov-Smirnov test (KS-test) (Hodges, 1958). The KS-test is used to judge whether two distinct sets of samples stem from the same distribution. Table 2 lists the p-values (p-value ∈ [0, 1]) of the tests. In general, high p-values (≥ 0.1) can be interpreted as a match of distributions, and low p-values indicate that the two sample sets stem from different distributions (Hodges, 1958). For all three data sets, the p-values of the FSNF indicate that the historical and the generated data do not follow the same distribution. In contrast, the p-values for PCF16 for PV, PCF6 for wind, and PCF5 for load demand are ≥ 0.1, indicating a good representation of the historical data distribution. The PCF10 for wind and the PCF16 for demand show p-values ≥ 0.05 and ≥ 0.01 which still indicate good matches of the distribution and significantly better matches compared to the FSNF. The PCF62 for PV gives a lower p-value and hence lower confidence that the data stems from the historical data distribution. This result matches the observations in Figure 5, where we observed a slightly worse fit from PCF62 compared to PCF16. Still, there is a significant improvement compared to the FSNF. Furthermore, the KS-test confirms our observation of increasing quality of fit for the 99% CEV compared to the 99.9% CEV.
The KS-test of the reference models, returns high p-values for all datasets for the Copula-generated scenarios, while only the wind capacity factor shows a high p-value among the W-GAN-generated scenarios. In general, the KS-test indicates good matching distributions only for Copulas and for the PCFs with low numbers of principal components.
In conclusion, the PCF returns significantly better fits of the distribution compared to the FSNF. Furthermore, the PDF fits by the PCF are at least as good or better compared to the literature methods. Considering fewer principal components appears to enhance the learning of the distributions, and accepting CEV values around 99% does not compromise the overall match of the distribution.
We use the power spectral density (PSD) based on the Welch transform (Welch, 1967) to investigate in the frequency domain whether the FSNF and the PCF can reproduce the fluctuational behavior of the original time series. Figures 6 shows the PSD of the target distribution, the FSNF, and the PCF generated scenarios for the three considered data sets, respectively.
For the PV data, the target distribution shows high amplitudes (> 10 −4 ) for low frequencies up to  (Welch, 1967). Top: historical data (target, solid lines), generated data from full space normalizing flow (FSNF, dashed lines), and generated data from PCF (PCF, dotted line and dash-dotted lines). Bottom: historical data (target, solid lines), generated data from the Copula (Copula, dotted lines), and generated data from W-GAN (W-GAN, dash-dotted lines). Left: PV capacity factor, center: wind capacity factor, right: demand factor.
by the FSNF. In contrast, PCF16 produces lower amplitudes than the target distribution in the regime of frequencies > 1 2 h , which indicates the filtering of some of the low amplitude fluctuation in the historical data set. The PCF62 shows a good fit for the true behavior of the target up to the highest frequencies.
The PSD analysis for the wind data shows a good match of the frequency behavior for the data generated from the PCF6 and PCF10. The FSNF generated scenarios show a general underestimation of the amplitudes. Note that the wind data exhibits higher fluctuation on all frequencies compared to the PV data. Therefore, the fluctuations at high-frequencies are represented in the principal components, and no filter effect can be observed. As the overall PSD shape is matched, the FSNF appears to find the right fluctuation behavior only with consistently lower amplitudes. Here, the narrower distribution described by the FSNF (see Figure 5) leads to a smaller range of scenarios and, therefore, lower amplitudes.
As for the wind scenarios, the PSD analysis of the load demand data in Figure 6 shows a good match of the PCF-generated scenarios to the historical data. The FSNF matches the overall PSD, but there are lower amplitudes between frequencies of > 1 8 h and > 1 2 h . The PSDs of the W-GAN-generated scenarios show an increase in amplitude for frequencies between 1 1 h and 1 0.5 h for all three datasets, indicating noisy scenarios. Furthermore, the PSDs of the PV capacity factor and the demand factor of the W-GAN-generated scenarios are higher compared to the historical scenario PSDs, i.e., the W-GANs generate scenarios with unrealistically high fluctuations. The Copulagenerated scenarios reproduce the PSDs of the wind capacity factor and the demand factor well. However, the Copula-generated PV capacity factor scenarios show high amplitudes for frequencies > 1 5 h . In conclusion, the PSD analysis of the PCF-generated scenarios clearly shows improved results for all data sets due to the dimensionality reduction compared to the FSNF. Some effects are specific to the PCA dimensionality reduction of the given data, e.g., the filter effect observed for the PV data and the noise in the case of the load demand data. Among all considered scenario generation approaches, the PCF is the only approach that does not result in noisy behavior or otherwise higher fluctuations for all three datasets.
Finally, we investigate whether the PCF recovers the dimensions with zero variance that occur during the nighttime hours of the PV scenarios. Figure 7 shows the time between midnight and 4 am of five randomly selected scenarios from the historical data, the normalizing flow models, and Copula and W-GAN-generated scenarios. The FSNF-generated scenarios clearly show noisy behavior. On the other hand, PCF16 and PCF62 do not show any noise and thus preserve the zero variance feature of the respective dimensions. The W-GAN scenarios show high fluctuations without any structure, while the Copula scenarios appear to identify the zero variance of the nighttime hours correctly. In addition to Figure 7, Table 3 lists the mean and variance for the time steps between midnight and 4 am. The mean and variance values for PCF16 and PCF62 confirm the observations from Figure 7. Figure 7 and Table 3 highlight the 'smeared-out' characteristic of the learned distribution as described in Section 3.1 and by Brehmer and Cranmer (2020). The results show that the FSNF is unable to detect dimensions of zero variance and does not accurately fit the manifold distribution, while PCF detects the manifold and correctly reproduces its features. In contrast to Figure 7, the mean and variance values of the Copulagenerated scenarios highlight that the Copula does not identify the zero variance characteristic of the nighttime hours. It appears that the Copula reproduces the zero variance characteristic for most but not  In conclusion, the PCF is the only method among the considered approaches that consistently identifies the zero variance characteristic of the PV scenarios during nighttime.

Conclusion
In this paper, we illustrate the issues associated with learning the probability distribution of electricity time series scenarios using normalizing flow density models. We find that these distributions lie on lower-dimensional manifolds due to the high autocorrelation between time steps. We show that standard normalizing flows are ill-suited to learn these manifold distributions and, instead, induce ill-conditioned or singular Jacobians that interfere with the normalizing flow training. However, this problem was previously not addressed in the literature on scenario generation, and we argue that this has led to some spurious results. To mitigate the problems arising from data manifolds, we use dimensionality reduction. Here, we exploit the isometric characteristic of the principal component analysis (PCA). We show theoretically that the PCA does not influence the density estimation since its log-Jacobian determinant is by design always zero. Thus, we avoid having to balance reconstruction loss and likelihood maximization. Furthermore, no expensive Jacobian or matrix inverse computations are necessary, which makes PCA very computationally efficient. Critically, neither the explicit inverse nor the omittable Jacobian determinant apply to most other dimensionality reduction techniques.
We apply the combined principal component flow (PCF) next to a standard full space normalizing flow (FSNF) to learn the distribution of PV capacity factors, wind capacity factors, and load demand scenarios from Germany from 2013 to 2015. The results show that the PCF can generate realistic PV, wind, and load demand scenarios despite a significant dimensionality reduction, while the FSNF overestimates the probability density function (PDF) in areas of high density and underestimates the tails. Moreover, the PDF fits improve with 99% cumulative explained variance (CEV) as opposed to 99.9% CEV, as the training problem becomes better conditioned. Besides the PDF, the power spectral density (PSD) analysis reveals that PCF recovers the general periodic behavior of the time series. As observed for the PV scenarios, PCA can introduce a potentially desired filter effect. On the other hand, there may be a trade-off between accurate distribution fits and inclusion of all the details in the data set. Therefore, it is critical to test how PCA-based dimensionality reduction affects the data and the final selection of principal components should always be made considering the desired application of the scenarios.
In addition to our analysis of the PCF-generated scenarios, we run a comparison to two more established methods, namely Copulas and Wasserstein-GANs. Our results highlight that the PCF performs either similarly good or better in all considered metrics compared to the literature methods. In fact, the PCF is the only method that shows good matches in the PDFs, scores high p-values in the KS-test, does not have any unexpected fluctuation, and recovers the zero variance characteristic of the photovoltaic time series during the nighttime.
In conclusion, our results highlight the importance of considering the inherent latent dimensionality of the data when using normalizing flow generative models, and that the combination of PCA dimensionality reduction and normalizing flows can build powerful distribution models for energy time series scenarios.


Covariance Kalman Geometric Graph Based Feature Extraction And Bernoulli Kernel Classifier For Plant Leaf Disease Prediction

As far as the agricultural domain is concerned, one of the most hot research areas of analysis is accurate prediction of leaf disease from the leaf images of a plant. The prediction of agricultural plant diseases bymeans of the image processing techniques will hence reduce the dependence on the farmers to safeguard their agricultural land and also their products. However, with the presence of noise, the leaf disease prediction is said to be hindered. To address this issue, in this paper, Covariance Kalman Geometric Graph-basedBernoulliClassifier (CKGG-BC) for Plant leaf disease prediction is proposed. The CKGG-BC method is split into three parts. To start with the plant leaf image provided as input, the Covariance Kalman Filtered Preprocessing modelintroduced for the image enhancement. Second, Geometric Graph-based Segmented Cooccurrence Feature Extraction model is applied to the preprocessed image to accurately segment the infected leaf areas and followed by which extracting the accurate infected leaf areas. Finally, Bernoulli Online Multiple Kernel Learning Classifier is applied for accurate plant leaf disease prediction with minimum classification error. The proposed method provides a significant refinement with respect to state-of-the-art methods. Even under complex background conditions, i.e., in the presence of noise, the averageaccuracy of the proposed method is said to be improved and hence paves mechanism for prediction of plant leaf disease in a significant manner. Experimentalresults exhibit the effectiveness of the proposed method in terms of computational overhead, accuracy, true positive rate and classification error respectively.

Introduction
The development of plant diseases has negative reactions on agriculturalproduction, and if the plant diseases are not detected timely, it will result in a growth in food insecurity. In specific, the predominant crops like, rice, maize, etc., are indispensable for undertaking the food supply and agricultural production. The preliminary warning andforecast are the foundation of significant avoidance and control for plant diseases.They play critical parts in the agricultural production management and decision-making process. To the date till, visual observationsof knowledgeable producers are still the predominant pattern for plant diseasedetection in rural areas of developing countries.
AlexNet and GoogleNet CNNwas proposed in [1] to present a significant soybean diseases identification method on the basis of a transfer learning model using pretrainedAlexNet and GoogleNet convolutional neural networks(CNNs). Here, five-fold cross-validationstrategy was utilized.Initially, the preprocessed images wereapplied as input to the pre-trained GoogleNetCNN architecture. Here, retraining was performed on the preprocessed images with the purpose of classifying into four class species of objects from the input disease data set, therefore resulting in the accuracy improvement.
Despite improvement observed in the accuracy, with the presence of noise, the accuracy factor is said to be compromised. To address this issue in this work, Covariance Kalman Filtered Preprocessing model are applied as preprocessor that with the assistance of Covariance Kalman filter the principal components are obtained and only with this preprocessed images, further analysis is said to be carried out. In this way, the accuracy of leaf disease prediction is said to be improved even in the presence of noise.
An Inception model by Visual Geometry Group for deep convolutional network, called, INC-VGGNet was proposed in [2] for plant leaf disease identification. To start with a pre-trained model obtained from the massive dataset was considered as input and then transferred to the particular task trained by our own data. Here, VGGNet pre-trained on ImageNet and Inception module were selected, erstwhile of initiating the training from the starting

Research Article Research Article Research Article
stage by initializing the weights in a random manner. Here, the weights were initialized with the aid of pre-trained networks, therefore contributing to validation accuracy.
Though validation accuracy was improved, geometric properties were not analyzed, therefore, contributing to classification error. To address this issue, in our work, Geometric Graph-based Segmented Co-occurrence Feature Extraction model is first applied then to the resultant segmented infected leaf areas, the prediction is made. In this manner, the classification error involved in prediction of leaf diseases is said to be reduced significantly.
Motivated by the above said issues, in this work, Covariance Kalman Geometric Graph-basedBernoulliClassifier (CKGG-BC) for Plant leaf disease prediction is proposed. This study aims to introduce Bernoulli Online Multiple Kernel Learning Classifier asan approach for classifying three types of leaves, i.e., pepper, tomato and potato according to sample leaf images. This study presents three main contributions in plant disease classification: (1) Implementation of the Covariance Kalman Filtered preprocessing modelby using the covariance between the two vectors (i.e., state and observation vectors) on plant leaf dataset for enhancing the image even in the presence of noise.
(2) To design a Geometric Graph-based Segmented Co-occurrence Feature Extraction algorithmby first obtaining the Geometric Pixel to accurate segment the infected areas and extract the geometric properties of the infected leaf areas by means of Graph-based Segmented Co-occurrence Feature Extraction.
(3) Accurate and timely prediction of leaf disease using Bernoulli Online Multiple Kernel Learning Classification algorithm (4) The proposed method is evaluated on plant disease dataset using well-known metrics, such as, computational overhead, accuracy, true positive rate and classification error with the results compared using stateof-the-art methods.
The organization of the remaining paper is as follows. Section 2 provides highlights of state-of-the-art plant leaf disease prediction methods. Section 3 elaborates in detail the proposed method, Covariance Kalman Geometric Graph-basedBernoulliClassifier (CKGG-BC) for Plant leaf disease prediction. Section 4 showcases the experimental settings with a detailed discussion of the proposed method in Section 5. Section 6 concludes the work.

Related Work:
Plant disease has become a crucial warning food security globally. A significant amount of losses are said to occur due to plant disease globally each year. In the current situation, there is a requirement for efficiently detection of the plant disease at an early stage so that plant diseases can be controlled significantly, therefore enhancing the agro ecosystem sustainability.
In [3], a novel automatic method for tassel detection was proposed using a color attenuation prior model that in turn removed the saturation present in the images via saturation graph. Followed by which the area of interest was detected by means of an Itti visualattention detection algorithm. Finally, the false positives present in the images were discarded via texture features and vegetation indices. However, measures were not taken to monitor large farms. To address this issue, an optimization method based on particle swarm optimization was proposed in [4] for detecting the disease in sunflower leaf. A review of neural network techniques via hyperspectral data for detecting disease in plant leaf was investigated in [5].
Swift and precise plant disease detection is evaluative to increasing agricultural productivityin a feasible manner. To be specific, speaking in a conventional manner, human experts have been depended upon to discover anomalies in plants generated by different factors, to name a few being, pesticides, deficiencies caused due to malnutrition or deteriorated weather conditions. However, it is found to be laborious process involving cost, time and in certain cases also proven to be impossible. In [6], a thorough review of current works conducted in the area of crop pesticide and recognition of disease with the aid of image processing and machine learning techniques were proposed. An elaborate review on deep learning techniques for leaf disease identification was investigated in [7].
Plant diseases are not only a warning to securitization of food globally, but can also prove to be involved catastrophic effects for small scale farmers whose day to day livelihoods heavily depend on healthy crops. In the developing country like India, more than 80 percent of the agricultural production is produced by small scale farmers. Hence, plant leaf disease identification in the early stage plays a main role for sustaining the productivity. In [8], multiple convolutional neural networks were applied to improve the validation accuracy involved in grape disease identification. Yet another novel classifier algorithm utilizing sine cosine algorithm based rider neural network was proposed in [9] to enhance the accuracy involved in classification. In [10], a deep convolutional neural network was applied to 14 crop species and the disease was diagnosed in a significant manner.
Owing to the high nutritional and medicinal aspect involved in apples it is said to be considered as the most productive fruit globally. Despite its familiarity and increased usage, numerous diseases are said to occur routinely on a large scale in the production of apple, thereby resulting in considerable economic losses. Hence, the timely and efficient apple leaf detection is said to be pivotal corroborating the healthy development of the appleindustry and therefore has become a hot research area in the field of agricultural. A deep CNN model was proposed in [11] for faster detection of disease and also in an accurate manner. Gray level co-occurrence matrix was utilized in [12] to denote the diseased part and also classification of disease was performed in a precise manner utilizing random forest. Yet another method to minimize the convergence iterations utilizing rectified linear unit functions was proposed in [13].
With the high-speed evolution of the smart farming, plant disease identification is said to become digitalized and also datadriven,therefore ensuring state-of-the-art decision support, analysis in a smart manner and ensuring prompt planning. A mathematical model of plantdisease detection and recognition using deep learning was presented in [14] that in turn helped in enhancing the accuracy of detection and training efficiency. An investigation was performed in [15] by training the convolutional neural network for disease identification in plants based on segmentation, therefore ensuring timely detection.
In the recent years, there are escalating tendencies of utilizing deep learning for detecting disease present in plant leaf. However, their executions may be laborious and cumbersome process involving images with adequate resolutions. In developing countries like India however, with restricted internet connectivity, methods that would accomplish well even when data with low resolutionare utilized are necessitated. A new residual network as branches to DCNN architecture was designed in [16] to minimize the issues related to gradient and therefore ensuring prompt detection. Yet another method to evaluate the detection and tracking performance utilizing Kalman filter with data association algorithm was proposed in [17].
Numerous visual computing based methods have been proposed in the recent years with the objective of predicting the plant leaf disease in an early manner. Despite early detection achieved, accuracy with which the detection was carried out is still said to be considered as a challenging issue. To address this issue, a novel hybrid method involving neuro fuzzy classifier was carried out in [18], therefore contributing to improved detection accuracy. Image processing technology was utilized in [19] for improving image recognition accuracy.
Motivated by the above research works conducted in the field of agriculture for early plant leaf detection even with the presence of noise, in this work a method called, Covariance Kalman Geometric Graph-basedBernoulliClassifier (CKGG-BC) is proposed to accurately predict the plant leaf disease. Theproposed method is described in detail in the forthcoming section.

Methodology
In the recent years, digital signal processing methods have earned appeal for the plant leaf disease detection. These digital signal processing methods includes diverse steps such as image preprocessing, segmentation, feature extraction and classification of disease class.During the acquisition of data or plant leaf, undesired noise gets added to the original plant leaf resulting in the noisy plant leaf image collection. This noisy plant leaf image may deteriorate the leaf image analysis quality. To address this issue, Covariance Kalman Geometric Graph-basedBernoulliClassifier (CKGG-BC) method is proposed that accuracy predicts the plant leaf disease even in the presence of noise in a computationally efficient manner. Figure 1 shows the block diagram of Covariance Kalman Geometric Graph-basedBernoulliClassifier (CKGG-BC) method.

Figure 1 Covariance Kalman Geometric Graph-based Bernoulli Classifier (CKGG-BC) method
As shown in the above figure, three different steps are carried out. To start with, image pre-processing is applied as the preliminary step.This proposed Covariance Kalman Geometric Graph-basedBernoulliClassifier (CKGG-BC) method utilized Covariance Kalman Filter as noise reduction in the preprocessing step, which also tends to enhance the quality of image. Second, Geometric Graph-based Segmented Co-occurrence Feature Extraction model is applied to the preprocessed image with which accurate segmentation of infected leaf areas and geometric properties of leaf area are extracted in a computationally efficient manner. Finally, robust leaf disease prediction is made by means of Geometric Graph-based Segmented Co-occurrence Feature Extraction. The elaborate description of the proposed method is given below.

Covariance Kalman Filtered preprocessing model
The objective of image enhancement lies in enhancing the perception or viewpoint of information in leaf images for human viewers. In this paper, Covariance Kalman Filtered Preprocessing model are applied as preprocessor. Figure 2 shows the block diagram of Covariance Kalman Filtered preprocessing model.

Figure 2
Block diagram of Covariance Kalman Filtered preprocessing model As given in the above block diagram, the Covariance Kalman Filtered preprocessing model initially utilizes the state equation of the linear leaf image database to evaluate the stateof the leaf image database in an optimal manner by means of the input and output of the system. As the observed leaf image database includes the effects of noise and interference, the optimal estimate is obtained by means of an advanced filtering mechanism. The Kalman filter utilizes the state equation " " and the observation equation " " toperform preprocessing. The recursive filtering model is utilized to evaluate the mean square error ofthe leaf image at the next moment, and then accurately predict the leaf disease even in the presence of noise. Let us assume the state equation " " and the observation equation " " of a linear leaf image system as given below.
From the above equations (1) and (2), " " and " −1 " represents the state features of leaf images obtained at two different timestamps " ", " − 1", " , −1 " representing the state transition matrix , " " denoting the observation matrix at timestamp " " with " −1 " and " " forming the state noise and observation noise respectively. Then, the state vector for the corresponding leaf image is mathematically expressed as given below.

=
, , , , , ℎ (3) From the above equation (3), the state vector representation " " is obtained on the basis of the position " " and velocity " ", width " " and height "ℎ" on the " " and " " axis of the leaf image. In a similar manner, the observation vector for the corresponding leaf image is mathematically expressed as given below.

=
, , , ℎ (4) Based on the observation vector, error covariance prediction equation is estimated in our work by means of analysis of principal components. The objective behind the application of analyzing the principal components for error covariance prediction is to remove highly correlated features and to enhance visualization. Let plant leaf " = 1 , 2 , … . , " represent a column (i.e. pixel column) vector with a dimension of " ", each column with mean and variance is represented as given below.
Then, from the above equations (5) and (6), with state and observation vectors " " and " ", the covariance between the two vectors is mathematically evaluated as given below.
From the above equation (), the population (representing the plant leaf) mean vector " " and population covariance matrix "Σ" is formulated as given below.
With the objective of designing a deterministic model so that to reduce the noise the state estimation, is updated as given below.
The pseudo code representation of Kalman Filtered Principal Component Preprocessing is given below.

Algorithm 1 Kalman Filtered Principal Component Preprocessing
As given in the above Kalman Filtered Principal Component Preprocessing algorithm, the objective remains in obtaining the noise minimized preprocessed plant leaf image by utilizing two different functions. First, a modified Kalman Filter utilizing error covariance prediction. Followed by which the Principal Components are utilized to enhance the image via preprocessing, therefore reducing the memory involved (i.e., computational overhead) in leaf disease prediction, owing to only the principal components utilized for further processing.

Geometric Graph-based Segmented Co-occurrence Feature Extraction model
Upon completion of the pre-processing, the second stage in accurate leaf disease prediction is segmentation. It is utilized in identifying the regions in the leaf image that areprobably to qualify as diseased infected regions, therefore clarifying later stages.In this paper, aGeometric Graph-based Segmented Co-occurrence Feature Extraction model to segmentation is performed with the objective of identifying the regions in the leaf image infected with disease. In our work, accurate prediction of leaf disease is performed in two stages. The first stage does the task of region of interest (ROI) identification by eliminating the background using Geometric Pixel Graph-based optimized segmentation. On the other hand, the second stage performs with the diseased infected region recognition by means of Co-occurrence Feature Extraction.
Let " = , " with vertices " ∈ ", the preprocessed leaf images to be segmented, and the edges " , ∈ " representing the adjacent vertices pairs. As far as segmentation of preprocessed leaf images are concerned, the elements in " ", represent the pixels, whereas weight of an edge is measureof dissimilarity between any two pixels connected by that edge in terms of color and shape. To accelerate this, Geometric Pixel Graphbased optimized segmentationusing feature extractors (color and spatial distribution) is implemented toproduce a new region of interest image denoted by " , " as show in figure.

Figure 3 Block diagram of Geometric Graph-based Segmented Co-occurrence Feature Extraction model
As illustrated in the above Geometric Graph-based Segmented Co-occurrence Feature Extraction model, two different processes are carried out separately. In the Geometric Pixel Graph-based optimized segmentation, segmentation " " is a separation of " ∈ " into regions " ∈ 1 , 2 , … . , " such that each region corresponds to a connected region in a graph. The region of interest identification is made by means of Simple Linear approach that utilizes channel information (i.e., color and spatial distribution) and is mathematically represented as given below. = 1 − 2 2 + 1 − 2 2 + 1 − 2 2 (10) From the above equations (10) and (11), the color channel segmented pixels for preprocessed leaf image " " is obtained on the basis of the lightness " 1 , 2 " and two color channels " 1 , 2 & 1 , 2 " respectively. In a similar manner, the spatial distributed segmented pixels for preprocessed leaf image " " is obtained via two spatial axis " 1 , 2 " and " 1 , 2 " respectively. Finally, with the aid of color channel segmented pixels and spatial distributed segmented pixels, the actual region of interest is obtained from equation (12) by eliminating the background, considered to be of least significance.In the second stage, the diseased infected region recognition" , " is identified by means of Graph-based Segmented Co-occurrence Feature Extraction.First Graph-based Segmentation is applied to the identified region of interest as given in equation (12). This is mathematically formulated as given below.
Next with the above segmented images as given in equation (13), feature extraction for extracting the geometric properties of the infected leaf areas is obtained using the co-occurrence matrix is mathematically formulated as given below.
From the above equation (15), the offset " ", " " refers to the channel and spatial distribution operator that is being applied to the " , ℎ " of the co-occurrence matrix. As a result of this, the geometric portions of the leaf images being infected are obtained in an accurate manner with improved true positive rate. The pseudo code representation of Geometric Graph-based Segmented Co-occurrence Feature Extraction is given below.

Algorithm 2 Geometric Graph-based Segmented Co-occurrence Feature Extraction
As given in the above Geometric Graph-based Segmented Co-occurrence Feature Extraction algorithm, the objective remains in extracting the geometric properties of the infected leaf areas via accurate segmentation of the infected leaf areas with maximum accuracy and true positive rate. To achieve this objective, first Geometric Pixel is applied to the preprocessed leaf image as input. Next, with the segmented infected leaf areas, relevant feature for leaf disease prediction are extracted by means of Graph-based Segmented Co-occurrence Feature Extraction, therefore attaining the required objective.

Bernoulli Online Multiple Kernel Learning Classifier
Finally, with the extracted geometric properties of the infected leaf areasBernoulli Online Multiple Kernel Learning Classifier model is applied for leaf disease prediction with minimum classification error. The Bernoulli Online Multiple Kernel Learning Classifier learns a classifier for a given kernel(segmented infected image), combines classifiers by linear weights in such a manner limiting the number of segmented infected image in each single kernel classifier, therefore minimizing the classification error. Figure 4 shows the block diagram of Bernoulli Online Multiple Kernel Learning Classifier model.
To limit the number of segmented infected image to be included in each single kernel classifier, asampling rule is proposed that determines if an incoming instance should be a segmented non-infected image by performing aBernoulli trial as given below.
From the above equation (18), based on the Bernoulli trial " ∈ 0,1 " indicates that a new segmented noninfected image should be added to update classifier upon " = 1", with " " and " " representing the ratio of segmented infected image to total received instances respectively. Finally, the optimal margin classification error for kernel " 1 , 2 , … , " with respect to collection of training samples " = , , ℎ ∈ , " is formulated as given below The pseudo code representation of Bernoulli Online Multiple Kernel Learning Classifier is given below.

Algorithm 3 Bernoulli Online Multiple Kernel Learning Classifier
As given in the above Bernoulli Online Multiple Kernel Learning Classification algorithm, the objective remains in predicting the leaf disease with minimum classification error. To achieve this objective, for each segmented infected image, optimal collections of " " kernels are evaluated based on Multiple Kernel Learning. Next, with the purpose of reducing the classification error, instead of applying conventional Online Multiple Kernel Learning Classifier, in our work, a Bernoulli trial is applied that not only minimizes the computational overhead incurred by considering only the required segmented infected image and therefore reducing the classification error.

Experimental settings
Experimental evaluation of the proposed Covariance Kalman Geometric Graph-basedBernoulliClassifier (CKGG-BC) and existing methods AlexNet and GoogleNet CNN [1], INC-VGGN [2] are implemented using Python using plant village dataset https://www.kaggle.com/emmarex/plantdisease/discussion [20]. This dataset is used for Plant leaf Disease Detection which comprises a variety of plant's RGB plant leaf images like, pepper, potato, tomato, with different sizes. Different unique numbers of both healthy leaf images and disease affected images are obtained as input from the dataset and experimental evaluation is carried out accordingly in terms of computational overhead, accuracy, true positive rate and classification error respectively.

Performance analysis of computational overhead
The first significant metric for leaf disease prediction is computational overhead. Lesser, the computational overhead, higher is the change of predicting the leaf disease and hence greater is the probability of safeguarding the plant. The computational overhead in our work is estimated as given below.
From the above equation (21), the computational overhead " " is measured based on the plant leaf image involved in simulation " " and the memory consumed in leaf disease prediction " ". It is measured in terms of kilobytes (KB).The performance of the proposed CKGG-BCmethod was comparedwith that of a previous plant leaf detection method AlexNet and GoogleNet CNN [1], INC-VGGN [2] in terms of computational overhead. The comparison is presented in table 1 which indicates that the proposed CKGG-BC outperformedthe state-of-the-art methods.  methods. When using our proposedCKGG-BC, "20" plant leaf including both healthy and affected images acquired as input, the computational overhead was observed to be "400 ", "500 " using [1] and "600 " using [2] respectively. Though increasing the number of plant leaf images causes an increase in the computational overhead, but with the simulation analysis results provided as above, a sharp decrease is said to be observed using CKGG-BC upon comparison with [1] and [2]. The minimization of computational overhead in CKGG-BC is due to the application of Covariance Kalman Filtered Preprocessing model. With this model, an integration of filtering and principal component analysis was taken into consideration during the preprocessing that in turn improved in the enhancement of image. Only with this enhanced image, further analysis was carried out, therefore minimizing the computational overhead using CKGG-BC by 32% compared to [1] and 45% compared to [2] respectively.

Performance analysis of accuracy
The second metric of significance for plant leaf disease prediction is the accuracy rate. Higher the accuracy, efficiency of the overall method is said to be. In other words, accuracy is measured as given below.
From the above equation (22), the accuracy rate " " is measured based on the plant leaf disease correctly predicted " " to the overall plant leaf " " involved in the simulation. The accuracy is measured in terms of percentage (%). The performance of accuracy of the proposed CKGG-BCmethod was comparedwith that of previous state-of-the-art methods AlexNet and GoogleNet CNN [1], . The comparison is presented in table 2 which indicates that the proposed CKGG-BC outperformedthe state-of-the-art methods in terms of accuacy.

Figure 6
Graphical representation of accuracy Figure 6 given above presents the plant leaf disease prediction accuracies for plant leaf images involving both health and infected images. In Figure 6, one can see that prediction accuracy can be improved if images are transformed usingCKGG-BCmethod. Compared to the state-of the-artmethods AlexNet and GoogleNet CNN [1], INC-VGGN [2]tested in this experiment, CKGG-BCmethod performs betterand improves accuracy to a greater extentthan the other state-of-the-art methods, [1] and [2]. Though a smart accuracy deviation was said to be observed, simulation analysis "20" numbers of healthy and infected plant leaf image provided as input, "85%" of accuracy was found using CKGG-BCmethod, "80%" using [1] and "75%" using [2] respectively. The reason behind the accuracy improvement using CKGG-BCmethod was due to the application of Geometric Graph-based Segmented Co-occurrence Feature Extraction model. With this model, initially geometric features were utilized for segmenting the infected leaf areas and then Co-occurrence Feature Extraction was performed for extracting geometric properties of infected leaf areas. This in turn improved the accuracy rate using CKGG-BCmethod by 5% compared to [1] and 8% compared to [2] respectively.

Performance analysis of true positive rate and classification error
Finally, the true positive rate and classification error is analyzed in this section. True Positive rate refers to the ratio of positives (healthy plant leaf as identified with healthy and infected plant leaf identified as infected) that are correctly identified. It is estimated as given below = * 100 (23) From the above equation (23), the true positive rate " " is measured based on the number of true positives " " and the total number of infected plant leaf " " considered for simulation. It is measured in terms of percentage. Finally, the classification error is measured as given below.

= * 100(24)
From the above equation (24), the classification error " ", is measured based on the samples incorrectly classified " " to the total number of samples considered " " for simulation. It is expressed in terms of percentage (%).   First, as far as true positive rate is concerned, higher amount healthy plant leaf were identified as healthy and infected plant leaf were identified as infected using the proposed CKGG-BC method upon comparison with [1] and [2]. The reason behind the improvement was due to the application of Geometric Graph-based Segmented Cooccurrence Feature Extraction algorithm. By applying this algorithm, first, the geometric properties of the infected leaf areas were significantly extracted via accurate segmentation of the infected leaf areas. Next, the Geometric Pixel were utilized and applied to the preprocessed leaf image to identify the segmented infected leaf areas. Finally, relevant feature were extracted for leaf disease prediction using Graph-based Segmented Co-occurrence Feature Extraction, therefore improving the true positive rate using CKGG-BC by 7% compared to [1] and [2].
Next, the classification error using CKGG-BC method was found to be comparatively lesser than [1] and [2]. The reason behind the reduction in classification error was due to the application of Bernoulli Online Multiple Kernel Learning Classification algorithm. By applying this algorithm, optimal collections of kernels were estimated on the basis of multiple kernel learning for only the segmented infected image. Next, instead of using the traditional Online Multiple Kernel Learning Classifier, in our work, a Bernoulli trial is applied that reduces the classification error using CKGG-BC method by 43% compared to [1] and 26% compared to [2] respectively.

Conclusion
In this paper, we proposed a novel Covariance Kalman Geometric Graph-basedBernoulliClassifier (CKGG-BC) method for plant leaf disease prediction. In pre-processing, to enhance the image and improve the quality of image for further analysis, principal components of filtered images are utilized and based on the principal components, advance filtering is performed. AGeometric Graph-based Segmented Co-occurrence Feature Extraction method is used for the segmentation of diseasedportion and extracting the geometric properties accordingly. Finally, Bernoulli Online Multiple Kernel Learning Classifier is applied for timely and precise leaf disease prediction with minimum error even in the presence of noise. According to our experimentalresults, CKGG-BC outperforms state-of-the-art methods interms of classification performance.Based on Geometric Graph-based Segmented Co-occurrence Feature Extraction and Bernoulli Online Multiple Kernel Learning Classifier,CKGG-BC can not only enhance true positive rate and accuracy,but also reduce the computational overhead and classification error, making it very practical in the field of agriculture applications.